HTTPS edge
Traefik terminates public HTTPS, redirects HTTP, and routes by hostname. The portfolio and Pong have separate public routes.
visitor → edge>_ reliability / systems notes
Reliability,
with receipts.
A public, deliberately bounded view of how the Belacca platform is built, delivered, protected, and improved.
Evidence boundary: this is an architecture note, not a live status page. The policy and evidence pipeline exists, and the external monitoring path publishes sanitized status.json and durable slo.json artifacts from outside the native cluster. Current service levels are calculated from the good and bad observations already available; the page shows the observed count and coverage beside each number. Once the history spans 30 days, the calculation uses the latest rolling 30-day window. Recovery, notification, and deployment assertions are rendered from the sanitized evidence ledger below, with source and verification timestamps; it distinguishes verified evidence, bounded evidence, outstanding work, and declared configuration. Native Prometheus is private diagnostic telemetry, not external availability proof. The 99% availability target per public service is an internal objective, not an SLA; optional authenticated dashboard and Flux checks, external cluster-down monitoring, health-aware failover, and real failure drills remain unclaimed.
01The current
The current
shape.
Small, explicit boundaries are easier to reason about than a diagram that promises everything.
Static site
The deployment declares two Caddy-backed site replicas for HTML, CSS, JavaScript, and discovery assets. /health is the probe endpoint; runtime readiness remains an external observation concern.
Game path
Pong routes through a gateway to its static frontend and Go lobby API. A room receives its own Kubernetes Pod for the live game.
edge → gateway → roomSimple by default
The portfolio is a static application. Its only runtime application request is the same-origin analytics proxy; there is no CMS, browser-to-backend application API, login, or client-side framework.
One writer on purpose
The Pong lobby uses one API replica because its room state lives in single-writer SQLite on persistent storage. That is a known availability and scaling boundary, not hidden magic.
Ephemeral game Pods
Each game room is represented by a dedicated Pod and Service with resource limits and a deadline. Waiting, finished, failed, and orphaned rooms are cleaned by the lobby.
Useful, not invasive
Visitor analytics are sent through /count to the self-hosted GoatCounter service. The browser does not contact an analytics vendor directly, and the dashboard is separate from the public site.
02Changes travel
Changes travel
as evidence.
The desired production action is a reviewed change, not an improvised shell session.
- 01
Commit
A change lands in the application repository and is reviewed as source. The portfolio SLO contract and its validators are reviewed with the change.
- 02
Test and build
GitHub Actions runs the site tests, builds the Caddy image, and publishes a commit-addressed GHCR tag.
- 03
Record
The workflow writes the immutable SHA tag to the deployment Kustomization, keeping the intended release in Git.
- 04
Reconcile
Flux watches the application repository and applies the site deployment on its configured reconciliation interval.
- 05
Rollback
The documented rollback is a reviewed Git revert followed by reconciliation.
scripts/gitops-rollback-check.shproves that the desired immutable image tag returns to the known-good state in an isolated repository; live Flux reconciliation still requires an operator-run production drill and is not claimed here. Published images now carry a registry SBOM and GitHub Artifact Attestation provenance; no admission or Flux verification is configured.
03Boundaries
Boundaries
before bravado.
The public surface says what is protected without turning an operational map into a treasure map.
Browser to site
The site is served over the TLS route and has Caddy security headers for content type sniffing, framing, referrers, permissions, and content policy. The CSP keeps scripts, styles, connections, and forms on the same origin.
Operations stay separate
Headlamp is a separately routed, authenticated shared-admin surface in native production. Its fixed ServiceAccount is bound to the built-in cluster-admin role; the Dex/OAuth2 Proxy allowlist authenticates the operator at the front door but does not provide per-user Kubernetes RBAC or impersonation. It is not embedded in this site, and no credentials, tokens, or cluster control plane are shipped to visitors.
Room lifecycle stays internal
Pong room lifecycle callbacks are addressed through the internal application service rather than the public gateway. Public WebSocket-compatible traffic reaches the game path; optional WebTransport follows a separate reviewed UDP contract. Neither exposes the control callback.
Controls still to add
Published images now receive a registry SBOM and GitHub Artifact Attestation provenance. Default-deny network policy, automatic attestation verification at reconciliation, and a mandatory vulnerability gate remain separate platform controls; they must not be inferred from the current headers or GitOps flow.
04Measure what
Measure what
users feel.
Reliability is a feedback loop: journey → signal → objective → budget → evidence → recovery.
The loop
- JourneyDefine the user action that matters.
- SLICount good and total events at the boundary.
- PolicySet an internal objective and report the measured level from available observations, then use the latest complete rolling window once it exists.
- BudgetUse missed reliability as a delivery trade-off.
- EvidenceConnect alerts, deployments, and incident notes.
What is observed
The portfolio SLI contract and external probe cover the health and homepage journey, with optional alias/path-preserving redirect assertions. The separate external status publisher's external user-journey checks now cover the portfolio, Pong, and analytics collector: analytics uses /status plus same-origin /count, and representative portfolio aliases are checked from outside the native cluster. GoatCounter remains a diagnostic dependency for the portfolio SLI: analytics failure is excluded from the primary event when the page remains available, and a failing /count request must not remove the primary page journey. Authenticated dashboard and Flux checks remain pending until an operator-managed least-privilege identity is provisioned out of band.
- portfolio
- External user-journey checks cover the portfolio.
- Pong
- External user-journey checks cover the create, join, and WebSocket-compatible real-time path. Native WebTransport is an optional path, not a current public-ingress guarantee.
- analytics collector
- External user-journey checks cover the public analytics collector.
- authenticated dashboard
- Authenticated dashboard checks await a reviewed synthetic identity.
Evidence boundary: portfolio-slo.json defines the portfolio journey, while belacca-status publishes sanitized status.json current-state evidence and slo.json measured service levels. Each current level is good observations / (good + bad observations); unknown or missing slots are displayed as coverage context and never count as success. Recovery and notification claims are not inferred from the SLO artifact; they are linked in the evidence ledger with their own timestamps and limitations. Native Prometheus is private diagnostic telemetry, not external availability proof or a public SLO calculation. The 99% availability objective per public service is internal, not an SLA, and budget exhaustion does not by itself change public status.
05Measured
Measured
service levels.
Loading the latest sanitized SLO evidence.
- No measured service-level observations are available.
06Recovery has
Recovery has
limits.
Naming the failure domain is more useful than displaying an unearned uptime number.
Recovery evidence is bounded
Live upload and isolated restore verification is recorded for Pong, GoatCounter, and Dex. The destination, retention policy, and source-consistency gates are separate claims; the evidence ledger below keeps those boundaries visible instead of presenting backup capability as current service health.
Therefore: the 24-hour RPO target remains documented, not an SLA or completed retention guarantee.
Safe first response
Runbooks start with non-destructive inspection, preserve protected state, check user-facing routes and Flux health, and use a reviewed Git revert for application rollback. Native services support host restart recovery; Alertmanager pages the operator through Telegram for configured in-cluster signals, while no real failure drills are claimed.
A separate controlled-drill recovery P95 under six minutes remains unproven.
Close the recovery loop
The external publisher, durable SLO artifact, expanded unauthenticated journeys, and reliable off-cluster backup destination are deployed. Live uploads and isolated restore verification now pass for Pong, GoatCounter, and Dex. Next reliability work is to build the 14-day retention history (the long-term monthly archive was dropped in a 2026-09-02 cost relaxation), complete isolated application restore rehearsals where required, test operator notifications, complete the rolling 30-day measurement window, provision authenticated dashboard/Flux journeys, add independent external cluster-down monitoring, health-aware failover, and repeatable failure drills. A live Flux rollback drill also needs production operator evidence. Each remaining item needs runtime proof before it becomes a claim here.
planned ≠ deployed
07Evidence
Evidence
ledger.
Loading the sanitized reliability evidence ledger.
- Reliability evidence will appear with the next validated ledger.
08 / QUESTIONSPrefer a system
Prefer a system
you can inspect?
Source, manifests, and the limits behind this page are public.
Inspect the platform