>_ reliability / systems notes

Reliability,
with receipts.

A public, deliberately bounded view of how the Belacca platform is built, delivered, protected, and improved.

Current state: this is an architecture note, not a live status page. Automated SLO telemetry, off-cluster backups, and an externally generated incident surface are not deployed yet.

01

The current
shape.

Small, explicit boundaries are easier to reason about than a diagram that promises everything.

01 / EDGE

HTTPS edge

Traefik terminates public HTTPS, redirects HTTP, and routes by hostname. The portfolio and Pong have separate public routes.

visitor → edge
02 / PORTFOLIO

Static site

Two NGINX-backed site replicas serve HTML, CSS, JavaScript, and discovery assets. /health is the probe endpoint.

edge → site service
03 / PONG

Game path

Pong routes through a gateway to its static frontend and Go lobby API. A room receives its own Kubernetes Pod for the live game.

edge → gateway → room
portfolio / stateless

Simple by default

The portfolio is a static application. Its only runtime request is the same-origin analytics proxy; there is no CMS, browser-to-backend application API, login, or client-side framework.

NGINX2 replicas/health
pong / stateful boundary

One writer on purpose

The Pong lobby uses one API replica because its room state lives in single-writer SQLite on persistent storage. That is a known availability and scaling boundary, not hidden magic.

GoSQLiteRWO storage
rooms / bounded runtime

Ephemeral game Pods

Each game room is represented by a dedicated Pod and Service with resource limits and a deadline. Waiting, finished, failed, and orphaned rooms are cleaned by the lobby.

WebSocketsper-room Podcleanup
analytics / first party

Useful, not invasive

Visitor analytics are sent through /count to the self-hosted GoatCounter service. The browser does not contact an analytics vendor directly, and the dashboard is separate from the public site.

same origincookie-freeaggregate data
02

Changes travel
as evidence.

The desired production action is a reviewed change, not an improvised shell session.

  1. 01

    Commit

    A change lands in the application repository and is reviewed as source.

  2. 02

    Test and build

    GitHub Actions runs the site tests, builds the NGINX image, and publishes a commit-addressed GHCR tag.

  3. 03

    Record

    The workflow writes the immutable SHA tag to the deployment Kustomization, keeping the intended release in Git.

  4. 04

    Reconcile

    Flux watches the application repository and applies the site deployment on its configured reconciliation interval.

  5. 05

    Rollback

    The documented rollback is a reviewed Git revert followed by reconciliation. Published images now carry a registry SBOM and GitHub Artifact Attestation provenance; no admission or Flux verification is configured.

03

Boundaries
before bravado.

The public surface says what is protected without turning an operational map into a treasure map.

public surface

Browser to site

The site is served over the TLS route and has NGINX security headers for content type sniffing, framing, referrers, permissions, and content policy. The CSP keeps scripts, styles, connections, and forms on the same origin.

private control

Operations stay separate

The Kubernetes dashboard is a separately routed, authenticated surface with read-only access. It is not embedded in this site, and no credentials, tokens, or cluster control plane are shipped to visitors.

internal callbacks

Room lifecycle stays internal

Pong room lifecycle callbacks are addressed through the internal application service rather than the public gateway. Public WebSocket traffic reaches the game path; it does not expose the control callback.

honest gap

Controls still to add

Published images now receive a registry SBOM and GitHub Artifact Attestation provenance. Default-deny network policy, automatic attestation verification at reconciliation, and a mandatory vulnerability gate are not configured. They must not be inferred from the current headers or GitOps flow.

04

Measure what
users feel.

Reliability is a feedback loop: journey → signal → objective → budget → evidence → recovery.

slo method / proposed

The loop

  1. JourneyDefine the user action that matters.
  2. SLICount good and total events at the boundary.
  3. SLOAgree a target and time window with stakeholders.
  4. BudgetUse missed reliability as a delivery trade-off.
  5. EvidenceConnect alerts, deployments, and incident notes.
candidate signals / not live

A sensible first cut

portfolio
Successful external HTTPS checks for the homepage and /health.
Pong lobby
Successful create and join journeys, followed by an accepted WebSocket session.
delivery
Flux source and application reconciliation health as an operational diagnostic.

Not deployed: there is currently no Prometheus-compatible application metric layer, SLO recording rule, burn-rate alert, or external monitor feeding this page. Targets, windows, owners, and paging policy remain to be agreed.

05

Recovery has
limits.

Naming the failure domain is more useful than displaying an unearned uptime number.

backup boundary / current

Manual and local

Pong and analytics retain SQLite data on single-writer, node-local persistent storage. Operator documentation contains manual consistency-and-copy procedures, but there is no scheduled encrypted off-cluster backup, retention policy, or verified restore job today.

Therefore: no public RPO or RTO promise.

recovery practice / current

Safe first response

Runbooks start with non-destructive inspection, preserve protected state, check user-facing routes and Flux health, and use a reviewed Git revert for application rollback. Incidents are not currently backed by an automated public status feed or paging service.

This page is not an incident channel.

planned / explicitly not live

Close the loop

Next reliability work is to add externally generated sanitized status data, user-journey SLOs and burn-rate alerts, encrypted off-cluster backups, restore verification, and repeatable failure drills. Each item needs runtime proof before it becomes a claim here.

planned ≠ deployed

06 / QUESTIONS

Prefer a system
you can inspect?

Source, manifests, and the limits behind this page are public.

Inspect the platform