Skip to content
Ali Akbari
Menu

opsledger

This website measures its own operation

Here is the evidence, where it comes from, what it cannot see, and how you can check it yourself.

What this is

opsledger is a small Go service I built for this site. It turns nginx's request stream into availability and latency indicators, counts its own data loss, and publishes daily reports in which every number can be recomputed from the observations it was computed from.

Two observers, deliberately different. The origin view comes from the production host and is self-reported. The external probe runs on GitHub-hosted machines, outside the host, and sees the site as a visitor does. Daily reports and releases are signed with Sigstore, keylessly, as the GitHub workflow that produced them.

Live evidence

The last 28 completed UTC days, as the host computes them now.

Computed by the production host from its own observations (unsigned). Signed daily reports add the external probe.

Loading live values from /api/v1/ops/status, /api/v1/ops/summary and /api/v1/release. Without JavaScript, open those URLs directly.

Architecture

opsledger data flowOn the production host, nginx sends one record per request over a Unix datagram socket to opsledger, which aggregates per minute into SQLite and serves completed days through a read-only API; nginx's own request counter is sampled to count lost records. On GitHub-hosted runners, an external probe checks the site over HTTPS, a daily workflow assembles the host's day with the probe runs, recomputes every value, signs the report and commits it to the ops-reports branch, and the CI release job signs the tested image digests that deploy.sh verifies before deploying.Production hostGitHub-hosted runnersdatagramsreconcilecompleted daysartifactssigned manifestchecksnginxone record per requestopsledgerqueue → aggregator → SQLiteRead-only API/api/v1/ops · /releasedeploy.shverifies, then deploysstub_statusnginx's request counterExternal probeHTTPS via the CDNops-publish (daily)assemble · recompute · signops-reportssigned daily JSONCI release jobsigns tested digests
Figure: opsledger. The host reports what nginx saw; the probe checks from outside; the publisher recomputes and signs both.

  1. 1Record

    nginx sends one record per request to a Unix datagram socket: time, method, path, status, duration and three flags. No address, user agent, query string or referrer.

  2. 2Aggregate

    opsledger (Go) reads the socket into a bounded queue; a single aggregator goroutine counts outcomes and exact SLI totals per minute and keeps an exponential latency histogram, flushed to SQLite in WAL mode. A full queue drops and counts, it never blocks nginx.

  3. 3Reconcile

    Once a minute opsledger reads nginx's own request counter (stub_status), so records lost between nginx and the ledger are counted, not assumed away.

  4. 4Probe

    A scheduled workflow on GitHub-hosted runners checks the public site over HTTPS through the CDN: status, content, snapshot consistency between pages and API, certificate validity.

  5. 5Publish

    Every day a workflow takes the host's completed day and the probe runs, assembles the report, rebuilds the 28-day summary, recomputes every value, signs both files with Sigstore and commits them to the ops-reports branch.

  6. 6Release

    CI signs a manifest of the exact image digests it tested; deploy.sh verifies that signature with a pinned verifier before it changes anything, and refuses unsigned or altered releases.

Definitions

Objectives apply over a rolling 28 days.

IndicatorDefinitionObjective
Origin availabilityOrganic, non-bot requests for pages, navigation payloads and the API that reached nginx. 5xx is bad; firewall blocks (403), rate-limit refusals (429) and client disconnects (499) are counted separately and excluded.99.5%
Origin latencyShare of those requests (excluding 5xx) whose nginx $request_time is within 300 ms for pages and navigation payloads, 100 ms for the API.99%
Probe successShare of external probe runs in which every check passed.99.5%
CompletenessShare of the requests nginx counted (stub_status) whose records reached the ledger.99%

What it cannot see

  • Responses served from the CDN's cache never reach the origin and are not counted.
  • An outage before nginx (DNS, CDN, network, host down) leaves no origin records; only the probe can see it.
  • The probe samples every 30 minutes, and GitHub may delay or skip runs. Zero failures in n runs bounds the failure rate below about 3/n at 95% confidence; it is not continuous uptime.
  • Origin data is reported by the host it describes. Recomputation proves the statistics follow from the published observations, not that the observations are true.
  • Signatures prove which workflow published which bytes and when; they do not prove the measurements are correct.

Verify it yourself

The API is public and read-only:

curl -s https://aliakbari.dev/api/v1/ops/summary
curl -s https://aliakbari.dev/api/v1/ops/daily/$(date -u -d yesterday +%F)
curl -s https://aliakbari.dev/api/v1/release

Recompute every published value from a directory of reports; any discrepancy exits non-zero:

go run ./services/ledger/cmd/ledger recompute <reports-dir>

Check a report's signature and the workflow that signed it:

ledger verify-bundle -artifact <day>.json -workflow .github/workflows/ops-publish.yml

Check that a release was signed by CI at its commit:

cosign verify-blob --bundle release.json.sigstore.json \
  --certificate-identity https://github.com/akynte/portfolio/.github/workflows/ci.yml@refs/heads/main \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-github-workflow-sha <commit> release.json

The source repository and the signed reports branch are private for now, so signed artifacts are available on request until the repository is published. The live API above is public.

The design, the trade-offs and what I learned are in the opsledger case study.