opsledger — Verifiable Operational Evidence

Overview

opsledger answers one question about this website with numbers a reviewer can check: is it up, and how fast is it? Two observers look at the site. nginx on the server sees every request that reaches it; a probe on GitHub's machines sees the site from the internet, as a visitor would, every 30 minutes. A Go service turns nginx's stream of requests into per-minute counts and latency histograms, and every day a workflow combines both views into a report, recomputes every number in it, signs it and publishes it.

The same pipeline now signs every release: CI signs the exact image digests it tested, and the deploy refuses anything it cannot verify.

Live from this site

Architecture

• Ingestion. nginx writes one small JSON record per request to a Unix datagram socket (its syslog transport). The record has a time, method, a path capped at 192 bytes, status and duration, and three flags nginx computes itself: bot user agent, Next.js navigation, probe. No address, user agent, query string or referrer ever reaches the ledger, and every client-controlled value is mapped to a fixed form in nginx first, so no request can make its own record unparseable.
• Concurrency. One goroutine drains the socket and never blocks; a bounded queue feeds one aggregator goroutine that owns all state, so there are no locks. When the queue is full, records are dropped and counted. Shutdown stops HTTP, then the reader, then drains the queue and writes the partial minute.
• Aggregation. Per minute and per class (page, navigation, API, static, …): outcomes, status classes, an exact count of requests within the latency objective, and a base-2 exponential histogram whose bucket indices are computed with integer arithmetic, so merges and percentiles are identical on every machine.
• Completeness. Once a minute the ledger reads nginx's own request counter. The difference between nginx's count and the records received is the loss in transit, measured independently of the ledger's own counters.
• Storage. SQLite in WAL mode with migrations, a size cap, 35-day retention, and recovery by replacement; published reports are the durable record.
• Isolation. nginx serves pages with or without the ledger; deploys never wait on it.

Reliability method

• Indicators. Origin availability counts organic, non-bot page, navigation and API requests; 5xx is bad; firewall blocks, rate-limit refusals and client disconnects are excluded and reported. Latency counts requests within 300 ms (pages) or 100 ms (API) of nginx's $request_time. A probe run succeeds only if all eight checks do: pages, PDF, API, snapshot consistency, certificate.
• Uncertainty. Every ratio carries its n and a 95% Wilson interval; with zero probe failures the exact bound 1 − 0.05^(1/n) is published as well. 28 days of 30-minute probes (1,344 runs) without a failure bound the failure rate below about 0.22%: not enough to claim 99.9%, and the reports say so. Percentiles are published as bucket intervals with a flag when fewer than ten observations lie above them.
• Reports. Each day is one JSON document of raw hourly observations plus every derived value. ledger recompute validates the schemas, checks the raw data's consistency and rebuilds every derived value with the same code; it fails on tampered counts, intervals, percentile bounds, coverage, probe results, windows and replaced files. ADR-0017

Supply chain

Build and test jobs moved from the self-hosted runner on the production host to GitHub-hosted runners, so pull-request code no longer runs where the site is served. A separate job, the only one allowed an OIDC token, signs a manifest of the tested image digests with Sigstore after every test has passed, and checks its own signature against impostors: wrong commit, workflow, repository, branch, trigger, digest, and a tampered manifest. The deploy verifies the release again on the host with a verifier pinned by digest outside the release and refuses it before anything changes. ADR-0015

Results so far

All figures below are lab measurements on one machine (Docker Desktop, 16 CPUs), not production:

• Loss accounting is exact. A burst of 200,000 requests at about 130,000 requests/s lost 1,603 records with the kernel's default queue of 10 datagrams per socket. The reconciliation reported 1,603 missing; nginx had logged 1,603 failed sends.
• Tuned by measurement. With net.unix.max_dgram_qlen 512, three identical bursts at 136,000–144,000 requests/s lost nothing. The ledger used 16–20 MiB of memory throughout.
• Tests. Unit, property, race and fuzz tests for the parser and histograms; an end-to-end test of the real command (records in, SIGTERM, nothing accepted lost, report recomputes, tampering detected); Sigstore negative tests against a virtual CA; and fail-closed deploy tests against a local registry.

No production figure exists yet: the service has not been deployed, and this page will not show one until it has been measured.

Private analytics, kept apart

Visitor analytics are a separate, private system: self-hosted Umami behind two first-party paths, with forged fields refused by the firewall, no cookies, and the owner's access only through an SSH tunnel. The public reports never contain visitor data, and the analytics are never used as evidence. ADR-0018

Limitations

• Not yet deployed: there are no production measurements, and the claims are capped at IMPLEMENTED.
• The origin view is reported by the system it describes, and cannot see failures before nginx.
• The external probe samples every 30 minutes while the repository is private; outages between runs are not observed.
• Signatures prove which workflow published which bytes; they do not prove the observations.
• The repository is private; the signed artifacts are in a private branch until it is published.

Lessons

• The measurement needs its own measurement: without nginx's counter, the lab loss would have been invisible, because the ledger's own counters showed zero drops.
• Privacy decisions belong in the transport: classifying bots and capping paths inside nginx means the ledger cannot leak what it never receives.
• An honest interval is more useful than a precise-looking percentage; small samples are the normal case for a personal site.