Skip to content
Ali Akbari
Menu

SRE

Production incident investigation

Implementedenvironment:ProductionEmploymentSelf-reported · private sourceemployment-incident-review

Claim

Investigated a production reliability incident, traced it to unbounded outbox retries and session advisory locks behind PgBouncer transaction pooling, bounded the retries in 11 of 13 event publishers and began replacing the locks.

Status

implemented
Capped at Implemented: the code, tests and records are private, so this is self-reported and cannot be verified publicly.
production
Real users or traffic: employment history, or a project with real usage.
employment
Real work for an employer, sanitized. Source code is proprietary.

Source

Company work. The source code and internal records belong to the company and cannot be published; any figures shown are sanitized. So this claim is self-reported: it stops at Implemented and links to no artifacts.

Limitations

  • The write-up is sanitized; the internal root-cause report is not published.
  • When my work ended, the pooler-safe lock had replaced the old locks in one service; four others still used them.
  • Most connection resets in the incident window could not be attributed with the logging in place.
  • Internal report and data: private.

Appears in

Related claims

Implementedenvironment:ProductionSelf-reported · private source

Event backbone with outbox, idempotency and dead-letter handling

Built the event backbone on Kafka with Avro schemas in a Schema Registry and a transactional outbox in each service, plus idempotent order placement and dead-letter handling.

BackendEmployment · no public artifactsEvidence