Skip to main content

Operability baseline audit

A per-product STOP audit of whether the estate can actually be run: monitoring, alerting and incident runbooks, checked against the observability standard's baseline. It produces two outputs deliberately kept apart: gaps go to the remediation backlog, and the runbooks that have earned their keep are preserved as capital — not rewritten because they're old.

The audit, per product​

  1. Monitoring — do the golden signals exist per service? Are dashboards looked at or ornamental (last-viewed dates tell you)? Are any SLOs stated anywhere, even informally?
  2. Alerting — do alerts exist, route somewhere accountable, and actually fire (test one)? Count alerts routing to leavers' inboxes and dead channels — on legacy estates this is routinely where the pages go. Note the noise ratio: an alert channel everyone has muted is a gap, not a control.
  3. Incident runbooks — do they exist, can on-call find them, and when were they last followed in anger? Flag procedures that live only in one engineer's head — on an insurance estate the overnight batch (renewals, bordereaux, MTA processing) is the classic case: fails silently, restarted by the one person who knows how.
  4. Incident history — walk the last few incidents: was diagnosis done from dashboards or from guesswork and SSH? This cross-references the MTTR figure from the DORA baseline and explains it.

Outputs​

  • Gaps → the remediation backlog — scored with everything else (WSJF), each linked to its evidence. A product with no error alerting on the policy-issuance path scores itself.
  • Surviving runbooks → the Flow-state library — a runbook that has steered a real incident is proven capital. Migrate it into the runbook template format and keep it current through the Flow-state loop; rewriting working runbooks from scratch is negative-value work and loses the hard-won detail.
  • A baseline snapshot per product — the operability before-photo, alongside the DORA one, feeding the metrics platform so post-RAPID operability claims are measured.

Why it runs in STOP​

Agents accelerate building, not firefighting a product you can't see into. The observability standard guarantees new builds ship diagnosable; this audit is the same bar pointed at the legacy estate — and its runbook harvest is what Support inherits instead of a quarterly knowledge dump.

Access and owners needed

The audit needs read access to each product's monitoring and alerting tooling and its incident ticket history, plus an ops owner per product to nominate the "surviving" runbooks — that call is a judgement about what has actually worked, and it can't be made from the outside.

Standards referenced: Google SRE (golden signals, SLOs), ITIL 4 (incident management practice).