Resilience & chaos testing
Proves the operability that Production Readiness asserts. Deliberately deferred (P2) so it doesn't slow the first builds — but scheduled, so degraded modes, backups and rollback are proven under controlled failure, not assumed until a real incident tests them for us.
What we test
- Dependency outages — a third-party data feed down, a queue backing up, a database failover. The product should degrade the way its NFR catalogue says it does: a broker portal that can't reach a rating service should queue the MTA and say so, not fall over.
- Degraded-mode behaviour — feature-flag kill switches, queue-and-retry paths, read-only fallbacks. Verify the right alerts fire (observability) and the runbook steps actually work when followed cold.
- Backup/restore — a backup that has never been restored is a hope, not a control. Restore into a clean environment and verify integrity, on a cadence, against the runbook's recovery section.
- Rollback — re-proven periodically, not only at first release. Production Readiness checks it once; releases and schema changes since then can quietly break it.
Quarterly game days
- Support and DevOps at the table. A game day is as much a rehearsal of detection, escalation and runbook accuracy as of system behaviour — the people who take the 3am call practise on a controlled afternoon (support playbook).
- Scenarios: one from the runbook's known-failure-modes table (does the documented response work?), one novel (what haven't we thought of?).
- Pre-production first. Experiments run against pre-production until confidence is earned. Any experiment touching production needs an explicit blast-radius limit, an abort condition, and named-human authorisation — deliberately breaking production is on the irreversible list.
- Measure: time to detect, time to correct diagnosis, runbook accuracy, escalation path followed. These are the operability numbers the quality view on the metrics platform tracks.
- Deposit the capital. Every finding lands somewhere permanent before the day closes: a runbook correction, a gotcha, a tuned alert, a new regression or resilience scenario automated where cheap. A game day that only produces a document has failed.
The deliberate deferral
Chaos testing before there's anything in production is theatre, and resilience engineering during the first build is scope the three-month window can't afford. So: nothing before first production release; first game day inside the first quarter of production operation; quarterly thereafter. For StudyBuddy that means the schedule is set at Production Readiness — the gate records the date, so the deferral is a plan, not a drift.
Whether (and which) experiments may ever run against live production — versus pre-production only — needs a DevOps + Support + Leadership decision per product, recorded when the first game day is scheduled. Client-facing insurance workloads may reasonably stay pre-production-only.
Standards referenced: Principles of Chaos Engineering, Google SRE (DiRT-style game days).