Skip to main content

AI-generated-test review

The biggest risk with agent-written tests is that they're written to pass, not to verify. This standard is the checklist for telling an honest test from a decorative one.

What to check​

  • Assertion strength — no tautologies (expect(x).toBe(x)), no snapshot-everything that asserts nothing meaningful. Each test states a specific expected behaviour.
  • Valid oracle — the test asserts the requirement, not the implementation. A test that just re-encodes what the code does will pass forever and catch nothing.
  • Not testing the mock — assertions are against real behaviour, not against the test doubles the test set up.
  • Edge cases actually covered — the PFD's positive/negative/boundary examples (acceptance-criteria standard) each have a test.

The hard rule​

Any agent change to an existing test requires explicit human approval. New tests can be reviewed on their merits, but an agent quietly weakening an existing test to make a change pass is exactly how regressions slip through — so that path is closed by default.

How it's enforced​

  • Mutation score (Test pyramid & coverage) catches non-asserting tests mechanically.
  • Human review at the merge gate covers oracle validity and test-the-mock — judgement a machine can't fully make.
  • Standards referenced: mutation testing, ISTQB CT-AI.