Skip to main content

Golden-task evaluation harness

The machinery behind the word "proven." One harness that scores tools, models, prompts and skills against a fixed set of real, anonymised tasks — so every claim ("this model is better", "this prompt works", "promote this skill") is measured, not asserted.

The task set​

A fixed, versioned set of real tasks drawn from our own work, anonymised:

  • Feature on the scaffold — build a small feature in the project scaffold to spec.
  • Legacy bug fix — a real defect in an anonymised legacy snapshot.
  • Test generation — produce honest tests for a given module (scored on mutation, not coverage).
  • Refactor — a safe refactor behind existing tests.

Each task ships with fixtures and an automated grader.

What's scored​

  • First-pass acceptance — did the output pass without correction?
  • Gate pass rate — did it clear the merge gate checks?
  • Cost — tokens / time to a passing result.

What the harness gates​

  • Tool/model selection — feeds the tooling scorecard and every radar ring-move for AI tooling.
  • Prompt promotion — a prompt moves trial → proven only on harness evidence.
  • Skill release — a skill's golden scenarios run here before it's published.

Rules​

  • Same tasks, same graders, every time — comparability is the whole point.
  • Anonymised — no client data or PII in the task set (UK GDPR); synthetic or scrubbed only.
  • Versioned — the task set is itself in Git; changes to it are decisions worth noting.