AI · 6 MIN READ
You cannot ship AI without an evaluation set
Every stalled AI project we have seen stalled at the same question: how do you know it works? Here is the cheapest useful answer.
AB19 Innovations
In short
AI features need an evaluation set the way code needs tests. Build a small labelled dataset early, score changes against it in CI, and treat prompt edits as deployments requiring a passing run.
A prototype gets built in a week and impresses everyone. Six months later it is still not in production. The reason is almost always the same, and it is not technical difficulty — it is that nobody can answer the question "how do we know it works?" with anything more than a demo.
Start with fifty examples
People delay building an evaluation set because they imagine it needs thousands of carefully labelled records. It does not. Fifty real inputs with expected outputs, chosen to cover the cases you actually care about, will catch most regressions and immediately change how the team works.
- Twenty typical cases — the bread and butter.
- Twenty known-hard cases — the ones that failed during the prototype.
- Ten cases where the correct output is a refusal or an escalation.
Score in CI, not in a notebook
An evaluation that lives in someone’s notebook runs when they remember. An evaluation in the pipeline runs on every change. Treat a prompt edit exactly like a code change: it needs a passing run before it merges.
PR opened
└─ run eval suite (50 cases)
├─ score >= baseline → allowed to merge
└─ score < baseline → blocked, with the failing cases listedThe blocked case is the valuable one. Without it, a prompt tweak that fixes the bug in front of you and quietly breaks four other things ships on a Friday afternoon.
Grade what you actually care about
Exact-match scoring punishes correct answers that are worded differently. Human grading does not scale. Model-graded evaluation works reasonably well when the rubric is specific — but the rubric has to be written down and agreed, not improvised per run.
Where it matters most, grade structurally. If the output is meant to be a typed record, validate the schema and compare fields. Structured output is easier to build, easier to score and easier to use downstream than prose.
An evaluation set is not quality assurance overhead. It is the thing that lets you change the system without fear, which is the only way anything reaches production.