Test Automation for AI: Scaling Evaluation Without Scaling Headcount

Manual evaluation does not scale past the first few weeks. Someone reads twenty outputs, forms an impression, and the impression is neither comparable to last week's nor defensible to anyone else.

Automation replaces the impression with a number. The design question is which parts can be automated honestly, and which cannot.

Automate in this order#

1. Structural checks. Does it parse? Are fields present and correctly typed? Is the arithmetic consistent? Deterministic, instant, and they catch most real breakage. If you automate nothing else, automate these.

2. Regression cases. Every production failure, re-run on every change. Cheap and compounding.

3. Refusal and safety behaviour. Should-refuse cases, permission boundaries, known injection attempts. Binary outcomes, so no judge required.

4. Rubric scoring. Semantic quality via a model judge. Useful, requires calibration, and belongs last because it is the least reliable.

Not automated: exploratory testing, usability judgement, and deciding what "good enough" means. Automation checks what someone already thought of.

The pipeline#

cases/ ──▶ runner ──▶ outputs/ ──▶ scorers ──▶ report
  │                       │           │           │
fixed set             archived    structural    per-case
+ metadata            for later    + rubric      deltas

Two design choices matter more than the rest.

Archive the outputs, not just the verdicts. When you investigate a regression six weeks later, a pass/fail record tells you nothing. The output tells you everything.

Report per-case deltas. An aggregate that holds steady while three cases break and three improve is exactly how a regression ships. The aggregate is for the dashboard; the diff is for the human.

Model-as-judge: where it works#

Works well for narrow, closed questions:

• "Is this claim supported by the text below? yes/no"
• "Does this output contain personal data? yes/no"
• "Does this answer address the question asked? yes/no"

Works poorly for broad quality: "rate this 1–10", "is this good?" Scores drift between runs and between versions.

Rules that make it trustworthy:

1. Use a different model from the one being tested. A model grading its own output shares its blind spots.
2. Pin and version the judge. Changing it changes your results — it is a model too.
3. Calibrate against humans on a sample before trusting it, and re-calibrate periodically.
4. Keep the rubric to three or four dimensions. Long rubrics score inconsistently.
5. Ask for a verdict and a reason. The reason is what lets you spot a judge that is wrong.

Keeping it fast#

A suite that takes an hour gets skipped under deadline pressure, which is precisely when it matters.

• Parallelise. These are independent API calls; run them concurrently up to your rate limit.
• Cache on prompt + model + input. If none changed, the result cannot have.
• Tier it. Structural checks on every commit; full semantic suite nightly.
• Prune. A case that has never failed and plausibly never will is costing money on every run.
• Fail fast on structural. No point paying a judge to score malformed JSON.

What to put in CI#

That last row is the one teams forget to wire up, and model changes are the most common cause of silent degradation.

The metrics worth alerting on#

• Pass rate drop on the golden set — block
• Consistency drop — 10/10 becoming 7/10 is a real regression even at the same nominal result
• Cost per completed task rising at flat quality — something is thrashing
• Refusal accuracy falling — it now answers things it used to decline. Dangerous and rarely watched
• Latency p95 — user-visible before quality is

Building it incrementally#

You do not need a platform. A workable first version is a folder of JSON cases, a script that runs them, and a second script that scores structurally. That is an afternoon, and it is already more than most teams have.

Add rubric scoring when structural checks stop finding things. Add CI integration when the suite is trusted enough that people will act on a red build. Order matters: a suite wired into CI before it is trusted just teaches people to bypass CI.

FAQ#

What framework should we use?#

Start without one. The mechanics are a loop, an API call and some assertions. Adopt a framework once you know what you need from it — most teams that pick one first end up fighting its assumptions.

How do we handle flaky cases?#

Report the pass rate rather than a binary. A case passing 7 times in 10 is information about the system, not noise to be suppressed — usually it means the prompt is underspecified.

Can this run without hitting a live API?#

Structural checks on archived outputs, yes. Anything measuring current behaviour needs live calls. Cache aggressively and tier the suite so the expensive part runs less often.

How much should an evaluation suite cost to run?#

Small relative to production spend. If evaluation costs more than a few percent of your inference bill, you are running too many cases too often — prune, cache and tier.

Who owns the suite?#

QA owns the standard and the thresholds; engineering owns keeping it green. What does not work is suite ownership sitting with nobody, at which point it decays into a red build everyone ignores.

What else is coming for Test Automation