Benchmarking AI Systems: Comparable, Reproducible Measurement

A benchmark is a measurement designed so that two results can be compared. Most AI benchmarking fails that test — not because the numbers are wrong, but because the conditions were not controlled enough for the comparison to mean anything.

If you cannot say exactly what was held constant, you have a number, not a benchmark.

What has to be pinned#

Every one of these changes the result. Leave any unpinned and your comparison is invalid:

That last row is the one people omit. A benchmark result without a date is not reproducible even against the same version string.

Why published benchmarks mislead#

Contamination. As a benchmark ages, its problems appear in training data. Scores rise without capability rising. A model scoring well on a two-year-old benchmark may simply have seen it.

Peak conditions. Published figures come from careful harnesses at settings you will not reproduce.

Different harnesses, same name. Two sites reporting "SWE-bench Verified" may prompt, parse and score differently. Scores are not comparable across sources — only within one.

Optimisation pressure. Anything used for marketing gets optimised for, sometimes at the expense of capabilities nobody measures.

Task mismatch. The deepest problem. A benchmark measuring GitHub issue resolution says nothing about your invoice extraction.

Public benchmarks are useful for one thing: ruling models out. They cannot rule one in for your task.

Building one that decides something#

Fix the input set and never change it silently. If you add cases, that is version 2 and results are not comparable to version 1. Say so in the report.

Run each case multiple times. Five is usually enough to expose variance. Report the distribution, not just the mean — a model that averages 80% by scoring 100/100/40/100/60 is not the same as one scoring 80 five times.

Randomise order if there is any state between runs.

Score deterministically wherever possible. Every check you can make mechanical is a check that does not drift.

If a model scores, pin and version the scorer. Changing the judge changes the results, and it is easy to forget the judge is also a model.

Include cost and latency. A benchmark measuring only quality will select an expensive, slow model for a task where a cheap one sufficed. Report cost per completed task and p50/p95 latency alongside quality, always.

Reporting so it can be trusted#

A result nobody can reproduce is an opinion. Publish, at minimum:

Benchmark:    invoice-extraction v3
Date:         2026-08-04
Cases:        40  (30 typical, 6 edge, 4 should-refuse)
Runs:         5 per case
Model:        <exact version string>
Temperature:  0.2
Prompt:       v7 (commit abc123)
Scorer:       deterministic (schema + arithmetic) + rubric v2

Result:       structural 100%  ·  semantic 87% (σ 4.1)
              refusal accuracy 90%
              cost $0.014 per completed task
              p50 1.2s / p95 3.8s

Anyone reading that can tell what was measured and whether it applies to them. A single headline percentage cannot.

Comparing fairly#

Same inputs, same scorer, same runs. Obvious and routinely violated.

Do not tune one candidate more than another. If you spent a day on model A's prompt and ten minutes on model B's, you have benchmarked your effort.

Report the ties as ties. When two candidates are within the noise, say so rather than declaring a winner on a tenth of a point.

Start from the cheapest. Model pricing spans a thousand-fold range; quality on well-specified tasks spans far less. The question is which is the cheapest that passes — see Model Evaluation.

When to re-run#

On model version change, on prompt change, on scorer change, and on a schedule regardless. A benchmark run once is a snapshot; a benchmark run repeatedly is a control.

FAQ#

How is this different from evaluation?#

Evaluation asks "is this good enough for my task?" Benchmarking asks "how do these compare under identical conditions?" Evaluation has a threshold; benchmarking has a ranking. Most teams need evaluation more than they need benchmarking.

Should we publish our benchmark results?#

Only with the full method, and only if you would be comfortable with someone reproducing it. A published number without conditions attached is marketing, and it invites reasonable people to call it that.

Can we trust a vendor's benchmark?#

Treat it as directional and check what was pinned. If the report omits temperature, prompt, run count or date, it cannot be reproduced — which does not make it false, but does mean it is not evidence.

How many runs is enough?#

Five per case is a reasonable default. If variance is high, more. If a result changes your decision, run enough that the confidence interval does not overlap the alternative.

Our benchmark says A wins but users prefer B. Which is right?#

The users. The benchmark is measuring something other than what they value — usually latency, verbosity or tone, none of which appear in a quality score. Add what they care about to the benchmark.

What else is coming for Benchmarking