Model Evaluation: Choosing a Model With Evidence

Model selection is usually made on a leaderboard, a demo, or whichever provider the team already has an account with. All three are guesses.

The alternative takes an afternoon: build a small evaluation set from your own work, run every candidate against it, and pick the cheapest one that passes.

Why public benchmarks do not answer your question#

They measure tasks that are not yours. SWE-bench measures resolving GitHub issues in open-source Python. If your task is extracting fields from invoices, that score tells you nothing useful.

The gaps between leaders are inside the noise. When two models sit a tenth of a point apart, choosing between them on that basis is choosing on nothing.

Benchmarks leak. As a benchmark ages, its problems appear in training data. Scores rise without capability rising.

They report peak, not typical. Published figures come from careful harnesses at settings you will not reproduce.

The one thing benchmarks are genuinely useful for: ruling models out. A model far below the others on reasoning is unlikely to surprise you on your reasoning task.

The evaluation set#

Twenty to fifty cases from real work. Each needs an input and a definition of a correct outcome.

Include the hard cases deliberately:

That third row is the one that compounds. Every production failure becomes a permanent case, and after a year the suite is precisely shaped to where your system actually breaks.

Scoring#

Deterministic checks first. Does it parse? Are required fields present? Does the arithmetic hold? Is there no personal data? These are cheap, fast, and catch most real breakage.

Rubric scoring for the rest. Three or four dimensions — faithfulness, completeness, tone. Longer rubrics score inconsistently, including when a model does the scoring.

Always a pass rate, never a single run. Run each case several times. Variance is data: a model that is right 100% at temperature 0 and 70% at your production setting is not the model you thought you were choosing.

Comparing candidates properly#

Run every candidate through the identical suite and tabulate:

Start from the cheapest and move up only when something fails. Model prices span a thousand-fold range while performance on well-specified business tasks spans far less. For classification, extraction, routing and summarisation, the difference between a frontier model and a mid-tier one is frequently invisible and the cost difference is not.

The right question is not which model is best. It is what is the cheapest model that passes.

The criteria that are not on any leaderboard#

Often decisive:

• Data handling terms — is your input used for training? This alone rules out options for regulated work
• Latency, which matters more than accuracy for anything interactive
• Rate limits at your actual volume
• Regional data residency
• Deprecation policy — how long do you have when a version is retired?
• Context window, if you genuinely need it

Re-evaluation#

A model version change is a change requiring a full run. Providers update models, sometimes silently. A prompt tuned to one version can behave differently on the next.

Therefore: pin the model version, and treat an upgrade as a code change that must pass the suite before shipping.

Re-run also when your data distribution shifts, when a new failure mode appears in production, and on a schedule regardless — quarterly is a reasonable floor.

The organisational payoff#

A team with an evaluation suite can switch models in an afternoon when the market moves, and knows immediately whether the switch was an improvement.

A team without one re-litigates the choice every quarter on vibes, and cannot tell whether last month's change helped.

That capability — not the current model choice — is the durable asset.

FAQ#

How many cases do we need?#

Twenty is enough to be useful; fifty is comfortable. Quality matters far more than quantity — twenty real cases covering your actual edge conditions beat two hundred synthetic ones.

Can a model evaluate another model's output?#

For narrow, specific questions, yes and it works well. For broad quality judgements it is unreliable. Use a different model than the one being tested, and calibrate against human scoring on a sample before you trust it.

Should we fine-tune instead of switching models?#

Rarely. Fine-tuning teaches style and format, not facts. If the problem is that the model does not know your information, the answer is retrieval — see RAG Testing — not training.

How do we compare cost fairly?#

Cost per completed task, including retries and failures. Cost per call flatters models that need several attempts, and per-token pricing hides the model that needs a longer prompt to work.

What if the cheapest model passes everything?#

Use it, and consider that your evaluation set may be too easy. Add harder cases from real failures — but do not upgrade the model just because a cheaper one succeeding feels wrong.

What else is coming for Model Evaluation