AI Testing: Autonomous Test Generation and Evaluation

Status: methodology published, tool not built. We wrote the AI Testing Center — 13 topic guides with checklists, examples and sample reports. We have not built an autonomous test generation product, and this page explains why the obvious version is not worth shipping.

What automated test generation actually produces#

Point a model at a codebase and ask for tests, and you get tests. They compile, they pass, and coverage rises.

Read them and a pattern appears: they mostly assert that the code does what the code does. The model infers intent from the implementation, so a function with a bug produces a test asserting the buggy behaviour. Coverage went up, defect detection did not, and now the bug has a test protecting it.

That is the central difficulty. A test is a statement of intended behaviour, and intent is not recoverable from implementation alone. It lives in requirements, in conversations, in the head of whoever asked for the feature.

Where generation genuinely helps is where intent is available: given a specification, generate the cases — including the boundaries and error paths that humans reliably skip. That is a narrower and much more defensible use.

The judge problem#

Testing a system whose output varies between runs means you cannot assert equality against a golden output. So you need something to decide whether an output is acceptable — usually another model.

That judge has the same weaknesses as the system under test. It can be influenced by the same content, it is inconsistent between runs, and it produces confident verdicts either way. A test suite whose oracle is unreliable does not fail loudly; it reports a pass rate that means less than it appears to.

This is why we did not build a tool. A testing product whose core component is an unmeasured judge sells assurance rather than providing it.

Where LLM-as-judge is defensible#

It is not useless — it needs to be treated as an instrument requiring calibration:

Measure the judge against human labels first. Take a few hundred outputs, have people label them, and measure agreement. If the judge agrees with humans 70% of the time, every number it produces afterwards carries that error. Skipping this step is the standard mistake.

Judge narrow properties, not overall quality. "Does this answer contain a claim not supported by the provided sources?" is answerable. "Is this a good answer?" is not.

Use it for relative comparison rather than absolute scores. Which of two versions is better is substantially more reliable than a score out of ten.

Report the pass rate with the run count. A single run of a non-deterministic system is an anecdote.

Keep a human-reviewed set that never changes, so you can detect the judge drifting when its model is updated underneath you.

What actually works, and is boring#

The methods that hold up are unglamorous:

Property-based assertions. Not "the output equals X" but "the output cites only supplied sources", "the output is valid JSON matching this schema", "the refusal appears when the corpus cannot answer". These are deterministic and cheap.

A fixed task set, re-run. Twenty to a hundred representative tasks with known-good outcomes, run repeatedly, tracking the pass rate over time. The absolute number matters less than the direction.

Regression on real failures. Every bad output anyone reports becomes a permanent test case. Within months the suite reflects your actual failure distribution rather than an imagined one.

Measuring components separately. For retrieval systems, retrieval hit rate independent of generation. If retrieval is at 60%, the system's ceiling is 60% and no prompt work raises it — and an end-to-end score hides that entirely. See RAG.

Adversarial input as a fixed suite. Once an attack works, it is a test forever.

What we built instead#

The AI Testing Center: 13 topic guides, each with a checklist, a worked example and a sample report, covering evaluation design, judge calibration, regression suites, red teaming and reporting.

Methodology rather than a tool, because the tool would have been the easy half. The reason teams struggle here is not the absence of a test runner; it is not knowing what a defensible evaluation looks like — and a tool that answers the wrong question confidently makes that worse.

What we would need before building a tool#

• A judge whose agreement with human labels we measure and publish, not assume
• Output that distinguishes "tested and held" from "we found nothing"
• Deterministic property checks as the foundation, with model judgement as a labelled layer above
• A report format that never states a system is correct

The last two are what most tools in this space get wrong, and the fourth is commercially awkward: "our coverage found nothing, which is not the same as nothing being there" does not sell as well as a green tick.

FAQ#

Do you offer an AI testing tool?#

No. We publish the methodology free. There is no product.

Can AI write my unit tests?#

It can write a lot of them quickly, and they will mostly assert current behaviour rather than intended behaviour. Useful for coverage of boilerplate; not a substitute for tests written from a specification. Always confirm a generated test fails against broken code — a surprising share do not.

How do you test something that answers differently every time?#

Assert properties rather than exact outputs, run a fixed task set repeatedly, and report a pass rate with the number of runs. Reserve model judgement for narrow, calibrated questions.

Is LLM-as-judge reliable?#

Only as reliable as its measured agreement with human labels — and almost nobody measures it. Do that first, keep the questions narrow, and prefer comparisons to absolute scores.

What is the single most valuable thing to measure?#

For retrieval systems, retrieval hit rate separately from answer quality. It sets the ceiling, and an end-to-end metric hides it. For agents, whether the caps and stopping conditions actually fire when tested.

How large should the evaluation set be?#

Start with twenty real cases with known answers and grow it from reported failures. Twenty representative cases beats two hundred invented ones, because the invented ones reflect what you imagined rather than what users do.

Why publish the methodology instead of selling a tool?#

Because the methodology is the scarce part, and a tool built on an uncalibrated judge would undermine the thing it claims to provide.

Related Articles#

Work through the AI Testing Center for the full methodology, testing for general principles, and AI agents for evaluating systems that act.

What else is coming for AI Testing