QA — Charter

QA exists to find out whether the thing works before customers do. In an AI-operated company the role gains a second, harder job: testing systems that do not give the same answer twice.

That is not a variation on existing testing. The usual contract — same input, same output — does not hold, so the techniques have to change.

What this role owns#

Quality strategy. What gets tested exhaustively, what gets a glance, and the reasoning. You cannot test everything; deciding deliberately rather than by running out of time is the job.

Release certification. The judgement that something is fit to ship, and the authority to say it is not.

AI evaluation. Whether models, prompts and agents behave — a distinct discipline from functional testing.

Test data. Realistic in shape and volume, lawful, and repeatable.

Escaped defect analysis. Every bug a customer found is information about the process, not just about the code.

KPIs#

Escaped defect rate is the one that cannot be gamed. Everything else can be made to look good.

Testing non-deterministic systems#

The part that is genuinely new, and where most teams have no method:

Test properties, not exact output. Does the JSON parse? Are required fields present? Do line items sum to the stated total? Is there no personal data in the response? These hold across all valid phrasings.

Score qualities against a short rubric. Faithfulness, tone, completeness — three or four dimensions. Long rubrics score inconsistently, including when a model does the scoring.

Measure a pass rate, not a single run. Run each case several times. A prompt that passes every time at temperature 0 and 60% of the time in production is a system nobody understands yet.

Test the refusals. Give it tasks it should decline. It should stop and say so, not invent a completion.

Every production failure becomes a permanent case. This single habit does more for long-run quality than any evaluator design.

Full method in the AI Testing Center and Prompt Testing.

AI agents in this function#

Test case generation agent — expands a requirement into positive, negative, boundary and non-functional cases. A test analyst decides what is missing.

Evaluation harness — runs the AI test suites on every prompt or model change, scores against the rubric, reports the delta per case rather than only the aggregate.

Exploratory assistant — proposes edge cases from the specification. Useful precisely because it suggests things nobody thought of.

Defect triage agent — classifies and routes incoming defects, flags duplicates.

What stays human: deciding what "good enough" means, release certification, exploratory testing, and judging usability. Automation checks what someone already thought of; people find what nobody did.

SOPs#

• Entry criteria — testing does not start until the build is deployed and smoke-passes. Starting early produces noise that discredits the effort.
• Exit criteria — agreed before testing, or "are we done?" becomes a negotiation.
• Defect severity — defined, with a response for each level.
• AI evaluation cadence — run on every prompt change, every model version change, and weekly regardless. A model upgrade is a change requiring a full run.
• Escaped defect review — for each: which control should have caught this?

Templates#

Test Plan · QA Checklist · Security Checklist · traceability matrix · evaluation rubric.

Workflows#

In: requirements and acceptance criteria · builds · defects from support · prompt and model changes.

Out: release certification or refusal · defect reports with reproduction steps · evaluation results per version · quality trend.

Handoffs: Engineering for fixes · PMO for release readiness · Security for findings with a security dimension · Support for known issues at release.

FAQ#

Can AI replace testers?#

It replaces a good deal of test execution and case generation. It does not replace deciding what matters, judging whether something is usable, or the adversarial instinct that finds what nobody specified. Those are the parts that were always the skill.

How do you test something that gives different answers each time?#

Assert properties rather than strings, score qualities against a rubric, and measure a pass rate across repeated runs. You are testing a distribution, not a value.

What is the biggest testing mistake?#

Too many end-to-end tests. They are slow, fail for environmental reasons, and a failure tells you something broke somewhere. Teams stop trusting them, then stop reading them.

Should QA be able to block a release?#

Yes, and it should be exercised rarely. A QA function that can never say no is a reporting function, and everyone learns to treat its output as advisory.

What else is coming for QA