Red Teaming AI: Adversarial Testing as a Discipline

Automated tests check what someone already thought of. Red teaming exists to find what nobody did.

For AI systems this matters more than for conventional software, because the failure modes are genuinely novel and the people who built the system are the least likely to imagine them — they know how it is supposed to be used.

What red teaming is and is not#

It is: a structured, time-boxed attempt to make the system behave badly, by people trying to break it rather than confirm it works.

It is not: running the security test suite again. If a case is already automated, it is regression testing. Red teaming is the search for the next category.

The output should be a regression suite, not a report. A report gets read once. Every finding turned into a permanent test case protects you forever, and that is the only durable product of the exercise.

Scope it before you start#

An unscoped red team produces a long list of things that do not matter.

Define:

The system boundary. Which components, which tools, which data.

What "bad" means here. Not generic misbehaviour — specific to this system. Disclosing another customer's data. Taking a financial action without approval. Producing advice that creates liability. Write these down first.

Rules of engagement. Test environment or production? Real customer data or synthetic? What is out of bounds? Who is told, and when?

Duration. Time-boxed. Two focused days beats an open-ended exercise that never concludes.

Who should do it#

Not the people who built it. They test the paths they designed.

Good compositions mix: someone who understands the domain and knows what would be damaging; someone with a security background; someone who has never seen the system; and — often the most productive — someone who is genuinely sceptical that it works.

The domain expert matters more than people expect. A security specialist finds injection. A claims handler finds that the system approves a claim it should have escalated, which is the finding that costs money.

Method#

Start from consequences, not techniques. Ask "what is the worst thing this system could do?" then work backwards to how you would cause it. Technique-first red teaming finds technique-shaped problems.

Work in pairs. One drives, one records. Findings are lost otherwise.

Record everything, including near-misses. A prompt that almost worked is a lead for the next session and a test case in its own right.

Escalate progressively. Start with plausible misuse, move to deliberate attack, then to combinations. Many real failures need two things to line up.

Try the boring attacks. Asking directly for something it should refuse works more often than elaborate jailbreaks.

Directions worth covering#

Recording a finding usefully#

A finding nobody can reproduce is a story. Each needs:

• Exact input, verbatim
• What happened, verbatim
• Why it matters — consequence, not just surprise
• Reproducibility: how many attempts out of how many
• Severity, based on what an attacker could cause

That reproducibility figure matters. Something that works once in twenty is different from something that works every time, and the fix differs accordingly.

Afterwards#

1. Triage by consequence, not novelty. The clever finding that causes nothing ranks below the dull one that discloses data.
2. Fix architecturally where possible. Most findings are excessive agency wearing a costume, and the fix is narrower tools rather than better prompting.
3. Every finding becomes a permanent test case. This is the point of the exercise.
4. Re-run the suite on every model change — resistance is a property of the model.
5. Schedule the next one. Quarterly for anything customer-facing. New capabilities create new categories, and your automated suite only knows about the old ones.

FAQ#

How often should we red team?#

Quarterly for customer-facing systems, and always after adding a tool or capability. Adding a tool widens blast radius, which is a bigger change than altering behaviour.

Can we automate it?#

The findings, yes — they belong in the regression suite. The search for new categories does not automate well, because automation only explores what it was told to. Automated adversarial generation is a useful supplement and not a replacement.

Do we need external specialists?#

Helpful and not essential to start. Internal people from another team, given clear scope and permission to be adversarial, find a great deal. Bring in specialists once the obvious layer is covered.

What if we find something serious mid-exercise?#

Stop and report immediately — do not save it for the write-up. Agree that rule before starting, so nobody hesitates.

Is red teaming worth it for a small internal tool?#

Scale it down, do not skip it. Half a day with two people asking "what is the worst this could do" finds most of what a week would, at the point where fixing it is cheap.

What else is coming for Red Team