AI Security: Automated Red Teaming and Monitoring

Status: not built. No tool, no product, no demo. This is an assessment of the problem and of why the obvious version of the solution is not worth shipping.

The problem with automated red teaming#

Automated red teaming against an AI system means generating adversarial inputs, running them, and judging whether the system did something it should not.

The generation part is easy. The judging part is the entire difficulty.

To decide automatically whether an output was a failure, you need a judge — usually another model. That judge has the same weaknesses as the system under test, can be manipulated by the same content, and produces confident verdicts either way. A red-teaming tool that reports "0 successful attacks" is indistinguishable from one that is failing to recognise successful attacks.

This is why we have not built one. A security tool that gives false assurance is worse than no tool, because it converts an unknown risk into a believed-safe one.

The threat categories worth knowing#

Prompt injection remains the defining problem. Any system that reads content — web pages, documents, tickets, emails, tool output — can receive instructions written for it. In a chatbot the damage is a bad answer; in an agent holding tools, the damage is an action. There is no reliable filter, because the attacker also writes text.

Indirect injection is the harder form: the attack is not in the user's message but in a document the system retrieves later. Testing the user-facing input misses it entirely.

Data exposure through retrieval. If retrieval is not filtered by the requesting user's permissions at query time, the system is a search engine over everything, exposed through a chat box. This is a breach, not a quality issue, and it is one of the most common real findings.

Agent-specific risks — goal hijacking, tool misuse, memory and context poisoning — are recognised distinctly from the general model risks, and the published guidance now separates agentic application risks from the older model-focused lists. If you are running agents, read both.

Model output as an injection vector into your own systems. Output rendered into a page, passed to a shell, or used to build a query is untrusted input. This is an old class of vulnerability wearing new clothing, and it is frequently overlooked because the output came from your own system.

What actually reduces risk today#

Not a scanner. Architecture:

Reduce blast radius. Sort tools by reversibility. Reading is cheap; sending, paying, deleting and changing infrastructure are different in kind. Irreversible actions need confirmation.

Do not let read content trigger tools directly. The dangerous combination is: reads untrusted content, holds a consequential tool, acts without confirmation. Break one of the three.

Filter retrieval by the requesting user's permissions, at query time. After generation is too late.

Give the system its own scoped, revocable credentials, never a shared or human account, and log every use.

Enforce caps outside the system. Spend, steps, rate. An agent can reason past its own instructions; it cannot reason past a supervisor that refuses the call.

Treat output as untrusted. Escape it, validate it, never pass it unchecked into a shell, a query or a page.

Each of these is a design decision, which is why they are cheap early and expensive later.

What automated testing IS good for#

Not a security verdict, but genuinely useful as regression detection:

Known-attack regression suites. Once a specific attack has worked against your system, encode it and re-run it forever. This is high-value and unambiguous — the attack either reproduces or it does not, so no judge model is required.

Permission tests. Deterministic and important: does user A's query ever retrieve content only user B should see? This can be tested reliably, and it catches the highest-severity class.

Refusal consistency. Does the system refuse the same category consistently across runs? Varies more than teams expect.

Output handling. Whether generated content is escaped correctly is ordinary application security testing and fully automatable.

The pattern: automate what has an objective answer, and use humans for what requires judgement.

What we would need before building a tool#

• A judging approach whose failure rate we could measure, not assume
• A benchmark of real attacks with known outcomes to calibrate against
• Clear separation between "we tested this and it held" and "we found nothing"
• An honest report format that never states a system is secure

The third is where most tools in this space are weakest, and the fourth is a commercial problem as much as a technical one — a report saying "no findings, which may mean no vulnerabilities or may mean our coverage missed them" is harder to sell than a green tick.

FAQ#

Do you offer AI security testing?#

Not as a product. This page describes why the automated version is harder than it appears. For independent evaluation of AI systems more broadly, see AI Testing.

Can prompt injection be fixed?#

Not eliminated with current architectures. It is mitigated by design: limit what the system can do in response to content it reads, require confirmation for consequential actions, and keep untrusted content clearly separated from instructions. Anyone claiming a filter that solves it is describing something the attacker gets to write around.

Is a guardrail model enough?#

It raises the bar and it is not a boundary. Guardrail models are themselves models — they can be manipulated, and they fail silently. Use them as one layer, never as the control that permits a risky capability.

What is the single highest-value control?#

Permission-filtered retrieval, for systems over internal documents. It is the difference between an assistant and an unintended search engine over everything the organisation holds — and it is a breach when it goes wrong, not a bug.

How do we test our own system without a tool?#

Start with the four automatable checks above, add every real attack you encounter to a regression suite, and get a person to spend a focused day trying to make it misbehave. That day usually outperforms a scanner.

Should we run a bug bounty for our AI system?#

Only once the basics are in place, or you will pay for findings you could have prevented. When you do, scope it explicitly — AI systems produce a long tail of "made it say something odd" reports that consume triage capacity without reducing risk.

Are the published risk lists worth following?#

Yes, as a checklist of categories rather than as a strategy. Note that agentic application risks are now treated separately from model risks; if you run agents, the agent-specific list is the more relevant one.

Related Articles#

See cyber security for the organisational controls, AI agents for bounded autonomy, and RAG for permission-filtered retrieval.

What else is coming for AI Security