AI red teaming for LLM applications
How to red team an LLM application rather than a model: scope it to your system, build the test set from ATLAS and OWASP, and measure something durable.
AI red teaming is adversarial testing of an AI system: deliberately trying to make it do the thing it is not supposed to do, and recording what worked.
The word has been stretched to cover two quite different activities, and conflating them wastes most of the budget spent on it.
- Model red teaming probes a model's safety behaviour — will it produce disallowed content. This is largely the provider's job, and repeating it tells you little about your deployment.
- Application red teaming probes your system: your prompts, your retrieval corpus, your tools, your permissions, your output handling. This is where your risk actually is, and it is the exercise worth running.
If a proposal you are reading is entirely about getting a model to say something offensive, it is testing someone else's product.
Scope it to the system, not the model
Write the scope as a list of things that must not happen, in your system's own terms. For example:
- A user must not be able to retrieve content they cannot open in the source system.
- Text in an indexed document must not be able to cause a tool call.
- The system prompt and tool schemas must not be recoverable.
- No request may exceed the per-tenant token budget.
- Model output must not reach the browser, the database or the shell without the same handling as any other untrusted string.
Each of those is testable, has a pass/fail, and stays meaningful when the underlying model changes. "The model should be safe" is none of those things.
Build the test set from published taxonomies
Two sources, used for different purposes:
- MITRE ATLAS catalogues real-world adversary tactics and techniques against AI-enabled systems, in the style of ATT&CK. Use it to enumerate techniques so your test set has coverage rather than whatever the tester thought of on the day.
- The OWASP Top 10 for LLM Applications gives you the risk classes to organise findings against, which is what makes the report legible to people who were not in the room.
Then add the two categories that only you can write: your own domain abuse cases, and your own indirect-injection channels — every content source your system will read that someone outside your organisation can write to.
Automate the regression, keep humans for discovery
Automated harnesses generate and mutate attack prompts at volume, and they are genuinely useful for two things: establishing a baseline attack success rate, and catching regressions when a prompt, a model version or a retrieval change ships. Wire them into CI and treat a rise in success rate like a failing test.
They are weak at novelty. The findings that matter — a chain of three individually-harmless steps, an abuse of your specific business logic, a permission that is wrong in a way no generic prompt would reveal — come from a human who understands the domain. Budget accordingly: automation for coverage and repeatability, people for discovery.
Measure attack success rate, and say how you measured it
The metric worth reporting is attack success rate: successful attempts over total attempts, broken down by technique and by objective. Report it with the things that make it comparable — the model and version, the temperature, the number of attempts allowed per objective, and who judged success and how.
Three cautions:
- An ASR from a single attempt per prompt is not comparable with one that allowed fifty. Multi-attempt success rates are far higher, and multi-attempt is what a real attacker does.
- If a model judges success, the judge is itself manipulable and has its own error rate. Sample and check it against human judgement.
- Report per technique. A headline "3% success" that hides 60% on the one technique that reaches your payment tool is worse than no number.
Turn findings into controls, not into prompt edits
The most common bad outcome of an AI red team is a list of blocked strings and a longer system prompt. Those close the specific inputs found and nothing else, and they will be bypassed by the next paraphrase.
For each finding, ask what would have stopped the class: a narrower tool scope, an authorisation check in code, a confirmation step, an output-handling fix, a permission corrected at the source. Fix at that level, then re-run the original attack to confirm — and keep it in the regression set forever.
Cadence
At minimum: before first production release, and after any change to the model or version, the tool set, the retrieval corpus policy, or the system prompt. In practice a continuous automated suite plus a periodic human engagement is the shape that works, because model versions move underneath you whether or not you changed anything.