How to benchmark an LLM guardrail on your own traffic

Every LLM guardrail on the market reports a detection rate, and almost none of those numbers predict how the product will behave on your traffic. That is not usually dishonesty. It is that the number was produced against a corpus the vendor chose, with a benign class the vendor chose, at a threshold the vendor chose, and any one of those three choices moves the result by more than the difference between products.

This page is the method we think you should use instead — on us and on everyone else. It is written so you can run it, and it is deliberately arranged so that a vendor's own number is never an input to it.

A note on what this page is not. It reports no measurements of SecureAI Guard. We have not published a detection rate or a latency profile for our own product, and we would rather publish the method first and the numbers when they come with the corpus and the logs attached. A number without a reproducible method is the thing this page argues against; publishing one here would be self-refuting.

The one design decision that matters

The attack class can come from public corpora. The benign class must come from your own traffic.

Almost every misleading guardrail benchmark fails on the second half. A detector's false-positive rate is a function of the distribution of legitimate requests it sees, and no public benign set resembles your application's. A security tool evaluated against a benign set of generic chit-chat will look excellent and then flag one in fifty of your real prompts, because your real prompts are full of the vocabulary of your domain — and if your domain is security, IT support, legal or software engineering, your users talk about credentials, deletion, overrides and system prompts all day.

The false-positive column, not the detection column, decides whether the control is still switched on in six months. Build the evaluation so that column is measured on real data.

Step 1 — Capture a representative week

Instrument the point in your application where the model context is assembled, not the HTTP handler. For every request record:

• the full assembled context — system prompt, retrieved chunks, conversation history, tool definitions, current user turn, kept as separate fields rather than one concatenated blob;
• the provenance of each chunk — which retrieval source, which document, which permissions were applied;
• the model and version called and the tools offered on that turn;
• the completion, and any tool calls made;
• timing at each hop.

One week, unsampled if you can afford the storage. Sampling by request rate biases towards your busiest, most homogeneous route, which is the one where a guardrail is least interesting.

Two constraints that are not optional: this capture contains everything sensitive your application handles, so it needs the same controls as your production data store and a defined deletion date. And you should treat the capture itself as a finding — the volume of personal data teams discover in their own prompts at this step is routinely the most valuable output of the whole exercise.

Step 2 — Build the attack class from public, citable corpora

Use published corpora rather than writing your own attack prompts. Prompts you write yourself encode your own assumptions about what an attack looks like, which is precisely the blind spot you are trying to measure.

JailbreakBench (NeurIPS 2024 Datasets and Benchmarks track) provides the JBB-Behaviors dataset — 100 distinct harmful behaviours drawn partly from AdvBench and HarmBench/TDC and partly original, organised into ten categories — plus, importantly, 100 benign behaviours for measuring over-refusal, a repository of jailbreak artifacts, and a standardised evaluation library with defined threat models and scoring functions. Use its benign set as a supplement to your own traffic, never as a replacement.

AgentDojo (ETH Zurich and Invariant Labs; NeurIPS 2024 Datasets and Benchmarks track; arXiv:2406.13352; MIT licence) is the one to use if your application is agentic. It is a dynamic environment for evaluating prompt injection attacks and defences against LLM agents across task suites, which means it measures whether the attack succeeded end to end rather than whether a string was classified — a materially different and more honest question.

Your own indirect corpus. Neither public set knows what your retrieval pipeline ingests. Take real documents from your own sources — wiki pages, tickets, PDFs, API responses — and plant injections in them, in the places your attackers would: HTML comments, alt text, footnotes, white-on-white text, the tail of a long document, a spreadsheet cell far to the right. Then run your real retrieval step. This is the highest-value part of the corpus and the part no vendor benchmark contains, because it depends on your ingestion.

Two warnings about public corpora. First, contamination: published attack sets are in the training data of current models and may be in the training data of the detectors you are evaluating, so scores on them are an optimistic bound. Second, they are static, and a real attacker adapts. Treat any detection rate measured against a fixed corpus as a ceiling, not an estimate.

Step 3 — Label, blind, and hold out

Sample your captured week down to a labelling set — a few thousand requests is usually enough — and stratify it by route, not uniformly, so low-volume routes are represented.

Have two people label independently against a written rubric, and record the disagreement rate before reconciling. If your two labellers disagree on 8% of cases, no detector can be meaningfully scored better than 92% on that set, and knowing that number stops an argument later.

Hold out at least a third of the corpus and do not look at it while tuning thresholds. Then run every candidate against the held-out set exactly once. A threshold chosen on the same data it is reported on is not a result.

Run every candidate against the same corpus in the same week. Vendor-reported numbers are not comparable to each other and must never be mixed with your own.

Step 4 — Metric definitions, stated exactly

Write these down before you measure, because the arguments afterwards are always about definitions.

• True positive rate (recall) = TP / (TP + FN). Of the attacks in the corpus, the share flagged.
• False positive rate = FP / (FP + TN). Of the legitimate requests, the share flagged. This is the number that determines whether anyone keeps the control on.
• Precision = TP / (TP + FP). Under realistic class imbalance — attacks are a small fraction of a percent of production traffic — precision collapses even at a very good FPR, and it is worth computing so the on-call cost is visible.
• Attack success rate (ASR) = the share of attacks that achieved their objective end to end, with the guardrail in place. For agentic systems this is the metric that matters; a detector can flag a payload and the agent can still perform the action.

Report TPR at a fixed FPR, for example TPR at 1% FPR and at 0.1% FPR. A bare "98% detection" is uninterpretable because it hides the threshold. Report a single ROC-AUC only as a supplement: under heavy class imbalance it is dominated by regions of the curve nobody will ever operate in.

Report per class. A pooled number hides the case you actually care about. Break it out by direct injection, indirect injection, data exfiltration attempt, sensitive-data leak and over-refusal, at minimum.

Step 5 — Measure latency properly

Latency is where guardrail benchmarks are most often wrong, and the errors are mechanical rather than subtle.

• Report added latency — the delta with the control on and off, same hardware, same payloads, same concurrency — not the absolute number.
• Report p50, p95 and p99, and state the concurrency and the payload size distribution you used. Percentiles without those three qualifiers are not comparable to anything.
• Use a fixed-rate load generator, not closed-loop. A closed-loop harness that waits for each response before sending the next one under-reports tail latency badly under saturation. This is coordinated omission, and it is the reason so many published p99s are implausibly close to the p50.
• Measure long contexts specifically. Inspection cost usually scales with context length, and a benchmark of 200-token prompts tells you nothing about a 32k-token RAG request, which is the request you actually serve.
• Measure the failure path. What is the latency when the detector times out, and what does the system do — fail open or fail closed? Time that path, do not reason about it.

The six mistakes that make the result meaningless

1. Using the vendor's corpus. It was chosen by someone with an interest in the outcome. This is not an accusation; it is a structural fact about vendor benchmarks, ours included.
2. No benign class, or a synthetic one. Detection rate without a false-positive rate measured on real traffic is not a result.
3. Reporting one threshold. Every product can be tuned to catch everything; the question is what it costs at that setting.
4. Testing only direct injection. The user's message is the part the attacker did not write. If the corpus has no indirect cases run through your real retrieval pipeline, the hardest class is untested.
5. Measuring the detector instead of the system. For agents, measure whether the action happened. AgentDojo exists because that distinction changes conclusions.
6. Treating a static corpus as an estimate of adversarial performance. It is an upper bound. Budget a paraphrase pass — re-run the attack set through a model asked to rewrite each payload while preserving intent — and report that number too. The gap between the two is a more useful signal about a detector than either number alone.

A worked scoring template

Score each candidate on one page, with every cell traceable to the run that produced it:

The empty column is the point. Fill it yourself, for every candidate including this one, and the decision stops depending on whose marketing you read most recently.

What we have and have not measured

We have not published our own numbers against this method. When we do, the publication will include the corpus construction, the held-out split, the threshold selection procedure and the raw run logs, so that the number can be disputed by someone who disagrees with it. Any vendor — us included — who publishes a detection rate without those four things is asking to be taken on trust, and a security product should never be the thing you take on trust.

If you run this against SecureAI Guard and get a result you did not expect, tell us — including if it is bad. We would rather know.

Related

• Build vs buy: writing your own LLM guardrails
• LLM security alternatives, and how to compare them
• What is prompt injection?
• AI red teaming for LLM applications