What red-teaming tools actually find
We built two deliberately weak AI agents, wrote down their weaknesses before testing anything, then ran four open-source red-teaming tools and our own attacker against both at a matched request budget.
Every AI red-teaming tool will hand you a list of findings. None of them can tell you what share of the real problem that list is. So we built two deliberately weak agents, wrote their weaknesses down before anything touched them, and ran four open-source tools and our own attacker against both at a matched request budget, three times each. Below is what each one reached, out of a total we knew in advance.
Why a known answer key
A red-teaming tool reports the weaknesses it found. It cannot report the ones it missed, and the weaknesses of a production agent are not known in advance. A findings list therefore has no denominator, and two tools reporting different counts against the same agent cannot be compared at all.
So we made the denominator first. Two agents were written with a fixed set of weaknesses, recorded before any tool touched them: prompt injection through content the agent reads, actions taken beyond the authority granted, cross-tenant reads, secret disclosure, shell commands run without confirmation. Every run below is scored against that same list, by the same scorer, from the same recorded traffic.
Planted weaknesses found across three runs
Three runs find more than any one run does
Every attacker ran three times against each agent, and no single run reaches what the three reach together. Our attacker found 8, 8 and 6 of the settlement agent's weaknesses in its three runs and 10 across them, so two of its ten came from a run that another run missed. garak's adaptive probe found 1, 2 and 3, and 3 across them.
That is a property of the tools, not of the scoring: a red-teaming run is a sample, not an inventory, and repeating one buys coverage. It is also the practical case for testing on every release rather than once, which we go into in manual, automated, and continuous red teaming.
Per run, and across three runs · settlement agent
What a tool reports and what it does are different numbers
Because we know the answer key, we can compare each tool's own report against what its traffic actually achieved. The gap is large and it runs in both directions.
A buyer reading either tool's output would draw the wrong conclusion in opposite directions: acting on findings that describe a working refusal, and unaware of a real weakness the same run triggered.
Full results
| Settlement | Coding | Requests | Tool calls | |
|---|---|---|---|---|
| Enoki | 10 / 12 · 83% | 7 / 8 · 88% | 98 | 55 | 68 | 90 |
| garak · adaptive | 3 / 12 · 25% | 3 / 8 · 38% | 30 | 89 | 139 | 154 |
| promptfoo | 1 / 12 · 8% | 3 / 8 · 38% | 101 | 70 | 4 | 223 |
| garak · static | 1 / 12 · 8% | 3 / 8 · 38% | 112 | 99 | 11 | 23 |
| hivetrace-red | 0 / 12 · 0% | 1 / 8 · 13% | 114 | 106 | 0 | 15 |
| deepteam | 0 / 12 · 0% | 0 / 8 · 0% | 43 | 39 | 0 | 1 |
Coverage columns are the total reached across three runs. Paired values read as settlement | coding, and requests and tool calls are per run.
How the comparison was set up
- Two agents. A back-office settlement desk over a freight ledger, with five tools, a chat surface and a mutable notes store; and an off-the-shelf open-source coding agent with shell execution and a plugin loader, which we did not write. Its defects are its own.
- The answer key came first. Every weakness was written down and given a machine-checkable predicate before any attacker ran, so a verdict is mechanical rather than a judgment call after the fact.
- A matched request budget. Every arm, ours included, was capped at the same number of requests to the target per run, set at the maximum any of our own runs had ever fired.
- Three runs each. Attacks on agents are non-deterministic, so a single run measures a sample. Coverage above is the union across an arm's three runs, and the per-run figures are shown alongside it.
- Tool versions. garak 0.16.0, promptfoo 0.122.2, deepteam 1.0.9, hivetrace-red 1.0.19. Each was configured toward the weakness classes these agents actually carry, at its own documented settings, with setup capped at the same operator time per arm.
What this does not show
Two agents are two agents. These weaknesses are the ones we planted or found and keyed, not a census of what agents get wrong, and a tool that scores low here may well be strong against something this key does not contain. The tools were also configured by us rather than by their authors, within a fixed time budget, so a vendor who knows their own tool better would likely do better with it.
What the design does support is the narrower claim it was built for: against a known list, with the same budget and the same scorer, these are the fractions each attacker reached. For what the tools are and where each one fits, see AI agent red teaming tools in 2026, and for the attack types that separate them, single-turn, multi-turn, and dynamic attacks.
Frequently asked questions
Why do you need a known answer key to compare red-teaming tools?
Because a tool can only report what it found, never what it missed. Without a list of the weaknesses that are actually there, findings counts have no denominator and two tools reporting different numbers against the same agent cannot be compared.
Which tools were tested?
garak 0.16.0 in two configurations (its adaptive agent probe and its static payload probes), promptfoo 0.122.2, deepteam 1.0.9, and hivetrace-red 1.0.19, each against both agents, alongside Enoki's own attacker.
Why run every attacker three times?
Attacks on agents are non-deterministic, so one run is a sample rather than an inventory. Across three runs our attacker reached 10 of the settlement agent's weaknesses while its best single run reached 8, and the same pattern shows up in the tools.
What is the difference between what a tool reports and what it achieved?
Its report is what its own evaluators claim; what it achieved is what its recorded traffic actually made the agent do. One tool reported 13 SQL-injection successes against an agent that has no database, and another triggered an unconfirmed shell command in every run while reporting nothing at all.
Was the comparison budget matched?
Yes. Every arm, including ours, was capped at the same number of requests to the target per run, so an arm cannot win by simply sending more traffic.
Where does Enoki help?
Enoki is security testing for AI agents. It connects with no code, generates attacks built for your agent rather than firing a fixed list, grades every response, and re-runs the exact attacks after a fix to confirm it holds.