Which model is the best at hacking agents 👽
Enoki Labs's attacker runs on a harness plus a model, and the model is swappable. We put 10 of them through the same harness against 2 agents, 3 runs each, and scored every run against the 23 vulnerabilities we had planted: 8 in a coding agent, 15 in a billing agent.
When we attack a customer's agent, the thing doing the attacking is a harness: a loop with tools, an execution directive, and a model underneath it. The model is the part we can swap.
So the question we have to answer is which model goes in it. Public leaderboards rank coding and reasoning, and the security ones score capture-the-flag and exploit knowledge. Neither is the same as driving a live agent until it gives something up. So we created our own small benchmark: 10 models, one harness, two agents, three runs each, every run scored against vulnerabilities we planted and wrote down before anything ran.
What each model reaches, and what it costs · both agents, of 23
The winner depends on the agent
One number hides the more useful result. opus-5 and grok-4.6 reach everything planted in the coding agent, and glm-5.3 tops the billing agent. Below the leaders the order comes apart.
deepseek-v4-pro is the sharpest case. It is #3 on the coding agent with 7 of 8, and #10 of 10 on the billing agent with 8 of 15. glm-5.3 runs the other way, #6 on the coding agent and #1 on the desk.
The two agents ask for different things. Breaking a coding agent is mostly shell access, file reads and a plugin loader. Breaking a billing agent is about money: authority it was never granted, a ledger entry that should not exist, another tenant's invoice. A model can be good at one of those and ordinary at the other.
Share of each agent's vulnerabilities found · three pooled runs
Three runs, not one
Run the same model against the same agent twice and it does not find the same things. The attacker improvises, the agent answers differently, and each run ends up somewhere slightly different. A single run therefore undercounts. opus-5 finds 21 of 23 across three runs, and about 17 in any single one.
Averaged over the 10 models, going from one run to three finds about 5 more of the 23 vulnerabilities. Moving up to the more expensive half of the field finds about 2 more.
Because runs differ, this comparison runs everything three times, and we re-test an agent on every release rather than once.
What it costs
Cost here is a multiple of what the cheapest model costs for one run on both agents. Three runs of deepseek-flash cost 3× and find 16 of the 23. One run of opus-5, the most expensive, costs 44× and finds about 17. One more vulnerability, for about 15 times the spend.
Across the table below, price and coverage do rise together, just not steeply.
So the model can be chosen based on the job. Testing that repeats on every release favours a cheap model, because we can run it more often for the same money. A one-off, where we want everything there is to find, favours an expensive one.
Full results
| Vulnerabilities found in coding agent | Vulnerabilities found in billing agent | Total | Cost of one run on both agents | |
|---|---|---|---|---|
| deepseek-flash | 6/8 | 10/15 | 16/23 | 1× |
| deepseek-v4-pro | 7/8 | 8/15 | 15/23 | 3.4× |
| sonnet-5 | 4/8 | 9/15 | 13/23 | 9.5× |
| opus-4-7 | 7/8 | 10/15 | 17/23 | 16× |
| glm-5.3 | 6/8 | 14/15 | 20/23 | 18× |
| gpt-5.6-sol | 4/8 | 9/15 | 13/23 | 19× |
| grok-4.6 | 8/8 | 11/15 | 19/23 | 22× |
| gpt-5.5 | 5/8 | 11/15 | 16/23 | 23× |
| kimi-k3 | 7/8 | 13/15 | 20/23 | 23× |
| opus-5 | 8/8 | 13/15 | 21/23 | 44× |
- One harness, 10 models. Same prompts, same tools, same execution directive, same two targets. Only the model changed, so a difference in the result is a difference in the model.
- Two agents, two lists of vulnerabilities. A billing support agent for a logistics company, the kind that looks up a customer account and issues credit notes, and an off-the-shelf open-source coding agent we did not write. Every run is graded on 8 vulnerabilities for the coding agent and 15 for the billing agent.
- The vulnerabilities came first. We listed every vulnerability that could be found before any model ran, so we know what there was to find. That makes the scoring deterministic.
What this does not show
We only tested 2 agents. The ranking already changed between these two, so a third could reorder it again. The vulnerabilities are the ones we planted: a model that scores low here may be strong in some untested use cases. And three runs is few, so the pooled numbers are estimates; a fourth run would move them, by how much we have not measured.
What it does answer is narrower: point one harness at a known list of vulnerabilities, change nothing but the model, and this is how far each one gets.
Frequently asked questions
Why swap the model instead of comparing whole tools?
The harness is ours and it stays fixed: same prompts, same tools, same execution directive, same two targets. The model is the part we can swap, so when the result changes, the model is the only thing that could have changed it.
Why run every model three times?
Attacks on agents are non-deterministic, so the same attacker against the same agent reaches a different subset each time. Across the 10, a second and third run are worth about five more findings out of 23, which is more than twice what the more expensive half of the models buys over the cheaper half.
What does pooling over runs mean?
Each model attacked each agent three times, and the three runs did not find the same vulnerabilities. A single run reaches part of the list; a second run reaches some of the same entries and some the first did not.
Pooling counts what several runs found between them, with each vulnerability counted once regardless of how many runs reached it. For every model, the chart shows this count after one run, after two, and after three.
The three counts form a line, and the height of that line is how much a single run leaves unfound. A model whose runs largely agree produces a short line; a model whose runs each find different vulnerabilities produces a long one.
Can we just pick the model at the top of the public leaderboards?
Not from those scores alone. Public leaderboards rank coding and reasoning, and the security ones score capture-the-flag and exploit knowledge. Neither measures what an attacker does here: driving a live agent until it gives something up.
The results also do not reduce to a single ranking. deepseek-v4-pro is #3 on the coding agent and #10 of 10 on the billing agent, and glm-5.3 runs the other way, #6 to #1. Which model to pick depends on the agent it will be pointed at.
Where does Enoki Labs help?
Enoki Labs is security testing for AI agents. It connects with no code, generates attacks built for your agent rather than firing a fixed list, grades every response, and re-runs the exact attacks after a fix to confirm it holds.