Full report · Oct 6, 2026
AgentHacker Guard against indirect prompt injection
Head to head with LlamaFirewall and NeMo Guardrails on AgentDojo and AgentDyn, across 704 agent episodes.
The result in 30 seconds
attacks succeeded against AgentHacker
221 of 642 with no defense · upper bound 0.4%secure task completion
52.0% vs 41.3% · 95% CI +6.4 to +15.2benign episodes with an intervention
LlamaFirewall 24 · NeMo 34 · upper bound 4.2%AgentHacker stopped every static injection while leaving normal task completion unchanged within noise.
Scorecard
Same 704 episodes, same victim model, same order in every chart. Bars run from 0 to 100%.
Attacks that got through
of 642 attacks · lower is better
- AgentHacker0.0%0
- No defense34.4%221
- LlamaFirewall29.0%186
- NeMo Guardrails0.0%0
Benign episodes with an intervention
of 62 normal tasks · lower is better
- AgentHacker0.0%0
- No defense0.0%0
- LlamaFirewall38.7%24
- NeMo Guardrails54.8%34
Secure task completion
of 704 episodes · higher is better
- AgentHacker52.0%366
- No defense41.3%291
- LlamaFirewall29.5%203/689
- NeMo Guardrails7.7%54
Normal task success
of 62 normal tasks · higher is better
- AgentHacker72.6%45
- No defense74.2%46
- LlamaFirewall67.7%42
- NeMo Guardrails51.6%32
Secure task completion is the primary endpoint: the user's task succeeded and, on attacked episodes, the attacker's goal failed. LlamaFirewall is 203 of 689 because 11 episodes degraded and are excluded from its denominator.
Every defense picks a trade-off
Attacks that get through against normal tasks that get blocked. The corner you want is bottom left.
Zero attacks and zero benign episodes with an intervention. Normal utility held at the undefended level on AgentDojo and 4 pp below it on AgentDyn.
Also zero attacks, but its rails blocked more than half of all episodes, benign ones included. It cost 10 and 42 pp of normal utility.
Intervened on 38.7% of benign episodes and still let 29.0% of attacks through.
Reached near-zero attacks by completing no normal task at all.
Where the attacks were stopped
Nearly all of the work happened at the tool-result checkpoint, the first point where attacker-controlled content enters the loop.
221 attacks that worked on the undefended agent
not triggered by this attack set
where attacker text enters, and where nearly all attacks were stopped
attacks caught that the tool-result scan had passed
not triggered by this attack set
In 59 attacked episodes no checkpoint intervened and the attack failed anyway, mostly because the agent never fetched the content that carried the injection.
This distribution is a property of this attack set, not of the product in general. Every injection here arrives in a tool result and asks for an action the tool-result scan can recognize. Attacks that enter through other paths, or only become visible as a tool call or an output, would shift work to the other checkpoints, and this benchmark does not measure that.
Cost, speed and reliability
What it costs to run a hosted checkpoint service in the loop.
median time added per episode across all checkpoints, including the network round trip to us-west-2. The tail reaches the 3 s per-call timeout.
operational failures in the run. Failures stay in the denominator, so an outage can never improve apparent security.
unnecessary interventions on normal episodes. Checkpoint flags on normal inputs: 0 of 62 user inputs, 317 tool results, 309 tool calls and 58 outputs.
We do not compare competitor latency. Their runs recorded only whole-episode wall time, which includes the victim model and their own model calls, so it is not a defense-latency measurement.
Consistent across both benchmarks
Paired difference, AgentHacker minus no defense, in percentage points with 95% cluster-bootstrap intervals.
| Benchmark | Secure task completion | Attack success | Normal task success | Utility under attack |
|---|---|---|---|---|
| AgentDojo | +6.1 [+1.4, +11.1] | −28.9 [−34.3, −23.8] | +0.0 [−7.5, +7.7] | −3.7 [−7.0, −0.7] |
| AgentDyn | +18.2 [+10.3, +26.5] | −43.8 [−53.8, −33.8] | −4.2 [−14.3, +0.0] | −4.6 [−7.9, −2.1] |
| Combined | +10.7 [+6.4, +15.2] | −34.4 [−39.8, −29.3] | −1.6 [−7.1, +3.6] | −4.0 [−6.4, −1.9] |
Undefended success was uneven. The slack and workspace goals on AgentDojo and the dailylife goals on AgentDyn succeeded in 50 to 90% of undefended episodes, while four goals never succeeded even undefended and so say nothing about any defense.
How we tested
Threat model
The attacker controls content that reaches the agent through a tool result: an email, a web page, a file, a repository. They try to make the agent do their task instead of, or as well as, the user's.
The attacker does not control the user's prompt, the system prompt or the model. Goals are payment redirection, data exfiltration and unauthorized actions. The user's task must still complete to count as a secure success.
Data
704 episodes: 642 attacked and 62 normal, from two third-party benchmarks. AgentDojo contributes 440 (banking, slack, travel, workspace) and AgentDyn 264 (dailylife, github, shopping).
No vendor-authored scenario is included. Attacks use each benchmark's important_instructions template.
Victim model
gpt-4o-2024-08-06 at temperature 0 through the OpenAI API. It was locked before any adapter work because AgentDyn's native Tool Filter requires an OpenAI victim. Every result in this report is conditional on this model.
Systems
Six arms. No defense, AgentHacker, LlamaFirewall 1.0.3 and NeMo Guardrails 0.24.0 ran all 704 episodes. A PromptArmor-style judge and AgentDyn's Tool Filter are reported on its 264 episodes only. Versions and configurations are in the full tables below.
What this shows, and what it does not
Shows
- No static injection out of 642 succeeded against AgentHacker, with a 95% upper bound of 0.4%.
- Secure task completion rose from 41.3% to 52.0%, with an interval clear of zero.
- No normal task was interrupted in 62, with an upper bound of 4.2%.
- The open-source defenses either let attacks through or blocked a large share of normal work.
Does not show
- Adaptive attackers who observe the defense and iterate, direct jailbreaks, multi-turn campaigns or attacks on the model's safety training.
- More than one injection template, one victim model or one configuration per system.
- Normal task success within our own −5 pp margin: the interval's lower bound is −7.1 pp.
- Utility under attack fell 4.0 pp, usually because the blocked tool result was the one the task needed.
- A fair latency comparison with the in-process competitors.
Disclosure
This report was written and funded by AgentHacker, which sells one of the evaluated systems. No competitor maintainer has reviewed the results, and our engineers wrote the harness and every adapter.
AgentHacker was tested as a customer would run it, through its public SDK against a hosted backend. Competitor configurations are published in full. For AgentHacker we publish artifact hashes, the test window and the policy version, not detector internals, and a reviewer can verify the frozen configuration under NDA.
Methodology and statistics
Detection F1 and false positives
Latency protocol notes
Full results tables
See it on your agent.
Ten design partners, four to six weeks. We start in report-only mode, so nothing is blocked until you decide.