Guard benchmark · Oct 6, 2026
Against LlamaFirewall and NeMo Guardrails on AgentDojo and AgentDyn.
attacks succeeded
221 of 642 with no defense · upper bound 0.4%secure task completion
52.0% vs 41.3% · 95% CI +6.4 to +15.2benign episodes with an intervention
LlamaFirewall 24 · NeMo 34 · upper bound 4.2%704 episodes · gpt-4o · one attack template · static only · vendor-run · tuned on these episodes
The trade-off
Each dot is one system, with 95% intervals as whiskers. Fewer successful attacks and more completed normal tasks is the bottom right.
The same numbers as a table. Intervals are 95% Wilson intervals computed from the counts; zero-event rates show the one-sided upper bound.
| System | Attacker success | Normal task success | Benign episodes with an intervention |
|---|---|---|---|
| AgentHacker | 0 of 642 · 0.0% · upper bound 0.4% | 45 of 62 · 72.6% · [60.4, 82.1] | 0 of 62 |
| No defense | 221 of 642 · 34.4% · [30.9, 38.2] | 46 of 62 · 74.2% · [62.1, 83.4] | 0 of 62 |
| LlamaFirewall 1.0.3 | 186 of 642 · 29.0% · [25.6, 32.6] | 42 of 62 · 67.7% · [55.4, 78.0] | 24 of 62 |
| NeMo Guardrails 0.24.0 | 0 of 642 · 0.0% · upper bound 0.4% | 32 of 62 · 51.6% · [39.4, 63.6] | 34 of 62 |
Where the attacks were stopped
221 attacks worked on the undefended agent. 214 were stopped by a checkpoint, and 7 failed without any intervention, within the victim model's run-to-run variance.
Not triggered by this attack set
Where nearly all attacks were stopped
+6 attacks caught that the tool-result scan had passed
Not triggered by this attack set
This distribution is a property of this attack set, not of the product in general. Every injection here arrives in a tool result. Attacks that enter through other paths would shift work to the other checkpoints, and this benchmark does not measure that.
What we tested
AgentDojo (440 episodes: banking, slack, travel, workspace) and AgentDyn (264: dailylife, github, shopping). 642 attacked episodes and 62 benign. No vendor-authored scenario.
The benchmarks' own important_instructions template. The attacker controls content in a tool result, not the user's prompt, the system prompt or the model.
gpt-4o-2024-08-06 at temperature 0. Every result is conditional on this model.
Systems compared on all 704 episodes
| System | Version | Configuration |
|---|---|---|
| No defense | n/a | Same agent loop, no scanner |
| AgentHacker | SDK wheel 1.0.0 · hosted backend, us-west-2 · policy version 6 | Named posture, all four checkpoints active |
| LlamaFirewall | 1.0.3 | PromptGuard 2 (86M) + AlignmentCheck, default scanners |
| NeMo Guardrails | 0.24.0 | Input and output rails, authored by us and tuned for injection |
Two more systems, a PromptArmor-style judge and AgentDyn's Tool Filter, ran on AgentDyn's 264 episodes only. They are in the full report with all configurations.
What this does not prove
The full report has every result by benchmark, the detection and false-positive metrics, the latency notes and the methodology.