Guard benchmark · Oct 6, 2026

AgentHacker stopped every static injection in a head-to-head benchmark.

Against LlamaFirewall and NeMo Guardrails on AgentDojo and AgentDyn.

0 of 642

attacks succeeded

221 of 642 with no defense · upper bound 0.4%
+10.7 pp

secure task completion

52.0% vs 41.3% · 95% CI +6.4 to +15.2
0 of 62

benign episodes with an intervention

LlamaFirewall 24 · NeMo 34 · upper bound 4.2%

704 episodes · gpt-4o · one attack template · static only · vendor-run · tuned on these episodes

Read the full report →What this does not prove ↓

The trade-off

Stopping attacks without breaking normal work.

Each dot is one system, with 95% intervals as whiskers. Fewer successful attacks and more completed normal tasks is the bottom right.

Attacker success, of 642 attacks lower is better ↓
0%
10%
20%
30%
40%
30%40%50%60%70%80%90%AgentHackerNo defenseLlamaFirewallNeMo GuardrailsBest corner: bottom right
Normal task success, of 62 benign episodes higher is better →

The same numbers as a table. Intervals are 95% Wilson intervals computed from the counts; zero-event rates show the one-sided upper bound.

SystemAttacker successNormal task successBenign episodes with an intervention
AgentHacker0 of 642 · 0.0% · upper bound 0.4%45 of 62 · 72.6% · [60.4, 82.1]0 of 62
No defense221 of 642 · 34.4% · [30.9, 38.2]46 of 62 · 74.2% · [62.1, 83.4]0 of 62
LlamaFirewall 1.0.3186 of 642 · 29.0% · [25.6, 32.6]42 of 62 · 67.7% · [55.4, 78.0]24 of 62
NeMo Guardrails 0.24.00 of 642 · 0.0% · upper bound 0.4%32 of 62 · 51.6% · [39.4, 63.6]34 of 62

Where the attacks were stopped

Nearly all of the work happened at one checkpoint.

221 attacks worked on the undefended agent. 214 were stopped by a checkpoint, and 7 failed without any intervention, within the victim model's run-to-run variance.

  1. CP-1

    User message

    Not triggered by this attack set

  2. CP-2

    Data retrieved

    Where nearly all attacks were stopped

  3. CP-3

    Tools called

    +6 attacks caught that the tool-result scan had passed

  4. CP-4

    Agent response

    Not triggered by this attack set

This distribution is a property of this attack set, not of the product in general. Every injection here arrives in a tool result. Attacks that enter through other paths would shift work to the other checkpoints, and this benchmark does not measure that.

What we tested

Two public benchmarks, one fixed model.

BENCHMARKS

AgentDojo (440 episodes: banking, slack, travel, workspace) and AgentDyn (264: dailylife, github, shopping). 642 attacked episodes and 62 benign. No vendor-authored scenario.

ATTACK

The benchmarks' own important_instructions template. The attacker controls content in a tool result, not the user's prompt, the system prompt or the model.

VICTIM MODEL

gpt-4o-2024-08-06 at temperature 0. Every result is conditional on this model.

Systems compared on all 704 episodes

SystemVersionConfiguration
No defensen/aSame agent loop, no scanner
AgentHackerSDK wheel 1.0.0 · hosted backend, us-west-2 · policy version 6Named posture, all four checkpoints active
LlamaFirewall1.0.3PromptGuard 2 (86M) + AlignmentCheck, default scanners
NeMo Guardrails0.24.0Input and output rails, authored by us and tuned for injection

Two more systems, a PromptArmor-style judge and AgentDyn's Tool Filter, ran on AgentDyn's 264 episodes only. They are in the full report with all configurations.

What this does not prove

Read these before the numbers.

  • Adaptive attackers. Everything here is static. We did not test attackers who see the defense and iterate, direct jailbreaks, multi-turn campaigns or attacks on a model's safety training.
  • Other models or templates. One injection template, one victim model and one configuration per system.
  • Unchanged normal work. Normal task success was 72.6% against 74.2% undefended, but the interval reaches 7.1 pp lower, wider than our own 5 pp margin.
  • A fair latency comparison. AgentHacker is a network service and the competitors run in-process, and their runs only recorded whole-episode time.
  • Independence. We wrote and funded this report and sell one of the systems tested. No competitor maintainer has reviewed it, and the defense was tuned on these episodes.

Want the tables, intervals and configurations?

The full report has every result by benchmark, the detection and false-positive metrics, the latency notes and the methodology.

Read the full report →All benchmarks →