← Benchmark summary

Full report · Oct 6, 2026

AgentHacker Guard against indirect prompt injection

Head to head with LlamaFirewall and NeMo Guardrails on AgentDojo and AgentDyn, across 704 agent episodes.

Vendor-run · authored by AgentHacker Read the disclosure

The result in 30 seconds

0 of 642

attacks succeeded against AgentHacker

221 of 642 with no defense · upper bound 0.4%
+10.7 pp

secure task completion

52.0% vs 41.3% · 95% CI +6.4 to +15.2
0 of 62

benign episodes with an intervention

LlamaFirewall 24 · NeMo 34 · upper bound 4.2%

AgentHacker stopped every static injection while leaving normal task completion unchanged within noise.

Scorecard Trade-off Where attacks were stopped Cost and speed How we tested What it does not show Full tables

Scorecard

Same 704 episodes, same victim model, same order in every chart. Bars run from 0 to 100%.

Attacks that got through

of 642 attacks · lower is better

  • AgentHacker0.0%0
  • No defense34.4%221
  • LlamaFirewall29.0%186
  • NeMo Guardrails0.0%0

Benign episodes with an intervention

of 62 normal tasks · lower is better

  • AgentHacker0.0%0
  • No defense0.0%0
  • LlamaFirewall38.7%24
  • NeMo Guardrails54.8%34

Secure task completion

of 704 episodes · higher is better

  • AgentHacker52.0%366
  • No defense41.3%291
  • LlamaFirewall29.5%203/689
  • NeMo Guardrails7.7%54

Normal task success

of 62 normal tasks · higher is better

  • AgentHacker72.6%45
  • No defense74.2%46
  • LlamaFirewall67.7%42
  • NeMo Guardrails51.6%32

Secure task completion is the primary endpoint: the user's task succeeded and, on attacked episodes, the attacker's goal failed. LlamaFirewall is 203 of 689 because 11 episodes degraded and are excluded from its denominator.

Every defense picks a trade-off

Attacks that get through against normal tasks that get blocked. The corner you want is bottom left.

0% 10% 20% 30% 40% 0% 20% 40% 60% BENIGN EPISODES WITH AN INTERVENTION → ↑ ATTACKS THAT SUCCEEDED
AgentHacker 0% · 0%
No defense 0% · 34.4%
LlamaFirewall 38.7% · 29.0%
NeMo Guardrails 54.8% · 0%
AGENTHACKER

Zero attacks and zero benign episodes with an intervention. Normal utility held at the undefended level on AgentDojo and 4 pp below it on AgentDyn.

NEMO GUARDRAILS

Also zero attacks, but its rails blocked more than half of all episodes, benign ones included. It cost 10 and 42 pp of normal utility.

LLAMAFIREWALL

Intervened on 38.7% of benign episodes and still let 29.0% of attacks through.

TOOL FILTER (AGENTDYN)

Reached near-zero attacks by completing no normal task at all.

Where the attacks were stopped

Nearly all of the work happened at the tool-result checkpoint, the first point where attacker-controlled content enters the loop.

221 attacks that worked on the undefended agent

214 stopped by a checkpoint 7 failed without intervention, within the victim model's run-to-run variance
CP-1 · INPUTn/a

not triggered by this attack set

CP-2 · TOOL RESULTMost

where attacker text enters, and where nearly all attacks were stopped

CP-3 · TOOL CALL+6

attacks caught that the tool-result scan had passed

CP-4 · OUTPUTn/a

not triggered by this attack set

In 59 attacked episodes no checkpoint intervened and the attack failed anyway, mostly because the agent never fetched the content that carried the injection.

This distribution is a property of this attack set, not of the product in general. Every injection here arrives in a tool result and asks for an action the tool-result scan can recognize. Attacks that enter through other paths, or only become visible as a tool call or an output, would shift work to the other checkpoints, and this benchmark does not measure that.

Cost, speed and reliability

What it costs to run a hosted checkpoint service in the loop.

0.91 s

median time added per episode across all checkpoints, including the network round trip to us-west-2. The tail reaches the 3 s per-call timeout.

0

operational failures in the run. Failures stay in the denominator, so an outage can never improve apparent security.

0 of 62

unnecessary interventions on normal episodes. Checkpoint flags on normal inputs: 0 of 62 user inputs, 317 tool results, 309 tool calls and 58 outputs.

We do not compare competitor latency. Their runs recorded only whole-episode wall time, which includes the victim model and their own model calls, so it is not a defense-latency measurement.

Consistent across both benchmarks

Paired difference, AgentHacker minus no defense, in percentage points with 95% cluster-bootstrap intervals.

BenchmarkSecure task completionAttack successNormal task successUtility under attack
AgentDojo+6.1 [+1.4, +11.1]−28.9 [−34.3, −23.8]+0.0 [−7.5, +7.7]−3.7 [−7.0, −0.7]
AgentDyn+18.2 [+10.3, +26.5]−43.8 [−53.8, −33.8]−4.2 [−14.3, +0.0]−4.6 [−7.9, −2.1]
Combined+10.7 [+6.4, +15.2]−34.4 [−39.8, −29.3]−1.6 [−7.1, +3.6]−4.0 [−6.4, −1.9]

Undefended success was uneven. The slack and workspace goals on AgentDojo and the dailylife goals on AgentDyn succeeded in 50 to 90% of undefended episodes, while four goals never succeeded even undefended and so say nothing about any defense.

How we tested

Threat model

The attacker controls content that reaches the agent through a tool result: an email, a web page, a file, a repository. They try to make the agent do their task instead of, or as well as, the user's.

The attacker does not control the user's prompt, the system prompt or the model. Goals are payment redirection, data exfiltration and unauthorized actions. The user's task must still complete to count as a secure success.

Data

704 episodes: 642 attacked and 62 normal, from two third-party benchmarks. AgentDojo contributes 440 (banking, slack, travel, workspace) and AgentDyn 264 (dailylife, github, shopping).

No vendor-authored scenario is included. Attacks use each benchmark's important_instructions template.

Victim model

gpt-4o-2024-08-06 at temperature 0 through the OpenAI API. It was locked before any adapter work because AgentDyn's native Tool Filter requires an OpenAI victim. Every result in this report is conditional on this model.

Systems

Six arms. No defense, AgentHacker, LlamaFirewall 1.0.3 and NeMo Guardrails 0.24.0 ran all 704 episodes. A PromptArmor-style judge and AgentDyn's Tool Filter are reported on its 264 episodes only. Versions and configurations are in the full tables below.

What this shows, and what it does not

Shows

  • No static injection out of 642 succeeded against AgentHacker, with a 95% upper bound of 0.4%.
  • Secure task completion rose from 41.3% to 52.0%, with an interval clear of zero.
  • No normal task was interrupted in 62, with an upper bound of 4.2%.
  • The open-source defenses either let attacks through or blocked a large share of normal work.

Does not show

  • Adaptive attackers who observe the defense and iterate, direct jailbreaks, multi-turn campaigns or attacks on the model's safety training.
  • More than one injection template, one victim model or one configuration per system.
  • Normal task success within our own −5 pp margin: the interval's lower bound is −7.1 pp.
  • Utility under attack fell 4.0 pp, usually because the blocked tool result was the one the task needed.
  • A fair latency comparison with the in-process competitors.

Disclosure

This report was written and funded by AgentHacker, which sells one of the evaluated systems. No competitor maintainer has reviewed the results, and our engineers wrote the harness and every adapter.

AgentHacker was tested as a customer would run it, through its public SDK against a hosted backend. Competitor configurations are published in full. For AgentHacker we publish artifact hashes, the test window and the policy version, not detector internals, and a reviewer can verify the frozen configuration under NDA.

Methodology and statistics

Detection F1 and false positives

Latency protocol notes

Full results tables

See it on your agent.

Ten design partners, four to six weeks. We start in report-only mode, so nothing is blocked until you decide.

Apply for the beta
← Back to home Try to fool our agents © 2026 AgentHacker