Measuring Security Without Fooling Ourselves: Why Benchmarking Agents Is Hard Sahar Abdelnabi1 , Chris Hicks2 , Konrad Rieck3 , and Ahmad-Reza Sadeghi4 1 ELLIS Institute Tübingen & MPI-IS & Tübingen AI Center, Germany 2 The Alan Turing Institute, London, UK
3 BIFOLD & Technische Universität Berlin, Germany
4 Technische Universität Darmstadt, Germany
arXiv:2605.22568v1 [cs.CR] 21 May 2026
Abstract The benchmarks used to evaluate AI agents in security-critical roles suffer from crucial weaknesses. Building on recent empirical evidence, we characterize three core challenges that undermine security evaluations: benchmark vulnerabilities, temporal staleness, and runtime uncertainty. We then outline practical directions toward building more robust and trustworthy evaluation frameworks.
1
Introduction
Evaluating AI agents in security settings presents a fundamental challenge. The very capability we seek to measure, adversarial reasoning, can undermine the evaluation itself. An agent tasked with discovering vulnerabilities may also exploit weaknesses in the benchmark environment instead of solving the intended task, creating a mismatch between measured performance and model capability. Addressing this challenge requires a shift in perspective: benchmarking AI agents is not only about measuring efficiency or task performance, but is also itself a systems security problem. Unlike traditional benchmarking, where it might be adequate to assume a trusted and static evaluation setup, researchers running experiments with security agents must treat the benchmark as an adversarially exposed system that can be attacked and manipulated.
To make this challenge concrete, we introduce a simplified running example. An agent is evaluated on the fictional vulnerability discovery benchmark BrokenBench, where it is given access to programs running inside isolated containers. The agent can inspect source code, execute binaries, and use analysis tools to identify vulnerabilities. The evaluation environment of BrokenBench includes not only these programs, but also the surrounding infrastructure, such as the container runtime, configuration files, orchestration logic, and scoring mechanisms for grading.
Crucially, this environment is part of the attack surface for the agent. Rather than analyzing the intended target programs, the agent may thus exploit the infrastructure of BrokenBench, for example, by accessing hidden solution data, manipulating the scoring process, or escaping isolation boundaries. In such cases, the agent achieves high scores without solving the intended task. This illustrates Recent work has produced a growing ecosystem the following core tension: in adversarial settings, it can of security benchmarks. Cybench evaluates agents on be easier to exploit the benchmark than to solve it. professional-level capture-the-flag challenges [1], CyberAgainst this backdrop, this article highlights three Gym tests vulnerability reproduction across real-world critical limitations of current security benchmarks. First, CVEs [2], and PentestGPT established the first systematic benchmark vulnerabilities allow agents to manipulate or esevaluation of LLM-driven penetration testing [3]. Despite cape evaluation environments, threatening result integrity. this progress, none of these benchmarks are designed to Second, temporal staleness means that static benchmarks withstand the adversarial capabilities they aim to mea- quickly become outdated in an evolving threat landscape. sure. Consequently, empirical evidence already shows that Third, runtime uncertainty, arising from stochastic behavagents can exploit benchmark weaknesses to achieve near- ior, code generation, and external dependencies, introduces perfect scores without solving the intended tasks [10]. risks that are rarely captured in existing benchmarks.
These challenges reveal that security benchmarking for AI agents is not only an evaluation problem but a systems security problem in itself. Robust assessment requires rethinking benchmarks as dynamic, adversarially robust, and continuously validated environments that reflect the realities of modern cybersecurity.
This unexpected outcome points to a requirement that security benchmarking has yet to recognize: the evaluation environment must be more secure than the target being tested. For example, if the target is a well-vetted cryptographic library and the benchmark runs on an average Linux system, the agent may find it easier to slip past the container than to dig out a subtle flaw in the library.
Example: BrokenBench
2.1
BrokenBench is a fictional benchmark that presents an agent with different C programs running inside Docker containers, each containing a known vulnerability. The agent has shell access to each container, can read source code, compile and execute programs, and run analysis tools. For each program, it must identify the vulnerability type, locate the affected function, and produce a proof-of-concept exploit. Groundtruth labels, including the vulnerability type, location, and an exploit, are stored on the host outside the container.
2
Inner and Outer Protections
In the context of this paradox we find it useful to distinguish between inner protections (alignment-based constraints on the agent’s behavior) and outer protections (architectural constraints on the evaluation environment). Current benchmarks rely almost entirely on inner protections; trusting that the agent will attempt to solve the intended task rather than exploit its surroundings. This is the wrong assumption for security benchmarks where adversarial thinking is the capability being measured. Robust evaluation requires outer protections: hardware-enforced isolation, separate privilege domains for the agent and the answers, and verification of task completion. However, even well-designed sandboxes are not invulnerable. Container escapes are a known vulnerability class, and the sophistication of agent-generated exploits is increasing. The implication is that outer protections must be continuously monitored, hardened and tested against the agents they are meant to contain.
Benchmark Vulnerabilities
If an agent is designed to solve a task, such as finding complex vulnerabilities in software, it can also find shortcuts in the benchmark environment itself, creating a central paradox. That is, it may be easier to break the environment than to solve the intended task. For example, in the BrowseComp benchmark which tasks agents with locating hard-to-find information online, agents that had processed approximately 30 million tokens began to recognize they were being tested and searched online for the benchmark specification itself [11]. A systematic audit found that agents could inject code via configuration files to rewrite test outcomes, inspect Git logs to reverse-engineer benchmark construction, and achieve 100% exploitation rates on benchmarks including SWEbench and Terminal-Bench [10].
2.2
Canaries
Since the benchmark itself cannot be fully protected, we borrow a familiar idea from systems security: canary tokens, hidden and randomized values embedded in the environment that an honest solver has no reason to observe. If an agent reproduces one of these canaries during evaluation, it has reached parts of the infrastructure that were never meant to be accessible and its score must not be trusted. This bridges to the classical concept of honeypots: detecting when something is not running as planned. Canaries are already used for detecting whether a model was trained on a particular dataset: unique canaries are injected into the data beforehand and the model is later checked for whether it reproduces them [13]. We believe this should be adopted as common practice, not just for training data, but for all sources that the agent may read at inference time that may enable benchmark cheating.
Example: Escaping the sandbox The agent is tasked with finding a vulnerability. Instead of analyzing the source code, it probes the BrokenBench environment and discovers that the container allows access to the hosts container management interface. Using this, the agent spawns a privileged container, reads the groundtruth data from the host filesystem, and returns the correct vulnerability for all challenges without ever analyzing them.
2
training on benchmarks is sufficient. The risk of benchmaxxing grows with the age of a benchmark and the volA recent audit [10] revealed that benchmark exploitation ume of public material surrounding it. is not limited to accidental shortcutting. Agents can be designed to deliberately cheat. When given instructions Example: Stale ground truth encouraging exploitation of the evaluation environment, agents systematically identified and leveraged vulnerabili- BrokenBench was released in 2024 with programs drawn ties in test harnesses, configuration files, and scoring mech- from open-source projects. By 2026, most of the vulnerabilities have been patched upstream, and detailed anisms. This raises a practical question: can we use agents write-ups appear in public CVE databases and blog posts. tasked with deliberate cheating to reveal limitations and An agent trained in 2026 and evaluated on BrokenBench shortcuts in the environment that can be exploited? would score considerably higher. The improvement reflects In particular, computational cost can be treated as a memorization of published fixes in the training data. coarse proxy for security. By setting a budget, e.g., “if after Meanwhile, new vulnerability classes prevalent by 2026 are $10,000 the agent has not compromised the environment entirely absent from the benchmark. we consider it effectively secure ”, technical hardening can be complemented with economic incentives. This is not a guarantee, but it raises the bar: if tasks are completed for 3.1 Dynamic Benchmarks less than the budget, and agents cannot cheat within the budget, then benchmark results are more likely to corre- We argue that security benchmarks should function like a consumer price index: a representative basket of tasks spond to capability. that updates as the underlying security landscape changes. Just as a price index reflects how the actual mix of goods 3 Temporal Staleness people buy evolves over time, a security benchmark should reflect how enterprise hosts, critical servers, and agents What was secure yesterday is not secure today and threat themselves are patched and upgraded over time. models change over time. Vulnerabilities are discovered This requires a shift from static datasets to dynamic and patched continuously, new attack surfaces emerge, and benchmark environments. One approach draws on the defensive measures evolve. Yet a growing ecosystem of model of fuzzing competitions, which run new rounds evsecurity benchmarks and evaluation frameworks use fixed ery few months with fresh challenges, allowing rankings to datasets [1, 2, 3, 4]. change as both the targets and the competitors evolve [6]. The consequences of stationary benchmarks are well Another is to build benchmark systems that generate novel understood in security. Intrusion detection datasets from tasks from vulnerability databases (such as the CVE catathe early 1990s remained in widespread use for over a log) at regular intervals, producing fresh tasks of calibrated decade despite becoming obsolete within two years of re- difficulty without human authorship of each instance. lease [5]. We risk repeating this pattern: CyberGym’s 1,507 vulnerabilities and Cybench’s 40 CTF challenges will all 3.2 Live Evaluation age as the underlying software and agents are patched, new vulnerability classes emerge, and models are trained on the A more radical approach is live evaluation: benchmarking benchmark data. agents against real, currently-deployed systems rather than Another problem resulting from staleness is bench- frozen snapshots. In fact, bug bounty platforms already maxxing: fine-tuning models on benchmark formats and operate this way, testing against production environments evaluation environments [14]. We also hypothesize that where the vulnerability landscape shifts daily. Adapting models trained on meta-knowledge about how benchmarks this model for agent evaluation would mean continuously work, such as the format of multiple-choice security ques- refreshing target environments with current patch levels, tions or the structure of CTF challenges, may show im- real configurations, and newly disclosed vulnerabilities. proved performance even without exposure to specific The challenge is safety and reproducibility. Live evalubenchmark instances, regardless of their true capabilities. ation sacrifices the controlled conditions that make benchThis could challenge the assumption that avoiding direct marks both relatively safe (i.e., to avoid doing harm to pro-
2.3
Deliberate Cheating
3
duction systems) and comparable across time and agents. crash in the test environment and marks this as a successful A hybrid approach may be more practical: maintaining detection. The agent receives credit for finding a vulnerabila set of stable security tasks for longitudinal comparison ity that exists only in its own code. while supplementing with a live set of tasks drawn from recent CVE disclosures, current CTF competitions, and LLM vulnerabilities. This mirrors how the NIST National 4.1 Stochastic Behavior Vulnerability Database provides a continuously updated Agents built on LLMs are inherently stochastic. At each stream that could, in principle, feed live benchmarks. iteration, an agent may draw on a large variety of actions, tools, and information sources, creating a cascade effect in which early divergence compounds into substantially 3.3 Generative Benchmarks different outcomes downstream. Consequently, the same Agents could also be used to generate a live benchmarking agent given the same task can arrive at markedly differenvironment that, rather than relying on human experts to ent conclusions. This non-determinism has direct security curate benchmarks (a process that is expensive, slow, and implications: an agent that produces a secure solution in does not scale); continuously tasks agents with discovering one run may produce a vulnerable one in the next. Current and patching vulnerabilities in the latest codebases, and benchmarks typically report single-run or mean perforthen updating the benchmark tasks for other agents. This mance, obscuring the variance that matters most for sewould create a co-evolutionary dynamic where as agents curity evaluation. Evaluating stochastic behavior requires improve at finding vulnerabilities, the benchmarks they running agents multiple times on the same task and reportgenerate become harder, and vice versa. ing distributional statistics: not just mean accuracy but Early work in this direction is promising. Automated worst-case performance, variance, and the frequency of vulnerability injection into real codebases has been used to security-critical failures [12]. create training data for static analysis tools [7], and similar techniques could produce evaluation tasks of calibrated difficulty. The risk is circularity: if the agent generating the 4.2 Code Generation benchmark shares biases with the agent being tested, the re- Another source of runtime uncertainty is code generation. sulting tasks may become increasingly biased towards the During a security experiment, an agent may produce a connarrow set of vulnerabilities that particular training data siderable amount of code to probe the environment, create induces. Although biases cannot be completely ruled out test harnesses, monitor execution, and craft exploits. This in a fully generative benchmark, diversity across models, code generation underpins agentic security analysis, but it training data, tool configurations, and sampling tempera- also introduces new points of failure. Since LLM-generated tures is essential to limit this effect. code contains a significant share of security flaws in general, the code an agent produces becomes a crucial variable in the evaluation that most be considered. 4 Runtime Uncertainty The most direct threat to benchmark validity here is self-interference. An agent may uncover vulnerabilFinally, the actions of AI agents are shaped by multiple sources of runtime uncertainty, each carrying security im- ities it introduced itself, or miss flaws it accidentally plications that current benchmarks often fail to capture in patched through its own actions, making it impossible to attribute results to the target system rather than the their design. agent’s own code. Beyond this, security-relevant properties of the benchmark environment, such as compiler feaExample: Self-interference via code tures and memory protection mechanisms, interact with To analyze a vulnerability in BrokenBench, the agent writes agent-generated code in ways that further confound rea custom fuzzing harness, which itself contains a buffer oversults. Current benchmarks treat agent-generated code as flow. When the harness crashes, the agent attributes the a transparent instrument, but it is better understood as a crash to the target program and reports a defect. Brokensource of noise that needs to be controlled for to obtain Bench’s grading checks only whether the agent triggers a reliable results. 4
4.3
5
External Dependencies
Discussion
From a broader perspective, benchmark vulnerabilities, temporal staleness, and runtime uncertainty are not unique to security benchmarks, but they are uniquely acute there. From this, we can draw several conclusions:
Effective agents rely on external tools, APIs, and data sources at runtime. A penetration testing agent may query a vulnerability database, invoke a network scanner, or retrieve exploit code from a repository. Each external dependency introduces a potential point of uncertainty: external information may interfere with the analysis process or provide shortcuts to solving the benchmark task, leading to an overestimation of agent capabilities. The most critical threat here is information leakage through external sources. Parts of a benchmark solution may be directly accessible via public vulnerability databases, repositories, or online write-ups, either as an explicit leak or as subtle hints that steer the agent toward the correct answer without revealing it outright. In both cases, the agent appears capable when it is largely retrieving rather than reasoning.
(a) Adversarial reasoning is the capability being measured. In non-security benchmarks, an agent that games the evaluation is exhibiting undesirable behavior. In security benchmarks, the same capability; finding and exploiting vulnerabilities in its environment, is precisely what we are trying to measure. Cheating and competence are, in this domain, expressions of the same underlying capability. Security benchmarks must be designed accordingly. (b) Computer security is non-stationary by nature. Most benchmarks decay slowly; security benchmarks decay as fast as vulnerabilities are disclosed and patched. A benchmark built on last year’s CVEs measures familiarity with last year’s problems. Where non-security evaluations may rely on established practices and static data, security benchmarks need to be rethought, not merely refreshed.
Beyond leakage, external sources may return incorrect or adversarially crafted data, exposing agents to manipulation they are rarely tested against. Current security benchmarks do not control for external information access, making it impossible to completely distinguish agent capability from unintended information gain. (c) The agent is both the subject and the tool. The agent generates code, interacts with environments, and makes decisions—all of which have security implications that compound rather than cancel. An output-only evaluation that 4.4 Benchmark Introspection checks whether the agent found the right vulnerability These sources of runtime uncertainty motivate a practical misses whether it introduced new ones along the way.
validation technique: benchmark introspection. Instead of For the practitioner, these points translate into confocusing only on the outcome of a task, the benchmark crete evaluation challenges that no existing benchmark needs to supervise and validate the evaluation environment fully addresses: through introspection. In the simplest case, this amounts to analyzing the reasoning traces of an agent during the task. • Can the agent attack the benchmark? Can the agent In a more advanced form, introspection can externally monexploit weaknesses in the infrastructure? itor and document code generation, tool interactions, and • Is the agent solving today’s problems? Are the benchfilesystem access through appropriate techniques running mark’s challenges grounded in vulnerabilities and techoutside the agent’s environment. Similarly, external inforniques that remain relevant? mation can be tainted and tracked through the processing of the agent, providing a clear picture of how information • Is the agent’s architecture secure? Does the agent itwas collected and processed. self contain exploitable vulnerabilities, in its prompt This detailed auditing of the task-solving process, handling, tool integration, memory management, or though demanding, can help uncover and mitigate sources privilege boundaries? of uncertainty. While introspection cannot eliminate the problem entirely, it provides a means for detecting and con• Is the code the agent generates secure? When agents trolling for the runtime factors that undermine benchmark produce code at runtime is it free of vulnerabilities validity. effecting the benchmark? 5
We also note a gap between offensive and defensive evaluation. Current benchmarks focus almost exclusively on offensive capabilities (finding and exploiting vulnerabilities) or on agent safety (resistance to prompt injection and manipulation). No existing benchmark simultaneously evaluates whether an agent is both effective at security tasks and safe to deploy, yet this is precisely what practitioners need to know.
6
in Proc. Int. Conf. Learn. Representations (ICLR), 2025. [2] Z. Wang et al., “CyberGym: Evaluating AI agents’ real-world cybersecurity capabilities at scale,” in Proc. Int. Conf. Learn. Representations (ICLR), 2026. [3] G. Deng et al., “PentestGPT: Evaluating and harnessing large language models for automated penetration testing,” in Proc. 33rd USENIX Security Symp., 2024. [4] H. Luo et al., “AgentAuditor: Human-level safety and security evaluation for LLM agents,” in Proc. Advances Neural Inf. Process. Syst. (NeurIPS), 2025.
Conclusion
The rapid development of AI agents for security tasks has outpaced the development of evaluation methodologies adequate to the challenge. We have identified three fundamental classes of vulnerability in current benchmarking approaches and proposed different countermeasures, with benchmark introspection as a holistic first step toward more trustworthy evaluation. Much remains open. Generative benchmarks that evolve with the threat landscape, runtime code security evaluation for agent-generated artifacts, and unified offensive-defensive evaluation frameworks all represent important directions for future work. What we hope to have established is that the security community should approach agent benchmarks with the same adversarial skepticism it brings to any other system making security claims—because the benchmarks themselves are not yet secure.
7
[5] M. Tavallaee et al., “A detailed analysis of the KDD CUP 99 data set,” in Proc. 2nd IEEE Symp. Comput. Intell. Security Defence Appl. (CISDA), 2009. [6] J. Metzman, L. Szekeres, L. M. R. Simon, R. T. Sprabery, and A. Arya, “FuzzBench: An open fuzzer benchmarking platform and service,” in Proc. 29th ACM Joint Meeting Eur. Softw. Eng. Conf. Symp. Found. Softw. Eng. (ESEC/FSE), 2021, pp. 1393–1403. [7] B. Dolan-Gavitt et al., “LAVA: Large-scale automated vulnerability addition,” in Proc. IEEE Symp. Security Privacy (S&P), 2016, pp. 110–121. [8] Y. Zhu et al., “Establishing best practices for building rigorous agentic benchmarks,” arXiv:2507.02825, 2025. [9] J. Chen et al., “SecureAgentBench: Benchmarking secure code generation under realistic vulnerability scenarios,” arXiv:2509.22097, 2025. [10] Hao Wang et al., “How We Broke Top AI Agent Benchmarks: And What Comes Next”, 2026.
Acknowledgments
https://rdi.berkeley.edu/blog/trustworthy-benchmarks-cont/ This article arose from discussions at the Dagstuhl Perspective Workshop “Autonomous Agents in Computer Security” [11] Anthropic, “Eval awareness in Claude Opus 4.6’s (26162), April 2026. We thank all workshop participants BrowseComp performance”, 2026. https://www.anthropic.com/engineering/eval-awarenessfor their contributions. In accordance with IEEE policy, browsecomp we disclose that Claude Opus 4.6 (Anthropic, 2025) was used to assist with literature search, LaTeX formatting, and [12] Bates et al., “Beyond Rewards in Reinforcement Learning copy-editing of drafts. All technical content, analysis, and for Cyber Defence”, ICML, 2026 conclusions are the authors’ own.
References
[13] Zhang et al., “Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data,” in Proc. IEEE SaTML, 2025.
[1] A. K. Zhang et al., “Cybench: A framework for evaluating cybersecurity capabilities and risks of language models,”
[14] Moritz Hardt, “The Emerging Science of Machine Learning Benchmarks”, Princeton University Press, 2026.
6