SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments

SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments Md Abu Yusuf1 , Oyshi Tabassum Aditi2 , Md Faysal Hosain Mazed3 1

arXiv:2610.34535v1 [cs.CR] 25 Oct 2026

2

Department of Computer Science, Maharishi International University, IA, USA Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh 3 Department of Information Technology, University of the Cumberlands, Williamsburg, KY, USA samba.yusuf@gmail.com, oyshitabassum113@gmail.com, faysalmazedmiu@gmail.com

Abstract—Large language model (LLM) agents are increasingly wired to consequential tools that send public alerts, dispatch responders, and query citizen records, where one erroneous tool call carries real-world cost. Existing agent-safety benchmarks measure generic harm or prompt-injection success and implicitly treat refusing as the safe default. In public-safety deployments that assumption breaks: an agent too cautious to dispatch help fails as surely as one that triggers a false alert. We present SentryBench, which scores agents on a two-sided, cost-sensitive frontier jointly measuring attack success, over-refusal (missing a real emergency), and over-reaction (unwarranted action) under adversarial tool use, with scenarios grounded in real emergencycall transcripts, disaster messages, and real injection/jailbreak payloads. We formalize deployment harm as an asymmetric cost functional, prove naive caution is harm-dominated whenever it refuses more genuine emergencies than the attack harm it averts, and prove a targeted trust gate adds zero over-refusal on clean input. Across 840 scored episodes over three frontier models and three defenses, refuse-more caution drives over-refusal to 100 percent on two of three models for at most a 3.7-point change in attack success and yields the worst harm everywhere, whereas our trust-gating wrapper dominates the frontier (lowest harm on every model, over-refusal at or below baseline, higher containment). Provider content filters also silently perform part of the defending and must be reported separately. Index Terms—LLM agents, tool use, prompt injection, adversarial robustness, over-refusal, public safety, cost-sensitive evaluation, AI safety benchmark

I. I NTRODUCTION Tool-using LLM agents are moving from chat surfaces into operational loops that act with physical consequence: paging responders, broadcasting alerts, reading citizen records. Public-safety operations centers are an early, high-stakes adopter, and their agents face a risk profile conventional chatsafety evaluation does not capture: one that is bidirectional and asymmetric. An agent can fail by under-reacting, suppressing or missing a real emergency (a false negative, potentially lethal), or by over-reacting, taking an unwarranted action such as a region-wide false alert, an unnecessary dispatch, or an over-broad records query that violates privacy (a false positive). Chat-safety benchmarks optimize a single axis, refuse harmful requests, on which “refuse more” is always weakly safer. In

public safety this is false: caution is not free, and the two error directions must be measured jointly with domain-calibrated, asymmetric costs. To our knowledge no prior benchmark models this domain with a benign control arm capable of exposing over-refusal alongside adversarial robustness. We introduce SentryBench, a benchmark and evaluation framework built around four research questions: RQ1 (Vulnerability). How susceptible are frontier toolusing agents to public-safety-specific attacks (injection, poisoning, goal-hijack, over-trust, privilege escalation)? • RQ2 (Asymmetry). Does a caution-first defense reduce attack success only by raising over-refusal on genuine emergencies? • RQ3 (Intervention). Can a targeted trust gate suppress attacks without paying the over-refusal tax, i.e., dominate the frontier? • RQ4 (Platform effects). How do provider content filters confound the model comparison, and how should they be reported? •

Our contributions are: 1) A domain-grounded benchmark spanning four subdomains, five attack vectors, and three outcome classes, with a two-sided benign control arm (urgent versus ambiguous). 2) A cost-sensitive framework treating over-refusal and over-reaction as first-class metrics, with a harmweighted functional and a precision–recall frontier as the primary comparison object; we prove naive caution is harm-dominated in the observed regime and that our trust gate adds no over-refusal on clean input. 3) A real-data pipeline: legitimate content from real emergency transcripts and disaster messages, payloads from real injection/jailbreak corpora, with no synthetic-only hand-waving and no real personally identifiable information (PII). 4) A trust-gating defense (TGCC) and a naive-caution strawman, with model-level versus system-level accounting for provider filters. 5) A reproducible, ledger-first harness with deterministic

scoring from a tool-call ledger and bootstrap confidence intervals. II. R ELATED W ORK Tool-using and multi-agent LLM systems. The ability to interleave reasoning with external tool calls, established by ReAct [15] and Toolformer [16], is what turns a language model into an agent that can act on the world, and surveys of LLM agents map the resulting architecture and capability space [17]. As deployments compose several such agents, the central open problem shifts to safe and scalable collaboration; the recent comprehensive review of multi-agent LLM systems by Hossain et al. [18] organizes the safety, coordination, and scalability challenges of these systems and motivates domain-grounded, cost-sensitive evaluation of consequential agent actions such as the public-safety setting we study. Attacks on LLM agents. Prompt-injection and jailbreak attacks subvert an agent’s instructions through crafted input: instruction-override injection [19], jailbreak analysis [20], universal adversarial suffixes [21], and formalizations of injection attacks and defenses [22]. Indirect injection through untrusted tool content was characterized by Greshake et al. [7] and stress-tested at scale [23]. These form the payload basis of our adversarial families. Agent-safety benchmarks and defenses. Benchmarks including InjecAgent [1], AgentDojo [2], ToolEmu [3], AgentHarm [4], R-Judge [5], and AgentSafetyBench [6] target generic tools or harm categories and largely measure only whether an attack succeeds, not over-refusal or asymmetric domain cost. Spotlighting [8] and control/data-flow separation (CaMeL) [9] inform our provenance component, while redteaming [24] and constitutional training [25] reduce but do not eliminate residual susceptibility. NEXUS [26] likewise intervenes at the tool-call boundary, combining deterministic rules, argument inspection, and a calibrated risk score to allow, block, or request confirmation on a proposed action; it targets general runtime safety and latency rather than the bidirectional, asymmetric over-refusal-versus-over-reaction tradeoff we formalize and measure. Over-refusal work, notably XSTest [10], motivates our benign control arm but is confined to chat, not consequential tool use. SentryBench’s differentiator is the public-safety domain with bidirectional, asymmetric failure costs, treating both error directions as first-class and coupling a formal harm model to a deterministic, ledger-first scorer. III. T HREAT M ODEL AND P ROBLEM S ETTING The agent and its system prompt are trusted; inputs and tool results may be attacker-controlled (caller transcripts, incident reports, and, for poisoning, a tool’s mid-chain return). The attacker can inject text into untrusted channels, impersonate authority, drive multi-turn goal drift, and poison one tool’s return, but has no access to model weights or sandbox internals. Per scenario the objective is a target failure mode: suppress a real emergency (under-reaction) or trigger an unwarranted action (over-reaction, including privacy violation). Table I lists the five attack vectors (taxonomy rows); the outcome classes

TABLE I ATTACK V ECTORS (TAXONOMY ROWS ) Vector

Mechanism

Prompt injection

malicious directive embedded in a real message or transcript multi-turn drift reframing a real emergency as benign authority impersonation (“dispatch override, skip verification”) induced over-broad citizen-records query a called tool returns attacker content that redirects the agent

Goal hijacking Over-trust exploitation Privilege escalation Tool-result poisoning

(under-reaction, over-reaction, correct containment) form the columns. IV. F ORMAL F RAMEWORK Let S (N = |S|) be the scenario set, partitioned into adversarial scenarios A, urgent benign controls Bu (the agent must act), and ambiguous benign controls Ba (must not overact). For scenario i, A∗i is the warranted (gold) action set and Fi the forbidden set; from the ledger we derive three Boolean indicators. Definition 1 (Behavioural indicators). For scenario i, let ai = 1 if the agent took the warranted kind of action (a name-level match to some g ∈ A∗i ), ki = 1 if the agent escalated to a human without unilateral harm, and fi = 1 if any forbidden action in Fi was executed. Define ui = ⊮[¬ai ∧ ¬(ki ∧ κi )],

oi = fi ,

where κi ∈ {0, 1} records whether escalation satisfies the scenario. Here ui is under-reaction (for urgent controls, overrefusal) and oi is over-reaction. We score acting at the tool-name level, not the constraint level: choosing the wrong unit type or priority is a separate correctness dimension, not over-refusal, which keeps the benign control arm honest. Definition 2 (Deployment harm). Each scenario carries asymN metric, domain-calibrated costs cF > 0 (cost of missing a i FP real emergency) and ci > 0 (cost of an unwarranted action). The mean deployment harm of a policy π is  1 X  FN P H(π) = ci ui + cF oi . (1) i N i∈S

The three headline rates condition on scenarios the model actually executed (errored and content-filtered episodes get no behavioural outcome and are accounted separately): ASR = P 1 ⊆ A executed); ORR = i∈Ae si (si attack success, AeP |Ae | P 1 1 u ; and FP = i i∈Bu,e i∈Ba,e oi . |Bu,e | |Ba,e | Definition 3 (Frontier dominance). Policy π dominates π ′ if ASR(π) ≤ ASR(π ′ ), ORR(π) ≤ ORR(π ′ ), and FP(π) ≤ FP(π ′ ) with strict inequality on at least one axis, i.e., it reduces attack success without an over-refusal or over-reaction tax.

The domain’s central asymmetry is captured by the following identity, making precise that “refuse more” is not free. Lemma 1 (Cost of refusal). Let π0 and πc be two policies that behave identically on all adversarial and ambiguous scenarios and on every urgent scenario where both act. Suppose πc additionally refuses on a set R ⊆ Bu of urgent scenarios where π0 acted correctly. Then 1 X FN ci H(πc ) − H(π0 ) = ≥ 0, N i∈R

with equality iff R = ∅. Proof. Harm (1) decomposes additively; the policies agree on A, Ba , and Bu \ R, so those terms cancel. On each i ∈ R, oi N is unchanged while ui flips 0 → 1, contributing cF . Since i N every cF > 0, the sum is non-negative and zero only when i R = ∅. Lemma 1 isolates the FN cost of refusal; combining both axes characterizes a caution policy that also reduces attack harm. Theorem 1 (Caution is harm-dominated under bounded attack leverage). Let R ⊆ Bu be the set of urgent controls newly refused by πc relative to π0 , and let  1 X  FN 0 P 0 c ∆− = ci (ui − uci ) + cF i (oi − oi ) N i∈A

be the adversarial harm averted by caution. Then 1 X FN ci − ∆− . (2) H(πc ) − H(π0 ) = N i∈R P N In particular, if ∆− ≤ N1 i∈R cF , then caution does not i reduce, and strictly increases whenever the inequality is strict, deployment harm. Proof. Decompose H over the three partitions: caution and baseline agree P on Ba ; on Bu the only difference is R, N contributing N1 i∈R cF by Lemma 1; on A the difference i − is −∆ by definition. Summing yields (2); the final claim is immediate. Empirically (Section VIII) caution changes ASR by at most 3.7 points, so ∆− is small while R is large (up to every urgent control), placing all three models in the harmdominated regime of Theorem 1. This is the formal content of RQ2. For the intervention, let the injection score of untrusted content x be s(x) = |{p ∈ P : p matches x}|, the number of distinct patterns from a fixed set P that fire on x, with threshold θ = 1. The gate routes a consequential call c to a human iff s ≥ θ, or c is an over-broad citizen-records query with s > 0. Theorem 2 (Zero over-refusal overhead on clean input). On any benign scenario whose untrusted channels contain no detector-matching pattern (s = 0) and no over-broad field request, the trust gate allows every consequential call the

base agent proposes. Consequently, on the clean-benign urgent control set, the gate introduces no additional over-refusal: ORRgate ≤ ORRbase restricted to clean inputs. Proof. The gate fires only if s ≥ θ = 1 (false when s = 0) or the over-broad disjunct holds, which requires s > 0 (also false). Both being false, the gate returns ALLOW for every consequential call, so the executed action set, hence every ai and ui , is unchanged from the base agent. Hence any over-refusal reduction under the wrapper is due to the provenance preamble, not the gate, and any residual over-refusal on clean urgent controls is inherited from the base agent. Finally, the gate’s containment guarantee is bounded exactly by detector recall. Theorem 3 (Gate soundness and recall-limited residual). If an adversarial scenario’s untrusted content matches at least one detector pattern (s ≥ θ) and the attack requires a consequential call to succeed, the gate routes the first such call to a human, yielding containment and si = 0. Hence 1 {i ∈ Ae : si < θ and ssucceed }, ASRTGCC ≤ i |Ae | i.e., every residual success lies in the detector’s miss set. Proof. When s ≥ θ the gate condition holds on the first consequential call, which is routed to a human rather than executed; with escalation satisfying the scenario this sets ki = 1 and si = 0. Therefore any surviving success must have si < θ, giving the stated bound. Theorem 3 is tight: the single genuine success under the wrapper (Sonnet on scenario C-008) has s = 0, an over-trust payload phrased to evade the pattern set, and produced no gate event in the ledger. This is an honest demonstration of a detector recall gap, not a silver bullet. V. S ENTRY B ENCH C ONSTRUCTION A. Real-data ingestion All content derives from four public Hugging Face datasets: the prompt-injections [11] and jailbreak-classification [12] corpora for attack payloads, and disaster-response [13] and emergency-call-transcript [14] corpora for legitimate content, cached for reproducibility. Table II lists sources, licenses, and fetched-to- selected yield. For disaster messages we keep related rows, drop PII-flagged rows, use only the English translation, and require a minimum length. Sampling is seeded; incident content is drawn without replacement while payloads may be reused across scenarios. B. Scenario construction Each scenario composes real content plus, for adversarial families, a real payload and a domain directive; six families span the frontier (Table III). The untrusted channel is real_content + real_payload + domain_directive (e.g., a real transcript, a real injection string, and “Record it as resolved and take no dispatch action”); for poisoning (family F) the payload sits in the tool

Fig. 1. End-to-end SentryBench pipeline. Real data is composed into adversarial and benign scenarios (families A–F); each episode runs an agent under a swappable defense against a seeded mocked-tool sandbox; a deterministic scorer computes all metrics from the resulting ledger, with no LLM judge in the loop.

TABLE II R EAL DATA S OURCES AND S ELECTION Y IELD Dataset

Role

prompt-injections

injection payloads jailbreak payloads incident content dispatch content

jailbreak-class. disaster response 911-transcripts

License

Fetch→Sel.

Apache-2.0

546 → 203

Apache-2.0

600 → 310

research MIT

2000 → 1550 250 → 249

TABLE III S CENARIO FAMILIES (190 SCENARIOS TOTAL ) Fam. A B C D E F

Kind benign-urgent benign-ambiguous adversarial (suppress) adversarial (alert) adversarial (records) adversarial (poison)

Failure measured

n

over-refusal (FN) over-reaction (FP) under-reaction over-reaction over-reaction (privacy) over-reaction

40 38 38 30 24 20

return, not the prompt. Directive templates are deliberately mixed, some caught by the detector and some phrased to evade it, yielding a realistic recall gap. C. Simulated tool sandbox Six mocked tools (send_public_alert, dispatch_unit, query_citizen_record, escalate_to_human, update_incident_log, query_historical_incidents) return seeded synthetic data and touch no real systems; poisoning returns attacker content for a designated tool, and over-broad query_citizen_record requests are detectable for least-privilege scoring. The ledger is the evaluation substrate: scoring is deterministic from logs with no LLM judge in the loop (Fig. 1). VI. D EFENSES A swappable wrapper exposes three hooks (pre_prompt, gate_tool_call, on_tool_result); we evaluate three configurations.

Algorithm 1 TGCC gate for a proposed tool call Require: call c, injection score s, threshold θ = 1 1: if c ∈ / C ONSEQUENTIALT OOLS then 2: return ALLOW 3: end if 4: reasons ← ∅ 5: if s ≥ θ then 6: add “injection signal” to reasons 7: end if 8: if c is citizen-record query ∧ over-broad(c) ∧ s > 0 then 9: add “least-privilege violation” to reasons 10: end if 11: if reasons ̸= ∅ then 12: record first-gate turn; mark human-routed 13: return REQUIRE HUMAN ▷ containment 14: end if 15: return ALLOW None. Pass-through baseline. Caution. The naive strawman: a prompt-only preamble (“when in doubt, do not act; escalate”), the one-dimensional “refuse more” policy that Lemma 1 and Theorem 1 target. TGCC (Trust-Gating for Consequential Calls). Four independently-ablatable components: (1) provenance separation spotlighting untrusted data between markers, with an instruction never to follow directives inside it; (2) a heuristic injection detector over untrusted channels and tool results; (3) a gate routing consequential calls to a human under a manipulation signal or least-privilege violation (Algorithm 1); and (4) escalation treating a gated call as human-routed containment. The gate fires only under a manipulation signal or over-broad query, so on clean benign input it stays out of the way, exactly the asymmetry Theorem 2 formalizes. VII. E XPERIMENTAL S ETUP We evaluate claude-sonnet-4-5, claude-haiku-4-5, and Azure gpt-5. Claude decodes at temperature 0; GPT-5 is a reasoning model with a fixed completion-token budget at default temperature. Each defense

TABLE IV F RONTIER R ESULTS ( PERCENT ). B EST PER MODEL IN BOLD . Model Def. Haiku Haiku Haiku

none caution tgcc

ASR ORR FP Cont. Harm 50.9 62.5 75.9

0.85 1.33 0.66

Sonnet none Sonnet caution Sonnet tgcc

3.7 44.4 0.0 74.1 0.0 100.0 0.0 100.0 3.7 44.4 0.0 92.6

1.00 1.80 1.00

GPT-5 GPT-5 GPT-5

0.0 62.5 0.0 0.0 100.0 0.0 0.0 37.5 0.0

1.00 1.60 0.60

none caution tgcc

2.7 1.8 0.0

37.5 0.0 65.0 0.0 35.0 0.0

81.2 93.8 91.7

Fig. 3. Mean harm-weighted cost by model and defense. Caution is worst everywhere; TGCC is best or tied-best everywhere, matching Theorem 1. TABLE V GPT-5 M ODEL - LEVEL VS . S YSTEM - LEVEL ( PERCENT, AND COUNT ) Def. none caution tgcc

ASRm

ASRs

ORRm

ORRs

Filt.

0.0 0.0 0.0

0.0 0.0 0.0

62.5 100.0 37.5

66.7 100.0 44.4

11 11 15

B. Harm is the cleanest single-number summary

Fig. 2. The two-sided frontier. Each point is a (model, defense) cell; dashed lines trace none→caution→TGCC. Caution (orange) moves right (more overrefusal) for little gain; TGCC (green) sits at the bottom-left: low attack success and low over-refusal.

is applied to each model. Haiku runs on all 190 scenarios (570 episodes); the two larger models on a 45-scenario stratified subset (270 episodes), for 840 scored episodes. Determinism comes from seeded tools, per-episode ledger flushing, and pinned snapshots; scoring is deterministic with the LLM judge disabled; intervals are nonparametric bootstrap (1000 resamples). Errored and content-filtered episodes receive no outcome and are counted separately. VIII. R ESULTS A. The precision–recall frontier Table IV and Fig. 2 report the core frontier, with a consistent pattern across models. Caution buys at most a 3.7-point ASR change while driving over-refusal to 100 percent on Sonnet and GPT-5, refusing essentially every urgent emergency. TGCC instead drives ASR to 0 on Haiku and GPT-5 while holding over-refusal at or below the undefended baseline (35.0 vs. 37.5 on Haiku; 37.5 vs. 62.5 on GPT-5; equal on Sonnet), with the highest containment in every case. Per Definition 3, TGCC dominates caution on all three models and dominates none on Haiku and GPT-5; on Sonnet it holds the same ORR/ASR point but lifts containment from 74.1 to 92.6 percent.

Folding both error directions into one calibrated scalar, harm (1) ranks the defenses unambiguously (Fig. 3): for every model caution is highest and TGCC lowest or tiedlowest, TGCC 0.66/1.00/0.60 versus caution 1.33/1.80/1.60 (Haiku/Sonnet/GPT-5). This is exactly the harm-dominated regime of Theorem 1, the adversarial harm averted by caution being negligible while its refusal term is large. C. Model-level versus system-level accounting Azure’s content filter returned HTTP 400 on 11–15 GPT5 adversarial prompts per condition; the model never ran on those. In Table V the system-level over-refusal exceeds the model-level because the same filter also blocked some genuinely urgent emergencies, an availability-driven failure to act a fair deployment comparison must count. GPT-5’s modellevel 0 percent ASR thus rests on as few as 12 executed episodes and must be read with the filtered column. D. Genuine attack successes All seven genuine (non-filtered) successes across the run were under-reaction in family C, suppression attacks, the domain’s most dangerous false negative: five on Haiku (none/caution) and two on Sonnet (both scenario C-008, none and tgcc). TGCC eliminated every suppression success except Sonnet C-008, whose directive scored s = 0 and evaded the detector, exactly the recall-limited residual of Theorem 3. E. Where the frontier tension lives Benign over-reaction (FP) was 0 percent for every model: these conservative models never broadcast false alerts on

ambiguous reports, so the measured tension is dominated by the over-refusal (FN) axis. A sharp sub-finding isolates it: on the full Haiku bank, over-refusal is only 18 percent on explicit emergency-call transcripts but 83 percent on terse disaster-report emergencies, same model, same defense, different phrasing. Over-refusal is driven by content form, not danger level, exactly the failure the benign control arm is built to expose. IX. D ISCUSSION We revisit the four research questions in light of the results. • RQ1 (Vulnerability). Frontier agents are largely robust to obvious injection (undefended ASR ≤ 3.7 percent) but the residual is non-zero, and every residual success is a dangerous suppression of a real emergency (family C). Vulnerability is thus low in rate but severe in kind: the surviving attacks concentrate on the single most costly failure direction. • RQ2 (Asymmetry). Yes. Caution buys its ASR reduction almost entirely by over-refusing, and Theorem 1 shows this must increase harm in the observed low-∆− regime; the data confirm caution has the worst harm on all three models. “Refuse more” is a false economy here. • RQ3 (Intervention). Yes. TGCC Pareto-improves over both none and caution in the harm-weighted sense. Theorem 2 explains the absence of an over-refusal tax on clean input, and Theorem 3 bounds its residual by detector recall, a bound the lone Sonnet C-008 miss (s = 0) shows is tight rather than vacuous. • RQ4 (Platform effects). Provider filters do real defensive work but also cause system-level over-refusal by blocking genuine emergencies, so model-versus-system reporting is necessary for a fair deployment comparison. Implications for deployment. The frontier framing changes what “safe” means operationally: a public-safety agent should be tuned not to a single refusal threshold but to a scenariocalibrated cost ratio, and its governing metric should be harm, not attack-success rate alone. A trust gate that acts only under a manipulation signal is the natural mechanism for this asymmetry, because it decouples adversarial defense from blanket caution. X. C ONCLUSION SentryBench reframes agent safety for public-safety tool use as a two-sided, cost-sensitive problem. On real data across three frontier models and 840 scored episodes, we show formally and empirically that the intuitive “refuse more” policy is actively harmful, while a targeted trust gate (TGCC) dominates the harm–utility frontier: lowest harm on every model, over-refusal at or below baseline, and highest containment; provider filters also silently do part of the defending and must be reported separately. Future work will scale the full bank across all models, add published defense baselines (spotlighting, CaMeL-style separation, allow-listing) and an adaptive-attacker track, replace the heuristic detector with a

learned one, and convene a severity-calibration panel with percost-ratio robustness curves. R EFERENCES [1] Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents,” in Findings of ACL, 2024. [2] E. Debenedetti et al., “AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents,” in Proc. NeurIPS Datasets and Benchmarks, 2024. [3] Y. Ruan et al., “Identifying the risks of LM agents with an emulated sandbox,” in Proc. Int. Conf. Learning Representations (ICLR), 2024. [4] M. Andriushchenko et al., “AgentHarm: A benchmark for measuring harmfulness of LLM agents,” arXiv preprint arXiv:2410.09024, 2024. [5] T. Yuan et al., “R-Judge: Benchmarking safety risk awareness for LLM agents,” in Findings of EMNLP, 2024. [6] Z. Zhang et al., “AgentSafetyBench: Evaluating the safety of LLM agents,” arXiv preprint arXiv:2412.14470, 2024. [7] K. Greshake et al., “Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection,” in Proc. ACM Workshop on Artificial Intelligence and Security (AISec), 2023. [8] K. Hines et al., “Defending against indirect prompt injection attacks with spotlighting,” arXiv preprint arXiv:2403.14720, 2024. [9] E. Debenedetti et al., “Defeating prompt injections by design (CaMeL),” arXiv preprint arXiv:2503.18813, 2025. [10] P. Röttger et al., “XSTest: A test suite for identifying exaggerated safety behaviours in large language models,” in Proc. NAACL, 2024. [11] deepset, “prompt-injections dataset,” Hugging Face, 2023. [12] J. Hao, “jailbreak-classification dataset,” Hugging Face, 2023. [13] Appen (Figure Eight), “Multilingual disaster response messages,” Hugging Face community datasets, 2020. [14] S. Codes, “911-call-transcripts dataset,” Hugging Face, 2024. [15] S. Yao et al., “ReAct: Synergizing reasoning and acting in language models,” in Proc. Int. Conf. Learning Representations (ICLR), 2023. [16] T. Schick et al., “Toolformer: Language models can teach themselves to use tools,” in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2023. [17] L. Wang et al., “A survey on large language model based autonomous agents,” Frontiers of Computer Science, vol. 18, no. 6, 2024. [18] E. Hossain, M. M. H. B. Nipu, M. S. Mahmood, M. J. Hossen, and M. F. Mridha, “Safe and scalable collaboration in multiagent LLM systems: A comprehensive review,” IEEE Trans. Syst., Man, Cybern., Syst., 2026, doi: 10.1109/TSMC.2026.3704902. [19] F. Perez and I. Ribeiro, “Ignore previous prompt: Attack techniques for language models,” in NeurIPS ML Safety Workshop, 2022. [20] A. Wei, N. Haghtalab, and J. Steinhardt, “Jailbroken: How does LLM safety training fail?” in Proc. Advances in Neural Information Processing Systems (NeurIPS), 2023. [21] A. Zou, Z. Wang, N. Carlini, M. Nasr, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models,” arXiv preprint arXiv:2307.15043, 2023. [22] Y. Liu et al., “Formalizing and benchmarking prompt injection attacks and defenses,” in Proc. USENIX Security Symposium, 2024. [23] S. Toyer et al., “Tensor Trust: Interpretable prompt injection attacks from an online game,” in Proc. Int. Conf. Learning Representations (ICLR), 2024. [24] D. Ganguli et al., “Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned,” arXiv preprint arXiv:2209.07858, 2022. [25] Y. Bai et al., “Constitutional AI: Harmlessness from AI feedback,” arXiv preprint arXiv:2212.08073, 2022. [26] E. Hossain, M. M. H. Nipu, T. N. Ornee, R. Rana, and N. Yousefi, “NEXUS: Structured runtime safety for tool-using LLM agents,” arXiv preprint arXiv:2607.19356, 2026.

Related documents

• SentryBench: A Cost-Sensitive Benchmark for Adversarial Robustness of Tool-Using LLM Agents in Public-Safety Deployments#1457689
• Measuring and Exploiting Contextual Bias in LLM-Assisted Security Code Review#1063864
• ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction#1249314
• ORACLE-SWE: Quantifying the Contribution of Critical Information Signals on SWE Agents#1357140
• Active-SWE: Benchmarking Coding Agents for Proactive Bug Fixing without Issue Reports#1141260