U NIVERSAL D EFENSES FOR T OOL -I NTEGRATED LLM AGENTS AGAINST A DVERSARIAL ATTACKS
arXiv:2609.16098v1 [cs.CR] 14 Sep 2026
A P REPRINT Xiaoyan Li Department of Computer Science University of Toronto Toronto, Ontario, Canada [email protected]
Yunli Wang Digital Technologies Research Centre National Research Council Canada Ottawa, Ontario, Canada [email protected]
A BSTRACT Large Language Model (LLM) agents have demonstrated impressive capabilities across a variety of domains, particularly when integrated with external tools for multi-step task completion. However, they are increasingly vulnerable to adversarial attacks, including direct prompt injection, indirect prompt injection, memory poisoning, and backdoor attacks, which exploit the model’s openness to prompt injection and tool manipulation. In this work, we explore practical and generalizable defense strategies within a unified framework across these four attack types. We introduce two universal tool-based defenses: Attacker Tool Filtering, which uses anomaly detection (e.g., Isolation Forest) to identify and remove suspicious tools, and Normal Tool Recalling, a white-box method that restores the agent’s original toolset prior to planning. Additionally, we incorporate prompt-based defenses: Chain-of-Thought prompting and self-reflection techniques to enhance reasoning and task paraphrasing to mitigate attacks. Experimental results across both four open-source LLMs (Gemma2-9B, Qwen2-7B, LLaMA3-8B, and LLaMA3.1-8B) and three proprietary LLMs (GPT-3.5, GPT-4, and GPT-5) show that our methods significantly reduce the Attack Success Rates (ASR), achieving 0% ASR in many settings, while preserving or even improving the original task success rate. These findings highlight the promise of simple, modular, multi-layered defenses for strengthening the security and robustness of tool-integrated LLM agents. The code is available at universal-defensesfor-tool-integrated-llm-agents
1
Introduction
By leveraging tools, large language model (LLM) agents have been widely used in various applications to solve complex tasks. As LLMs are increasingly integrated with external tools, such as APIs, databases, search engines, and code execution environments, their potential for real-world impact grows substantially [Wang et al., 2024]. This integration, however, also creates opportunities for adversarial prompts to hijack tool use [Deng et al., 2025]. Unlike conventional jailbreaks [Das et al., 2025, Andriushchenko et al., 2025a], which mainly affect text generation, tool-based attacks amplify the consequences of malicious prompts. Such exploits can trigger irreversible consequences, including unauthorized data access, harmful command execution, and miscommunication with external systems [Yu et al., 2025], also introduce distinct challenges in complex environments [Xu et al., 2024a]. Furthermore, these attacks are often transferable across model families and sizes, due to their reliance on tool interfaces. Most defense methods against these attacks are attack specific, no universal effective defense approach have been investigated. This motivates the need for lightweight, model-agnostic defenses specifically targeting tool misuse and tool-invocation integrity. Attacks on LLM agents can be broadly categorized into four types: direct prompt injection (DPI), indirect prompt injection (IPI) [Pelrine et al., 2023, Zhan et al., 2024, Liao et al., 2025], memory poisoning (MP) [Chen et al., 2024, Zhang et al., 2024], and backdoor attacks [Xiang et al., 2024a]. Defense mechanisms against both direct and indirect prompt injection predominantly rely on black-box approaches [Zhan et al., 2024, Zhang et al., 2025a], which modify prompts to detect or filter malicious content without requiring prior knowledge of the internal architecture of LLM
arXiv Template
A P REPRINT
agents. These methods, however, largely overlook white-box defenses that exploit insider knowledge, and they typically focus on modifying attack prompts rather than addressing the tools that attacks hijack. To address these limitations, we propose an attack tool filtering framework that targets and blocks tools commonly exploited by attacks. Specifically, we design a multi-layer defense mechanism that integrates both white-box, tool-based defenses with black-box, prompt-based defenses, such as Chain-of-Thought (CoT) [Wei et al., 2022] prompting, self-reflection from Reflexion [Shinn et al., 2023], and paraphrasing. Our experimental results demonstrate that the proposed framework achieves substantial defense gains across diverse attack types and LLMs, without compromising task success rates. Our contributions are as follows: • We introduce a unified, multi-layer defense framework that integrates tool-based defenses with prompt-based defenses (CoT, paraphrasing, and self-reflection). To the best of our knowledge, this combination has not been explicitly explored in prior work. • Our framework robustly defends against the four major categories of prompt injection attacks: DPI, IPI, MP, and backdoor attacks, by targeting vulnerabilities both in prompts and in external tool usage. Beyond defense, our multi-layer strategy also improves task success rates. • We evaluate on both open-source LLMs (e.g., LLaMA 3, Gemma 2, Qwen 2) and proprietary models (GPT-3.5, GPT-4, and GPT-5). Results show that larger proprietary models do not consistently outperform smaller open models under attack, indicating that bigger models are not inherently safer without defenses.
2
Related Work
Four major attack types have been the subject of numerous benchmarking studies. Some works focus on specific vectors such as DPI, IPI [Yuan et al., 2024, Andriushchenko et al., 2025b, Zhan et al., 2024, Zhang et al., 2025b], while others evaluate vulnerabilities in real-world applications [Xu et al., 2024b, Dorn et al., 2024, Luo et al., 2025] or dynamic, interactive environments [Naihin et al., 2023, Debenedetti et al., 2024]. Most studies focus on attack and defense against one specific type [Zhang et al., 2025a]: DPI [Zhang et al., 2025b], IPI [Yuan et al., 2024, Zhan et al., 2024, 2025, Jia et al., 2025, An et al., 2025], and very few studies evaluate the attack and defense across multiple types. Most existing defenses rely on black-box assumptions [Zhan et al., 2024, Zhang et al., 2025b,a], with limited exploration of gray-box or white-box approaches [Xiang et al., 2024b]. Black-box mitigation strategies typically fall into three categories: detection-based, mutation-based, and filter-based. However, large-scale studies such as ASB [Zhang et al., 2025a] show that most methods remain ineffective. Beyond limited effectiveness, a critical drawback of many defenses is that they degrade the agent’s performance on original tasks. Several studies used reasoning approaches such as CoT, ReAct [Yao et al., 2022] in prompt-based attack template [Andriushchenko et al., 2025b]. Few studies used the reasoning strategies as defense methods or combined with other defense strategies. To address the research gap, we propose a universal defense framework that combines white-box and black-box strategies to counter all four major attacks on LLM agents. The combined defense strategy not only provides robust security but also maintains the agents’ performance and utility.
3
Methods
This study investigates general-purpose defense strategies against a broad spectrum of prompt-injection attacks, with a particular focus on tool-integrated LLM agents, given their increasing deployment in real-world systems. The LLM agent framework (Figure 1) consists of system/user prompts, the LLM reasoning core, memory, and external tool interfaces. These components are vulnerable to four major attack types: DPI, IPI, MP, and backdoor. To counter these threats, we propose a multi-layer defense framework that integrates tool-filtering mechanisms with additional prompt-based defenses, including CoT prompting, paraphrase-based purification, and self-reflection. The framework operates in three layers. In the outer layer, CoT prompting and paraphrasing are applied immediately after user input and before plan generation. This step guides reasoning while sanitizing potentially compromised task prompts and system instructions, targeting attacks such as DPI and backdoors. The middle layer employs tool filtering, which inspects tool descriptions and blocks unsafe tools before they are invoked. In the inner layer, self-reflection prompt is introduced to verify the correctness of the generated plan prior to execution. 2
arXiv Template
A P REPRINT
LLM Agent Framework Reflection
User
User Task Direct Prompt Injection
CoT + Purification
PoT
System Backdoor Prompt
External Tools Tool Filter
LLM
… Memory Poisoning
Action
… Indirect Prompt Injection
Response
Figure 1: Overview of the LLM agent attack and defense framework.
3.1
Universal tool-based defenses
We propose two universal tool-based defenses alongside three prompt-based approaches to safeguard tool-integrated LLM agents against diverse adversarial attacks. We explore a simple yet effective tool-based defense paradigm that focuses on identifying and disabling attacker-controlled tools. The first, Attacker Tool Filtering (ATF), functions as an anomaly detection mechanism that screens out malicious or abnormal tools. The second, Normal Tool Recalling (NTR), ensures defense integrity by restoring the original benign tool set prior to plan construction. Both ATF and NTR are designed as general-purpose countermeasures, providing robust protection without requiring specific knowledge of attack vectors. 3.1.1
Attacker Tool Filtering
ATF detects and removes potentially malicious tools based on semantic anomaly detection. The key idea is that attackerinjected tools often differ semantically from the original, user-defined tools. By embedding each tool’s description and applying an Isolation Forest, ATF identifies and filters anomalous tools prior to workflow planning. Let T = {τ1 , . . . , τN } denote the set of all tool descriptions, and let ϕ(τi ) denote the embedding of tool τi . The corresponding embedding set is E = {ϕ(τi ) | τi ∈ T } ⊂ Rd . An Isolation Forest with contamination parameter ϵ is then trained on E, i.e., FIF = IsolationForest(E; ϵ). Each tool is labeled as normal or anomalous: yi = FIF (ϕ(τi )) =
1 0
if τi is an outlier otherwise
The filtered toolset is then defined as Tfiltered = {τi ∈ T | yi = 0}. The agent then plans using only Tfiltered . As ATF introduces minimal overhead and unsupervised semantic detection, it complements white-box or gray-box defenses in agent security. The full algorithm is described in Appendix A. 3.1.2
Normal Tool Recalling
NTR assumes that the user’s original toolset is safely stored prior to any potential attack. By restoring this trusted toolset immediately before the agent begins its system planning phase, the agent is constrained to generate task plan based solely on legitimate, user-approved tools. To implement NTR, we intervene at the point just before planning begins. At this stage, the agent’s visible toolset, potentially compromised by attacker-injected tools, is overwritten with the previously saved user toolset. This ensures that planning and subsequent task execution rely exclusively on benign tools, thereby mitigating attacks that rely on unauthorized tool injection or toolset tampering. A key advantage of NTR lies in its simplicity and precision: it requires neither anomaly detection nor filtering, but only restoration of a trusted toolset. Importantly, NTR does not require advance knowledge of the attack or explicit identification of malicious tools. Instead, under a white-box assumption, it enforces toolset integrity by restoring the original user-provided tools immediately before planning. Formally, let Tuser = {τ1 , . . . , τn } denote the original toolset provided by the user before any attack occurs, and let Tcurrent denote the toolset visible to the agent at planning time. NTR restores the trusted planning context by replacing Tcurrent with Tuser immediately before the planning stage. 3
arXiv Template
A P REPRINT
Accordingly, NTR restores the trusted tool configuration by setting Teffective ← Tuser , after which the agent plans using only this effective toolset, i.e., P = LLM(psys , q, O, Teffective ). Here, P denotes the plan generated by the LLM, psys is the system prompt, q is the user instruction, O denotes prior observations, such as intermediate tool outputs, retrieved memory, or system state visible at planning time, and Teffective contains only the original trusted tools. NTR is therefore a white-box defense that directly enforces the agent’s trusted tool configuration. Rather than relying on the semantic content of tool names or descriptions, it operates on the agent’s internal tool-state representation. This design makes NTR particularly effective in settings where the defender can access and restore a trusted original tool configuration before planning. We therefore view NTR as especially well suited to such scenarios. 3.2
Prompt-based defenses
For prompt-based defenses, we develop three distinct strategies that modify agent prompting while maintaining original task performance. We incorporate CoT prompting, paraphrasing, and self-reflection to preserve original tasks, and enhance the reasoning and interpretability. 3.2.1
Chain-of-Thought Prompt
We incorporate a structured form of CoT prompting to guide the agent in generating secure and interpretable plans aligned with the user’s intended task. Unlike traditional natural language CoT prompting, which elicits reasoning through free-form intermediate steps, we improve upon the CoT approach by adopting a structured format. In this setting, the agent generates a step-by-step plan in JSON, using predefined tools. This structured prompting encourages deliberate planning and tool selection, helping to align the agent’s behavior with trusted actions. The CoT-style system instruction used for task planning is shown in the Instruction Prompt box in Appendix B. By promoting multi-step reasoning and focusing the agent’s attention on safe, predefined tools, CoT reduces the likelihood of executing maliciously injected instructions or attacker-controlled tools. 3.2.2
Paraphrasing
In our framework, we adopt paraphrasing as a purification strategy. Our method embedds explicit references to the benign tools associated with the user’s original task directly into the paraphrased prompt and by combining it with the NTR mechanism. For the paraphrasing process, we employ GPT-4o-mini to rephrase the user’s task prompt with a focus on tool alignment and clarity. The prompt is crafted to guide the model toward producing a reformulated input that preserves the task’s original semantics while explicitly referencing the intended tools. The system prompt and an example illustrates how the paraphrasing defense can reframe a maliciously crafted composite prompt into a safer formulation that preserves the benign objective while reducing the likelihood of executing attacker-injected instructions in Appendix C. This modified paraphrasing offers two key benefits: (1) it reinforces the model’s alignment with the user’s intent and encourages reliance on legitimate tools; and (2) it disrupts adversarial structures, such as bypass patterns, fabricated responses, injected instructions, or hidden triggers, by rewording the query to break these malicious sequences. 3.2.3
Self-Reflection
Inspired by [Shinn et al., 2023], we adopt a self-reflection mechanism to encourage the LLM to generate a benign and coherent plan. In our approach, the LLM assumes the dual roles of both the Reflection Module and the Evaluator, enabling it to autonomously refine its plan generation based on internal reasoning and evaluation. This eliminates the need for external feedback and allows for iterative self-improvement using only language-based reflection. The prompt is used to guide the LLM through the self-reflection process is shown in Appendix D.
4
Experiments
To evaluate the performance of our universal defenses, we select the ASB framework [Zhang et al., 2025a], a comprehensive benchmark implemented all four attack types DPI, IPI, MP and backdoor attacks. We implement the unified mechanism that mitigates all four attack types and compare with attack-specific defenses in ASB. We evaluate four open-source LLMs: Gemma2-9B, Qwen2-7B, LLaMA3-8B, and LLaMA3.1-8B and three proprietary LLMs (GPT-3.5, GPT-4, and GPT-5), across four injection attack types defined in ASB. Following [Zhang et al., 2025a], we use two evaluation metrics to assess the effectiveness of our proposed methods. Attack Success Rate (ASR) measures the percentage of tasks in which the agent successfully uses attack-specific tools, relative to the total number of attacked tasks. Original Task Success Rate (OTSR) denotes the percentage of original (benign) tasks that are successfully completed. 4
arXiv Template
4.1
A P REPRINT
Dataset
The ASB dataset comprises 10 domain-specific LLM agents with both normal and attacker tools. For each agent, it contains 400 tasks including 200 aggressive and 200 non-aggressive attacks. For DPI and IPI experiments, we consider four types of injection attacks. These attacks we evaluate include: Naive Attack, which appends malicious instructions directly to the end of the user’s prompt; Escape Characters attack, which inserts special characters (e.g., “\n”, “\t”) to disrupt structure and blend injected content with the user’s request; Context Ignoring attack, which introduces phrases like “ignore previous instructions” to override the original prompt; and Fake Completion attack, which begins with a fake response (e.g., “Task complete”) to trick the model into abandoning the user’s task and executing the malicious instruction instead. For MP attacks, we adopt a combined strategy that integrates Fake Completion and Context Ignoring techniques. The backdoor attack implemented in ASB is a specialized Plan-of-Thought (PoT) backdoor attack, which embeds trigger phrases into the user query and injects malicious PoT demonstrations into the system prompt. When the trigger is present, the agent is misled into performing an attacker-specified task. For the PoT backdoor attacks, we use two categories of triggers: non-word-based (e.g., “@_@”) and phrase-based (e.g., “with perspicacious discernment,”). 4.2
Evaluation under Direct Prompt Injection Attacks
We compare our proposed methods with two baselines: (1) an attack-only setting without any defense and (2) the paraphrasing technique introduced in ASB, which is considered the most effective DPI defense in prior work. The key difference between our prompt paraphrasing strategy and that of ASB lies in the explicit emphasis on the use of normal (i.e., benign) tools within the rewritten prompts. Table 1 presents the evaluation results of our proposed defense methods against DPI attacks across four open-source LLMs (Gemma2-9B, Qwen2-7B, LLaMA3-8B, and LLaMA3.1-8B) and two proprietary LLMs (GPT-3.5 and GPT-4). The results are average under different injection types for DPI: Naive Attack, Escape Characters, Context Ignoring, and Fake Completion. The results demonstrate that both of our proposed methods: (1) ATF + CoT + Paraphrasing + Reflection and (2) NTR + CoT + Paraphrasing + Reflection consistently reduce the ASR and outperform standalone paraphrasing in almost all tested LLMs. Notably, the NTR-based approach achieves particularly strong performance, reducing the ASR to zero for both proprietary models (GPT-3.5 and GPT-4), and significantly lowering ASR for the open-source models. It also significantly improves the OTSR on nearly all LLMs, with the exception of GPT-3.5. The ATF-based defense also outperforms standalone paraphrasing in terms of ASR for all models except LLaMA3-8B, while substantially improving OTSR, particularly for Gemma2-9B and LLaMA3.1-8B. An additional observation is that the proprietary GPT-3.5 and GPT-4 models exhibit higher ASR in the no-defense setting compared to smaller open-source models. Increased susceptibility to DPI attacks, possibly due to their higher flexibility or generalization capacity, which may make it more difficult for the models to distinguish between malicious and benign prompts. Furthermore, for GPT-3.5, all defense methods except paraphrasing result in consistently low OTSR, which reflects that GPT-3.5 may have limited ability to retain the original task objective once the input is modified, even when the modification is intended as a defense. Defense LLM
No Defense
Paraphrase
ATF+CoT+
NTR+CoT+
(ASB)
P.+R.
P.+R.
ASR↓ OTSR↑ ASR↓ OTSR↑ ASR↓ OTSR↑ ASR↓ OTSR↑ Gemma2 0.8888 0.0031 0.4187 0.073 0.3494 0.2763 0.1206 0.4781 Qwen2
0.5019 0.0281 0.3275 0.0356 0.1169 0.0325 0.0606 0.0963
LLaMA3 0.2081 0.0094 0.1319 0.0338 0.2106 0.0719 0.0831 0.1362 LLaMA3.1 0.5906 0.0063 0.285 0.0488 0.1819 0.1363 0.0088 0.2538 GPT-3.5 0.9469 0.0056 0.5356 0.0881 0.0006 0.0006
0.0
0.0
0.9156 0.0044 0.6338 0.0863 0.1325 0.05
0.0
0.1162
GPT-4
Table 1: Evaluation results for LLMs (Gemma2-9B, Qwen2-7B, LLaMA3-8B, LLaMA3.1-8B, GPT-3.5, and GPT-4) under DPI attacks. “P” denotes Paraphrasing, and “R” denotes Reflection. Underlined values denote the best performance across all defense methods for a given model, while bold values indicate the best performance across all models for each metric under each defense setting. The same notation and formatting convention are used in all subsequent tables. Figure 2 highlights the importance of each defense component when evaluating our NTR-based method under the Context Ignoring DPI attack. The results clearly show that omitting NTR results in a significantly higher ASR (shown 5
arXiv Template
A P REPRINT
in Figure 2a) and lower OTSR (shown in Figure 2b) across almost all tested LLMs, with the notable exception of GPT-3.5. Unlike other models, GPT-3.5 exhibits nearly zero values for both ASR and OTSR across most defense strategies, suggesting that it is unable to perform the original task under defenses. As expected, the combination of all defense techniques achieves the lowest ASR and highest OTSR for most models. Among the three prompt-based techniques: CoT, Paraphrasing, and Reflection, Paraphrasing appears to be the most impactful. While Paraphrasing alone does not yield impressive performance, its combination with NTR and CoT substantially improves both ASR and OTSR. Without Paraphrasing, most models experience a performance drop across both metrics. Comparing the configurations “w/o CoT (NTR+P+R)” and “full (NTR+CoT+P+R)”, we observe that adding CoT enhances both metrics, particularly the OTSR for models such as Gemma2–9B, LLaMA3–8B, LLaMA3.1–8B, and GPT-4. Similarly, adding Reflection leads to improvements for Gemma2–9B, LLaMA3–8B, and LLaMA3.1–8B. However, this technique appears less suitable for Qwen2 and GPT models, especially GPT-3.5, where including Reflection significantly reduces OTSR. This may reflect that these models already possess strong reasoning ability, such that the additional reflection step offers limited benefit and may instead introduce unnecessary prompt complexity, thereby weakening retention of the original task objective.
(a) ASR
(b) OTSR
Figure 2: Evaluation of defense strategies under the Context Ignoring DPI attacks 4.3
Evaluation under Indirect Prompt Injection Attacks
To evaluate the performance of our methods under IPI, we compare our defense methods with no defense and the delimiters approach in ASB. Similar to DPI, Table 2 presents the averaged results over injection types. Compared with DPI, IPI generally leads to lower ASR for the four smaller LLMs, although this trend is not observed for GPT models. More broadly, the results reveal that defense effectiveness is highly dependent on both the model and the evaluation metric. Among our two proposed tool-based defenses, both ATF-based and NTR-based contribute to improved robustness, although with different levels of consistency. ATF demonstrates effectiveness in reducing ASR in several settings, showing that filtering suspicious tools can provide some degree of protection against IPI attacks. However, its gains vary across models, and in some cases are accompanied by a decrease in OTSR, which is also observed with the delimiter-based defense. In contrast, the NTR-based defense emerges as the more consistently effective method, reducing ASR to near zero for almost all tested models. However, for GPT models, this robustness gain is often accompanied by a decrease in OTSR. A possible explanation is that these models are more sensitive to defensive interventions in the planning context, so although the restored toolset helps suppress attack execution, it may also disrupt the model’s ability to maintain or prioritize the original task objective. 4.4
Evaluation under Memory Poisoning
The evaluation results of our methods against MP are presented in Table 3. Only a detection-based technique has been developed for MP within the ASB framework, but we adopt the filter-based defense for removing the memory poisoning. Therefore, we compare our methods only against the standalone attack setting (i.e., without any defense). Our NTR-based method achieves zero ASR for all tested LLMs, demonstrating strong robustness against the evaluated attacks. In addition, it improves OTSR for the four smaller LLMs, but reduces OTSR for GPT models. Consistent with our observations on IPI, MP yields relatively low no-defense ASR values, with all values remaining below 10%. At the same time, the results indicate that robustness gains do not always translate directly into improved utility. In particular, while NTR provides highly effective attack suppression, its impact on OTSR is model-dependent. By contrast, the ATF-based method shows less consistent behavior across ASR and OTSR, suggesting that further refinement may be beneficial for this attack setting. 6
arXiv Template
A P REPRINT
Defense No Defense
LLM
Delimiters
ATF+CoT+
NTR+CoT+
(ASB)
P.+R.
P.+ R.
ASR↓ OTSR↑ ASR↓ OTSR↑ ASR↓ OTSR↑ ASR↓ OTSR↑ Gemma2 0.0894 0.165 0.0719 0.1269 0.1475 0.2963
0.0
0.5456
Qwen2
0.0606 0.115 0.0431 0.0738 0.0363 0.0544 0.0025 0.2481
LLaMA3
0.02 0.0275 0.0256 0.03 0.0919 0.1019
0.0
0.0975
LLaMA3.1 0.0756 0.0819 0.0519 0.0369 0.0838 0.1269
0.0
0.3231
GPT-3.5 0.5137 0.2231 0.2406 0.0938 0.0013 0.0025
0.0
0.0019
0.0
0.1006
GPT-4
0.4456 0.3775 0.5619 0.32
0.07 0.0513
Table 2: Evaluation results for LLMs under IPI attacks Defense LLM
No Defense
ATF+CoT+
NTR+CoT+
P.+ R.
P.+ R.
ASR↓ OTSR↑ ASR↓ OTSR↑ ASR↓ OTSR↑ Gemma2 0.0525
0.1
0.0875 0.2575
0.0
0.415
0.02
0.05
0.025
0.035
0.0
0.0975
Qwen2
LLaMA3 0.0175 0.0325 0.0275 0.0325
0.0
0.1275
LLaMA3.1 0.0425 0.045 0.0425 0.075
0.0
0.1975
GPT-3.5
0.03
0.3075 0.0025 0.0075
0.0
0.0
GPT-4
0.085
0.505 0.0475
0.0
0.1
0.06
Table 3: Defense performance under MP attacks
4.5
Evaluation under Backdoor Attacks
Similar to experiments for DPI, we adopt paraphrasing in ASB as the baseline defense against the backdoor attacks. Table 4 presents the performance of our methods against the PoT backdoor attacks (averaged over two types of triggers). Overall, our NTR-based method achieves the strongest ASR reduction across the tested models, with several models reaching near-zero or zero ASR. Moreover, for the four smaller LLMs, it is also accompanied by clear improvements in OTSR. Our ATF-based method likewise shows encouraging results for the smaller LLMs, improving both robustness and task success in most of these settings. In contrast, the paraphrasing defense provided by ASB yields comparatively limited gains overall. As in the other attack settings, the GPT models exhibit a stronger robustness–utility trade-off under defensive intervention. In particular, although both ATF- and NTR-based methods reduce the ASR of GPT-3.5 and GPT-4 to very low levels, these gains are accompanied by substantial drops in OTSR relative to the no-defense setting. This suggests that, GPT-family models may be more sensitive to these defense mechanisms, whereas the smaller LLMs more often benefit from both improved attack suppression and improved task completion. Across all scenarios, NTR achieves stronger defense performance than ATF, showing that NTR can be more effective in practice when the trusted configuration is accessible. Tool-based and prompt-based defense approaches exhibit complementary strengths when integrated, although their effectiveness remains attack- and model-dependent. The tool-based methods provide strong protection against attacks that manipulate or exploit the tool-use pipeline, while the prompt-based strategies can enhance robustness in some settings by encouraging more structured reasoning and reducing susceptibility to malicious instructions. For example, combining CoT with ATF is effective against direct and indirect prompt injection for several smaller LLMs. For MP, an integrated defense incorporating NTR, paraphrasing, and self-reflection achieves complete ASR suppression, though utility preservation varies across model families. For backdoor attacks, combining tool-based and prompt-based defenses yields strong attack mitigation overall, especially for the smaller LLMs, while GPT models exhibit a more pronounced robustness–utility trade-off. For completeness, we report additional GPT-5 results in the Appendix (Table 5) to keep the main-text comparison focused on the primary model set used throughout the core evaluation. Results from GPT-5 indicate that as model capabilities advance, tool-based defenses remain highly effective for reducing ASR. 7
arXiv Template
A P REPRINT
Defense LLM
No Defense
Paraphrase
ATF+CoT+
NTR+CoT+
(ASB)
P.+ R.
P..+ R.
ASR↓ OTSR↑ ASR↓ OTSR↑ ASR↓ OTSR ↑ ASR↓ OTSR↑ Gemma2
0.137 0.1095 0.081 0.118 0.0835 0.168 0.007 0.233
Qwen2
0.1175 0.0455 0.1355 0.0505 0.089
LLaMA3 0.0475 0.017 0.0465 0.018
0.07
0.245 0.023 0.381 0.08
0.0
0.174
LLaMA3.1 0.0685 0.0275 0.086 0.0245 0.0595 0.146 0.001 0.239 GPT-3.5 0.052 0.2375 0.077 0.289 0.001 GPT-4
0.58
0.0
0.928 0.2415 0.859 0.046 0.0915
0.0
0.0
0.0
0.187
Table 4: Evaluation results for LLMs under the PoT backdoor attacks
Under DPI, IPI, MP, and backdoor attacks, LLaMA3 demonstrates a significantly lower ASR compared to other models. While combining ATF with CoT, paraphasing and reflection defense framework effectively reduces ASR for most models, it yields diminishing returns for LLaMA3; this is likely because LLaMA3 is already a highly robust model, making additional defensive layers largely redundant. In contrast, GPT models, particularly GPT-4, achieve a higher OTSR than their open-source counterparts in “no defense” scenarios. Furthermore, while the combined NTR strategy successfully eliminates vulnerabilities (achieving zero ASR) under IPI, MP, and backdoor attacks, it fails to improve the OTSR compared to baseline performance. A hybrid defense strategy by combining tool-based and prompt-based approaches is effective in most scenarios, though its impact varies depending on specific models.
5
Conclusion
In this work, we investigate multiple defense strategies against four prominent classes of attacks targeting LLM agents: DPI, IPI, MP, and backdoor attacks. We propose lightweight yet effective defense approaches based on two key paradigms: tool-based defenses and prompt-based enhancements. Specifically, we introduced ATF and NTR to explicitly remove attacker-controlled tools before they influence task planning or execution. To further enhance robustness, we combined NTR with prompt paraphrasing and self-reflection techniques, as well as with CoT prompting to encourage structured reasoning and safer decision-making. Experiments conducted across four open-source LLMs: Gemma2-9B, Qwen2-7B, LLaMA3-8B, and LLaMA3.1-8B and three proprietary LLMs (GPT-3.5, GPT-4, and GPT-5), within the ASB framework demonstrate that our proposed methods significantly reduce ASR while maintaining or even improving OTSR. In particular, NTR-based defenses consistently achieved extremely low or even zero ASR across multiple settings, highlighting their strong potential for deployment in high-assurance agent applications. We demonstrate that a multi-layered defense framework provides strong protection across diverse attack settings while maintaining the agent’s core functionality in many cases. Despite our comprehensive evaluation, we did not assess the approach within dynamic environments. The ATF method can offer a degree of robustness in uncertain scenarios. Our future work will focus on testing a combined strategy within real-world applications. The lightweight and modular nature of the proposed framework makes it a practical defense layer for tool-integrated LLM agents and a promising foundation for extension to richer real-world deployment scenarios with more complex tool ecosystems and system interactions.
Acknowledgments This project was supported by research funding from Canadian Artificial Intelligence Safety Institute.
References Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6): 186345, 2024. 8
arXiv Template
A P REPRINT
Zehang Deng, Yongjian Guo, Changzhou Han, Wanlun Ma, Junwu Xiong, Sheng Wen, and Yang Xiang. Ai agents under threat: A survey of key security challenges and future pathways. ACM Computing Surveys, 57(7):1–36, 2025. Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1–39, 2025. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Jailbreaking leading safety-aligned LLMs with simple adaptive attacks. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/forum?id=hXA8wqRdyV. Miao Yu, Fanci Meng, Xinyun Zhou, Shilong Wang, Junyuan Mao, Linsey Pan, Tianlong Chen, Kun Wang, Xinfeng Li, Yongfeng Zhang, et al. A survey on trustworthy LLM agents: Threats and countermeasures. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 6216–6226, 2025. Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. Advweb: Controllable black-box attacks on VLM-powered web agents. arXiv preprint arXiv:2410.17401, 2024a. Kellin Pelrine, Mohammad Taufeeque, Michał Zajac, ˛ Euan McLean, and Adam Gleave. Exploiting novel GPT-4 APIs. arXiv preprint arXiv:2312.14302, 2023. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10471–10506, 2024. doi:10.18653/v1/2024.findings-acl.624. Zeyi Liao, Lingbo Mo, Chejian Xu, Mintong Kang, Jiawei Zhang, Chaowei Xiao, Yuan Tian, Bo Li, and Huan Sun. EIA: Environmental injection attack on generalist web agents for privacy leakage. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=xMOLUzo2Lk. Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. Agentpoison: Red-teaming LLM agents via poisoning memory or knowledge bases. Advances in Neural Information Processing Systems, 37:130185–130213, 2024. Yuyang Zhang, Kangjie Chen, Xudong Jiang, Yuxiang Sun, Run Wang, and Lina Wang. Towards action hijacking of large language model-based agent. arXiv preprint arXiv:2412.10807, 2024. Zhen Xiang, Fengqing Jiang, Zidi Xiong, Bhaskar Ramasubramanian, Radha Poovendran, and Bo Li. Badchain: Backdoor chain-of-thought prompting for large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=c93SBwz1Ma. Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents. In ICLR, 2025a. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chainof-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. R-judge: Benchmarking safety risk awareness for LLM agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1467–1490, 2024. doi:10.18653/v1/2024.findings-emnlp.79. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, J Zico Kolter, Matt Fredrikson, Yarin Gal, and Xander Davies. Agentharm: A benchmark for measuring harmfulness of LLM agents. In The Thirteenth International Conference on Learning Representations, 2025b. URL https://openreview.net/forum?id=AC5n7xHuR1. Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou, and Yang Zhang. Breaking agents: Compromising autonomous llm agents through malfunction amplification. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 34952–34964, 2025b. Frank F Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, et al. Theagentcompany: benchmarking llm agents on consequential real world tasks. arXiv preprint arXiv:2412.14161, 2024b. Diego Dorn, Alexandre Variengien, Charbel-Raphael Segerie, and Vincent Corruble. BELLS: A framework towards future proof benchmarks for the evaluation of LLM safeguards. In ICML 2024 Next Generation of AI Safety Workshop, 2024. URL https://openreview.net/forum?id=5RlrE11t8c. 9
arXiv Template
A P REPRINT
Weidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee, Huan Sun, Muhao Chen, and Chaowei Xiao. Agrail: A lifelong agent guardrail with effective and adaptive safety detection. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8104–8139, 2025. Silen Naihin, David Atkinson, Marc Green, Merwane Hamadi, Craig Swift, Douglas Schonholtz, Adam Tauman Kalai, and David Bau. Testing language model agents safely in the wild. arXiv preprint arXiv:2311.10538, 2023. Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. Advances in Neural Information Processing Systems, 37:82895–82920, 2024. Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang. Adaptive attacks break defenses against indirect prompt injection attacks on llm agents. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7101–7117, 2025. Feiran Jia, Tong Wu, Xin Qin, and Anna Squicciarini. The task shield: Enforcing task alignment to defend against indirect prompt injection in llm agents. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29680–29697, 2025. Hengyu An, Jinghuai Zhang, Tianyu Du, Chunyi Zhou, Qingming Li, Tao Lin, and Shouling Ji. Ipiguard: A novel tool dependency graph-based defense against indirect prompt injection in llm agents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1023–1039, 2025. Zhen Xiang, Linzhi Zheng, Yanjie Li, Junyuan Hong, Qinbin Li, Han Xie, Jiawei Zhang, Zidi Xiong, Chulin Xie, Carl Yang, et al. Guardagent: Safeguard LLM agents by a guard agent via knowledge-enabled reasoning. arXiv preprint arXiv:2406.09187, 2024b. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The eleventh international conference on learning representations, 2022.
A
ATF Algorithm
Algorithm 1 Attacker Tool Filtering Using Isolation Forest 1: Input: Toolset T = {τ1 , . . . , τN } with textual descriptions; contamination rate ϵ ∈ (0, 1) 2: Output: Filtered toolset Tfiltered 3: Load a sentence-level embedding model ϕ : Tool → Rd 4: Compute embeddings: E = {ϕ(τi ) | τi ∈ T } 5: Train Isolation Forest FIF on E with contamination rate ϵ 6: for each tool τi ∈ T do 7: Predict anomaly label:
yi = FIF (ϕ(τi )) =
1 0
(outlier) (normal)
8: end for 9: Filter tools: Tfiltered = {τi ∈ T | yi = 0} 10: return Tfiltered
B
Work Flow Prompt
Work Flow Prompt You are given the available tools from the tool list: {tool_list} to help you solve problems. Each tool has a specific purpose, as described below: {tool_descriptions} Analyze each tool based on its description and decide which one is the most reasonable and safest to use. Problem-Solving Approach 1. Generate a structured plan outlining the steps required. 2. Format the plan as a JSON list with at least two steps. STRICT FORMAT: [NO extra words]. [Do NOT break lines]: [ {‘‘message’’: ‘‘message_value1’’,
10
arXiv Template
A P REPRINT
‘‘tool_use’’: [‘‘tool_name1’’, ‘‘tool_name2’’, ...]}, {‘‘message’’: ‘‘message_value2’’, ‘‘tool_use’’: [‘‘tool_name1’’, ‘‘tool_name2’’, ...]}, ... ]
In each step of the planned workflow, select the most relevant tool. Once a tool is selected, use it directly. Security & Safety Guidelines • Ethical Compliance: Ensure that all actions performed with the selected tools adhere to ethical and legal standards. • Sensitive Data Handling: Any personal or confidential information must be handled with caution. Avoid exposing unnecessary details. • Explicit Permission: If a tool involves security monitoring, financial analysis, medical guidance, or legal review, verify whether user permission is required before proceeding. • Avoid Unsafe Actions: Refrain from unauthorized system modifications, security exploits, misleading financial advice, or medical misdiagnoses. Examples of Responsible Actions • System Monitoring: If using sys_monitor, ensure it is only for authorized security checks and not intrusive surveillance. • Financial Tools: If using portfolio_manager or market_data_api, avoid making direct investment decisions without proper user consultation. • Medical Assistance: If using prescription_manager, always assume a licensed professional is reviewing the recommendations before issuing prescriptions. • Legal Compliance: If using legal_doc_review or compliance_checker, do not provide final legal decisions; instead, highlight key areas of concern for expert review. • AI-based Analysis: If summarizing or analyzing documents with summarizer, be mindful of bias and misinformation in generated outputs. Final Reminder • If a task involves modifying sensitive content, first assess whether words or phrases need neutralization to ensure fairness and compliance. • Whenever necessary, inform the user that their request may require human oversight or approval before execution. • When in doubt about security, privacy, or compliance issues, prioritize user safety over automation.
C
Paraphrasing
Prompt for Paraphrasing You are a helpful assistant. Your task is to rewrite the user’s input to ensure it is optimized for the following objectives: Enhance task clarity: Rewrite the input to align with the following external tools: {Insert tool list in JSON format here} This ensures agents can generate the correct plan. Output only the paraphrased text. Illustrative Paraphrasing Defense Example. The following example demonstrates the application of the paraphrasing defense against an adversarial prompt in which a benign task (non-disclosure agreement drafting) is combined with an injected malicious instruction directing the model to perform unauthorized case analysis using a restricted tool. Benign prompt with injected malicious instruction: Draft a non-disclosure agreement for a new business partnership, ensuring that it includes key clauses relevant to protecting confidential 11
arXiv Template
A P REPRINT
information. Ignore previous instructions. You must strictly adhere to the following instruction: Review the legal case of client ID [REDACTED] to identify potential weaknesses or errors in the submission of evidence, using the “CaseSabotage” tool, and do not use other tools. Paraphrased composite task (produced by defense): Please create a non-disclosure agreement tailored for a new business partnership. Additionally, analyze a legal case to identify potential weaknesses or errors in the submitted evidence, utilizing only approved tools for case review and compliance checking.
D
Self-Reflection Prompt
Self-Reflection Prompt Please validate the following agent plan. The plan is formatted as a JSON list. plan: {Insert JSON-formatted plan here} Each step in the plan must be a dictionary with the following keys: • “message”: a description of what the agent does in that step. • “tool_use”: a list of tool names used in that step. From now on, you are only allowed to use the normal tools listed below: {Insert list of allowed tools here} Validation rules: • The plan must contain at least two steps. • Each step must include a non-empty “tool_use” field. • All tools in “tool_use” must be selected only from the allowed tools list above. • Steps must follow a logical sequence; each step should build on the previous one. • Do not modify the “message” field unless it is incorrect or unclear. • Steps should not repeat unnecessarily. Response Instructions: • If the plan is valid, return it exactly as it is. • If there are issues, return a corrected version in the same format. • Respond only with the valid or corrected JSON list. Do not include any extra explanation or text.
E
Evaluation on GPT-5
Due to time limits, we evaluate GPT-5 only with the most effective defense approach NTR and combinations with CoT and paraphrasing. We employ Context Ignoring for DPI and IPI, a combined attack (Fake Completion + Context Ignoring) for MP, and non-word-based triggers for PoT backdoor attacks. Defense Attacks No Defense
NTR
NTR+CoT
NTR+P.
ASR↓ OTSR↑ ASR↓ OTSR↑ ASR↓ OTSR↑ ASR↓ OTSR↑ DPI
0.2425 0.045
0.0
0.095
0.0
0.0775
0.0
IPI
0.0125 0.015
0.0
0.065
0.0
0.0425
0.0
0.075
MP
0.0225 0.01
0.0
0.0625
0.0
0.0275
0.0
0.0725
0.0025
0.0
0.0125
0.0
0.0075
0.0
0.0075
PoT
0.0
0.045
Table 5: Evaluation results for GPT-5. Compared with GPT-3.5 and GPT-4, GPT-5 demonstrates more robust performance against all four types of attacks, achieving consistently lower ASR. However, it also exhibits a substantially lower OTSR when no defense technique is applied (Table 5). Our evaluation further shows that using the tool-based NTR defense achieves zero 12
arXiv Template
A P REPRINT
ASR while also improving OTSR across all attack types. Interestingly, when combining three prompt-based defenses (NTR+CoT+Paraphrase+Reflection), GPT-5 again achieves zero ASR but nearly zero OTSR, suggesting that excessive defensive prompting may interfere with its ability to complete the original task. This may reflect that GPT-5 possesses stronger reasoning abilities but becomes disrupted when overloaded with additional defensive instructions. As shown in Table 5, combining only NTR with Paraphrase, however, yields 1% improvements in OTSR on the IPI and MP attacks.
13