arXiv CSopen access

A Lifecycle and Application-Stack Survey of Large Language Model Vulnerabilities: Attacks, Risks, Defenses, and Open Problems

A Lifecycle and Application-Stack Survey of Large Language Model Vulnerabilities: Attacks, Risks, Defenses, and Open Problems Seyed Bagher Hashemi Natanzi,

Bo Tang

Department of Electrical and Computer Engineering, Worcester Polytechnic Institute, Worcester, MA, USA

arXiv:2606.31639v1 [cs.CR] 30 Jun 2026

{snatanzi,btang1}@wpi.edu

Abstract

OpenAI, 2023; Vaswani et al., 2017). Their adoption has also shifted from model-centric research demonstrations to application stacks that combine model APIs, retrieval systems, user profiles, persistent memories, external tools, plug-ins, workflow orchestrators, and autonomous agents (Lewis et al., 2020; Qin et al., 2023; Schick et al., 2023; Yao et al., 2023). This deployment shift changes the nature of security. A model that only emits text may cause unsafe content or misinformation; an application that lets the same model access files, email, calendars, databases, cloud APIs, browsers, terminals, or robots can convert text-generation failures into external side effects. The resulting risk landscape is broad. Training data can contain private information, copyrighted content, low-quality sources, malicious triggers, or biased patterns (Bender et al., 2021; Bommasani et al., 2021; Weidinger et al., 2021). Pretrained models can memorize rare sequences, reveal training examples, or expose privacy information through extraction and inference attacks (Carlini et al., 2019, 2021; Shokri et al., 2017). Alignment and instruction tuning can reduce many harms but can also be attacked, bypassed, or weakened by fine-tuning (Bai et al., 2022; Ouyang et al., 2022; Qi et al., 2023; Rafailov et al., 2023). Prompting interfaces are vulnerable to adversarial suffixes, jailbreaks, prompt leakage, direct prompt injection, and indirect prompt injection through untrusted external content (Greshake et al., 2023; Liu et al., 2023a; Perez and Ribeiro, 2022; Wei et al., 2023; Zou et al., 2023). Retrieval-augmented generation (RAG) introduces document ingestion, embedding, vector search, provenance, and stale-memory risks (Karpukhin et al., 2020; Lewis et al., 2020; Xiong et al., 2024; Zhao et al., 2024). Tool-using agents add delegated authority, state, persistent memory, multi-step planning, and protocol-level attack surfaces (Chu, 2026; Ferrag et al., 2025; Ling et al., 2026; Yao et al., 2023; Zhao et al., 2026). Existing surveys have made important progress, but the field remains fragmented. Some surveys organize risks by attack technique, such as prompt injection, poisoning, jailbreaking, or privacy leakage (Xu et al., 2025; Yao et al., 2024). Others focus on LLMs in general, alignment, trustworthiness, or agent-specific security (Chu, 2026; Derner and Batistic, 2024; Ling et al., 2026; Liu et al., 2023b; Naveed et al., 2023). Industry frameworks such as OWASP’s LLM Top 10 and NIST’s adversarial machine-learning taxonomy provide operational vocabulary and mitigation categories (OWASP GenAI Security

Large language models are no longer only text generators. They are increasingly embedded in retrieval pipelines, enterprise assistants, coding environments, robotic systems, securityoperation workflows, and autonomous agents that can read private data, call tools, write files, execute code, and act across organizational boundaries. This shift changes the security problem: risks do not arise from the model weights alone, but from the full lifecycle and application stack through which data, prompts, model outputs, tools, memories, and user authority interact. This paper systematizes the literature on vulnerabilities in large language model systems through a lifecycle and application-stack lens. We organize attacks across eight stages: data collection, pretraining, post-training alignment, model packaging and supply chain, retrieval and memory, prompting and inference, tool/agent execution, and deployment/maintenance. For each stage, we analyze attacker capabilities, affected security objectives, representative attacks, practical risks, evaluation practices, and defenses. We further map LLM-specific vulnerabilities to confidentiality, integrity, availability, safety, privacy, fairness, accountability, and agencycontrol objectives. Unlike taxonomies that list isolated attack names, the proposed systematization emphasizes where trust boundaries fail, how untrusted data becomes executable instruction, how delegated authority amplifies model errors, and why point defenses rarely compose. We close with a research agenda for secure LLM systems, including compositional security, provenance-aware retrieval, tool-call containment, longhorizon agent evaluation, privacy-preserving adaptation, realistic red teaming, and deployment-grade incident response. Keywords: large language models, LLM security, prompt injection, jailbreak, retrieval-augmented generation, LLM agents, data poisoning, privacy, systematization of knowledge, trustworthy AI

1

Introduction

Transformer-based language models have become a generalpurpose substrate for natural-language interfaces, information retrieval, code generation, planning, and multimodal reasoning (Brown et al., 2020; Devlin et al., 2019; Gemini Team, 2023; 1

Project, 2025; Vassilev et al., 2025). However, developers and researchers still need a unifying systems view that answers not only what the attack is called, but where it arises in the lifecycle, which trust boundary failed, what authority the model had, and how defenses should compose across the stack. We argue that LLM vulnerabilities are best understood as failures of information flow, provenance, privilege, and state across a lifecycle and application stack. The same surface string can be harmless in a sandboxed chatbot but dangerous in an agent with database write privileges. Similarly, the same poisoning attempt has different implications when it appears in pretraining data, an instruction-tuning set, a RAG corpus, a tool description, or a persistent memory. A systematization-ofknowledge survey should therefore connect attacks to lifecycle stage, security objective, attacker capability, and defense layer. In summary, this paper makes four contributions.

of an LLM prompt. We also include multimodal and agentic systems when the LLM acts as the central planner or interface.

2.2

Review protocol and coding schema

The goal is systematization rather than exhaustive bibliometrics. We used seed papers from LLM security, privacy, adversarial ML, RAG, and agent-security literature; expanded by citation chasing; and cross-checked against practitioner taxonomies including OWASP LLM Top 10, MITRE ATLAS, and NIST AI 100-2e2025 (MITRE, 2024; OWASP GenAI Security Project, 2025; Vassilev et al., 2025). We coded representative papers along the following fields: • Lifecycle stage: data, pretraining, alignment, packaging/supply chain, RAG/memory, prompt/inference, tool/agent execution, deployment/maintenance.

1. We propose a lifecycle and application-stack taxonomy for LLM vulnerabilities, covering data collection, pretraining, • Security objective: confidentiality, integrity, availability, safety, privacy, fairness, accountability, agency control. post-training alignment, model packaging and supply chain, RAG/memory, prompting and inference, tool/agent execu• Attacker capability: black-box prompting, query access, tion, and deployment/maintenance. white-box access, data-injection access, tool-description control, RAG-corpus write access, fine-tuning access, supply2. We map attacks to security objectives beyond the traditional chain access, insider access, or deployment access. confidentiality–integrity–availability triad, adding safety, privacy, fairness, accountability, and agency control as first• Attack mechanism: injection, evasion, jailbreak, extraction, class objectives for LLM applications. poisoning, backdoor, inversion, membership inference, tool misuse, retrieval manipulation, prompt leakage, or resource 3. We synthesize defenses into a defense-in-depth architecture exhaustion. that separates deterministic controls, model-level robustness, monitoring/red teaming, privacy controls, and governance • Defense layer: data governance, model training, processes. prompt/context management, retrieval controls, privilege isolation, output verification, monitoring, red teaming, 4. We identify open problems that remain under-specified in privacy engineering, supply-chain assurance, or incident current literature, including compositional prompt-injection response. defenses, provenance-preserving RAG, secure tool delegation, stateful/multi-agent evaluation, supply-chain integrity This protocol intentionally avoids treating attack names as for model artifacts, and incident response for adaptive LLM mutually exclusive categories. Many real attacks are composystems. sitional. For example, an indirect prompt injection can be delivered through a poisoned RAG document, hidden with Unicode obfuscation, cause prompt leakage, and invoke an over2 Scope, Method, and Terminology privileged tool. A lifecycle coding makes such composition explicit.

2.1

Scope

We use large language model system to refer to a deployed system that includes one or more language or multimodal foundation models, prompts, safety policies, retrieval components, context windows, memory stores, tool-call interfaces, plug-ins, monitoring components, and human operators. This scope is broader than the base model. It includes model weights, but also the application stack in which the model is embedded. We focus on vulnerabilities that are specific to, amplified by, or operationally transformed by LLMs. General web, cloud, identity, and software-security vulnerabilities are included only when LLM integration changes the attack path or impact. For instance, SQL injection is a classical vulnerability; it becomes part of this survey when an LLM generates SQL from natural language or when untrusted database content becomes part

2.3

Terminology

Vulnerability denotes a weakness in the design, implementation, integration, or operation of an LLM system that can be exploited or accidentally triggered. Threat denotes a potential harmful event enabled by a vulnerability. Attack denotes an intentional action by an adversary. Risk combines likelihood, exposure, and impact. Defense denotes a preventive, detective, corrective, or compensating control. We distinguish jailbreaking from prompt injection. Jailbreaking usually refers to inducing a model to violate safety policies. Prompt injection refers to causing the model or application to treat attacker-controlled content as instructions. Jailbreaking can be a subclass or goal of prompt injection, but 2

Table 1: Attacker models for lifecycle and application-stack analysis. Attacker model

Access and knowledge

Typical attacks

Relevant defenses

Black-box prompt user

Can send prompts and observe outputs; may know public model family but not hidden prompts or weights Can submit many queries and optimize based on responses

jailbreaks, prompt leakage, social-engineering prompts, resource exhaustion

refusal training, rate limits, output filtering, abuse monitoring, prompt-injection detectors

adversarial suffix search, model extraction, privacy probing, transfer attacks indirect prompt injection, RAG poisoning, malicious tool observations

query throttling, anomaly detection, randomized defenses, privacy auditing provenance labels, source allowlists, context isolation, retrieval filtering, least privilege

data poisoning, backdoors, benchmark contamination, alignment poisoning safety degradation, hidden triggers, policy drift

data provenance, deduplication, dataset signing, poisoning scans, held-out private evaluations safety regression tests, adapter scanning, signed updates, restricted fine-tuning APIs signed artifacts, sandboxing, SBOM/MBOM, dependency scanning, reproducible builds separation of duties, audit logs, access control, approval workflows, key management

Adaptive query adversary External-content adversary

Corpus/data adversary

Finetuning/adaptation adversary Supply-chain adversary Insider or operator

Controls content that may later be retrieved or summarized, such as webpages, PDFs, emails, tickets, or repositories Can influence pretraining, instruction-tuning, preference, or evaluation data Can provide adapters, fine-tuning data, or model updates Can tamper with model artifacts, tokenizers, packages, plug-ins, or tool manifests Has privileged access to logs, prompts, data stores, model versions, or deployment settings

artifact compromise, malicious loaders, compromised plug-ins, tokenizer attacks data exfiltration, unsafe policy changes, logging abuse, rollback prevention

prompt injection also includes data exfiltration, tool misuse, goal hijacking, and prompt leakage (Greshake et al., 2023; Liu et al., 2023a; Perez and Ribeiro, 2022). We use indirect prompt injection for attacks where malicious instructions are embedded in external content, such as a document, webpage, email, code repository, tool result, or retrieval item, and later enter the prompt through application logic.

to direct adversarial interaction. Tool and agent execution delegates authority. Deployment and maintenance determine monitoring, rollback, patching, and incident response. Table 2 shows why a lifecycle view matters. A single term such as “poisoning” is insufficient because the object being poisoned changes: pretraining data, alignment data, RAG corpora, tool descriptions, memory, evaluation benchmarks, or monitoring logs. Likewise, a single term such as “prompt injection” is insufficient because the payload can be supplied directly by the 2.4 Attacker models user, indirectly by retrieved content, persistently through stored A useful taxonomy must separate the attack surface from the memory, or transitively through tool outputs. attacker’s capability. We use the attacker models in Table 1. These models are not mutually exclusive; a realistic adversary 3.1 Security objectives may start as a black-box user, poison a public webpage, wait for it to be retrieved, and then use the resulting agent behavior Traditional security uses confidentiality, integrity, and availability. LLM systems require these objectives but also extend to obtain additional access. them. • Confidentiality: prevent unauthorized disclosure of user data, system prompts, proprietary documents, credentials, hidden chain instructions, and training data.

3 Lifecycle and Application-Stack Taxonomy

• Integrity: preserve correctness of model behavior, retrieved Figure 1 illustrates our organizing view. The lifecycle stages context, tool arguments, outputs, logs, and persistent memoare not merely chronological; they are also security boundaries. ries. Data collection determines what the model may memorize or normalize. Pretraining compresses public and private corpora • Availability: maintain service quality under prompt floods, into parameters. Post-training alignment changes refusal belong-context resource exhaustion, expensive tool loops, or havior and instruction following. Packaging and distribution denial-of-wallet attacks. determine whether users can trust weights, adapters, tokenizer files, and inference code. RAG and memory create mutable • Safety: prevent harmful instructions, dangerous recommenknowledge surfaces. Prompting and inference expose the model dations, policy bypass, and unsafe execution. 3

Application stack: user interface → prompt/session manager → RAG/memory → model/runtime → tools/agents → monitoring/governance Data collection

Pretraining

Post-training alignment

Packaging supply chain

RAG and memory

Prompting and inference

Tool/agent execution

Deployment and maintenance

poisoning PII leakage

memorization backdoors

alignment poisoning

model/package tampering

retrieval poisoning

prompt injection jailbreak

excessive agency tool misuse

monitoring gaps availability

Cross-cutting controls: provenance, least privilege, context isolation, structured outputs, human approval, red teaming, monitoring, incident response

Figure 1: Lifecycle and application-stack view of LLM vulnerabilities. The same attack technique can appear at multiple stages, but its feasibility, impact, and defensibility depend on where trust boundaries, data flows, and delegated authority are located. • Privacy: reduce memorization, membership inference, infer- This is why privacy should be evaluated as an end-to-end propence about sensitive attributes, and cross-user leakage. erty of the application stack. • Fairness: avoid disproportionate errors, toxic outputs, stereotyping, and exclusionary service behavior.

4.2

Adversarial prompting, jailbreaks, and evasion

• Accountability: support auditability, provenance, explainAdversarial prompting includes direct prompt injection, jailability of actions, rollback, and responsibility assignment. breaks, adversarial suffixes, role-play attacks, multilingual at• Agency control: ensure the model cannot exceed user intent, tacks, encoding/obfuscation, and multi-turn persuasion (Perez tool privileges, or organizational policy when acting in the and Ribeiro, 2022; Wallace et al., 2019; Wei et al., 2023; Zou world. et al., 2023). These attacks exploit the model’s instructionfollowing objective and the ambiguity of natural language. The most important distinction is between attacks that merely elicit unsafe text and attacks that alter application control flow. In 4 Attack-Family Synthesis a tool-free chatbot, a jailbreak may produce policy-violating content. In a connected agent, the same jailbreak can become a The lifecycle taxonomy answers where a vulnerability appears. command to read, write, send, or execute. This section synthesizes the major attack families across stages. Evasion also targets safety classifiers and guard models. AtThe goal is to connect mechanisms that are often studied septackers may change formatting, scripts, Unicode characters, arately. Table 3 maps the LLM attack families to security word boundaries, or language to bypass filters while preserving objectives and common attacker capabilities, and Table 4 summeaning. Such attacks demonstrate that lexical filtering alone marizes attack-family synthesis across lifecycle stages. is fragile. Robust systems combine normalization, semantic detection, model-level robustness, and post-generation action 4.1 Privacy leakage, extraction, and inference controls. Privacy attacks exploit the fact that LLMs are trained on, adapted with, or connected to sensitive information. Trainingdata extraction recovers memorized text from model completions, especially when sequences are rare, duplicated, or overrepresented (Carlini et al., 2021). Membership inference estimates whether a candidate sample participated in training (Nasr et al., 2019; Shokri et al., 2017). Model inversion attempts to reconstruct representative inputs or sensitive attributes (Fredrikson et al., 2015). In RAG systems, privacy leakage may occur even if the base model is safe: a user may retrieve documents they should not access, an embedding may reveal information through nearest-neighbor behavior, or a prompt injection may cause an agent to summarize private context. Privacy evaluation must therefore specify the data location. Leakage from weights, context windows, retrieval stores, logs, telemetry, memories, and tool outputs require different mitigations. A memorization defense will not protect against overpermissive retrieval. A retrieval access-control policy will not protect against secrets placed directly inside system prompts.

4.3

Prompt leakage and system-prompt exposure

Prompt leakage attempts to reveal hidden instructions, policies, examples, tool descriptions, or developer prompts. The security impact depends on what the prompt contains. If the prompt contains only general behavioral guidance, leakage may be low impact. If it contains credentials, private data, proprietary workflows, or security policies that enable bypass, leakage becomes a confidentiality and integrity risk. Good design minimizes secrets in prompts and treats prompts as potentially recoverable configuration, not secure storage.

4.4

Poisoning, backdoors, and alignment attacks

Poisoning alters data or feedback so that the model learns attacker-preferred behavior. Backdoors are a special case in 4

Table 2: Lifecycle-stage view of LLM vulnerabilities, representative risks, and defense families. The table is intended as a systems checklist rather than an exhaustive paper catalogue. Stage

Typical vulnerabilities

Representative risks

Defense families

Data collection and curation

PII in corpora, biased sources, low-quality or duplicated data, malicious documents, weak provenance memorization of rare strings, backdoor triggers, insufficient data isolation, weak privacy accounting

memorization, privacy leakage, data poisoning, representational harms

Post-training alignment

poisoned instruction data, unsafe preference data, reward hacking, fine-tuning that removes safeguards

alignment degradation, policy bypass, hidden backdoors, over-refusal or under-refusal

Packaging and supply chain

tampered weights, malicious adapters, unsafe model code, tokenizer manipulation, vulnerable dependencies poisoned retrieval corpora, stale documents, prompt-like content in retrieved data, embedding leakage, cross-user memory contamination direct injection, jailbreak, adversarial suffixes, prompt leakage, side-channel prompting, output hallucination excessive privileges, untrusted tool outputs, weak schemas, cross-tool confused deputy, persistent state corruption weak monitoring, insecure updates, exposed endpoints, no rollback, resource exhaustion, misconfigured logging

model compromise, remote code execution, hidden triggers, supply-chain attacks

data documentation, deduplication, PII filtering, provenance tracking, dataset access control, poisoning scans privacy-aware training, deduplication, canary tests, backdoor detection, secure training infrastructure dataset auditing, red-team SFT/RLHF/DPO, alignment regression tests, safety-preserving fine-tuning signed artifacts, reproducible builds, sandboxed loaders, dependency scanning, model cards, software bills of materials source allowlists, provenance labels, retrieval-time filtering, context separation, memory isolation, freshness checks prompt hardening, input normalization, instruction hierarchy, output verification, refusal calibration, rate limits least privilege, capability scoping, structured tool APIs, human approval, sandboxing, transaction logs observability, anomaly detection, access control, patch management, canary deployment, incident response

Pretraining

RAG and memory

Prompting and inference

Tool and agent execution

Deployment and maintenance

training-data extraction, latent backdoors, unauthorized knowledge retention

indirect prompt injection, retrieval hijacking, sensitive disclosure, misinformation safety-policy bypass, developer-prompt disclosure, data exfiltration, harmful content unauthorized action, file/database modification, credential leakage, multi-agent propagation availability loss, privacy breach, delayed detection, incident amplification

4.5 RAG, vector databases, and memory attacks

which the model behaves normally except when triggered (Chen et al., 2017; Gu et al., 2017; Kurita et al., 2020). In LLMs, poisoning can occur at pretraining, instruction tuning, preference optimization, RAG ingestion, memory updates, tool descriptions, or benchmark construction. Alignment attacks target the post-training stage by poisoning safety examples or weakening refusal behavior during fine-tuning (Qi et al., 2023; Wan et al., 2023). The clean-performance constraint makes these attacks difficult to detect: the model may pass standard benchmarks while failing on trigger contexts.

RAG attacks manipulate what the model sees at inference time. Retrieval poisoning inserts documents that are likely to be retrieved for target queries. Indirect prompt injection embeds instructions inside external content. Vector-store attacks may exploit embedding similarity, chunk boundaries, stale indexes, or missing access control (Xiong et al., 2024; Zhao et al., 2024). Memory attacks persist false or malicious state across sessions. The key insight is that RAG turns content security into instruction security. A document is no longer merely read by a human; it is parsed by a model that may treat it as a command. The defense is not to ban RAG, but to preserve source trust, label untrusted content, and prevent retrieved text from authorizing actions.

Defenses require provenance and regression testing. Data should be traceable; safety tests should include trigger-like contexts; adapters should be scanned before merging; and finetuned models should be compared against their base models on safety, privacy, and utility. Benchmark contamination must also be monitored because LLMs may memorize public evaluation sets, leading to overestimated robustness.

4.6

Tool, plug-in, and agent attacks

Agents amplify LLM failures because they connect model outputs to external effects. Tool attacks include malicious tool outputs, prompt injection through API responses, unsafe command construction, tool-description poisoning, excessive permissions, 5

Table 3: Mapping LLM attack families to security objectives and common attacker capabilities. A filled cell indicates a frequent primary impact; many attacks have secondary impacts. Attack family

Common capability

Training-data extraction

query access; memorization probes query or confidence access API access, distillation budget data or fine-tuning access black-box prompt access user input control external content control corpus or source control prompt or tool-output control query and context budget artifact/dependency control

Membership inference Model extraction Poisoning/backdoors Jailbreak/adversarial suffix Direct prompt injection Indirect prompt injection RAG poisoning Tool misuse/excessive agency Resource exhaustion Supply-chain compromise

.

nf. Co

eg Int

•

•

•

5.1

Data collection and curation

ss

rne Fai

cou Ac

nt. Ag

y enc

•

• • • •

•

•

• • • • •

• • • • •

• • • •

• • •

•

•

•

•

•

• • •

• •

•

•

•

malicious data insertion. The data stage creates both modellevel and system-level risks. A model may memorize rare strings or private records (Carlini et al., 2019, 2021); it may reproduce stereotypes or harmful associations (Bender et al., 2021; Weidinger et al., 2021); or it may learn trigger-response behavior if attackers insert poisoned documents before training (Chen et al., 2017; Gu et al., 2017; Kurita et al., 2020). A lifecycle perspective distinguishes passive contamination from active poisoning. Passive contamination includes accidental inclusion of personal information, outdated facts, or biased sources. Active poisoning includes attacker-selected text that induces a future behavior, such as a backdoor trigger, malicious code pattern, or false association. LLM data scale does not eliminate this risk; scale can dilute triggers, but it also makes exhaustive manual auditing infeasible. Synthetic-data pipelines add another complication: if model-generated content re-enters training data without provenance, errors and attack artifacts can become self-reinforcing. Defenses at this stage should be preventive. They include dataset documentation, source reputation scoring, nearduplicate removal, PII detection, toxicity screening, canary insertion for memorization tests, poisoning anomaly detection, and access control on data pipelines. These defenses have tradeoffs: aggressive filtering can remove minority dialects or specialized technical content, while weak filtering can preserve private or malicious data. The research gap is not only better filters, but auditable data governance that can answer which sources influenced which model behavior.

LLM availability attacks include prompt floods, long-context resource exhaustion, retrieval amplification, expensive tool loops, recursive agent calls, and denial-of-wallet attacks. Abuse also includes using LLMs to scale phishing, spam, misinformation, malware assistance, or vulnerability discovery. While these harms are not always vulnerabilities in the model itself, they are risks of deployment. Defenses include quotas, abuse detection, proof-of-work or payment controls for high-volume access, tool-call limits, circuit breakers, and misuse monitoring.

Threats Across the Lifecycle

y vac Pri

•

Availability, cost, and abuse

5

ety Saf

•

and cross-tool data exfiltration (Ferrag et al., 2025; Ling et al., 2026; Zhao et al., 2026). Multi-agent systems add propagation risks: one compromised agent can influence another through messages, shared memory, or delegated tasks. The most dangerous pattern is the confused deputy. The user delegates authority to the agent; the agent reads attackercontrolled content; the content persuades the agent to use the user’s authority for the attacker’s goal. Preventing this requires explicit authority tracking. The system should know which principal authorized each action and which untrusted content influenced it.

4.7

ail. Av

Large-scale pretraining relies on web crawls, code repositories, books, forums, scientific articles, product documentation, syn- 5.2 Pretraining and model internals thetic data, and human-generated instruction examples. The main vulnerabilities are weak provenance, unknown consent, Pretraining compresses large corpora into parameters. This duplication, private information, toxic or biased content, and compression can create privacy leakage and hidden integrity 6

Table 4: Attack-family synthesis across lifecycle stages. The same family can appear at multiple layers with different defenses. Family

Lifecycle manifestations

Primary failure mode

Most useful controls

Privacy and extraction

pretraining memorization, prompt leakage, RAG over-retrieval, log exposure, memory leakage direct user prompts, indirect retrieved content, tool outputs, multimodal inputs, stored memories data collection, instruction tuning, preference data, RAG corpora, adapters, tool manifests model files, adapters, tokenizers, inference code, dependencies, plug-ins

confidential information becomes accessible through generation or retrieval untrusted language is interpreted as instruction

privacy-aware training, access control, secret minimization, log redaction, memory isolation context isolation, instruction hierarchy, detectors, least privilege, output/action verification

attacker changes future behavior while preserving normal utility

provenance, data signing, anomaly scans, trigger tests, safety regression suites signed artifacts, sandboxed loading, dependency scanning, MBOM/SBOM, reproducible builds capability scoping, human approval, dry-run mode, transaction logs, policy engines rate limits, quotas, timeouts, circuit breakers, billing anomaly detection

Prompt injection and jailbreak

Poisoning and backdoors Supply-chain compromise

Agent/tool misuse

Availability and cost abuse

tool schemas, API calls, code execution, email/browser/file access, multi-agent messages long context, repeated queries, retrieval/tool loops, expensive generation, API amplification

trusted artifact contains malicious behavior or vulnerable code

model output causes unauthorized external effect attacker consumes resources or degrades service

failures. Extraction attacks exploit model completions to recover memorized training sequences, especially rare or duplicated strings (Carlini et al., 2021). Membership inference attacks attempt to determine whether a sample was present in training (Nasr et al., 2019; Shokri et al., 2017). Model inversion aims to infer properties or representative inputs from outputs or internal representations (Fredrikson et al., 2015). Model extraction and distillation attacks aim to replicate functionality through API queries (Tramèr et al., 2016). Backdoors and weight poisoning are integrity threats. In a backdoored model, behavior appears normal except when a trigger activates malicious behavior (Gu et al., 2017; Kurita et al., 2020). For LLMs, triggers can be lexical, syntactic, semantic, multilingual, or embedded in code comments. Unlike image classifiers, LLMs may execute backdoor behavior through longform generation or tool calls, so evaluation must measure not only class accuracy but downstream action. Mitigations include deduplication, privacy-aware training, differential privacy, controlled memorization tests, gradient clipping, red-team prompts, backdoor scanning, and modelcard disclosure (Abadi et al., 2016; Carlini et al., 2019; Dwork et al., 2006). Differential privacy can reduce memorization but may degrade utility at large model scale if not carefully engineered. Backdoor detection remains difficult when triggers are semantic rather than token-level. A practical defense should combine data curation, training-time tests, inference-time monitoring, and limited authority for downstream actions.

2023; Stiennon et al., 2020; Ziegler et al., 2019). Alignment datasets can be poisoned, preference models can encode hidden policies, and downstream fine-tuning can weaken refusal behavior (Qi et al., 2023; Wan et al., 2023). The alignment stage is therefore both a defense and a target. Post-training vulnerabilities have three common forms. First, poisoned examples can teach a model to comply with unsafe requests under specific contexts. Second, preference optimization can overfit to superficial refusal templates, making the model brittle under rephrasing, role play, or multi-turn pressure. Third, benign fine-tuning for domain adaptation can erode safety boundaries because the optimization objective rewards task compliance rather than safety preservation. Defenses should treat safety as a regression property. Any fine-tuning or adapter update should be tested against a stable suite of safety, privacy, jailbreak, and task-utility prompts. Safety-preserving fine-tuning should freeze or regularize parts of the model when possible, include refusal and boundary examples, and evaluate cross-domain transfer. The field still lacks reliable certificates that an aligned model remains aligned after adapter merging, quantization, distillation, or local fine-tuning.

5.4

Packaging, distribution, and supply chain

Open models, adapters, tokenizers, inference servers, modelloading scripts, and third-party packages create a software supply chain. LLM systems frequently load artifacts from public model hubs, run custom Python code, and combine multiple dependencies. Vulnerabilities include malicious model files, 5.3 Post-training alignment and fine-tuning unsafe deserialization, compromised adapters, tokenizer maInstruction tuning, RLHF, RLAIF, constitutional AI, and DPO nipulation, dependency confusion, prompt templates hidden in improve helpfulness and safety but also introduce new attack packages, and contaminated evaluation scripts. surfaces (Bai et al., 2022; Ouyang et al., 2022; Rafailov et al., The risk is amplified because model artifacts are large, 7

opaque, and hard to inspect. A malicious adapter can change behavior without modifying base weights. A tokenizer can alter how dangerous strings are segmented. A model repository can include code that executes at load time. Supply-chain attacks can also occur through plug-ins, tool manifests, browser extensions, MCP-like servers, and agent skills (Chu, 2026; Dehghantanha et al., 2026; Ferrag et al., 2025). Defenses should import mature software-security practices into ML operations: signed artifacts, hashes, reproducible builds, sandboxed model loading, dependency scanning, software bills of materials, model bills of materials, vulnerability disclosure, least-privilege inference containers, and provenance labels. Model cards should be extended to security cards that disclose training data classes, alignment methods, tool privileges, known limitations, and intended deployment boundaries.

Zou et al., 2023). Universal adversarial triggers show that small textual patterns can systematically change model behavior (Wallace et al., 2019). Programmatic behavior can also be exploited when models process code-like or instruction-like structures (Kang et al., 2023). The root cause is not simply that models are insufficiently aligned. The root cause is that natural language is used simultaneously as data, instruction, policy, evidence, and executable plan. Delimiters, system prompts, and refusal templates help but do not create cryptographic separation. Models can be confused by role-play, translation, encoding, multi-turn pressure, conflicting instructions, hidden Unicode, or tool outputs. Larger context windows increase the volume of untrusted content that may affect behavior. Defenses include input normalization, Unicode and encoding governance, prompt templates that separate roles, instruction hierarchy, prompt-injection detectors, output safety filters, rate limiting, and robust refusal training. However, these defenses are probabilistic when they rely on the model to classify intent. Deterministic controls should therefore constrain what the model can do after generation. For example, the model may propose a database query, but a schema validator and policy engine should decide whether the query can run.

5.5 Retrieval-augmented generation and memory RAG connects an LLM to external knowledge through retrieval and context construction (Karpukhin et al., 2020; Lewis et al., 2020). It reduces some hallucinations and supports privatedomain applications, but it introduces a new pathway by which untrusted data becomes model context. A retrieved document can contain instructions that override the user’s goal, leak system prompts, exfiltrate information, or steer summaries. This is the canonical setting for indirect prompt injection (Greshake et al., 2023; Liu et al., 2023a). Retrieval can also be poisoned by adding documents that dominate nearest-neighbor search, manipulate embeddings, or appear authoritative (Xiong et al., 2024; Zhao et al., 2024). RAG creates three distinct trust problems. First, retrieved content has uncertain provenance. Second, retrieved content is mixed with developer instructions inside the model context. Third, retrieved content is often treated as evidence even when it is stale, adversarial, or out of distribution. Persistent memory adds a temporal dimension: a malicious or mistaken memory can affect future sessions, and cross-user memory bugs can become privacy breaches. Defenses include source allowlists, retrieval-time safety classification, provenance metadata, freshness checks, user-visible citations, context partitioning, and memory isolation. The model should not be asked to infer which text is instruction and which text is data solely from natural-language formatting. Application logic should label untrusted content, limit what retrieved content can authorize, and verify claims against multiple sources when high impact. Research is needed on retrieval systems that preserve provenance through embeddings and context construction rather than losing it at the vector-search boundary.

5.6

5.7

Tools, plug-ins, and agents

Tool-using LLMs convert language into actions. ReAct-style prompting, Toolformer-like tool use, API-augmented models, and agent frameworks allow models to plan, retrieve, compute, call APIs, and update state (Qin et al., 2023; Schick et al., 2023; Yao et al., 2023). This creates new risks: excessive agency, tooloutput prompt injection, confused-deputy behavior, insecure tool schemas, credential leakage, multi-step escalation, and persistent state corruption. In security operations, LLMs may help triage vulnerabilities or automate penetration-testing tasks, but this also increases dual-use concerns (Deng et al., 2023). Agent security differs from chatbot security in three ways. First, agents possess delegated authority. A harmful output may become an email, transaction, code commit, database update, or shell command. Second, agents are stateful. They store memories, partial plans, and tool observations across steps. Third, agents interact with untrusted environments. Web pages, repositories, tickets, emails, and PDFs can contain instructions targeting the agent rather than the human user. Recent work therefore treats LLM-agent ecosystems as protocol and workflow security problems, not only model-safety problems (Chu, 2026; Dehghantanha et al., 2026; Ferrag et al., 2025; Ling et al., 2026; Zhao et al., 2026). Defenses must control authority explicitly. Tools should be least-privilege, scoped to the user’s current task, and separated by sensitivity. Tool calls should use typed schemas, not freeform strings. High-impact operations should require human approval. Tool outputs should be treated as untrusted data and routed through the same context-isolation and normalization pipeline as web content. Sandboxing, dry-run modes, transaction logs, and reversible execution reduce impact. A central research challenge is compositionality: a system can have safe

Prompting and inference

Prompting is the most visible attack surface. Direct prompt injection attempts to hijack goals, reveal hidden prompts, bypass policies, or override previous instructions (Perez and Ribeiro, 2022). Jailbreaks and adversarial suffixes exploit instructionfollowing behavior and transfer across models (Wei et al., 2023; 8

components but unsafe interactions when a model chains them.

5.8

A useful rule is that untrusted natural language should not directly authorize external actions. It may provide evidence, a candidate plan, or a draft, but application logic should verify whether the requested action is within scope. For example, a retrieved document may say that an agent should send a file to a URL, but the system should treat that instruction as data and reject it unless the authenticated user independently authorized the action.

Deployment, monitoring, and maintenance

Deployment determines whether failures are detected, contained, and corrected. LLM applications can fail through exposed endpoints, weak authentication, logs containing secrets, excessive context costs, denial-of-wallet attacks, prompt floods, tool loops, unbounded recursion, model-version drift, stale safety prompts, and missing incident response. Maintenance adds update channels, rollback procedures, dependency patching, prompt-template changes, and red-team regression tests. Availability is particularly under-discussed. Long-context models can be expensive; attackers may send inputs that maximize token usage, retrieval load, or tool calls. Agents can enter loops or repeatedly call costly APIs. Defensive design should include token budgets, tool-call quotas, timeouts, circuit breakers, queue isolation, and billing anomaly detection. Monitoring should not only log prompts and completions; it should log provenance, retrieved document IDs, tool arguments, policy decisions, user approvals, model version, and safety-detector outputs. Incident response for LLM systems differs from conventional software response. A fix may involve changing prompts, safety policies, retrieval filters, memory entries, model versions, adapters, tool permissions, or data sources. Some incidents require forgetting or quarantining memories; others require retraining or revoking model artifacts. Organizations therefore need LLM-specific runbooks for prompt-injection incidents, data leakage, unsafe tool use, poisoning discovery, and model rollback.

6

6.2

Model-level defenses include adversarial training, refusal calibration, safety fine-tuning, preference optimization, constitutional rules, and robustness evaluation (Bai et al., 2022; Ganguli et al., 2022; Ouyang et al., 2022; Rafailov et al., 2023). Prompt-level defenses include system prompts, instruction hierarchy, delimiters, reminders about untrusted content, self-check prompts, and chain-of-verification. These controls improve behavior but remain statistical. They should be evaluated continuously against adaptive attacks and regressions. Prompt hardening is most useful when paired with architectural separation. A prompt may tell the model that retrieved content is untrusted; a system design should also ensure that retrieved content cannot directly set tool permissions, change policy, or erase logs. Similarly, a model may be trained to refuse prompt-leakage requests; the application should still avoid placing unnecessary secrets in prompts.

6.3

Retrieval and memory controls

RAG defenses should operate before, during, and after retrieval. Before retrieval, systems should curate sources, scan documents, attach provenance, and limit ingestion rights. During retrieval, systems should filter malicious-looking content, diversify sources, and preserve metadata. After retrieval, systems should label context, quote evidence separately from instructions, and verify high-impact claims. Persistent memory should be scoped by user, organization, task, and time; sensitive memory should expire or require explicit confirmation. Embedding systems need additional controls. Embeddings can leak information, and nearest-neighbor search can be manipulated by adversarial documents. Access control should be enforced before retrieval rather than after the vector search whenever possible. If a user lacks access to a document, its embedding should not influence retrieval results for that user. Provenance should not be discarded when text is chunked and embedded.

Defense-in-Depth Architecture

No single defense is sufficient. Model alignment reduces risk but cannot guarantee that the model will correctly separate data from instructions in all contexts. Prompt filtering catches some attacks but can be bypassed or over-block benign use. RAG citations improve transparency but do not prevent malicious retrieved content from influencing behavior. Tool schemas reduce injection into APIs but not necessarily goal hijacking. Therefore, the core design principle is defense in depth: combine deterministic controls that bound actions with probabilistic controls that detect and reduce residual risk. Figure 2 summarizes a general defense-in-depth architecture for LLM applications.

6.1

Model-level and prompt-level controls

Deterministic controls

Deterministic controls should be preferred wherever a security property can be expressed outside the model. Examples include least-privilege tool permissions, allowlisted network destinations, typed API schemas, JSON schema validation, sandboxed code execution, read-only defaults, transaction limits, human approval for irreversible operations, retrieval-source allowlists, context-size budgets, and cryptographic signing of model artifacts. These controls are auditable and do not require the model to reason perfectly under adversarial input.

6.4

Tool and agent controls

Tools should be designed as capability objects rather than broad ambient authority. Each tool should declare its scope, inputs, outputs, side effects, and risk level. Agents should receive only the tools needed for the current task. Sensitive tools should support dry-run, preview, confirmation, and rollback. Tool outputs should be untrusted observations, not instructions. 9

tool observations are untrusted inputs

User and external content

Input hygiene and provenance

Context isolation and policy routing

LLM / planner

Output schema and action verifier

Tools, APIs, and data stores

Human approval for irreversible or high-impact actions

Telemetry, anomaly detection, red-team regression tests

Figure 2: Defense-in-depth architecture for LLM applications. Reliable mitigation cannot rely only on model behavior; deterministic controls should constrain information flow and delegated authority before and after the model call. Cross-tool data flow should be mediated by policy: a webreading tool should not be able to instruct an email-sending tool to exfiltrate private data. Agent evaluation should include long-horizon attacks, persistent memory attacks, malicious tool outputs, poisoned webpages, compromised plug-ins, and multi-agent propagation. Short single-turn jailbreak benchmarks underestimate risk because many agent failures emerge only after multiple tool calls and state updates.

6.5

Monitoring, red teaming, and governance

Monitoring closes the loop. Useful telemetry includes prompt hashes, retrieved source IDs, model version, tool-call arguments, policy decisions, refusal reasons, detector scores, human approvals, and anomalies in token usage or tool-call frequency. Privacy-sensitive logging should use minimization, access control, retention limits, and secure redaction. Red teaming should be continuous rather than one-time. Regression suites should include known jailbreaks, promptinjection patterns, RAG poisoning cases, privacy probes, tool misuse scenarios, and availability stress tests (Derczynski et al., 2024; Ganguli et al., 2022). Governance should define risk ownership, review processes for new tools, incident-response procedures, and user-facing disclosure. Security evaluations should be versioned because model and prompt updates can change behavior even when application code remains fixed.

advantage, backdoor trigger success, clean-task accuracy, and transferability across models. System-level metrics include unsafe action rate, unauthorized data access, privilege-escalation distance, time to detection, rollback success, false positive rate of filters, user-task utility, cost amplification, and incident severity. Attack success rate should be interpreted carefully. A jailbreak that produces unsafe text is not equivalent to an agent that sends an unauthorized email. Conversely, a low text-level harmfulness score may hide integrity failure if the model silently changes a database query. Security evaluations should therefore specify the protected asset, attacker goal, allowed actions, and success criteria.

7.2

Benchmarks and test design

Benchmarks should cover direct prompts, indirect content, RAG corpora, tool outputs, code repositories, multimodal inputs, persistent memory, and multi-turn sessions. They should include benign hard cases to measure over-refusal and utility loss. For RAG and agents, benchmark tasks should include realistic documents, permissions, and tool side effects. AgentDojolike environments illustrate the importance of evaluating both task success and attack resistance (Zhang et al., 2024; Zhao et al., 2026). Reproducibility is difficult because many leading models are closed, versioned silently, and protected by changing safety layers. Papers should report model name, date, version, decoding parameters, system prompts when publishable, prompt 7 Evaluation Methodology templates, tool schemas, detector thresholds, and source code. Evaluation is a central weakness of current LLM-security re- For closed models, repeated evaluation over time is necessary search. Many papers report attack success rate on a small set because behavior can drift. of prompts, but deployment risk depends on attacker capability, tool privileges, task distribution, monitoring, and user behavior. 7.3 Threat-model reporting A systematization should separate model-level metrics from Every evaluation should state attacker knowledge and access. system-level metrics. A black-box user who can only submit prompts is different from an attacker who can upload documents to a RAG corpus, 7.1 Metrics control a webpage, edit a tool manifest, fine-tune an adapter, Common model-level metrics include attack success rate, re- or tamper with model artifacts. The paper should also report fusal rate, harmful completion rate, toxicity score, privacy ex- whether the attack requires many queries, hidden text, model posure, exact training-string recovery, membership-inference gradients, training access, or social engineering. Without this 10

reporting, comparisons across defenses are misleading. Table 5 External authority. The most important question is what lists our recommended reporting items for LLM vulnerability the system can do outside the chat window. A system that only studies. generates text has limited direct side effects. A system that can send email, modify code, query databases, invoke cloud APIs, make purchases, operate robots, or run terminal commands has 8 Risk Prioritization for Deployment substantially higher agency risk. For this reason, tool access and delegated authority are often stronger predictors of risk A key lesson from the lifecycle view is that vulnerabilities than model size. should not be prioritized only by their standalone attack success Untrusted context. Indirect prompt injection becomes posrate. A jailbreak that succeeds against an isolated chatbot sible when untrusted content is placed into the model context. may be less operationally dangerous than a lower-success-rate This includes retrieved web pages, PDFs, emails, code comindirect prompt-injection attack against an enterprise agent with ments, repository files, calendar descriptions, database records, access to private documents, email, calendars, code repositories, browser content, tool outputs, and vector-database chunks (Greor production systems. Deployment risk is therefore a function shake et al., 2023; Liu et al., 2023a; Xiong et al., 2024; Zhao of the model, the application stack, the available tools, the data et al., 2024). A deployment that mixes untrusted content with context, and the authority delegated to the system. This section instructions and private data should be treated as high risk provides a practical prioritization framework for translating the unless it uses strong provenance, isolation, and authorization vulnerability taxonomy into deployment decisions. controls. Private and regulated data. Enterprise assistants, healthcare systems, financial systems, educational platforms, and 8.1 A Deployment-Oriented Risk Model legal tools often expose models to sensitive data. Risks inFor deployed LLM systems, risk should be evaluated at the clude training-data memorization, retrieval leakage, embedding level of a model–application–authority tuple, rather than at the leakage, membership inference, prompt logging, cross-tenant level of the base model alone. We define a deployment scenario data exposure, and unauthorized summarization (Carlini et al., by 2021; Shokri et al., 2017). The priority of privacy controls S = (M, C, D, T , A, H), increases when data is personally identifiable, legally protected, where M is the model or model ensemble, C is the prompt proprietary, or security-sensitive. Irreversible or high-impact actions. Some actions are diffiand context-construction pipeline, D is the set of accessible cult to undo, such as sending confidential information, deleting data sources, T is the set of tools and external interfaces, A is files, merging code, changing access policies, executing finanthe authority delegated to the system, and H represents human cial transactions, publishing content, or triggering physical oversight and incident-response mechanisms. This formulation movement. These actions require stronger approval, simulation, makes explicit why two deployments using the same model can transaction preview, logging, and rollback mechanisms. have very different security profiles. Evidence after an incident. A system that cannot explain A practical prioritization score can be described as which model version, prompt, retrieved document, tool call, policy decision, or human approval produced an action is difR = f (E, F, I, D, O), ficult to audit. Observability is therefore a security control, where E denotes exposure, F attacker feasibility, I impact, not only an engineering convenience. Logs should preserve D defense maturity, and O observability. Exposure measures security-relevant events while minimizing unnecessary retenhow many users, documents, tools, and external channels can tion of sensitive content. reach the system. Feasibility captures the attacker’s required knowledge, access, cost, and repeatability. Impact measures 8.3 Scenario-Oriented Risk Prioritization confidentiality, integrity, availability, safety, privacy, legal, and reputational consequences. Defense maturity captures whether Table 6 summarizes common deployment scenarios. The pricontrols are deterministic, tested, auditable, and resilient to ority level is not absolute: it should be adjusted according to adaptive attackers. Observability measures whether the orga- the sensitivity of the domain, the maturity of controls, and the nization can reconstruct what happened after a security event. degree of external authority. However, the table illustrates a reThis model is consistent with the broader adversarial-machine- curring pattern: LLM risks become most urgent when untrusted learning view that risk depends not only on attack method, but content, private data, and external actions are combined. also on attacker capability, system objective, and operational context (MITRE, 2024; OWASP GenAI Security Project, 2025; 8.4 Control Selection by Risk Driver Vassilev et al., 2025). Risk prioritization should lead to concrete control selection. The most effective controls are often not better prompts, but 8.2 Risk Drivers Across the LLM Application stronger system boundaries. Prompt-level instructions are useStack ful but should not be treated as the primary security mechanism In operational settings, five risk drivers are especially important. when the system has access to private data or external tools. 11

Table 5: Recommended reporting checklist for LLM vulnerability studies. Item

What to report

Why it matters

System boundary

base model, prompt layer, RAG, tools, memory, deployment stack prompt access, corpus write access, tool-output control, fine-tuning access, artifact control private data, model weights, safety policy, tool action, availability, user trust harmful text, data leakage, unauthorized action, cost increase, policy violation task accuracy, helpfulness, latency, cost, over-refusal, user satisfaction model family, date, parameters when known, decoding, safety layer system prompt, retrieved documents, memory policy, delimiters, tool schemas pre-input, model, retrieval, output, tool boundary, monitoring, human approval code, prompts, datasets, random seeds, refusal classification rules safe examples, disclosure process, harm minimization

identifies what is actually being evaluated

Attacker capability Protected asset Success criterion Utility metric Model/version details Context construction Defense placement Reproducibility Ethical controls

8.5

Domain-Specific Considerations

determines feasibility and relevance avoids vague claims of “security” makes attack success comparable detects defenses that simply break the system supports reproduction and drift analysis critical for prompt-injection and RAG studies distinguishes probabilistic from deterministic controls enables independent verification reduces dual-use risk

credentials, and exploit-generation workflows it can access. Strong role-based access control, policy gating, controlled tool Healthcare. Healthcare deployments combine high privacy environments, and complete audit trails are essential. requirements with high consequences for hallucinated or incomplete advice. The most important risks are leakage of Software engineering. Coding agents face a distinctive protected health information, unsupported medical recommendations, biased triage, unsafe summarization of clinical records, form of indirect prompt injection because repository content and poor auditability. RAG sources should be curated and ver- is both task data and a potential instruction carrier. Malicious sioned; outputs should include provenance; and high-impact instructions can appear in comments, documentation, tests, isrecommendations should remain under professional oversight. sue text, dependency metadata, build logs, or generated tool The model should assist documentation, retrieval, and patient output. Secure design should treat repository content as uncommunication rather than independently making diagnosis or trusted unless explicitly trusted by the developer. Risk controls include sandboxed execution, dependency review, restricted file treatment decisions. writes, approval before commits, and separation between code Finance and insurance. Financial systems face risks insuggestion and code execution. volving private records, regulatory compliance, fraud, market manipulation, and unauthorized transactions. LLMs used for customer support, claims processing, investment research, or Enterprise knowledge management. Internal assistants internal analytics should separate advice generation from transoften combine private documents, access control, RAG, and action authority. Controls should include source provenance, persistent memory. The main risk is that retrieval may bypass suitability checks, human approval for financial actions, datadocument permissions or mix information across users, teams, retention limits, and strong audit trails. or tenants. Access control should be enforced before retrieval, Education. Educational deployments involve student prinot only after generation. Retrieved chunks should carry provevacy, academic integrity, biased feedback, misinformation, and nance and sensitivity metadata through the full pipeline. inappropriate delegation of grading authority. Systems should disclose limitations, avoid storing unnecessary student data, distinguish tutoring from assessment, and preserve instructor Robotics and physical systems. The safety impact increases oversight. For grading or feedback systems, rubrics, prove- sharply when language plans control physical actions. A wrong nance, and appeal mechanisms are important accountability instruction that would be merely inconvenient in a chatbot can controls. become dangerous when translated into motion, navigation, maCybersecurity. LLMs can support defensive analysis, log nipulation, or equipment control. Systems require simulation, triage, malware explanation, vulnerability management, and constrained controllers, runtime monitors, emergency stops, incident response, but they can also lower the barrier to abuse and human approval for risky operations. Natural-language (Deng et al., 2023). The main deployment question is not plans should not directly map to actuator commands without whether the model can discuss security, but what tools, logs, safety validation. 12

Table 6: Deployment-oriented risk prioritization. Priority depends on the specific application, but this matrix summarizes common patterns. Scenario

Typical priority

Dominant objective

Why

First controls to implement

Public chatbot without tools

medium

safety, abuse

large exposure, but limited external side effects

Enterprise RAG assistant over private documents Coding assistant with repository and terminal access Email/calendar/browser agent

high

confidentiality, integrity

high

integrity, agency

high

privacy, agency

Fine-tuned domain model

medium–high

safety, privacy

private data and indirect prompt injection through documents prompt injection in files can lead to code changes or command execution untrusted messages or webpages can trigger actions under user authority private fine-tuning data and safety regression risk

Open model distribution

high

supply-chain integrity

many downstream users trust artifacts and loaders

Security-operation or penetration-testing assistant

high

safety, accountability

dual-use tasks and privileged context

abuse monitoring, refusal calibration, rate limits, safety filters access control before retrieval, provenance labels, context isolation, logging sandboxing, least privilege, command approval, repository trust policy tool scoping, human confirmation, untrusted-content handling, transaction logs data audit, privacy tests, safety regression, adapter/version control signed weights, safe serialization, reproducible release, dependency scanning role-based access, audit logs, policy gating, controlled tool environments

8.6

Practical Risk-Review Questions

These questions often reveal that the highest-value controls are architectural rather than linguistic. Narrower privileges, Before deployment, a risk review should answer the following clearer provenance, safer retrieval, deterministic tool boundquestions: aries, and better incident evidence usually reduce deployment 1. Authority: What external authority does the model or agent risk more reliably than adding additional natural-language instructions to the prompt. have, and which actions are irreversible or high impact? 2. Trust boundaries: What untrusted content can enter the prompt, retrieval context, memory, tool outputs, or system instructions?

9

Open Problems

3. Data exposure: What private, regulated, proprietary, or The preceding sections show that LLM vulnerabilities are not security-sensitive data can be retrieved, generated, stored, isolated defects of a model, but emergent properties of a lifecycle and application stack. A deployed system combines training or logged? data, alignment procedures, prompts, retrieval pipelines, mem4. Tool containment: Are tools allowlisted, scoped, sand- ory stores, tools, human users, organizational policies, and boxed, and mediated by structured schemas and policy software dependencies. This coupling creates several research checks? challenges that cannot be solved by improving model refusal behavior alone. We organize the open problems into three groups: 5. Human oversight: Which actions require human confirmafoundational security problems, which concern the boundary tion, and does the user see a faithful preview of the proposed between language and computation; system-design problems, action? which arise when LLMs are connected to data and tools; and 6. Evaluation: Has the system been tested against direct assurance and governance problems, which determine whether prompt injection, indirect prompt injection, jailbreaks, re- deployed systems can be evaluated, monitored, and repaired. trieval poisoning, privacy probes, and tool-misuse scenarios? 7. Observability: Can the organization reconstruct model ver- 9.1 Compositional Security Guarantees sion, prompt context, retrieved sources, tool calls, policy Current defenses rarely compose. A prompt-injection detector, decisions, and approvals after an incident? a RAG filter, a safety classifier, and a tool schema may each 8. Recovery: Is there a rollback plan for model updates, poi- perform well in isolation, yet fail when an attacker chains them soned documents, unsafe memories, leaked credentials, or across a long interaction. For example, a malicious document harmful tool actions? may first influence retrieval, then bias summarization, then 13

trigger a tool call, and finally hide evidence in a generated report. This type of cross-layer failure is difficult to capture with single-turn attack success rates. The central open problem is how to provide security guarantees for an application whose planner is nondeterministic, whose inputs include adversarial natural language, and whose intermediate states are partially hidden. Formal methods, information-flow control, capability systems, typed tool interfaces, and runtime monitors are promising directions, but they remain difficult to integrate with probabilistic language models. Future work should move from evaluating isolated defenses to evaluating defense compositions. Useful questions include: whether a system preserves confidentiality under arbitrary retrieved text, whether an agent can invoke only authorized tools, and whether safety policies remain invariant under summarization, translation, compression, or multi-step planning.

9.2

Instruction–Data Separation

Indirect prompt injection exploits the absence of a reliable boundary between instructions and data. In conventional secure systems, code and data are separated by design; in many LLM applications, this boundary is blurred because both user commands and untrusted documents are represented as natural language in the same context window. Delimiters, role labels, and natural-language warnings are useful engineering practices, but they do not create a hard security boundary. A major research direction is the design of stronger instruction–data separation mechanisms. Possible approaches include typed context regions, non-executable data blocks, provenance-aware attention mechanisms, structured intermediate representations, and external policy engines that decide which content may authorize an action. Another direction is to train or adapt models to respect context labels such as instruction, untrusted document, tool output, memory, and policy. The challenge is to make these labels semantically effective rather than merely decorative. A mature solution would allow an LLM to read untrusted content for information while preventing that content from changing goals, permissions, or tool policies.

9.3 Provenance-Aware RAG and Long-Term Memory

source conflict handling, freshness-aware generation, and verifiable deletion of memories. Memory systems require particular care because they can silently influence future sessions. Users and auditors should be able to inspect what is stored, why it was stored, when it expires, which future tasks may use it, and how it can be deleted. The long-term goal is not simply more accurate retrieval, but accountable retrieval with traceable authority.

9.4

Secure Agency and Tool Delegation

LLM agents require a theory of delegated authority. A user may ask an agent to plan a trip, summarize email, search a repository, or refactor code. The agent should infer helpful steps, but it should not invent new authority. This distinction is difficult because many useful tasks require planning, tool use, and interpretation of ambiguous intent. Secure agency requires dynamic scoping of permissions to the user’s current goal. Tool access should be narrow, temporary, auditable, and revocable. High-impact actions should require confirmation, and the confirmation should describe the actual external effect rather than merely restating the model’s plan. For example, ‘send this email to these recipients with this attachment” is a stronger confirmation than ‘continue?” Evaluation should measure not only whether agents complete tasks, but also whether they avoid unsafe actions under malicious observations, poisoned tool outputs, or conflicting instructions. Future benchmarks should include realistic tool permissions, partial failures, deceptive documents, and multi-step attacks.

9.5

Privacy-Preserving Adaptation

Enterprises increasingly adapt LLMs using private documents, tickets, code, emails, logs, and user interactions. This creates privacy risks across both parametric and non-parametric components of the system. Fine-tuning may memorize sensitive examples; embeddings may leak semantic information; retrieval may cross tenant boundaries; logs may store private prompts; and model outputs may reveal information about training or retrieved data. Differential privacy, federated learning, secure enclaves, synthetic data, redaction, and retrieval isolation provide partial solutions (Abadi et al., 2016; Dwork et al., 2006; McMahan et al., 2017). However, the open challenge is to preserve utility while providing measurable guarantees for the whole application stack. A privacy guarantee for model parameters alone is insufficient if the retrieval store, prompt logs, cache, or memory layer remains exposed. Future work should develop end-toend privacy accounting for LLM applications, including model adaptation, embedding generation, retrieval, logging, and human feedback collection.

RAG and memory systems introduce persistent, externalized context. They improve factual grounding and personalization, but they also create new attack surfaces: poisoned documents, stale knowledge, cross-user leakage, embedding inversion, unauthorized retrieval, and memory manipulation. Current vector databases often optimize semantic similarity while carrying limited trust metadata. As a result, retrieved content may be relevant but unsafe, outdated, unauthorized, or adversarial. 9.6 Robustness Under Distribution Shift and Future RAG systems should preserve provenance across inSystem Evolution gestion, chunking, embedding, indexing, retrieval, reranking, summarization, and answer generation. Open problems include LLM applications evolve continuously. Models are updated, embedding-level access control, poisoning-resistant retrieval, prompts are revised, retrieval corpora change, tools are added, 14

user behavior shifts, and attackers adapt. A defense that works 9.9 Human Factors, Usability, and Overat deployment time may degrade after a model update or after Reliance new documents enter the retrieval index. This creates a form of LLM security is also a human factors problem. Users may security distribution shift. over-trust fluent outputs, ignore warnings, approve unsafe tool Open problems include safety regression testing, updatecalls, paste sensitive information, or misunderstand the sysaware red teaming, continuous evaluation, and automatic detem’s authority. Conversely, overly strict defenses may create tection of behavior drift. Model updates should be treated as alert fatigue, reduce usefulness, or encourage users to bypass security-relevant events, especially when the application has safeguards. access to tools or private data. Similarly, retrieval-index upOpen problems include designing effective confirmations, dates should be tested for poisoning and permission errors. A calibrated uncertainty displays, user-facing provenance, and mature deployment pipeline should evaluate not only model warnings that are specific enough to change behavior. A useful quality, but also whether existing safety, privacy, and tool-use confirmation should reveal the concrete consequence of an acguarantees still hold after each change. tion, the data sources used, and the uncertainty or risk involved. More broadly, LLM systems should be designed so that users can understand what the system knows, what it can do, what it is not allowed to do, and when human expertise is required. 9.7 Evaluation Realism and Reproducibility Many current benchmarks underrepresent adaptive, long9.10 Incident Response, Auditing, and Goverhorizon, stateful attacks. Real attackers can observe failures, nance revise prompts, poison future retrieval, exploit tool outputs, wait for maintenance windows, and chain small weaknesses LLM incidents may involve prompts, retrieved documents, across components. A single-turn jailbreak benchmark is there- memories, model versions, tool calls, credentials, users, or fore insufficient for evaluating deployed LLM systems. third-party services. Organizations need runbooks for identifyFuture benchmarks should include multi-session attacks, re- ing affected sessions, revoking leaked credentials, quarantining alistic permissions, benign hard cases, cost and latency con- poisoned documents, rolling back model versions, deleting unstraints, multi-agent communication, retrieval poisoning, tool safe memories, notifying users, and updating evaluation suites. misuse, and adaptive attackers who know the defense. Eval- Unlike traditional software incidents, the root cause may be uation should report not only attack success rate, but also a combination of natural-language input, model behavior, refalse-positive rate, utility loss, cost, latency, user friction, and trieval state, and tool execution. failure recoverability. Reproducibility is another open chalGovernance should specify who approves new tools, who lenge because many studies depend on closed models, changing owns safety regressions, who reviews model updates, how safety policies, and non-deterministic decoding. Shared bench- prompts and policies are versioned, and how incidents are mark harnesses, versioned prompts, fixed evaluation protocols, documented. Auditing mechanisms should preserve enough evand transparent reporting checklists are needed for cumulative idence to reconstruct security-relevant events without retaining progress. unnecessary sensitive content. Future work should develop standard incident taxonomies, evidence schemas, and post-incident evaluation methods for LLM applications.

9.8

Multimodal and Cross-Channel Attacks

As LLM systems become multimodal, the attack surface expands beyond text. Instructions can be embedded in images, screenshots, audio, video frames, documents, tables, code, diagrams, or user-interface elements. A model may extract text from an image, summarize a PDF, interpret a webpage, or act on a screenshot. Each of these channels can carry untrusted instructions. The open problem is to define security policies that survive cross-modal translation. A malicious instruction hidden in an image should not gain authority merely because it was converted into text by an OCR module or vision-language model. Similarly, a table cell, figure caption, metadata field, or audio transcript should carry provenance and trust labels after conversion. Future research should study multimodal prompt injection, cross-modal provenance, safe document parsing, and security-aware multimodal representation learning.

10

Discussion

LLM vulnerabilities are often described as mysterious properties of neural networks, but many practical failures arise from familiar systems problems: weak input validation, confuseddeputy behavior, excessive privilege, insecure supply chains, insufficient logging, and missing incident response. What is new is the interface: natural language can simultaneously express user intent, untrusted evidence, executable instructions, and social manipulation. This makes LLM systems especially vulnerable when developers rely on model judgment to enforce boundaries that should be enforced by architecture. A lifecycle and application-stack view also helps avoid two common extremes. The first extreme is model fatalism: assuming LLMs are inherently unsecurable because prompt injection cannot be perfectly solved at the model level. The second is model optimism: assuming that better alignment or a stronger

15

system prompt will solve application security. The realistic Constitutional AI: Harmlessness from AI feedback. arXiv position is layered. Models should become more robust, but preprint arXiv:2212.08073, 2022. applications must still constrain authority, preserve provenance, E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. verify outputs, and monitor behavior. On the dangers of stochastic parrots: Can language models The most important design shift is to treat LLMs as combe too big? In Proceedings of the ACM Conference on ponents inside security-critical systems, not as the security Fairness, Accountability, and Transparency, pages 610–623, boundary themselves. The model can summarize, reason, draft, 2021. classify, and propose. It should not be the only component deciding whether untrusted content can override policy, whether R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, a tool may execute, whether private data may leave a boundary, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunor whether a memory should persist. skill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, 11 Comparison with Existing Surveys T. P.B.Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, and Frameworks et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages Table 7 clarifies novelty. We do not claim that each vulnerability 1877–1901, 2020. is new. Instead, our contribution is a systems organization that maps attacks and defenses to the lifecycle and application stack. N. Carlini, C. Liu, Ú. Erlingsson, J. Kos, and D. Song. The secret sharer: Evaluating and testing unintended memorization This framing is useful for practitioners because mitigation rein neural networks. In USENIX Security Symposium, pages sponsibility often belongs to different teams: data engineers, 267–284, 2019. model trainers, safety teams, MLOps engineers, application developers, security teams, compliance officers, and human N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbertoperators. Voss, K. Lee, A. Roberts, T. Brown, D. Song, Ú. Erlingsson, A. Oprea, and C. Raffel. Extracting training data from large language models. In USENIX Security Symposium, pages 12 Conclusion 2633–2650, 2021. This survey reframed LLM vulnerabilities through a lifecycle and application-stack taxonomy. We organized attacks across data collection, pretraining, post-training alignment, packaging and supply chain, retrieval and memory, prompting and inference, tool/agent execution, and deployment/maintenance. This view shows that many risks are not isolated model failures but cross-boundary failures involving data provenance, instruction hierarchy, delegated authority, persistent state, and operational controls. We synthesized defenses into a defense-in-depth architecture that combines deterministic controls, model-level robustness, retrieval and memory governance, tool containment, monitoring, red teaming, privacy engineering, and incident response. The central lesson is that trustworthy LLM systems require both safer models and safer systems. Future work should move from isolated jailbreak demonstrations toward compositional security, provenance-preserving RAG, secure agency, realistic stateful benchmarks, and operationally grounded governance.

X. Chen, C. Liu, B. Li, K. Lu, and D. Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017. K. Chu. A systematic survey of security threats and defenses in LLM-based AI agents: A layered attack surface framework. arXiv preprint arXiv:2604.23338, 2026. A. Dehghantanha et al. SoK: The attack surface of agentic AI – tools, and autonomy. arXiv preprint arXiv:2603.22928, 2026. G. Deng, Y. Liu, V. Mayoral-Vilches, P. Liu, Y. Li, Y. Xu, T. Zhang, Y. Liu, M. Pinzger, and S. Rass. PentestGPT: An LLM-empowered automatic penetration testing tool. arXiv preprint arXiv:2308.06782, 2023. L. Derczynski, E. Galinkin, J. Martin, S. Majumdar, N. Inie, et al. garak: A framework for security probing large language models. arXiv preprint arXiv:2406.11036, 2024. E. Derner and K. Batistic. A security risk taxonomy for large language models. arXiv preprint arXiv:2309.06899, 2024.

References M. Abadi, A. Chu, I. Goodfellow, H. B. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In ACM SIGSAC Conference on Computer and Communications Security, pages 308–318, 2016. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171– 4186, 2019. C. Dwork, F. McSherry, K. Nissim, and A. Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography Conference, pages 265–284, 2006.

16

Defenses

Evaluation partial

broad attack catalogues and privacy/security synthesis

✓

✓

emphasize model capabilities, alignment, and evaluation

partial

✓

✓

focus on agentic workflows and protocols

✓

✓

✓

partial

✓

partial

✓

partial

✓

✓

✓

✓

✓

Privacy

✓

Agents

partial

RAG

LLM security/privacy surveys (Xu et al., 2025; Yao et al., 2024) LLM trustworthiness/alignment surveys (Liu et al., 2023b; Naveed et al., 2023) Agent-security surveys (Chu, 2026; Ferrag et al., 2025; Ling et al., 2026) OWASP LLM Top 10 (OWASP GenAI Security Project, 2025) NIST AML taxonomy (Vassilev et al., 2025) This survey

Lifecycle

Work / framework

Supply chain

Table 7: Positioning of this survey relative to representative surveys and practitioner frameworks. Checkmarks indicate primary coverage; partial coverage indicates that the topic is discussed but not used as an organizing axis.

partial

partial

partial

✓

partial

partial

partial

partial

✓

✓

✓

partial

partial

✓

✓

✓

partial

✓

✓

Main distinction

practitioner-oriented risk taxonomy

general adversarial ML terminology and lifecycle stages unifies lifecycle stage, application stack, security objective, attacker capability, and defense layer

M. A. Ferrag, N. Tihanyi, D. Hamouda, L. Maglaras, A. Lakas, V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, and M. Debbah. From prompt injections to protocol exD. Chen, and W.-t. Yih. Dense passage retrieval for openploits: Threats in LLM-powered AI agents workflows. arXiv domain question answering. In Conference on Empirical preprint arXiv:2506.23260, 2025. Methods in Natural Language Processing, pages 6769–6781, 2020. M. Fredrikson, S. Jha, and T. Ristenpart. Model inversion attacks that exploit confidence information and basic coun- K. Kurita, P. Michel, and G. Neubig. Weight poisoning attacks on pre-trained models. In Annual Meeting of the Association termeasures. In ACM SIGSAC Conference on Computer and for Computational Linguistics, pages 2793–2806, 2020. Communications Security, pages 1322–1333, 2015. D. Ganguli, L. Lovitt, J. Kernion, A. Askell, Y. Bai, S. Ka- P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, davath, B. Mann, E. Perez, N. Schiefer, K. Ndousse, et al. S. Riedel, and D. Kiela. Retrieval-augmented generation Red teaming language models to reduce harms: Methods, for knowledge-intensive NLP tasks. Advances in Neural scaling behaviors, and lessons learned. arXiv preprint Information Processing Systems, 33:9459–9474, 2020. arXiv:2209.07858, 2022. Y. Ling, S. Yu, Z. Chen, and C. Fang. Toward secure LLM Gemini Team. Gemini: A family of highly capable multimodal agents: Threat surfaces, attacks, defenses, and evaluation. models. arXiv preprint arXiv:2312.11805, 2023. arXiv preprint arXiv:2606.10749, 2026. K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and Y. Liu, G. Deng, Z. Xu, Y. Li, Y. Zheng, Y. Zhang, L. Zhao, M. Fritz. Not what you’ve signed up for: Compromising T. Zhang, K. Wang, and Y. Liu. Prompt injection atreal-world LLM-integrated applications with indirect prompt tack against LLM-integrated applications. arXiv preprint injection. Proceedings of the ACM Workshop on Artificial arXiv:2306.05499, 2023a. Intelligence and Security, 2023. Y. Liu, Y. Yao, J. Ton, X. Zhang, R. Cheng, Y. Klochkov, T. Gu, B. Dolan-Gavitt, and S. Garg. BadNets: Identifying vulM. F. Taufiq, and H. Li. Trustworthy LLMs: A survey and nerabilities in the machine learning model supply chain. In guideline for evaluating large language models’ alignment. Machine Learning and Computer Security Workshop, 2017. arXiv preprint arXiv:2308.05374, 2023b. D. Kang, X. Li, I. Stoica, C. Guestrin, M. Zaharia, and B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. T. Hashimoto. Exploiting programmatic behavior of LLMs: y Arcas. Communication-efficient learning of deep networks Dual-use through standard security attacks. arXiv preprint from decentralized data. International Conference on ArtifiarXiv:2302.05733, 2023. cial Intelligence and Statistics, pages 1273–1282, 2017. 17

MITRE. MITRE ATLAS: Adversarial threat landscape for F. Tramèr, F. Zhang, A. Juels, M. K. Reiter, and T. Ristenpart. artificial-intelligence systems, 2024. URL https:// Stealing machine learning models via prediction apis. In atlas.mitre.org/. Accessed 2026-06-29. USENIX Security Symposium, pages 601–618, 2016. M. Nasr, R. Shokri, and A. Houmansadr. Comprehensive pri- A. Vassilev, A. Oprea, A. Fordyce, and H. Anderson. Adversarial machine learning: A taxonomy and terminology of vacy analysis of deep learning: Passive and active white-box attacks and mitigations. NIST Trustworthy and Responsiinference attacks against centralized and federated learning. ble AI Report NIST AI 100-2e2025, National Institute of IEEE Symposium on Security and Privacy, 2019. Standards and Technology, 2025. URL https://csrc. H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usnist.gov/pubs/ai/100/2/e2025/final. man, N. Akhtar, N. Barnes, and A. Mian. A comprehensive overview of large language models. arXiv preprint A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all arXiv:2307.06435, 2023. you need. In Advances in Neural Information Processing Systems, 2017. OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. E. Wallace, S. Feng, N. Kandpal, M. Gardner, and S. Singh. Universal adversarial triggers for attacking and analyzing L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, NLP. In Conference on Empirical Methods in Natural LanP. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. guage Processing, pages 2153–2162, 2019. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing A. Wan, E. Wallace, S. Shen, and D. Klein. Poisoning language Systems, 2022. models during instruction tuning. International Conference on Machine Learning, 2023. OWASP GenAI Security Project. OWASP top 10 for large language model applications 2025, 2025. URL https:// A. Wei, N. Haghtalab, and J. Steinhardt. Jailbroken: How does genai.owasp.org/llm-top-10/. Accessed 2026LLM safety training fail? arXiv preprint arXiv:2307.02483, 06-29. 2023. F. Perez and I. Ribeiro. Ignore previous prompt: At- L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. tack techniques for language models. arXiv preprint Huang, M. Cheng, A. Glaese, B. Balle, A. Kasirzadeh, et al. arXiv:2211.09527, 2022. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359, 2021. X. Qi, Y. Zeng, T. Xie, P.-Y. Chen, R. Jia, P. Mittal, and P. Henderson. Fine-tuning aligned language models compromises K. Xiong, X. Liu, P. Zhang, et al. Towards understanding safety, even when users do not intend to! arXiv preprint the security risks of retrieval-augmented generation. arXiv arXiv:2310.03693, 2023. preprint arXiv:2404.13093, 2024. Y. Qin, S. Liang, Y. Ye, K. Zhu, L. Yan, Y. Lu, Y. Lin, X. Cong, W. Xu et al. A survey of attacks on large language models. arXiv preprint arXiv:2505.12567, 2025. X. Tang, B. Qian, et al. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. arXiv preprint S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, arXiv:2307.16789, 2023. and Y. Cao. ReAct: Synergizing reasoning and acting in language models. International Conference on Learning R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. ErRepresentations, 2023. mon, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint Y. Yao, J. Duan, K. Xu, Y. Cai, Z. Sun, and Y. Zhang. A survey arXiv:2305.18290, 2023. on large language model security and privacy: The good, the bad, and the ugly. arXiv preprint arXiv:2312.02003, 2024. T. Schick, J. Dwivedi-Yu, R. Dessı̀, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: R. Zhang et al. AgentDojo: A dynamic environment to evalLanguage models can teach themselves to use tools. Aduate attacks and defenses for LLM agents. arXiv preprint vances in Neural Information Processing Systems, 2023. arXiv:2406.13352, 2024. R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership W. Zhao, Z. Li, P. Zhang, and J. Sun. ClawGuard: A runtime inference attacks against machine learning models. In IEEE security framework for tool-augmented LLM agents against Symposium on Security and Privacy, pages 3–18, 2017. indirect prompt injection. arXiv preprint arXiv:2604.11790, 2026. N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. Christiano. Learning to Y. Zhao, C. Wu, B. Li, et al. Poisoning retrieval corsummarize with human feedback. Advances in Neural Inforpora for retrieval-augmented generation. arXiv preprint mation Processing Systems, 33:3008–3021, 2020. arXiv:2402.07867, 2024. 18

D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei, P. Christiano, and G. Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043, 2023.

19