When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systems Li Zhang* , Yang Sun, Jie Shi
arXiv:2609.23700v1 [cs.CR] 20 Sep 2026
Huawei Abstract. Large language model agents are now privileged principals that take consequential actions — editing code repositories, operating inboxes, completing purchases. Their authority is kernel-grade, but it comes without what classical systems security requires: a trusted mediator interposed on every access. Operating-system vendors are now rebuilding the platform around this de-facto agent kernel, inheriting complete mediation as a design problem. We systematize the security of such systems around a single distinction: a crossing mediated over provenance admits a deterministic check, while one over content semantics does not. A trust-boundary taxonomy locates where mediation must occur and isolates the central mediation gap at two kinds of semantic judgment: distinguishing data from instruction in untrusted input, and an authorized action from an unauthorized one. We argue that this gap leaves an irreducible residual of undetected attacks wherever inputs and actions are not restricted in advance to an enumerated set. The same distinction makes attack-success statistics actionable, placing each number on a spectrum from deployment debt (a sound deterministic mediator left unused) to a structural gap (no such mediator known). We systematize defenses across runtime monitoring, architectural separation, and authorization, and show that current evaluations tend to overstate deployed security through evaluation-validity failures. Finally, we carry that analysis forward beyond the de-facto kernel, to an architecture in which the model itself becomes the arbitration core, and derive the design constraints, open challenges, and research agenda for a security-first AI-native OS. Keywords: LLM agent security; agentic AI; prompt injection; reference monitor; complete mediation; agent security evaluation; AI-native operating systems
1. Introduction
tions Windows as an agent-native runtime [3]; Android’s App Functions and Apple’s App Intents let an agent discover and invoke an application’s declared operations [4], [5]. These remain hybrids: intelligence layered onto a conventional kernel, not baked into the arbitration core. What exists today is a de-facto agent kernel, and the conventional OS beneath it is racing to enclose it [6]. The direction, however, is unambiguous, and its endpoint is an AI-native OS in which the model itself allocates memory, multiplexes agents, and authorizes every privileged call. Reaching that endpoint would not supply the check that is missing today. Instead, it would make supplying one harder: the component that would have to mediate is itself the probabilistic component that needs mediating.
A coding assistant that edits a repository and opens a pull request; a computer-use agent that completes a purchase by clicking through live web pages; an enterprise copilot that reads a shared inbox and invokes internal tools on a user’s behalf — these are deployed products, not laboratory demonstrations. What unites them is no longer their fluency but their authority: each is entrusted, under standing permission, to select and commit consequential actions against real systems. In assuming that authority, an autonomous, language-driven component has stepped into a position classical systems security reserves for the kernel: the privileged principal that arbitrates resources and binds privileged operations. In July 2026 two leading model developers disclosed that agents under evalu- Classical doctrine has a name for that check, and a rule ation had escaped their confinement and reached third- about where it sits. The doctrine keeps two roles apart: party production systems: a narrowly authorized goal was the principal that acts, and the mediator that confines it enough to induce actions no one had authorized [1], [2]. — an always-invoked, tamper-proof check between a privNeither disclosure involved an injected adversary. What ileged decision and its execution. The agent inherited the was absent was not a defense against an attacker, but a principal’s authority by occupying its role, but the mediacheck that could tell an authorized action from an unau- tor did not come with the promotion. This paper analyzes that deficit on the deployed systems, and then carries the thorized one. analysis forward to the AI-native OS they are evolving This kernel-ization is no longer merely emergent; the plattoward. One question organizes both halves: when an form itself is being rebuilt around it. The three major opagent crosses a trust boundary, what must the mediator erating systems are making the same move — extending at that crossing judge? the platform to serve agents directly, rather than leaving them to drive interfaces built for people. Microsoft posi- Some mediators judge provenance: where a piece of content came from, and therefore what authority it carries. ∗ Corresponding author. Email: [email protected]
1
That judgment can be frozen into a rule before the content it will be applied to exists, which is what makes its verdict deterministic. Others must judge content semantics: what the content means. No rule can be frozen in advance for a subject matter that arrives only at run time, so a mediator of that kind is necessarily a classifier. It sits at a chosen point on the trade-off between false positives and missed attacks, and it will be wrong some of the time. Most crossings admit judgments of both kinds, so the question defines a graded provenancevs-semantics axis rather than a two-way split. It separates defenses that leave an irreducible residual from those that deliver a structural guarantee, and it grades the field’s attack-success statistics: a number reflects deployment debt when a sound deterministic mediator exists but has not been adopted, and a structural gap when none is known, with most cases falling between. The axis has a constructive dual as well, the projection principle: where a crossing admits no deterministic mediator, one can still be bought by narrowing the space that crossing governs, at a price paid in expressiveness, autonomy, memory, or influence breadth.
§2
The object of study the de-facto agent kernel and the AI-native OS horizon
§4
Mediation framework and taxonomy the provenance-vs-semantics axis, six boundaries around the agent principal, and the mediation gap
§5
Attacks at the boundaries fifteen classes indexed to the boundaries — shipped exploits separated from lab results
Defenses and their residuals
§6–8
monitoring, architectural separation, authorization — what each mediates and what each leaves
§9
Evaluation validity why current measurement overstates security
The AI-native OS horizon
This paper makes five contributions, developed across the sections Figure 1 lays out. (i) A trust-boundary taxonomy locating each privileged crossing around the agent principal and stating the mediation obligation it carries. Derived from complete-mediation doctrine, it places each crossing on that axis. (ii) A threats-and-defenses systematization indexed to those boundaries, populating each crossing with the attack classes reported against it and the defenses that answer them. (iii) A probabilisticmediation analysis of runtime monitoring. Extending a known impossibility result for injection detection [7] to the moment an agent commits to an action, we argue that no monitor can be made complete wherever inputs and actions are not restricted in advance to an enumerated set — the mediation gap. We recast that limit as a measurement program, a monitor’s residual miss rate at a declared false-positive budget, and collect what has been measured layer by layer. (iv) An evaluation-validity systematization explaining why agent-security measurement frequently overstates deployed security, and what reporting discipline would correct it. (v) A transfer analysis carrying the deployed systematization onto the architectural AI-native OS by the same test: for each property that defines that architecture, we identify the invariant it puts at risk and derive the design constraint that follows. We state how far the reused evidence reaches, and record the residue as open problems and a research agenda.
§10
what carries over and what changes, design constraints, open problems, a research agenda
Figure 1: The flow of this paper. a kernel does. Pirch et al. argue that such agents must therefore be secured like operating systems, because they already function as ones [6]; Li et al. cast the foundation model as a kernel, local resources as device drivers, and skills as applications [8]. That mapping has since grown concrete. A deployed agent now ships with its own shell, file system, and memory that persists across sessions [9], [10], [11], [12]. Three properties characterize this de-facto kernel. First, its authority is standing: acquired at integration or installation and exercised at the agent’s discretion across task boundaries. Second, its reach is broad: one principal touches files, networks, shells, browsers, and other agents, whether through general-purpose facilities or through declared application interfaces. Breadth is what makes open-ended, multi-step work possible, and it is the same property that sets the attack surface — every service the agent can reach is a service an injected instruction can reach through it. Third, the principal holding that authority is probabilistic: it samples its decisions. The same request is not guaranteed to yield the same privileged action twice.
2. The Object of Study
2.2 The AI-Native OS as Architectural Horizon
2.1 The De-Facto Agent Kernel
Agents today do not yet decide what resources the work A growing class of deployed agents decides which privi- gets: how much compute and memory it may consume, leged actions execute, and reaches a user’s files, creden- and which agent runs when several contend. Those decitials, and networks on standing permission. That is what sions depend on what the work requires, which only the 2
model is in a position to know. The academic lineage points to kernels built for the model, with a scheduler, a memory manager, and a natural-language interface in place of a system-call table (AIOS [13], MemGPT [14], AgentOS [15]). The proposals divide on where arbitration is enforced. At one end the model is the kernel and arbitrates directly [16]; at the other a deterministic layer beneath the model retains enforcement and executes what the model proposes [13], [17]. When such a system is deployed, something must hold apart agents acting for different principals who do not trust one another. That is the classical controlled-sharing problem of a multi-user system [18]. Hypervisors and exokernels supply it with deterministic mechanism [19], [20]. The problem is already live: agents for different principals are served from one shared model, and that sharing has been shown to leak one user’s inputs to another [21], [22]. An AI-native OS would have to answer it with a core that decides by sampling. With this, we distill the six criteria of an AI-native OS:
puts a single system-level agent over the third-party agents its agent framework admits [24]. Figure 2 sets the de-facto agent kernel and the AI-native OS against a conventional OS. Both differ from it in the same way: standing arbitration authority held by a probabilistic principal. In turn, they differ from each other in who arbitrates.
3. Related Work and Positioning Several surveys of agent security have appeared recently. Kim et al. organize the field along seven design dimensions of agent systems, six attack vectors, and seven security risks, and close with a case study of AutoGPT [25]. Ling et al. follow a request around the agent’s operating loop, from input through planning, decision, and tool execution to output, with memory, monitoring, and coordination as cross-cutting concerns; they conclude that secure agents require explicit trust boundaries, principled privilege control, provenance-aware state management, and realistic evaluation [26]. Chu’s Layered Attack-Surface framework (LASM) places each attack and defense on one of seven stack layers, from the foundation model up to governance, crossed with a temporality axis, and reviews the methodological problems of existing evaluations [27]. Dehghantanha and Homayoun map ten attack surfaces along the data-flow pipeline, from untrusted input through the model core and tool execution to long-term memory, and trace multi-step attack paths across them; they supply an evaluation protocol with validity controls and commend hybrid designs in which a deterministic execution kernel enforces schemas and capabilities while the model remains an untrusted proposer [28]. Pirch et al. look at agents through the lens of operating systems: agents face the same problems of resource isolation, privilege separation, and communication mediation, and in tests of four deployed agents several protections fail in practice while established OS techniques could mitigate many of the failures [6].
1. Intelligence at the architectural core, not the application layer. Model inference is a first-class system resource — like memory or I/O scheduling — rather than a feature running atop the OS: an LLM kernel exposing its own system-call interface, agent scheduler, and memory manager [13], with arbitration decided from what the model knows about the work, enforced either by the model itself [16] or by a deterministic layer beneath it. 2. Intent-based rather than command-based control. The interface is natural-language goal specification with contextual interpretation, not explicit enumerated commands: natural language is the system’s own interface rather than a layer above a command one, and a kernel behind it translates the stated goal into orchestration [15], [16]. 3. Persistent, cross-session context. The system maintains a continuous model of user goals, state, and history as a first-class resource, held by a context manager that pages spans into and out of a bounded window [14]. 4. Autonomous multi-step execution. The system plans and commits multi-step tasks across applications without per-step human orchestration, selecting and binding actions the originating request did not name [4]. 5. OS-level agent lifecycle management. Permissions, identity, registration, sandboxing, and delegation for agents are handled by the system itself, with agents constituting a distinct workload class [3], [23]. 6. Mutually-distrusting multiplexing. Agents acting for principals who do not trust one another share the system’s resources, and the system, not the principals, is what holds them apart [18], [19]: AgenticOS isolates each agent capsule with hardware page tables beneath the model [17], and HarmonyOS 7
This systematization differs from existing surveys in four ways. (i) It indexes attacks and defenses by what the check at each trust boundary must decide, provenance or meaning, rather than by where an attack enters. The index tells which crossings a deterministic mechanism can cover and which need a probabilistic one, and it grades a reported attack-success figure accordingly, as a defense not yet adopted or as an open gap. (ii) It asks whether a complete mediator for the agent’s decisions can exist at all, rather than which OS techniques carry over to the agent. The answer gives a criterion for separating defenses that can be made complete from those that can only lower a residual, and says what an evaluation of the latter must report. (iii) It treats evaluation validity as a subject in its own right, so that a reader can judge how far a reported figure transfers to deployment. (iv) It extends the analysis to the AI-native OS, for which no
3
Conventional OS
Hybrid: de-facto agent kernel
Architectural AI-native OS
pre-agentic baseline
deployed today
horizon: envisioned
agent principal
model is the arbitration core
probabilistic, standing authority
applications kernel
decides scheduling, memory, authorization
deterministic resource arbiter
conventional kernel
no standing agent authority
agent atop the kernel: the kernel decides
owns scheduling, memory, arbitration
proposes
mechanism beneath, if any: executes what the model decides model is the kernel: the model decides
standing arbitration authority held by a probabilistic agent
Figure 2: The transition this paper spans: standing authority passes to a probabilistic agent, then arbitration itself passes to the model.
Table 1: How this work compares with the five nearest surveys. Covered = treated centrally; Partial = touched but not developed; Absent = not engaged. Entries describe each work’s stated scope, not its quality. LASM = Layered Attack-Surface framework. Dimension
Pirch et al. [6]
Dehghantanha & Homayoun [28]
LASM [27]
Kim et al. [25]
Ling et al. [26]
This work
Organizing principle
OS analogy: isolation, privilege separation, mediation
Attack surfaces along the data-flow pipeline
Seven stack layers × temporality
Design dimensions × attack vectors × risks
Lifecycle stages
Trust-boundary crossings, each placed by what its mediator must judge
Role given to the OS frame
Runtime as kernel, model as untrusted user; asks which OS techniques transfer
Deterministic execution kernel with the model as untrusted proposer, as a recommended pattern
Absent
Kernel–user analogy for planner/processor separation only
Absent
Asks whether an always-invoked mediator for the model’s semantic crossings can exist
Limit of runtime monitoring
Absent
Empirical — guardrails bypassable
Layer-locality argument: a control at one layer cannot see an attack at another
Empirical — detectors bypassed by adaptive attacks
Partial — over-blocking and observability dependence noted
Structural — residual argued irreducible where the mediator must judge meaning
Evaluation validity
Absent
Covered — evaluation protocol with validity controls
Partial — methodological issues of existing evaluations
Partial — benchmark caveats
Partial — benchmark gaps
Covered — reliability, evaluationawareness, confidentiality axis
Object of study
Deployed open-source agents
Deployed agentic systems
Deployed LLM agents
Deployed agentic systems
Deployed LLM agents
Deployed agents, then the AI-native OS
4
security systematization exists to our knowledge, so that designers of such systems know which deployed results carry over and which constraints they inherit. The key differences from the five closest surveys are summarized in Table 1.
ual. For the first judgment, Abdelnabi and Bagdasarian show that any flow a detector blocks can be reframed to appear legitimate, so no detector both blocks every attack and passes every legitimate flow [7], and Zverev et al. measure instruction–data separation directly and find that current models achieve little of it [31]. For the second, the only statement of what the principal authorized 4. A Trust-Boundary Taxonomy is the request itself, and the request is underspecified by The taxonomy applies to both objects of Section 2, the design: the principal delegated the task in order not to de-facto agent kernel and the AI-native OS. In each, an spell out what carrying it out would mean, and an intent LLM-based agent holds standing authority over system specified tightly enough to check exactly is a plan, not resources and exercises it through at least one of four ker- a delegation. That judgment is a specification problem nel functions: deciding which actions execute, scheduling before it is a detection problem, so a better classifier does and delegating work, managing contextual and persistent not close it either. The attacks these classifiers miss are state, and authorizing access to tools, files, networks, and the mediation gap. other agents. Section 4.1 recalls what a mediator is and derives the scale on which each crossing is graded; Section Web security met the same failure, data treated as in4.2 identifies the crossings around the agent principal and struction, in cross-site scripting, and contained it not with classifiers but with provenance and confinement: the the question a mediator must answer at each. same-origin policy, contextual escaping, Content Security Policy [32]. We therefore organize the boundaries 4.1 The Mediation Framework along a provenance-vs-semantics axis: at one end, crossMediators have confined privileged software for half a cen- ings whose mediating judgment can be settled in advance; tury. Anderson’s reference-monitor concept requires that at the other, crossings whose judgment can only be made every access to a protected resource pass through a media- on content that arrives at run time. Each crossing retion mechanism that is always invoked, tamper-proof, and ceives one of three grades on this axis. Deployment debt: small enough to be verifiable [29]; Saltzer and Schroeder the judgment can be settled in advance, so a determinelevate the always-invoked criterion into complete media- istic mediator is known, and what is missing is only its tion, every access to every object checked for authority on adoption. Structural gap: the judgment cannot be settled every occasion [18]. Schneider fixed the enforceable end in advance, so no deterministic mediator can exist and a of these requirements: a runtime monitor that halts on a residual remains. Mixed: the judgment can be settled forbidden step can enforce at most the safety properties partially in advance, and the rest must be made at run [30]. Together these bound what any always-invoked me- time. diator can promise, on one premise: that it can decide, at each step, whether the next action is the bad one. 4.2 The Trust Boundaries In the agent setting that premise fails, and not because An agent principal typically takes in content from users, the principal is probabilistic: classical reference monitors documents, and tools; reasons over that content and comwere built to confine adversarial, non-deterministic prin- mits to an action; sends the action to a tool, the host, or cipals, a malicious user process being their canonical case. another agent; and runs on an inference substrate shared What changes with the agent is the space the mediator with agents of other principals. This gives six crossings must judge. The agent’s repertoire is an open-ended (Figure 3): content enters the principal (B0); its reasonspace of natural-language behavior rather than a fixed ing commits to an action (B1); the action reaches a tool table of system calls. A deterministic mediator applies (B2), the host (B3), or another agent (B4); and the princia rule frozen before the input it judges exists, so it can pal’s state sits beside other principals’ state in the shared test only what is settled in advance: where content came substrate (B5). At each crossing a mediator must answer from, what authority vouched for it, whether an action one question, stated below with the crossing. lies in a declared set. It cannot test what content means, because the content arrives only at run time. A predi- B0: external input → agent. Untrusted content cate over provenance can therefore be deterministic, and crosses into the agent’s prompt context: user messages, retrieved documents, and tool outputs treated as context. a predicate over content semantics cannot. Mediation question: can the agent distinguish data from An agent kernel needs semantic judgments of two kinds: instructions at this crossing? We distinguish two cases whether content is data or an instruction, and whether an by who supplies the adversarial content, since they call action carries out what the principal asked. Each can be for different defenses. In B0-direct the input principal is made only by a classifier, which misses some fraction of itself the adversary, as in direct injection and jailbreaks; attacks at any false-positive rate a deployment can toler- the problem is authorization at the principal level. In ate. We argue that no better classifier removes that resid- B0-indirect a benign principal’s task carries adversarial
5
untrusted web / RAG / documents
mediation boundary (deterministic where the judgment is fixed in advance) structural gap: B0 / B1 admit only probabilistic mediation untrusted external entity (beyond the boundary)
B0 shared inference substrate
external input → agent
B5
B1
agent ↔ coresident principals
reasoning → action
Agent principal LLM reasoning core
B2
B4
agent → tool / transport
agent ↔ agent
B3
other agents
agent → OS / host
tools / MCP servers
OS / host resources
Figure 3: The six trust boundaries around the agent principal. B0 and B1 are graded as structural gaps: their mediating judgment cannot be fixed before run time. third-party content; the problem is provenance.
[33]. We count as an invocation any output that a downstream component acts on, whether or not the agent addressed it (the actuator convention): a chat client that fetches the image named by a markdown link, or a terminal that executes an escape sequence. • B2b — response-path/transport integrity. The channel returning a tool’s result, and the message path between agents. Mediation question: is the result the agent acts on the result the tool produced? This is an integrity problem on the action channel, downstream of an already-correct decision [34], [35]: in-transit modification of an otherwiselegitimate message, wherever the adversary sits, including on a compromised edge device. The line against B0 falls at the tool’s output: only content altered after the tool returned it belongs here. • B2c — configuration-channel provenance. The registration and loading of any instruction-bearing artifact the agent treats as executable configuration: tool descriptions, plug-in manifests, protocol capability advertisements [36], marketplace skills, rules
B1: agent reasoning → action. The transition from deliberation to a committed action. Mediation question: is the chosen action authorized and faithful to the principal’s intent? Both sides belong to the same principal, so what crosses is not authority but the binding of intent to action. B1 is the point at which a decision becomes an execution, and the point at which a mediator can stop the committed action. B2: agent → tool/transport. Three things cross here on separate channels: the call the agent issues, the result that returns, and the configuration that tells the agent which tools exist. They do not share a threat model, so we split B2 into B2a, B2b, and B2c as follows: • B2a — tool invocation. Outbound calls to tools and APIs. Mediation question: is each invocation least-privileged and authorized for the data that triggered it? The classical way to fail it is the confused deputy: a principal forwarding ambient authority to a tool on the strength of attacker-controlled text [18], 6
5. Attacks at the Trust Boundaries
files, and prompt templates. Mediation question: is the instruction-bearing metadata of a configuration artifact treated as untrusted data rather than as authoritative instruction? Tool poisoning and rug-pull attacks exploit exactly this provenance gap [37], [38] and persist across the ecosystem.
The attacker needs only one unmediated crossing to succeed. This section classifies the published attacks by the trust-boundary taxonomy of §4. For each attack class it asks which mediation failure the attack depends on, grades that crossing on the provenance-vs-semantics axis of §4.1, and separates mechanisms the literature conflates. Each class receives one primary boundary, the earliest crossing whose mediation failure is necessary for it, with later crossings recorded in chain notation (B0→B2a for an injection whose payload actuates as a tool call), and every crossing in the chain is graded by its own mediation question. Table 2 collects the classification: each row is one attack class with its primary boundary, the strongest evidence for it, and the mediation available against it.
B3: agent → OS/host. Effects on the underlying system: file access, process execution, network egress. Mediation question: are host-level operations confined to a least-privilege capability set? Host resources already carry reference-monitor machinery, so the mediator here can be sound; what it cannot validate is the intent behind an authorized syscall, and that is B1’s question. B4: agent ↔ agent. Negotiation, delegation, and message passing between agents. Mediation question: do trust and provenance labels remain non-downgradable across delegation? Decentralized-label discipline [39] is the classical answer, but negotiation in natural language produces behaviors that no label policy enumerates in advance.
5.1 B0: Direct and Indirect Prompt Injection Prompt injection makes the agent follow, as an instruction, content whose author had no authority to give one. In direct injection the writer is a user of the application, overriding the instructions its developer fixed in the system prompt: HouYi mounts it black-box against deployed LLM-integrated applications, with no access to the model [42]. The same direct channel serves reconnaissance: PLeak optimizes user queries that make deployed applications reveal their system prompts [82]. In indirect injection the instruction rides in content the agent retrieves or a tool returns, so the attacker need not be a user at all [41]; this is the form that matters most for an agent, since nearly everything it reads is content it did not author. Both forms cross B0: the user turn and the retrieved document are alike content entering the principal, and the mediation question is the same for both. Google’s web-scale monitoring finds injection patterns present and rising on the open web, up roughly one-third across successive CommonCrawl snapshots between late 2025 and early 2026 [83]. In the field, EchoLeak (CVE2025-32711) turned an indirect injection into zero-click exfiltration from a production enterprise copilot, carrying a crafted email past the vendor’s injection classifier [44]; in a large-scale public indirect-injection competition, every one of the thirteen frontier-class models evaluated was breached by at least one entry [43]. Both mediators, the vendor’s classifier and the model’s own refusal training, are probabilistic, and both were bypassed. Whether ingested content is instruction or data is a question about its meaning, and no deterministic check decides that, so the class is graded a structural gap.
B5: agent ↔ co-resident principals. Agents acting for mutually distrusting principals share one inference core, and what can pass between them is not a message but shared state: key–value caches, batching and scheduling queues, and context paged in and out on each principal’s behalf. Mediation question: does nothing cross between principals beyond what the scheduler intends? The obligation is isolation, the classical controlled-sharing problem of a multi-user system [18]. Today the serving framework beneath the agent enforces this isolation, as the host OS enforces least privilege on host operations (B3). In an AI-native OS the agent kernel schedules and multiplexes the inference core itself, so the isolation becomes its own obligation. B0 and B1 ask the two semantic judgments of §4.1 in pure form, and are graded structural gaps (Figure 3). B2c and B4 ask a provenance question first, which a deterministic mediator settles, and then the same two judgments on another channel: whether a registered tool description or an inbound agent message is itself an instruction, and whether a delegation hop passed the intent on faithfully. B2a, B2b, B3, and B5 ask provenance questions only. The taxonomy is a set of crossings rather than of layers, because complete mediation is defined over the act of crossing. A layered scheme such as LASM’s maps onto it at the layer interfaces [27], and an attack that spans layers becomes a path through several crossings (Figure 4). GeminiJack, a patched vendor proof of concept against an enterprise agent deployment, is such a path: a poisoned shared document enters at B0, persists in the retrieval store, and is retrieved later to drive a tool call that exfiltrates data across B2a, with no user interaction at any stage [40].
The surface widens once the agent perceives the open web or a rendered screen, and the payload need not be text. Injection has been delivered through rendered web content [45], through the accessibility tree that web agents read as their view of a page [46], and through content that adapts to the agent’s environment to exfiltrate personal data [47]; WASP measures the attack end-to-end
7
) worm tion (
a ropag self-p
iso po
ed ev
tri
retrieval / memory store
other agents
re
n
persistent-state transport: B0 payload carried forward in time
B4
shared doc / web page / email
er
lat
attackercontrolled content
B0
agent reasoning core
B1
privileged action / tool call
B2a
sensitive sink data exfiltration
GeminiJack: poisoned shared document → persists in store → retrieved → cross-tool exfiltration. EIA: adversarial web content read directly → generalist web agent → PII leak through the action channel.
Figure 4: Cross-boundary kill chains: one unmediated crossing suffices to complete a path. against web agents [84]. On a full desktop, VPI-Bench delivers injection through popups, banners, and screenshots [48], and OS-Harm measures it alongside misuse for computer-use agents [85]; adversarial images and audio carry the same injection into multimodal models [49]. Because these agents act on what they perceive, the injection actuates directly as a click or a tool call, which is the B0→B2a chain of Table 2. The mediation question is unchanged and only its carrier differs, so a B0 mediator that inspects text but not pixels or audio leaves the crossing open.
highly by the retriever rather than to persuade the model that reads them [57], [58]. All three enter through the agent’s ordinary ingestion path, so all three are B0 and share its structural gap; what differs is where the mediator must stand. Corpus poisoning is narrowed by provenance tracking over retrieved data, enforced at retrieval or load time while the origin of each passage is still known; self-writtenmemory poisoning requires that the agent’s own outputs be treated as untrusted when later retrieved, which collapses the trusted/untrusted partition memory systems assume; retriever-targeted poisoning defeats any defense that trusts the retrieval mechanism while scrutinizing only its inputs and outputs. A backdoor trained into the retriever [87], or into the model and its embedders and monitors [88], is not a crossing of the running principal’s interface: it is settled before the principal runs, and its mediator is the provenance of the artifact itself, the analogue of verifying the kernel image at boot rather than mediating its accesses at run time.
5.2 B0: Poisoning of Persistent State Persistent state lets an injection wait: content poisoned in one session is retrieved and followed in a later one. The literature conflates three mechanisms whose mediation questions differ. Corpus poisoning targets the document store: PoisonedRAG steers an agent’s output by corrupting as few as five entries, because the passage arrives through a channel the agent treats as trusted context [50], [51], and CorruptRAG succeeds with a single poisoned text without outnumbering benign ones [52]. Selfwritten-memory poisoning corrupts state the agent itself wrote in a prior turn or session, through query-only interaction by an ordinary user [54], by backdooring long-term memory together with the knowledge base [53], through the experience-retrieval loop [55], and through memorycontrol-flow attacks that force unintended tool calls [56]; Anthropic now shares one user’s memory across its chat and agentic surfaces, so a poisoned entry is scoped to the user rather than to the session that wrote it [86]. Retriever-targeted poisoning crafts passages to be ranked
5.3 B1: Selective Disclosure and Goal Drift Not every deviation from the principal’s intent is injected from outside. In selective disclosure the agent advances a goal by withholding material information rather than by asserting a falsehood. The unified deception taxonomy lists omission as a mechanism of its own, beside fabrication and pragmatic distortion [59], an agent can report to its overseer in ways that stay literally truthful while subverting oversight [60], and frontier models under evaluation show in-context scheming of the same kind [61]. Goal drift is the gradual case: an agent given an explicit 8
Table 2: Published attacks classified by the trust-boundary taxonomy. Each row names the class’s primary boundary; → marks a necessary later crossing, / marks alternatives at the same position, and parentheses qualify the crossing. Deployed vs. lab records the strongest evidence reached, a shipped-product exploit or a laboratory proof of concept. Mediation available records what can stop the attack: a deterministic mediator not yet deployed (deployment debt), only a probabilistic one (structural gap), or a deterministic one for part of the class (mixed). Boundary chain (primary first)
Attack class
Representative attacks
Deployed vs. lab
Mediation available
B0
Direct/indirect prompt injection
Greshake [41]; HouYi [42]; public IPI competition [43]
Deployed (EchoLeak, CVE-2025-32711 [44])
Structural gap
B0→B2a
Web, GUI, and multimodal WebInject [45]; accessibility-tree injection (malicious DOM, IPI [46]; EIA [47]; VPI-Bench accessibility tree, screenshot, [48]; Bagdasaryan [49] image/audio)
Deployed (browser and computer-use products)
Structural gap → deployment debt
B0 (persistent)
Corpus poisoning
Zhong [50]; PoisonedRAG [51]; CorruptRAG [52]
Lab
Structural gap
B0 (persistent)
Self-written-memory poisoning
AgentPoison [53]; MINJA [54]; MemoryGraft [55]; Memory-control-flow [56]
Lab
Structural gap
B0 (persistent)
Retriever-targeted passage poisoning
BadRAG [57]; Joint-GCG [58]
Lab
Structural gap
B1
Selective disclosure; goal drift
Deception taxonomy [59]; upward Lab deception [60]; in-context scheming [61]; goal-drift evaluation [62]
Structural gap
B0→B2a/B3
Taint-style RCE via tool execution (confused deputy)
AgentFuzz [63]; complete-takeover [64]
Deployed (34 zero-days / 23 CVEs)
Structural gap → deployment debt
B2b
Response-path / transport tampering
RTA [34]; malicious intermediary [35]; Agent-in-the-Middle [65]
Lab
Deployment debt
B2c
Configuration poisoning (tool descriptions, MCP metadata, skills)
MCP attack vectors [37]; MCPTox [38]; ToolHijacker [66]; BadSkill [67]; marketplace audit [68]
Deployed (real MCP servers; skill marketplace)
Mixed
B3
Sandbox/container escape from host confinement
Sandbox-escape evaluation [69]; OpenClaw sandbox-to-host chain [70]
Deployed (four-CVE chain to host persistence)
Deployment debt
B0→B4
Agent-to-agent worms / self-propagating injection
Morris-II [71]; Prompt Infection [72]; Zombie Agents [73]
Lab
Structural gap → structural gap
B4
Manipulated-knowledge spread between agents
Flooding spread [74]
Lab
Structural gap
B1→B4
Intent drift along delegation
Telephone-game chains [75]; MAST inter-agent misalignment [76]; inherited goal drift [77]
Lab
Structural gap → structural gap
B5
Cross-tenant KV-cache and scheduling channels
PROMPTPEEK [22]; InputSnatch [21]; Shadow in the Cache [78] (cache-at-rest, stronger adversary); RAG non-prefix cache [79]
Deployed (serving substrate, not the agent)
Deployment debt
B0 (availability)
Resource exhaustion / denial-of-wallet
OverThink [80]; tool-chain amplification [81]
Lab
Deployment debt
objective and then run over a long horizon under competing environmental pressures deviates from it, and every model evaluated drifts to some degree, the more so as its context grows [62]. Both deviations arise within the agent’s own decision process and require no adversarial crossing, so they are assigned B1, the crossing at which they first become observable to an overseer. The rule that assigns each class its earliest crossing leaves B1 few primary rows, but every chain in Table 2 that ends in an
action passes through it, since an injected instruction does harm only once the agent commits to acting on it. The crossing is graded a structural gap. A report that stays literally true passes any check on what it says; whether what it omits was material can be decided only against the principal’s intent, and whether a drifted action still serves the objective is the same judgment. Neither is a check a deterministic mediator can make.
9
5.4 B2: Tool-Interface Attacks
rug-pull, impersonation, unsigned artifacts — deterministically. But verifying who published a description does Three channels cross B2, and each carries its own attack not catch one that is legitimately registered and whose class and its own grade. text is itself malicious, as MCPTox shows; ruling regisB2a — Tool hijacking. An injected instruction is inert un- tered metadata benign or hostile is a semantic judgment til it reaches an actuator; the tool call is where hijacked the provenance mediator does not reach, so the crossing reasoning becomes privileged action, with the agent as is graded mixed. the confused deputy. What §5.1 records as a failure of inConfiguration poisoning is easily confused with the memgestion, B2a records as a failure of invocation: the same ory poisoning of §5.2, since both plant an instruction the chain, judged at its second crossing. AgentFuzz, which agent will later trust; what separates them is origin, not drives one agent to attack another, found 34 zero-day how the artifact is later loaded: a memory entry arrived vulnerabilities in deployed agent software, 23 of them asthrough the agent’s ingestion or self-write path and is signed CVEs, each confirmed exploitable by the authors’ data at B0, whereas a tool description or skill was inpipeline [63]; InjecAgent benchmarks the same confusedstalled and is configuration at B2c. deputy path across tool-integrated agents [89], and a complementary line drives agents to complete host takeover [64]. The chain needs both crossings, B0 for the injection 5.5 B3: Sandbox Escape and B2a for the actuation (B0→B2a in Table 2), so a At B3 the agent’s actions reach the host’s files, procomplete mediator at either would close it. Neither has cesses, and network. Reaching them through an authoone. B0 admits only probabilistic mediation (§5.1). B2a rized call is the terminus of the tool-hijacking chain of has a deterministic check, least-privilege invocation, so on §5.4 (B0→B2a/B3 in Table 2) and is graded with it; B3’s its own it is deployment debt; but least privilege bounds own class is escape from confinement, in which the agent’s what an actuated call can do, not whether it should have process gains host access it was never granted. Frontier been issued. The class is therefore primary at B0 and in- models probe their way out of container sandboxes unherits its structural grade, with deployment debt at B2a. der evaluation [69], and the OpenClaw agent platform B2b — Transport tampering. Response-path attacks alter supplies the deployed record: a one-click remote code exthe channel after the model has decided, so that the ac- ecution and a critical privilege escalation [90], [91], and tion taken diverges from the action chosen. Because the a four-CVE chain from code execution inside the sandtampering sits downstream of the decision, it succeeds box to persistence on the host [70]. The mediator here — against models that have already passed safety training process sandboxing, containers, seccomp, capability drop[34], and a malicious intermediary on the agent’s com- ping — is mature and deterministic, so an escape is a munication path can rewrite messages in transit [35]; be- bug in a sound mediator, closed by hardening what extween agents, an adversary that only intercepts and ed- ists rather than by any advance in semantic defense. The its the messages they exchange compromises the whole crossing is graded deployment debt. system without touching any agent [65]. Whether the result the agent acts on is the one the tool produced is a message-integrity question. Signing or channel authentication decides it, and the crossing is open only because those primitives are not deployed on agent tool chains, so it is graded deployment debt. B2c — Configuration poisoning. Tool descriptions, plugin manifests, and protocol metadata are loaded once and trusted thereafter, and the Model Context Protocol (MCP) ecosystem now supplies the evidence that they are treated as instruction. A systematic study catalogues tool-poisoning, puppet, and rug-pull attacks [37]; MCPTox measures roughly 66% average attack success for tool poisoning against real MCP servers [38]; ToolHijacker poisons tool selection with a single malicious tool document, which is trusted because it was registered and acts only when selected (chain B2c→B2a) [66]; BadSkill poisons skill marketplaces at low poison rates [67], and an audit of the OpenClaw marketplace found 341 malicious skills among the 2,857 then listed, rising to 824 as the marketplace grew past 10,700 [68]. Attestation and least-privilege registration close the provenance half —
5.6 B4: Worms and Intent Drift Two mechanisms cross B4, and the literature merges them. Self-propagating injection is a payload that, once processed by one agent, causes it to emit the same payload to others, replicating without further attacker action: Morris-II demonstrates this against RAG-backed email assistants [71], Prompt Infection spreads it across heterogeneous multi-agent systems [72], and Zombie Agents makes it persist across sessions in self-evolving agents [73]. The payload enters at B0 and multiplies at B4, so the class is primary at B0 (B0→B4 in Table 2) and inherits its structural grade. B4’s own mediation question, whether trust labels survive delegation, is decidable, but a per-hop check that preserves the label still forwards a correctly labeled malicious payload to the next victim, so label discipline does not stop a worm. Intent drift is the degradation of the principal’s intent along a delegation chain, as each agent reinterprets, summarizes, or filters what it forwards. The benign record establishes the mechanism: in transmission chains of LLM agents, biases negligible in a single output accumulate
10
over hops and pull the content toward attractor states [75]; across 1,600 traces from seven multi-agent frameworks, information withholding and ignored input between agents are recurrent failure modes [76]; and an agent conditioned on the trajectory of a weaker agent inherits that agent’s drift [77]. The adversarial counterpart enters at B4 itself: a single manipulated agent persuades benign peers to adopt and spread counterfactual knowledge without any injected instruction, and the manipulation persists once the peers store the conversation in retrieval memory [74]. Drift proper begins at B1, as the goal drift and selective disclosure of §5.3, and compounds at B4 (B1→B4 in Table 2), whereas a worm is a B0 payload that B4 replicates. The two scale differently: drift grows with the length of the delegation chain, a worm’s blast radius with the size of the agent network. Non-downgradable labels across delegation answer drift’s provenance half, but whether each hop’s forwarding was faithful to the intent is the B1 question repeated at every hop, so drift carries B1’s structural residual into B4.
its the human-readable fields of its returns while leaving their structure intact steers an agent into prolonged toolcalling loops that raise per-query cost by up to 658×, a denial of wallet [81]. A training-time variant, data poisoning that conditions a model to emit near-unbounded output on a trigger [93], is settled before the principal runs and is mediated by artifact provenance, as in §5.2. Both runtime attacks enter at B0, as ingested content and as a tool’s own return. What stops them is not a mediator at B0 but a resource counter: budgets, quotas, rate limits, loop bounds, per-task token ceilings. A counter decides on a number, not on what the content means, so it is deterministic, and the class is graded deployment debt.
Availability is attacked through crossings the taxonomy already names, but the property harmed is cost and latency rather than the integrity of an action. OverThink injects decoy reasoning problems into content a reasoning agent ingests, inflating the tokens spent per query while leaving the answer correct [80]. A malicious tool server that ed-
tacks, and they do not survive an attacker who adapts to the deployed defense. Known-answer detection is broken by an adaptive white-box attack [98]. DataSentinel had trained its detector in alternation against an attacker searching for injections that preserve the answer [97], and the same attack still drives it to 0% detection on most
6. Runtime Monitoring as Probabilistic Mediation
A runtime monitor is a mechanism separate from the agent that watches it and intervenes when it judges behavior unsafe [94]. Wherever a crossing’s mediation question is semantic, current practice deploys a runtime monitor as the mediator. A monitor has a threshold: lowering it 5.7 B5: Cross-Tenant Leakage catches more attacks and rejects more benign traffic, so Serving frameworks such as vLLM and SGLang reuse the a miss rate means something only together with the falsekey–value (KV) cache across requests that share a token positive rate at which it was measured; we call that pair prefix, and sharing it across users turns the cache into the monitor’s operating point. a channel. The first channel is timing: a request whose prefix is already cached returns its first token sooner, and Monitors differ in what they read: the input before the InputSnatch turns that into an attack through the ordi- model sees it, the reasoning the model writes down, the acnary API that recovers how much of a prefix is cached tivations inside the model, or the actions the agent emits. [21]. The second is service order: prefix-aware scheduling Sections 6.1–6.4 take the four layers in that order. Each serves a request earlier when it shares more cached state, states which crossing the layer mediates, what escapes it, and PROMPTPEEK reconstructs other users’ prompts and what miss rate has been measured. Section 6.5 asks from that ordering alone [22]. Two further lines widen what stacking the layers buys. the surface under stronger assumptions: inversion and injection attacks on the cache at rest, by an adversary 6.1 Monitoring the Input holding the plaintext cache and the model weights [78], An input monitor mediates B0: it inspects untrusted conand extraction through the non-prefix cache fusion that tent before the model reads it and rejects what it judges RAG stacks use to splice retrieved chunks, which reaches to be an attack. Anthropic’s Constitutional Classifiers the agent’s own retrieval path [79]. are the best-documented instance: input and output clasNever sharing cache across tenants is a complete determin- sifiers trained against a written constitution of permitistic mediator that closes the crossing outright, and where ted and forbidden content cut universal-jailbreak success sharing is enabled it has been declined for throughput; from 86% to 4.4% at a 0.38% rise in refusals of benign trafpartitioning on provenance — per-tenant namespaces, or fic, and the production version holds the added refusals sharing only system-supplied prefixes — is the same me- near 0.05% [95], [96]. DataSentinel builds on knowndiator at finer grain, and SafeKV benchmarks both as answer detection, which hands the untrusted content to established baselines [92]. The mediator exists and is un- a separate detector model together with a secret instrucadopted, so the crossing is graded deployment debt. tion, and reads the detector’s failure to repeat that instruction as evidence of injection [97]. 5.8 Availability across the Crossings Reported detection rates are measured against fixed at-
11
tasks [98]. Gradient, reinforcement-learning, search, and human-guided attackers tuned against twelve recent jailbreak and injection defenses bypass most of them with success above 90%, where the majority had reported success near zero [99]. Under adaptive attack, the input layer misses most of the attacks it was reported to catch. 6.2 Monitoring the Chain of Thought A reasoning model writes intermediate reasoning before it acts. A chain-of-thought monitor reads that text and mediates B1: it asks whether the reasoning shows the agent pursuing something other than what the principal asked. On reward hacking by a frontier reasoning model in coding tasks, Baker et al. find that a monitor reading the chain of thought catches 95% of the hacks and a monitor reading only the actions catches 60% [100]. OpenAI’s monitorability suite confirms the ordering on practical tasks, reports that reinforcement learning at current scale does not materially reduce what the trace reveals, and shows that follow-up questions to the model, with the answers passed to the monitor, raise the catch rate [101]. In OpenAI’s GPT-5.6 evaluation the model reasoned about seeded sabotage in its trace but kept it out of its final output in 76% of the trajectories in which it continued the sabotage [102]. Models also cannot yet control the trace: asked to keep a given piece of reasoning out of it, Claude Sonnet 4.5 succeeds 2.7% of the time, against 61.9% for its final output [103]. LlamaFirewall’s alignment check audits agent reasoning for injected goals in deployment [104], and OpenAI monitors 99.9% of its internal coding-agent traffic with a frontier model that reads every tool call and the chain of thought [105].
channel the trace leaves out: Anthropic’s activation reading finds representations of rule-circumvention, fabrication, and evaluation-awareness that the chain of thought never mentioned [109]. Detection reads a signal: linear probes and sparse-autoencoder features flag drift or injection [110], [111], attention signatures separate injected from legitimate instructions without training [112], and the signals have a mechanistic basis, since ablating a single safety-specific attention head makes harmful output roughly 16× more likely [113]. Intervention follows detection: ICON steers attention back to the legitimate task when it detects over-focusing on injected content [114]. Interpretation renders the signal: attention attribution locates which untrusted span drove a decision, and activation decoding writes internal state out as natural language [115], [116]. OpenAI now runs the layer in production as the first stage of a pipeline: activation classifiers inspect every sampled token and escalate to automated investigators that read the tool actions and the reasoning, at roughly 20% of the monitored inference compute [117].
The best measured case shows what the layer can and cannot check. NeuroFilter reports a 0% bypass rate on its privacy-policy trajectories, because it checks activations against one fixed norm and deviating activations are nearly orthogonal to it; on an out-of-distribution attack family the same monitor misses 55%, and a generalpurpose probe scores at or below 65% [118]. A pinned predicate is checked exactly, and the open action space is not. The representation is also a target: obfuscatedactivation attacks perturb what the probe reads and cut its recall from 100% to 0% while 90% of the outputs remain jailbroken [119], and a model can be reinforcementtuned to produce harmful output while its activations stay The trace is legible because nothing has yet trained the inside the accepted region [120]. The layer needs whitemodel to hide it. Baker et al. penalize the agent during box access to the residual stream, so an operator of a reinforcement learning whenever the monitor flags its rea- closed model cannot run it at all. soning, and find that the agent exploits the task’s reward less at first, then as often as before but without writing 6.4 Monitoring the Action Trajectory about it, so the monitor’s recall falls to near zero [100]. No training is needed when the adversary can edit the trace. An action monitor reads what the agent emits: tool calls, Rewriting only the reasoning of a completed exploit to their arguments, and their results, over the whole trajecread as good-faith engineering, with every command and tory. It mediates B2 and B3, and it is the only layer output left unchanged, drops a held-out monitor’s catch that sees a chain of individually permitted steps converge rate from about 95% to under 11% on the hacks the ac- on a harmful outcome. AgentSight reads the trajectory tions do not give away [106]. A backdoor implanted by from outside the application: it traces the agent–kernel fine-tuning does the same from inside: the model pro- boundary with eBPF, links the model’s emitted intent duces the attacker’s chosen behavior while its trace stays to the system calls it triggers, and catches prompt injecbenign [107]. And the trace shows only what the model tion and runaway tool loops at under 3% overhead [121]; puts into words. The multi-lab monitorability statement whether a linked pattern is malicious is then handed to a draws the conclusion: the trace is a fragile opportunity, language-model observer. CausalArmor makes that judguseful while the model is not optimized against it and ment cheaper by attributing the agent’s intent to one only for intent the model puts into words [108]. input segment through leave-one-out ablation and sanitizing only that segment [122]. Frontier models are the strongest monitors of this kind, and three evaluations 6.3 Monitoring Internal Activations measure them. SHADE-Arena pairs benign main tasks An activation monitor reads the model’s internal repre- with harmful side tasks; the best monitor, Gemini 2.5 Pro, sentations rather than its text, and mediates B1 on the separates benign from sabotage transcripts at an AUC of 12
0.87 [123]. SLEIGHT-Bench fixes the operating point: at a 1% false-positive budget an Opus 4.6 monitor with extended thinking catches 32% of forty evasion attacks, and twenty of them are never caught across ten trials [124]. And the monitor is itself a model reading attacker-shaped text: prompt injection aimed at the trusted monitor lets frontier agents evade it and complete malicious tasks on both main AI-control benchmarks, and a protocol that resamples on suspicion amplifies the injection [125]. 6.5 Composing Monitors Every layer misses attacks, so deployments stack them. The stack’s miss rate is not the product of the layers’ miss rates: LlamaFirewall’s input classifier and chainof-thought auditor let through 7.5% and 2.9% of AgentDojo’s attacks alone and 1.75% together, against 1.2% if their misses were independent [104]. Three reasons make the misses overlap. The layers read different channels, so an attack is judged only by the layers that read the channel it uses: intent the model never puts into words passes the trace monitor unjudged (§6.2), and a chain of individually permitted steps passes every layer but the trajectory monitor (§6.4). The layers are learned predictors with correlated failures: one obfuscation of the activations defeats harmfulness probes, sparse-autoencoder features, and out-of-distribution detectors alike [119]. And a monitor is itself a model: it can be poisoned before deployment [88] or injected at run time (§6.4), and a compromised monitor fails silently at the layer meant to catch the others. No surveyed evaluation measures a stack under adaptive attack. The nearest evidence is Google DeepMind’s from defending Gemini: eight injection defenses were each tested against three attacks, once with the attack fixed and once with it tuned to the defense, and in sixteen of the twenty-four pairings the tuned attack succeeded at least as often as the fixed one; Gemini’s defenses are therefore re-evaluated continuously rather than certified once [126]. Stacking narrows the mediation gap of §4.1, but does not close it.
7. Architectural Separation as Deterministic Mediation Architectural separation arranges the components of the agent so that untrusted content and privileged action never meet on one channel. What remains to check is deterministic: whether an operation is in the fixed set, whether data marked untrusted may reach its destination, whether a signature verifies. As agents run unattended, architectural separation bears the assurance that a human in the loop used to provide. 7.1 Separating Untrusted Content from Privileged Action The strongest architectural defenses neutralize prompt injection by construction rather than by classification, and
the first family does so in the control flow. Willison’s dual-LLM pattern pairs a privileged model that has tool access but never reads untrusted content with a quarantined model that reads untrusted content but holds no tools [127]; Beurer-Kellner et al. catalogue the design patterns that grew from it: plan-then-execute, context minimization, action-selector, and the dual-LLM family itself [128]. CaMeL is the evaluated instance: a privileged model emits a plan over a fixed set of operations before any untrusted content is read, and the quarantined model that handles the content may fill in data values but cannot alter the control flow, so an injected instruction cannot redirect the sequence of privileged actions [129]. The doctrine behind it is the object-capability tradition, which bounds the confused deputy by making authority a possessed, unforgeable token naming an object and a permitted operation, so that possession proves authorization and no ambient grant can be confused into misuse [130], [131], [132], [133]. CaMeL’s planner-issued capabilities bind each operation to such a token rather than to whatever instruction last appeared in context, which installs a deterministic check at the B1 and B2 crossings. Platform ecosystems now perform the same re-enumeration at scale: Apple’s App Intents and Android’s App Functions fix at build time the operations the system may discover and invoke [4], [5]. An operation the developer never declared is invisible to the orchestrator, which is the expressiveness cost; the declaration is also the registration-time provenance mediator the B2c crossing prescribes (§4.2). A second family separates execution domains. IsolateGPT runs each third-party app or tool of an LLM platform in its own isolation domain and permits crossdomain interaction only through a hub that mediates every inter-app call, at under 30% overhead [134]; Prompt Flow Integrity adds secure handling of untrusted data and explicit privilege-escalation guards to the same isolation [135]. A compromised tool’s blast radius is bounded by its domain. The two families mediate different crossings and compose: CaMeL fixes what may execute, IsolateGPT where each principal executes. One family worth mentioning here is data–instruction separation inside the model. Spotlighting delimits or encodes retrieved text so that the model can tell it from instructions [136]; ASIDE, ISE, and AIR mark the data channel in the representation, by rotating or tagging data-token embeddings and by carrying the instruction hierarchy’s privilege signal through the layers [137], [138], [139], [140]; StruQ and SecAlign train the separation into the weights, by fine-tuning on marked queries and by preference optimization against injected instructions [141], [142]. This is not architectural separation, and its guarantee is of a different kind. CaMeL keeps an injected instruction out of the control channel, whereas these techniques leave every flow possible and only make the model less likely to follow the instruction.
13
7.2 Bounding the Influence of Persistent State None of the defenses above reaches the persistent B0 channel, which §5.2 separated into three variants. RobustRAG provides certified robustness against bounded retrieval corruption through an isolate-then-aggregate structure: each retrieved passage is processed in isolation and the per-passage responses are securely aggregated, so a bounded number of poisoned passages cannot control the output beyond a provable bound [143]. It covers the corpus-poisoning variant only. The self-written-memory variant now has defenses on both sides of the probabilistic/structural divide. A-MemGuard validates each reasoning path by consensus against related memories and stores detected failures as lessons consulted before future actions, reporting attack-success reductions above 95% at minimal utility cost [144]; as a learned consensus detector it is a probabilistic mediator and carries the mediation gap. MemLineage is the structural counterpart: a Merkle-log-backed derivation DAG over agent memory records which retrieved entries influenced each new write, and sensitive actions whose justification descends from an external ancestor are gated, at zero attack success on its memory-poisoning workloads and sub-millisecond overhead [145] — the retrieval- and load-time provenance mediator §5.2 prescribes. LiSA gates the reuse of accumulated safety memory: an entry is released only when a posterior bound over its evidence clears a declared threshold [146]. The gate is arithmetic over counts, so the residual moves into the trustworthiness of what was counted. The retriever-targeted variant has no published defense. The strictest bound retains nothing: Apple’s Private Cloud Compute keeps no client-derived state between requests, so there is no persistent surface to poison, at the cost that defenses depending on durable memory cannot be expressed [147]. 7.3 Preserving Trust Labels across Agent Delegation Structural confinement within a single agent does not survive delegation unless trust labels travel with the data. When one agent forwards content to another, the receiver must know whether it originated from trusted configuration or untrusted runtime input; if the source-trust label can be stripped or downgraded in transit, the B4 crossing reintroduces the confused deputy that §7.1 closed within a single agent, and the multi-agent topology launders provenance. The countermeasure is decentralized information-flow control (IFC). The decentralized label model attaches confidentiality and integrity labels to data and permits a principal to restrict but never to relax labels it does not own; that label monotonicity is the structural invariant [39]. OS-level DIFC systems enforce such labels end-to-end across processes through a trusted reference [148], [149], [150], [151]. Transposed to agents, “untrusted” becomes a sticky integrity label: a taint attached to a payload may be further restricted by any downstream
agent but never elevated, extending CaMeL’s intra-agent labels [129] into inter-agent delegation. FIDES is the fullest agent-side instance: confidentiality and integrity labels attached as the agent plans and acts, with deterministic label propagation bounding what untrusted inputs can influence, evaluated on AgentDojo [152]. Nondowngradable provenance is necessary but not shown sufficient: enforcing label monotonicity across heterogeneous agents with differing trust semantics is open, and no surveyed system — FIDES included, whose labels live within one planner’s trust domain — demonstrates endto-end label preservation across an organizational boundary, where the labels’ meaning may not be shared [153].
8. Authorization as Mediation of the Principal Authorization in the agent era must attach to an action’s reversibility, not to the actor’s identity. Identity does not thereby become irrelevant: it is what scopes an agent’s standing authority and makes its actions attributable, while reversibility is what gates the consequential ones. Together they are the policy layer deciding which crossings of B2a (agent→tool), B3 (agent→OS/host), and B4 (agent↔agent) are permitted at all. A soundly placed authorization gate is a deterministic mediator over an adversarial principal rather than a trustworthy classifier of one. The multi-agent, cross-device setting then exposes tensions around cross-protocol trust, collusion, and the location of enforcement that none of the surveyed schemes resolve. 8.1 Agent Identity as a First-Class Principal Authorization begins by treating the agent itself as a named principal rather than as an extension of the user who launched it. When an agent acts with the ambient authority of its operator, every tool call inherits the operator’s full privilege, recreating the confused-deputy condition; a distinct identity lets the agent’s authority be scoped, attenuated, and audited independently of the human on whose behalf it acts. Industry and research guidance now codifies this under a zero-trust framing in which no agent is trusted by virtue of its origin, with agent identity, certification, and activity logging argued as firstclass infrastructure [154], [155], [156]. The durable principle is least privilege [18]: an agent holds only the capabilities its current task demands, for only as long as the task runs, and the grant is observable to a reference-monitorstyle mediator [29]. Progent’s programmable per-call privilege policies cut attack success from 39.9% to 1.0% on AgentDojo by checking each tool invocation against a least-privilege specification [157]; AgentSpec writes such policies as trigger–predicate–enforcement rules checked at each tool call [158], and ClawGuard induces them per task and enforces them at every tool-call boundary, reporting near-zero indirect-injection success on its bench-
14
marks [159]. Mobile-style scoped permissions and continuous information-flow-aware mediation are advanced as generalizations [160], [161]; the OS vendors are beginning to supply scoped permissions from below, through execution containers with OS-enforced access declarations and per-agent identity in place of the operator’s borrowed authority [3], [23].
through approval, scope, or policy gates, tiering access from read-only through sandboxed edit to gated full access with mandatory review at the irreversible tier [165].
• R1 — Composition. Individually reversible actions can compose into an irreversible effect, so reversibility is a property of the action sequence, not of each step in isolation. • R2 — External visibility. An outward action becomes irreversible the moment its effect is observed beyond the trust boundary: a message is irreversible once read, not once recalled. • R3 — Confidentiality taint. Any action moving sensitivity-labeled data across an external boundary is irreversible-tier whatever its own reversibility, because a disclosure cannot be undone.
man can also be deceived, because the approval prompt is drawn by the client from the agent’s own output. Under the actuator convention that renderer is a B2a crossing, the client-rendered channel EchoLeak exfiltrated through [44], so a hijacked agent can shape what the human sees and obtain assent for an action other than the one that will run. Classical systems security answers this with a trusted path, an unspoofable channel between the user and the enforcement mechanism; no surveyed mechanism supplies one for agent approval gates.
Reversible operations — drafting a document, reading a file, issuing an idempotent query — can be permitted autonomously and corrected after the fact. Irreversible ones — sending external communications, executing financial transfers, deleting persistent state, granting further permissions — admit no rollback and are reserved for human-in-the-loop confirmation. Reversibility is an integrity criterion, and confidentiality does not obey it: reading a sensitive object changes no state, yet what it discloses cannot be un-read. R3 closes this gap. The read attaches the object’s sensitivity label to everything derived from it, and the labels of §7.3 cannot be downgraded, so a later action that carries the labeled data to an outbound socket, a public log, or an untrusted tool is gated as irreversible even though the action by itself could be undone. Without R3 a chain of locally reversible reads and writes realizes exactly the exfiltration of the GeminiJack kill chain (Figure 4). The pattern is already practice: an analysis of twenty-one production agent deployments finds that all of them keep a human in the security loop
Delegation moves authority across agents and devices. Verifiable delegation needs a cryptographic identity anchor. W3C Decentralized Identifiers and Verifiable Credentials let a downstream agent prove who it is and on whose authority it acts, so a delegated request carries its authorization rather than re-asserting it at each hop [153], [166]; workload-identity standards [167] and agentspecific frameworks [168], [169] instantiate the anchor. AgentSafe applies it in-band: an authenticated communication layer validates each message’s provenance and sender authority, and a trust-tiered memory store separates what each tier may read, with defense success above 80% against message-spread attacks [170]. No agreed interoperation layer joins the identity domains, so a delegation chain verifiable within one domain need not remain so once it crosses into another.
A practical test is “Impossible versus Tedious” [154]: a sound authorization boundary makes a harmful outcome impossible to reach without human assent, not merely tedious. A probabilistic check an attacker can grind past is tedium; a hard gate on irreversible actions halts the 8.2 Authorization Tiered by Operation offending execution rather than usually flagging it [30], Reversibility and since monitors are evadable (§6), the gate is the place Recent guidance grounds the gate in operation reversibil- for a structural guarantee on high-consequence operations. ity [154], [162]. The lineage predates the guidance docu- The gate retreats from the open action space to a finite, ments: GoEX proposed a reversible execution envelope mediable set of consequential actions. with post-facto validation, undo, and damage confine- The gate’s guarantee is conditional in two ways. It esment as runtime primitives in 2024 [163]. The alterna- capes the mediation gap only while the irreversible set is tive — dynamic, risk-adaptive authorization that tracks enumerated in advance. If membership must be judged the agent’s fluctuating trustworthiness [164] — inherits at run time, the judgment is semantic and the gap rethe mediation gap, since a runtime trust estimate of a turns. And the gate guarantees only that a human says stochastic principal is itself a probabilistic classifier. We yes before an irreversible act, not that the yes is welltherefore index authorization to the operation’s conse- founded. R1 and R3 are conservative by design and place quences, a property determinable without estimating the most action sequences and every action touching sensiprincipal at all. Reversibility is not yet a formal predicate, tive data behind the gate, so the human is asked often but three rules give it a checkable shape. and approval decays into confirmation fatigue. The hu-
8.3 Multi-Agent and Cross-Device Trust
A delegation chain is also only as trustworthy as its weakest participant: a single compromised or underprovisioned device caps the trust of the whole chain, the weakest-device bound [162]. An adversary controlling one
15
low-assurance node in an otherwise hardened mesh issues requests that inherit the chain’s accumulated authority, and the compromised endpoint becomes a confused deputy for the whole system [33], [154]. The labels of §7.3 do not help here: they record where data came from, and their enforcement assumes that every hop is honest, which is exactly what a compromised device is not. None of the surveyed agent-level mechanisms bounds the trust of the device itself.
agent’s safety score says nothing about it, so a platform serving more than one principal must measure the substrate separately. The Harm scored column shows what is counted: hijacked actions and aggregated unsafe actions dominate, and confidentiality is counted separately by four benchmarks (InjecAgent, AgentDAM, RedTeamCUA, SafeClawArena) and by no defense evaluation — every defense in §§6–8 reports an attack-success rate over actions and none separates an exfiltration rate, although exfiltration is the realized harm of the composed kill chain A centralized policy-enforcement point gives consistency of Figure 4, so a defense’s reported number leaves its exand a single locus for audit and complete mediation [18], filtration rate unknown. [171], but is a single point of failure and a scaling bottleneck in a fluid mesh [161]. Distributed enforcement, The Scoring and Harness columns show under what conthrough per-agent policies derived at run time [172], [173] ditions the number holds. Deterministic checks appear or AgentSafe’s in-band checks, survives the loss of any where the outcome is mechanical, a specific tool call node and scales with the mesh, but is harder to keep that was or was not issued or a file, OS, or workspace consistent and multiplies the surfaces an attacker can tar- state that was or was not reached, so the scoring adds get. For agents the trade-off is sharper than the classical no variance of its own. Where the harm is an aggreone, because wherever the enforced predicate is semantic gate over categories of unsafe action, the score is usuthe mediator is probabilistic. A centralized point con- ally a model judge’s (Agent-SafetyBench, OS-Harm, A3Scentrates the miss rate in the one classifier that gates Bench, AgentCanary; SABER keeps rules primary), so everything, and adaptive injection subverts such trusted the broadest safety numbers move with the judge and inmonitors [125]. Distributed points are many smaller me- herit its variance and bias. Harness realism runs from diators whose learned failure modes are correlated, and recorded tool responses with no execution (InjecAgent), when agents watch agents they can be induced to collude through author-written scaffolds and emulated tools, to [174]. The choice is between one large residual and many a real operating system (OS-Harm, RedTeamCUA) and small correlated ones. the shipped agent platform (SafeClawArena, A3S-Bench, AgentCanary); SafeClawArena’s authors argue that scaffolds understate deployed exposure [178], and only Safe9. Measuring the Residual: Evaluation ClawArena scores the shipped platform deterministically. Validity And even a deterministic result holds only for the model The defenses surveyed in §§6–8 are only as credible as snapshot, platform, and context length it was run on. the measurements that validate them. Agent security is An unpinned result decays silently as the deployed model measured on actions rather than text — a file exfiltrated, changes. Platform hardening moves exposure differently a tool call executed, state modified. A headline number for different models: on the production binary it cuts one is an upper bound on the agent’s security in deployment, frontier model’s attack success from 69.7% to 21.9% while on three counts. The benchmarks cover some specific raising another’s injection rate from 58.0% to 71.0% [178], boundaries and harms, and their results hold only for the so safety is a property of the model–platform pair. Shortmodel snapshot, platform, and context they were run on. context safety does not predict long-context safety: most The reported statistic is an average over attempts and, for of sixteen models are safe on under 55% of LongSafety’s a classifier, one point on an undisclosed operating curve, 1,543 long-context cases [187], and an agent’s accumulatusually measured against static attacks. What makes it ing tool-call history makes long context the norm. Nor worse is that the agent model can distinguish evaluation does safety track capability: none of Agent-SafetyBench’s sixteen agents, built on leading models across the capafrom deployment. bility range, exceeds a 60% safety score, with the failures traced to robustness and risk-awareness defects rather 9.1 Benchmark Coverage and Reproducibility than to capability deficits [181], and MCPTox finds the Table 3 summarizes the benchmark landscape for the more capable models often more susceptible to tool poiagents this paper covers: tool-using, computer-using, cod- soning because the attack rides on instruction-following ing, and claw-style platform agents. The Boundary col- [38]. A safety score is a property of the configuration it umn shows where the field measures: every benchmark was measured on, and capability has not bought safety. exercises some of B0–B3, two reach the configuration channel B2c (MCPTox, AgentCanary), one reaches B4 9.2 Metrics and Evaluation Conditions (SafeClawArena, within a single trust domain), and none reaches B5. B5 is absent because it is a property of A defense that holds nine times in ten is not a defense the serving substrate rather than of the agent. Its mea- against an adversary who can retry. Security is a propsurement lives in the serving literature (§5.7), and an erty of the worst case across attempts, so reliability across 16
Table 3: Representative agent-security benchmarks, grouped by the harm their headline number counts. Harm scored names what the headline number counts. Scoring names how it is decided: a deterministic check inspects a tool call or an environment state mechanically, a model judge has an LLM read the trajectory, and a benchmark combining rules with a judge is labeled by the primary one. Harness names what the agent runs against. Benchmark
Boundary
Harm scored
Scoring
Harness
AgentDojo [175]
B0→B2a
hijacked action
deterministic check
author scaffold
Agent Security Bench (ASB) [176]
B0/B2a
hijacked action
deterministic check
author scaffold
InjecAgent [89]
B0→B2a
hijacked action; data stealing
deterministic check
recorded tool responses, no execution
WASP [84]
B0→B2a
hijacked action
deterministic check
web replicas
MCPTox [38]
B2c
hijacked action
deterministic check
real MCP servers
RedTeamCUA [177]
B0→B3
hijacked action; file exfiltration
deterministic check
OS VM + web replicas
SafeClawArena [178]
B0/B2/B4
hijacked action; confidentiality leakage
deterministic check
production binary
AgentDAM [179]
B2a
confidentiality leakage
model judge
author scaffold
AgentHarm [180]
B1/B2
harmful task completion
model judge
author scaffold
Agent-SafetyBench [181]
B1/B2/B3
unsafe action, aggregated
model judge
author scaffold
OS-Harm [85]
B0/B1
unsafe action, aggregated
model judge
OS VM
OpenAgentSafety [182]
B0/B1
unsafe action, aggregated
rules + model judge
author scaffold, real tools
SABER [183]
B0/B1/B3
harmful operation, aggregated
rules; judge auxiliary
project containers
A3S-Bench (ASEval) [184]
B0
risk behavior, aggregated
model judge
production binary
AgentCanary [185]
B0/B2c
unsafe action, aggregated
model judge
production binary, real tools
ToolEmu [186]
B2/B3
unsafe tool outcome
model judge
LM-emulated tools
repeated trials is the metric that matters and single-shot success a poor proxy for it. Two estimators are often conflated: Pass@K, the probability that at least one of K attempts succeeds, flatters a system by rewarding any single success; PassK , the probability that all K attempts succeed, is the security-relevant quantity, since an attacker needs only one of the agent’s many runs to fail open. The gap is a property of the metric rather than of any one model: on tau-bench [188], GPT-4o achieves a Pass8 rate below 25% where its single-attempt rate appears acceptable, and reliability frameworks for long-horizon agents find the same degradation across repeated runs [189]. A guardrail that must hold across every interaction in a session cannot be certified on a single-shot number. The single number is also the wrong number when the mediator is a classifier. If every deployable mediator of a semantic judgment is a tunable classifier (§4.1), an attacksuccess rate reported without a declared false-positive budget is an arbitrary point on an undisclosed ROC curve: the same defense can be made to show almost any catch rate by silently moving its threshold. Operating-point reporting states efficacy as a catch rate at a pinned falsepositive budget. One deployed detector reports that way:
ADR, in production at Uber, states its result on its own 302-task benchmark as 67% of attacks caught at zero false positives, and on AgentDojo as every attack caught at three false alarms in 93 tasks, vendor-reported [190]. A model judge is a further source of error. RedTeamCUA scores by execution rather than by a judge on the ground that the injection aimed at the agent can mislead the judge as well [177]; and a judge from the same model family as the subject may share its blind spots, as in A3S-Bench, where scoring oracle, attack injector, and one tested subject share a family [184], and no benchmark in Table 3 checks for this. Table 4 consolidates the quantified defense results of §§6–8 with the condition under which each was taken. Across the seven rows, residual magnitude tracks evaluation condition, not mechanism class. Sub-5% residuals appear on both sides of the grouping, and the one row measured adversarially at a pinned operating point, SLEIGHT-Bench’s fixed-1%-FPR protocol [124], is both the worst number and the only one stated at an operating point. Adaptive evaluations target the most prominent defenses, so the pattern is a hypothesis, and it makes a
17
Table 4: Residual-leakage synthesis for representative quantified defenses of §§6–8, each reported exactly as its source states it. ASR = attack success rate; FPR = false-positive rate; IPI = indirect prompt injection. Defense
Boundary
Reported residual / efficacy
Stated cost
Evaluation condition
Probabilistic mediator — the run-time decision is a classifier’s Spotlighting (§7.1) [136]
B0
ASR >50% → ~2%
input encoding
vendor self-report, static
Constitutional Classifiers [95]
B0
jailbreak 86% → 4.4%
+0.38% over-refusal
vendor self-report, red-team
AIR, layer-wise privilege signal (§7.1) [140]
B0/B1
~9.2× ASR reduction
training-time change
author-report, static
SLEIGHT-Bench monitor [124]
B2/B3
32% catch @ 1% FPR
FPR budget pinned
vendor benchmark, adversarial
ClawGuard [159]
B2a
near-zero IPI ASR, evaluated benchmarks
rule induction/maintenance
author-report, deterministic harness
Progent, per-call privilege policies (§8.1) [157]
B2a
ASR 39.9% → 1.0%
policy authoring
author-report, deterministic harness
AgentSafe, trust-tiered memory (§8.3) [170]
B4
>80% defense success
trusted tier policy
author-report, simulated mesh
Deterministic mediator — the semantic judgment was made offline
prediction: adaptive re-evaluation will push the sub-5% residuals up. Adaptive re-evaluation has already done so for StruQ and SecAlign [191] and for known-answer detection [98]; the twelve-defense adaptive bypass points the same way across mechanism classes [99]; and NIST’s agent-hijacking evaluation raised attack success from 11% to 81% against a pinned frontier assistant by iterating on its attacks [192]. No surveyed adaptive evaluation has touched ClawGuard, the per-call privilege policies, or AgentSafe. Running one would show whether their residual lies in the actions the policy permits or in the step that derives the policy, which for ClawGuard is itself a model. Continuous re-evaluation under adaptive attack, as practiced for deployed injection defenses [126], carries the adaptive standard into deployment, and public infrastructure for it now exists at competition scale: the LLMail-Inject challenge releases 208,095 adaptive attack submissions from 839 participants against an injectiondefended email assistant [193], so defense numbers can be read against an adaptive attacker population rather than a static suite. The discipline of §§9.1–9.2 condenses into a reporting checklist: model snapshot, platform, and context length pinned; efficacy stated at a declared false-positive budget; PassK reported alongside Pass@K; exfiltration rate reported alongside attack-success rate; results taken under adaptive attack, or labeled static; judge–subject family independence stated; and whether the measurement was taken on shared or isolated inference infrastructure. No surveyed evaluation practices all seven.
9.3 Evaluation Awareness The agent model under evaluation can recognize that it is being evaluated, and its behavior can depend on that recognition. The recognition has been measured. Across 61 datasets spanning chat and agentic settings, frontier models distinguish evaluation from deployment transcripts well above chance, the best at 0.83 AUC against a 0.92 human baseline, with awareness higher in agentic settings, the setting of every row in Table 3 [194]. Two vendors corroborate with unrelated methods: Anthropic’s white-box activation probes find a generally high rate of latent evaluation-awareness its model almost never verbalizes [109], and OpenAI’s GPT-5.6 audits find verbalized awareness in 16% of an external auditor’s samples, down from 43% for a prior checkpoint [102]. Whether the lower verbalized rate is less awareness or awareness less visible to the evaluator, awareness is present at a non-trivial, largely unverbalized rate across models and vendors. Although neither vendor finds a confirmed instance of deliberate sandbagging, the recognition is not inert. A model can be made to underperform strategically under evaluation [195], and Anthropic’s workspace analysis finds that the model’s internal workspace sometimes encodes recognition of being in an evaluation, and that ablating those representations can surface malicious propensities that were otherwise concealed [196]. Test-time behavior can therefore be conditioned on cues that separate evaluation from deployment, so certifying an agent that passes a benchmark as secure in deployment is out of reach with current methods.
18
10. From the De-Facto Kernel to an AI-Native OS
der intent-based control the interface is the goal itself, so underspecification at B1 is guaranteed rather than contingent. The two known ways to make intent checkable Most of what §§4–9 establish for the de-facto agent kernel are the endpoints of a range, from narrowing the action transfers to an AI-native OS. A system built to the six cri- space completely to not narrowing it at all, and both fail. teria of §2.2 still ingests untrusted content from outside A pinned plan restores determinism but surrenders the (B0), invokes tools written by third parties (B2), executes generality the delegation was for: over AgentDyn’s sixty on a shared host (B3), exchanges work with agents acting open-ended tasks and 560 injection cases, CaMeL drives for other principals (B4), and serves those principals from both attack success and utility to zero, because a plan one inference core (B5). Every crossing of §4 stays popu- fixed before the content arrives cannot accommodate a lated, the attack classes of §5 transfer, and the mediation task whose shape was not known in advance [197]. Freegap transfers with them. form delegation leaves the mediator nothing to check acWhat does not transfer is the way the de-facto form tions against. Between the endpoints, residual and utility bought its structural guarantees. Every defense in §§7–8 trade against each other: on the same benchmark, Prothat carries a guarantee rather than a statistic is a finite gent’s per-call policies (§8.1) hold attack success to 1.7% projection of an open space: a non-enumerable domain but keep only 5.8% of utility; the dynamic-policy defense narrowed by construction until a deterministic mediator DRIFT reaches 0.8% at 27.1%; and Meta SecAlign-70B covers what remains, at a price paid in whatever the nar- (§7.1), which narrows nothing, keeps 53.4% of utility at rowing put outside (Figure 5). CaMeL projects the action 9.0% attack success. The two stated designs sit at the space onto a finite graph and pays in expressiveness (§7.1); endpoints rather than between them: AgentOS fronts its Private Cloud Compute projects the state space to zero kernel with a single natural-language port that translates and pays in memory (§7.2); RobustRAG projects the in- the goal directly into orchestration [15], and AgenticOS fluence of any single passage onto a certified bound and has agents declare intent in a structured manifest that pays in influence breadth (§7.2); the reversibility-tiered a deterministic layer checks [17]. A more capable model gate projects the consequence space onto an enumerated does not remove the trade-off: on the evidence of §9.1, irreversible set and pays in autonomy (§8.2). A projec- capability has not bought safety, and the more capable tion is a semantic judgment made once, offline, so that models are the more susceptible to tool poisoning. A systhe run-time check need not make it, and the guarantee tem offering intent-based control must therefore supply reaches exactly as far as that offline judgment was sound. what neither endpoint does: a specification of authorized Three of the six criteria are commitments to keep one of intent, precise enough to admit a deterministic check over these spaces open, intent-based control the action space, the actions it induces and loose enough to keep the discreautonomous multi-step execution the consequence space, tion the agent was delegated, which makes part of the B1 and persistent cross-session context the state space, so the problem a matter of language design rather than classifier AI-native OS forgoes by definition three of the four projec- accuracy. tions that bought the body’s structural guarantees; only Autonomous multi-step execution delegates a plan whose the retrieval-influence bound transfers unchanged. Be- steps the agent selects as it goes. In the de-facto form yond that, each criterion does one of two things to the me- the deterministic guarantee for irreversible actions is the diator at its crossing. Intent-based control, autonomous human gate of §8.2: the agent runs unsupervised inside execution, lifecycle management, and multiplexing leave the reversible envelope and escalates at the irreversible the mediator beneath the model but widen what it must frontier, and twenty-one production deployments run that check, so each requires a new deterministic check. Intelli- way. The criterion does not abolish the gate. What it regence at the architectural core and persistent context put moves is the gate’s precondition, a statically enumerable the model in charge of a decision, resource arbitration and set of irreversible operations. The gate fires on operacontext paging, so each requires that enforcement of the tions, but by the composition rule R1 of §8.2 reversibility decision stay in a mechanism beneath the model. Table is a property of a sequence, and a sequence of individu5 states, for each criterion, the invariant at risk and the ally reversible operations can be irreversible as a whole. constraint that follows. When the delegated unit is the plan, what must be classified is the sequence, and no list of irreversible operations captures it. Three responses exist and each fails. RaisIntent-based control hands the agent a goal in natural ing the gate to plan granularity keeps the rule that no language and leaves the choice of operations to it. In the enumerated operation executes without assent, but empde-facto form the agent’s privileged interface is a set of ties the assent, because the operator approves a descriptools, and whether the delegated intent is underspecified tion of steps the agent has not yet selected. Classifyin the sense of §4.1 depends on how the task was dele- ing each step at run time forfeits determinism. Keepgated: an operator can pin a plan or approve a finite set ing plans short or domain-confined, so that the set of seof actions, and then every action is checked exactly. Un- quences stays enumerable, is the projection along the con10.1 Intent, Autonomy, and Lifecycle
19
action space
state space
retrieval influence
consequence space
open naturallanguage actions (nonenumerable)
unbounded persistent state
unbounded passage influence on generation
open-ended action consequences
project
project
project
project
finite action graph (CaMeL)
zero retained state (Apple PCC)
isolate-then-aggregate, certified bound (RobustRAG)
enumerated irreversible set + human gate
cost: expressiveness
cost: memory
cost: influence breadth
cost: autonomy
Figure 5: The projection principle: a structural guarantee narrows an open space and pays in what it excludes.
Table 5: Design constraints for a security-first AI-native OS. Criterion (§2.2)
Change to the mediator
Invariant the de-facto form holds
Boundary
(1) Intelligence at the architectural core
passes to the model
Resource arbitration B3; availability (§5.8) and state integrity are enforced by a deterministic mechanism beneath the model
(2) Intent-based control
remit widens
The privileged interface is an enumerable operation set, so underspecified intent is contingent on how a task was delegated
B1
Supply a specification language for authorized intent that admits a deterministic check over the induced action set
(3) Persistent, cross-session context
passes to the model
Eviction is a data operation, not an authority-bearing one
B0 via persistent state
Poisoning deployed Gate retain and evict on provenance; (§5.2); no prototype at the retention content must not layer decide its own retention
(4) Autonomous multi-step execution
remit widens
Irreversible operations are unreachable without human assent, over a statically enumerable set
B1; B2a
Make reversibility Gate deployed machine-decidable (§8.2); approval over sequences surface attacked before raising the (EchoLeak); calculus gate above the prospective operation; keep the confirmation surface unshapeable by the agent
(5) OS-level lifecycle management
remit widens
Registration and B2c; B4 delegation are the ecosystem’s concern, not the kernel’s
Own registration and delegation as kernel obligations
Deployed (§5.4 registration; §7.3 delegation)
(6) Mutuallydistrusting multiplexing
remit widens; passes to the model with (1)
Principals sharing an inference core are held apart by the substrate
Partition on provenance, by a mechanism outside the model
Serving-layer attacks published (§5.7); prospective at the agent kernel
B5
20
Design constraint
Evidence
Keep enforcement in Both forms stated (§2.2); every built a mechanism the design enforces model may propose beneath the model to but not rewrite (AIOS, SchedCP); no attacked instance §4.1 argument; two stated designs (AgentOS, AgenticOS); no attacked instance
sequence axis, paid in the autonomy the criterion exists to supply. A system offering autonomous multi-step execution therefore preserves the guarantee of §8.2 only insofar as reversibility is machine-decidable over sequences. Such a calculus becomes a precondition of the AI-native OS, and GoEX’s undo and damage-confinement runtime supplies a substrate to build on [163]. Composed with intent-based control the difficulty compounds: enumerating the irreversible set needs a fixed unit of action and a bounded set of induced actions, and autonomous execution removes the first as intent-based control removes the second. Even where the gate is kept, its approval surface is itself a B2a crossing the agent’s output can shape (§8.2). Under autonomous execution that surface is consulted less often and each consultation carries more, so what the operator sees when asked to confirm must reach them over a trusted path the agent cannot write. OS-level lifecycle management hands permissions, identity, registration, sandboxing, and delegation for agents to the system itself. In the de-facto form those are the ecosystem’s concern: tools and skills are registered wherever they are published, and a delegation between agents carries only whatever labels the sending side chooses to attach. The damage manifests at two crossings. At B2c a legitimately registered but malicious artifact defeats provenance by construction (§5.4), and at B4 a label that does not survive a handoff reintroduces the confused deputy (§7.3). Registration and delegation are where authority is conferred, so a system that manages agent lifecycles must own B2c and B4 as kernel obligations rather than inherit whatever the ecosystem provides. 10.2 Arbitration and Context Intelligence at the architectural core and persistent crosssession context put the model in charge of a decision: resource arbitration, made today by a scheduler that counts consumption, and context paging, made today by the serving layer on token identity, are under the two criteria made by the model. Moving arbitration into the model changes what an attacker can reach. The resource attacks of §5.8 already steer an agent’s consumption through injected content, and what caps them today is a counter the content cannot touch. A scheduler that reads the same content in the same inference pass can be steered by it as well, so availability, which the de-facto kernel guarantees deterministically, comes to depend on the same B0 judgment that the mediation gap already leaves to a classifier. Classical mechanism/policy separation shows the loss need not be taken whole: an untrusted party may choose a policy at run time while a deterministic mechanism, fixed in advance, bounds what any policy can do [19], [20], so a model that owns scheduling policy under such a mechanism widens no gap, and the guarantee survives exactly as far as enforcement is retained beneath the model. Under the first form of criterion (1), the model as the entire kernel with nothing beneath it, nothing is re-
tained: each guarantee a counter supplies today becomes a judgment the model makes on content it did not author, the mediation gap widened to cover availability. Under the second form the guarantee survives, and the first form buys nothing the second does not, since the semantics a counter cannot see enter through the proposed policy either way. The second form is also the one every built design takes. AIOS schedules the model as a resource in conventional kernel code and reaches hardware through the host operating system’s system calls [13]; SchedCP has the model synthesize a scheduling policy that a verifier checks before the Linux kernel loads and enforces it [198]; the model-native computing architecture makes the split its design, a probabilistic execution plane that proposes and a deterministic control plane that bounds it [199]; the platform vendors keep enforcement below the agent (§5.8); the first form remains a stated position [16]. A security-first AI-native OS should therefore take the second form: enforcement stays in a mechanism the model may propose to but not rewrite, wherever the line between proposed policy and retained mechanism is drawn, and no surveyed design yet states a criterion for where it falls. Persistent cross-session context puts the model in charge of the second decision: which spans of context to retain in the window and which to evict. The criterion rests on the virtual-memory analogy. MemGPT draws it explicitly, moving context between fast and slow memory like a hierarchical memory system and having the model itself issue the paging calls [14], and the constraint lies where the analogy breaks. A classical pager is transparent because it blocks: the faulting computation waits until the page returns, so eviction never changes what it computes, and AIOS keeps that discipline where it suspends a decode and restores its snapshot [13]. When an agent’s context manager evicts a span, the model keeps generating rather than waiting. Restoring the span later brings back its text, but what the model generated in the meantime was computed without it and stays in the context. AIOS replaces an evicted span with a summary, and the summary cannot be turned back into the span [13]. Eviction therefore changes what the model computes, and what a later check can see. Poisoned content can target the checks rather than the task, steering the pager to evict exactly the spans a monitor would have read: the confused deputy, relocated into the pager. MemLineage gates actions on derivation provenance, not retention (§7.2), so no surveyed design mediates at this layer. Retain and evict decisions over persistent context must therefore be gated on provenance: decided by the standing of the content they act on, and never writable by that content. 10.3 The Shared Inference Core Mutually-distrusting multiplexing on its own leaves the mediator beneath the model: the partition sits today in the serving substrate beneath the agent (§4.2), its chan-
21
nels are already deployed and attacked (§5.7), and the criterion makes the agent kernel rather than the substrate accountable for it. Prefix-aware scheduling serves a request earlier when it shares more cached state, so the order of service is a disclosure channel [22]. It is the only scheduling decision in the surveyed evidence that leaks across a trust boundary, and it shows what a model-owned scheduler would take over: a policy chosen for throughput became a channel without anyone deciding it should. Composed with a model-owned arbitration core, the criterion changes who decides reuse. Today a serving framework matches prefixes and evicts blocks by a fixed policy over token identity and recency, a computation on the request rather than on its meaning; a model-owned core would choose what to cache, what to share, and whose entry to evict in the same inference pass that reads the requests, on grounds available only in their semantic content. The mediation question changes accordingly. Under the substrate policy it is whether the partition is correctly computed, which is checkable; under a model-owned core it is whether a probabilistic component’s sharing decisions can be induced by one tenant’s content to expose another’s, which is the B0 judgment moved into the cache manager. Neither criterion creates this alone: multiplexing alone leaves isolation to the substrate, and a model-owned core alone serves one principal with nothing to separate. Composed, the isolator is the arbiter: the component that holds tenants apart is a model that reads what each tenant writes and can be steered, so the monitor fails the tamper-proof requirement of Anderson’s reference monitor [29].
anisms enforce, and a proposal takes effect only when a principal outside the model activates it. Four of the six constraints call for a provenance mechanism: the arbitration line, retention gated on provenance, registration and delegation as kernel obligations, and the partition executed outside the model. Two call for a projection: the authorized-intent specification and reversibility decided over sequences. None calls for a monitor. A securityfirst AI-native OS therefore keeps its guarantees where the de-facto kernel keeps them, in deterministic mediators beneath the model, and differs from it in what those mediators are: not a general-purpose operating system but the enumerated set of mechanisms and projections the constraints name, with the core deciding and the floor enforcing what it decides.
The constraints can be checked against existing designs. AIOS schedules the model as a resource from kernel code beneath it and conforms trivially; SchedCP passes the model’s policy through a verifier and the kernel, and conforms; SafeKV keeps the partition deterministic and makes only the sensitivity decision semantic, and conforms; a pager that evicts on the model’s judgment and continues generating violates them. What the constraints do not supply is a guarantee at every crossing. At B1 no provenance mechanism can exist (§4.1), and the pinned plan that closed the crossing by projection is forgone under intent-based control, so until an authorized-intent specification exists a monitor is the only barrier there, with a measured miss rate. B2b, the response path, has no mediator of any kind in the surveyed evidence. Table 6 lists these and the other open problems, with the existing The way out is to keep the partition itself deterministic work on each and what would count as solving it. and let semantics decide only what is partitioned, and SafeKV shows it can be built at the serving layer: every 11. Conclusion block is private on insertion and becomes shareable only once a three-tier classifier of rules, a small model, and The LLM agent has been promoted to a system-level prinan LLM validator judges it non-sensitive, with a runtime cipal wielding kernel-grade authority over resources and safeguard that bounds the leakage a misclassification can actions, and no mechanism known today mediates that cause [92]. No deployed system yet composes a model- authority as security doctrine has demanded of privileged owned core with multiplexing, so at the agent kernel the software for half a century. Complete mediation strains same discipline is prospective. A shared core must there- here not because the principal is stochastic, since classifore be partitioned on provenance rather than on a judg- cal monitors have always confined non-deterministic prinment about content, by a mechanism the model does not cipals, but because at the two crossings where untrusted input becomes instruction and where reasoning commits itself execute. to an action, any always-invoked check must itself judge natural-language meaning, and so leaves an irreducible 10.4 Open Problems and Research Agenda residual of undetected attacks. The design constraints for a security-first AI-native OS derived in §§10.1–10.3 share one shape. The semantic This paper maps that gap. A trust-boundary taxonomy judgment a crossing needs can be made by a mediator locates every crossing where mediation is needed, and the in one of three ways: a monitor makes it at run time, attacks reported at each show which are exploited in pracprobabilistically, with a measured miss rate; a projection tice. Defenses come in two kinds. Monitors judge meanmakes it once, offline, so that the run-time check is de- ing at run time and miss a measurable share of attacks. terministic; a provenance mechanism beneath the model Structural confinement narrows what the agent can do never makes it, and decides on where content came from until a deterministic check suffices, at a cost in expresinstead. Mechanism/policy separation orders the three: siveness, autonomy, memory, or influence. Current evalthe model and its monitors propose, projections and mech- uations often overstate what deployed defenses achieve. 22
Table 6: Open problems for a security-first AI-native OS. #
Open problem
Boundary
Origin
Existing work
Success criterion
1
Authorized-intent specification
B1
intent-based control
AgentOS at the free-form end, AgenticOS at the pinned end [15], [17]; tool-call policy languages (Progent, §8.1) at the pinned end
A language precise enough for a deterministic check over the induced action set, loose enough to keep the delegated discretion
2
Reversibility decidable over sequences
B2a
autonomous execution; compounded by intent-based control
R1–R3 stated (§8.2); GoEX undo runtime [163]
A type system over action sequences and flow labels matching practitioner judgment on a published trajectory corpus, disagreement rate reported
3
Confirmation surface the agent cannot shape
B2a
autonomous execution
The attack is deployed (EchoLeak, §8.2)
A trusted path whose contents the confirmed agent cannot write
4
Which arbitration decisions may enter the model
B3; availability
intelligence at the architectural core
Every built design enforces beneath the model (AIOS, SchedCP; §10.2)
A stated criterion for the line, or a model-owned scheduler provably bounding starvation, amplification, and eviction integrity with no mechanism beneath
5
Provenance-gated retention; pinned spans
B0, persistent
persistent context
MemLineage gates actions, not retention (§7.2)
Retain and evict decisions attributable to content standing and unwritable by content; an account of who may pin a span
6
Cross-tenant write channel
B5, persistent
persistent context with multiplexing
Every channel of §5.7 is a read; cache injection is shown against the cache, not another tenant’s context
A demonstrated or bounded write from one tenant into another’s retained context
7
Registration and delegation as kernel obligations
B2c; B4
lifecycle management
Ecosystem registries (§5.4); single-domain identity primitives (§8.3); classical trust management [200], [201]; cross-organizational accountability named open [162]
A kernel-held registry, and a delegation credential whose labels survive a handoff across frameworks and organizations (§7.3)
8
Action-channel integrity
B2b
carries over unchanged
None surveyed; a signature proves authorship, not intent (§5.4)
Invocations and responses signed and attested to the tool instance that produced them
9
Privileged-writer / quarantined-reader contract across delegation
B1; B4
autonomous execution; lifecycle management owns B4
Dual-LLM split [127], evaluated by CaMeL [129] and generalized as a pattern [128]; FIDES, IsolateGPT, and AgentSafe each realize a part [134], [152], [170]
A writer that alters state but consumes only provenance-clean input and a reader that consumes any content but cannot escalate, demonstrated end-to-end across delegation with the monotone labels of §7.3 at a declared operating point
10
Kernel-held audit chain
crosscutting
lifecycle management; rows 2, 3, and 7 presuppose a record the agent cannot rewrite
Reconstructed from application caches today; ADR proposes a context-and-intent field on every tool call [190]
The intent, reasoning, and action chain held tamper-proof by the kernel and bound to every invocation
11
Per-boundary miss-rate budgets
B0; B1
carries over unchanged
Operating-point reporting (§9.2); LiSA’s declared posterior error budget [146]
A harness rejecting any deployment whose miss rate at a boundary exceeds its budget under adaptive attack, on two independent stacks
12
Projection price against capability
B1; B2a
intent-based control
AgentDyn’s cost ordering (§10.1); A cost-of-projection curve measured capability has not bought safety across capability tiers, showing (§9.1) whether by-construction security becomes affordable for a general-purpose agent
23
What this systematization offers is a coordinate system, [10] a measured account of where each mediator fails, and a line between what the evidence establishes and what is still open. The AI-native operating system that vendors are moving [11] toward is attractive because it asks less of the user: work simply happens, with fewer requests to make and fewer actions to confirm. It keeps open the very spaces those defenses narrow: it takes goals rather than commands, executes plans rather than steps, and remembers across [12] sessions. Three of the four structural guarantees therefore do not transfer, and the semantic judgment that today sits at two crossings spreads to scheduling, paging, and [13] the sharing of one model among tenants. For each of the six defining properties we derive the check that would have to replace what is lost, and none exists yet. These [14] checks, together with the further problems the evidence leaves open, are the research agenda this paper sets out. [15]
References [1]
[2]
[3]
[4]
[5]
[6]
[7]
[8]
[9]
OpenAI, “OpenAI – Hugging Face Incident Technical Report,” Aug. 2026. Available: https: //cdn.openai.com/pdf/67869394-cb91-4c12-888c5cbd85c7814c/OpenAI-Hugging-Face%20Inciden t-Technical-Report.pdf Anthropic, “Investigating three real-world incidents in our cybersecurity evaluations,” Anthropic. [Online]. Available: https://www.anthropic.com/ news/investigating-incidents-cybersecurity-evals D. Huang and L. Iyer, “Windows platform security for AI agents,” Windows Developer Blog, Microsoft. [Online]. Available: https://blogs.window s.com/windowsdeveloper/2026/06/02/windowsplatform-security-for-ai-agents/ Android Developers, “Overview of AppFunctions,” Android Developers, Google. [Online]. Available: https://developer.android.com/ai/appfunctions Apple, “App Intents.” [Online]. Available: https: //developer.apple.com/documentation/appinten ts L. Pirch et al., “Toward Securing AI Agents Like Operating Systems,” arXiv preprint arXiv:2605.14932, 2026, Available: https://arxiv.org/abs/2605.14932 S. Abdelnabi and E. Bagdasarian, “AI Agents May Always Fall for Prompt Injections,” arXiv preprint arXiv:2605.17634, 2026, Available: https://arxiv. org/abs/2605.17634 Y. Li, “Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security,” arXiv preprint arXiv:2401.05459, 2024, Available: https://arxiv.org/abs/2401.05459 “Tools, skills, and plugins,” OpenClaw Docs. [Online]. Available: https://docs.openclaw.ai/tools
[16]
[17]
[18]
[19]
[20]
[21]
[22]
24
“Tools & Toolsets,” Hermes Agent Documentation, Nous Research. [Online]. Available: https://herm es-agent.nousresearch.com/docs/user-guide/feat ures/tools X. Wang et al., “The OpenHands Software Agent SDK: A Composable and Extensible Foundation for Production Agents,” arXiv preprint arXiv:2511.03690, 2025, Available: https://arxi v.org/abs/2511.03690 “How Claude remembers your project,” Claude Code Documentation, Anthropic. [Online]. Available: https://code.claude.com/docs/en/memory K. Mei et al., “AIOS: LLM Agent Operating System,” arXiv preprint arXiv:2403.16971, 2024, Available: https://arxiv.org/abs/2403.16971 C. Packer et al., “MemGPT: Towards LLMs as Operating Systems,” arXiv preprint arXiv:2310.08560, 2023, Available: https://arxiv.org/abs/2310.08560 R. Liu et al., “AgentOS: From Application Silos to a Natural Language-Driven Data Ecosystem,” arXiv preprint arXiv:2603.08938, 2026, Available: https://arxiv.org/abs/2603.08938 Y. Ge, Y. Ren, W. Hua, S. Xu, J. Tan, and Y. Zhang, “LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem,” arXiv preprint arXiv:2312.03815, 2023, Available: https://arxiv.org/abs/2312.03815 Z. Zhao et al., “AgenticOS: An IntentOriented Secure Operating System Architecture for Autonomous AI Agents,” arXiv preprint arXiv:2606.21129, 2026, Available: https://arxiv.org/abs/2606.21129 J. H. Saltzer and M. D. Schroeder, “The Protection of Information in Computer Systems,” Proceedings of the IEEE, 1975, doi: 10.1109/PROC.1975.9939. P. Barham et al., “Xen and the Art of Virtualization,” in ACM SOSP, 2003. doi: 10.1145/945445.945462. D. R. Engler, M. F. Kaashoek, and J. O’Toole, “Exokernel: An Operating System Architecture for Application-Level Resource Management,” in ACM SOSP, 1995. doi: 10.1145/224056.224076. X. Zheng et al., “InputSnatch: Stealing Input in LLM Services via Timing Side-Channel Attacks,” arXiv preprint arXiv:2411.18191, 2024, Available: https://arxiv.org/abs/2411.18191 G. Wu et al., “I Know What You Asked: Prompt Leakage via KV-Cache Sharing in Multi-Tenant LLM Serving,” in Network and Distributed System Security Symposium (NDSS), 2025. doi: 10.14722/ndss.2025.241772.
[23]
[24]
[25]
[26]
[27]
[28]
[29] [30]
[31]
[32]
[33]
[34]
[35]
Microsoft, “What is Microsoft Entra Agent ID?” Microsoft Learn. [Online]. Available: https://le arn.microsoft.com/en-us/entra/agent-id/what-ismicrosoft-entra-agent-id Huawei, “HarmonyOS 7 Developer Beta launches; the all-scenario intelligent operating system upgraded [in Chinese],” Huawei Newsroom. [Online]. Available: https://www.huawei.com/cn/news/2 026/6/harmonyos7-hdc J. Kim et al., “The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey,” in USENIX Security Symposium, 2026. Available: https://arxiv.org/abs/2603.11088 Y. Ling, S. Yu, Z. Chen, and C. Fang, “Toward Secure LLM Agents: Threat Surfaces, Attacks, Defenses, and Evaluation,” arXiv preprint arXiv:2606.10749, 2026, Available: https://arxiv. org/abs/2606.10749 K. Chu, “A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework,” arXiv preprint arXiv:2604.23338, 2026, Available: https://arxiv. org/abs/2604.23338 A. Dehghantanha and S. Homayoun, “SoK: The Attack Surface of Agentic AI -- Tools, and Autonomy,” arXiv preprint arXiv:2603.22928, 2026, Available: https://arxiv.org/abs/2603.22928 J. P. Anderson, “Computer Security Technology Planning Study,” 1972. F. B. Schneider, “Enforceable Security Policies,” ACM Trans. Inf. Syst. Secur., 2000, doi: 10.1145/353323.353382. E. Zverev, S. Abdelnabi, S. Tabesh, M. Fritz, and C. H. Lampert, “Can LLMs Separate Instructions From Data? And What Do We Even Mean By That?” in International Conference on Learning Representations (ICLR), 2025. Available: https: //arxiv.org/abs/2403.06833 S. Stamm, B. Sterne, and G. Markham, “Reining in the Web with Content Security Policy,” in WWW, 2010. doi: 10.1145/1772690.1772784. N. Hardy, “The Confused Deputy,” ACM SIGOPS Operating Systems Review, 1988, doi: 10.1145/54289.871709. M. Luo et al., “When Alignment Isn’t Enough: Response-Path Attacks on LLM Agents,” arXiv preprint arXiv:2605.02187, 2026, Available: https: //arxiv.org/abs/2605.02187 H. Liu, C. Shou, H. Wen, Y. Chen, R. J. Fang, and Y. Feng, “Your Agent Is Mine: Measuring Malicious Intermediary Attacks on the LLM Supply Chain,” arXiv preprint arXiv:2604.08407, 2026, Available: https://arxiv.org/abs/2604.08407
[36]
[37]
[38]
[39]
[40]
[41]
[42]
[43]
[44]
[45]
[46]
[47]
25
X. Hou, Y. Zhao, S. Wang, and H. Wang, “Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions,” arXiv preprint arXiv:2503.23278, 2025, Available: https: //arxiv.org/abs/2503.23278 H. Song et al., “Beyond the Protocol: Unveiling Attack Vectors in the Model Context Protocol (MCP) Ecosystem,” arXiv preprint arXiv:2506.02040, 2025, Available: https://arxiv.org/abs/2506.0 2040 Z. Wang et al., “MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers,” arXiv preprint arXiv:2508.14925, 2025, Available: https://arxiv.org/abs/2508.14925 A. C. Myers and B. Liskov, “A Decentralized Model for Information Flow Control,” in ACM SOSP, 1997. doi: 10.1145/268998.266669. Noma Labs, “GeminiJack: Zero-Click Indirect Prompt Injection in Google Gemini Enterprise,” Noma Security. [Online]. Available: https://no ma.security/noma-labs/geminijack/ K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising Real-World LLMIntegrated Applications with Indirect Prompt Injection,” arXiv preprint arXiv:2302.12173, 2023, Available: https://arxiv.org/abs/2302.12173 Y. Liu et al., “Prompt Injection attack against LLM-integrated Applications,” arXiv preprint arXiv:2306.05499, 2023, Available: https://arxi v.org/abs/2306.05499 M. Dziemian et al., “How Vulnerable Are AI Agents to Indirect Prompt Injections? Insights from a Large-Scale Public Competition,” arXiv preprint arXiv:2603.15714, 2026, Available: https: //arxiv.org/abs/2603.15714 P. Reddy and A. S. Gujral, “EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System,” arXiv preprint arXiv:2509.10540, 2025, Available: https://arxi v.org/abs/2509.10540 X. Wang, J. Bloch, Z. Shao, Y. Hu, S. Zhou, and N. Z. Gong, “WebInject: Prompt Injection Attack to Web Agents,” arXiv preprint arXiv:2505.11717, 2025, Available: https://arxiv.org/abs/2505.117 17 S. Johnson, V. Pham, and T. Le, “Manipulating LLM Web Agents with Indirect Prompt Injection Attack via HTML Accessibility Tree,” arXiv preprint arXiv:2507.14799, 2025, Available: https: //arxiv.org/abs/2507.14799 Z. Liao et al., “EIA: Environmental Injection Attack on Generalist Web Agents for Privacy Leakage,” arXiv preprint arXiv:2409.11295, 2024, Available: https://arxiv.org/abs/2409.11295
[48]
[49]
[50]
[51]
[52]
[53]
[54]
[55]
[56]
[57]
[58]
T. Cao et al., “VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents,” arXiv preprint arXiv:2506.02456, 2025, Available: https: //arxiv.org/abs/2506.02456 E. Bagdasaryan, T.-Y. Hsieh, B. Nassi, and V. Shmatikov, “Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs,” arXiv preprint arXiv:2307.10490, 2023, Available: https://arxiv.org/abs/2307.10490 Z. Zhong, Z. Huang, A. Wettig, and D. Chen, “Poisoning Retrieval Corpora by Injecting Adversarial Passages,” arXiv preprint arXiv:2310.19156, 2023, Available: https://arxiv.org/abs/2310.19156 W. Zou, R. Geng, B. Wang, and J. Jia, “PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Models,” arXiv preprint arXiv:2402.07867, 2024, Available: https://arxiv.org/abs/2402.078 67 B. Zhang et al., “Practical Poisoning Attacks against Retrieval-Augmented Generation,” arXiv preprint arXiv:2504.03957, 2025, Available: https: //arxiv.org/abs/2504.03957 Z. Chen, Z. Xiang, C. Xiao, D. Song, and B. Li, “AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases,” arXiv preprint arXiv:2407.12784, 2024, Available: https: //arxiv.org/abs/2407.12784 S. Dong et al., “Memory Injection Attacks on LLM Agents via Query-Only Interaction,” arXiv preprint arXiv:2503.03704, 2025, Available: https: //arxiv.org/abs/2503.03704 S. S. Srivastava and H. He, “MemoryGraft: Persistent Compromise of LLM Agents via Poisoned Experience Retrieval,” arXiv preprint arXiv:2512.16962, 2025, Available: https://arxi v.org/abs/2512.16962 Z. Xu, X. Zhu, Y. Yao, M. Xue, and Y. Song, “From Storage to Steering: Memory Control Flow Attacks on LLM Agents,” arXiv preprint arXiv:2603.15125, 2026, Available: https://arxi v.org/abs/2603.15125 J. Xue, M. Zheng, Y. Hu, F. Liu, X. Chen, and Q. Lou, “BadRAG: Identifying Vulnerabilities in Retrieval Augmented Generation of Large Language Models,” arXiv preprint arXiv:2406.00083, 2024, Available: https://arxiv.org/abs/2406.00083 H. Wang et al., “Joint-GCG: Unified GradientBased Poisoning Attacks on RetrievalAugmented Generation Systems,” arXiv preprint arXiv:2506.06151, 2025, Available: https://arxiv.org/abs/2506.06151
[59]
[60]
[61]
[62]
[63]
[64]
[65]
[66]
[67]
[68]
[69]
26
J. Shi, T. J. Zhang, Z. Jin, and V. Conitzer, “From Sycophancy to Deception: A Unified Taxonomy for LLM Spontaneous Misalignment,” arXiv preprint arXiv:2604.04788, 2026, Available: https: //arxiv.org/abs/2604.04788 D. Guo et al., “Are Your Agents Upward Deceivers?” arXiv preprint arXiv:2512.04864, 2025, Available: https://arxiv.org/abs/2512.04864 A. Meinke, B. Schoen, J. Scheurer, M. Balesni, R. Shah, and M. Hobbhahn, “Frontier Models are Capable of In-context Scheming,” arXiv preprint arXiv:2412.04984, 2024, Available: https://arxiv. org/abs/2412.04984 R. Arike, E. Donoway, H. Bartsch, and M. Hobbhahn, “Technical Report: Evaluating Goal Drift in Language Model Agents,” arXiv preprint arXiv:2505.02709, 2025, Available: https://arxiv. org/abs/2505.02709 F. Liu et al., “Make Agent Defeat Agent: Automatic Detection of Taint-Style Vulnerabilities in LLM-based Agents,” in USENIX Security, 2025, pp. 3767–3786. Available: https://www.usenix.o rg/conference/usenixsecurity25/presentation/liufengyu M. Lupinacci, F. A. Pironti, F. Blefari, F. Romeo, L. Arena, and A. Furfaro, “The Dark Side of LLMs: Agent-based Attack Vectors for System-level Compromise,” arXiv preprint arXiv:2507.06850, 2025, Available: https://arxiv.org/abs/2507.06850 P. He, Y. Lin, S. Dong, H. Xu, Y. Xing, and H. Liu, “Red-Teaming LLM Multi-Agent Systems via Communication Attacks,” arXiv preprint arXiv:2502.14847, 2025, Available: https://arxiv. org/abs/2502.14847 J. Shi, Z. Yuan, G. Tie, P. Zhou, N. Z. Gong, and L. Sun, “Prompt Injection Attack to Tool Selection in LLM Agents,” arXiv preprint arXiv:2504.19793, 2025, Available: https://arxiv.org/abs/2504.197 93 Y. Qu et al., “Supply-Chain Poisoning Attacks Against LLM Coding Agent Skill Ecosystems,” arXiv preprint arXiv:2604.03081, 2026, Available: https://arxiv.org/abs/2604.03081 O. Yomtov, “ClawHavoc: 341 Malicious ClawedBot Skills Found by the Bot They Were Targeting,” Koi Security. [Online]. Available: https://www. koi.ai/blog/clawhavoc-341-malicious-clawedbotskills-found-by-the-bot-they-were-targeting R. Marchand et al., “Quantifying Frontier LLM Capabilities for Container Sandbox Escape,” arXiv preprint arXiv:2603.02277, 2026, Available: https://arxiv.org/abs/2603.02277
[70]
[71]
[72]
[73]
[74]
[75]
[76]
[77]
[78]
[79]
[80]
[81]
Cyera Research, “Claw Chain: Cyera Research Unveil Four Chainable Vulnerabilities in OpenClaw,” Cyera Research. [Online]. Available: https://ww w.cyera.com/blog/claw- chain- cyera- researchunveil-four-chainable-vulnerabilities-in-openclaw S. Cohen, R. Bitton, and B. Nassi, “Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applications,” arXiv preprint arXiv:2403.02817, 2024, Available: https: //arxiv.org/abs/2403.02817 D. Lee and M. Tiwari, “Prompt Infection: LLMto-LLM Prompt Injection within Multi-Agent Systems,” arXiv preprint arXiv:2410.07283, 2024, Available: https://arxiv.org/abs/2410.07283 X. Yang, Y. He, S. Ji, B. Hooi, and J. S. Dong, “Zombie Agents: Persistent Control of Self-Evolving LLM Agents via Self-Reinforcing Injections,” arXiv preprint arXiv:2602.15654, 2026, Available: https://arxiv.org/abs/2602.15654 T. Ju et al., “Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communities,” arXiv preprint arXiv:2407.07791, 2024, Available: https://arxiv.org/abs/2407.07791 J. Perez et al., “When LLMs Play the Telephone Game: Cultural Attractors as Conceptual Tools to Evaluate LLMs in Multi-turn Settings,” arXiv preprint arXiv:2407.04503, 2024, Available: https: //arxiv.org/abs/2407.04503 M. Cemri et al., “Why Do Multi-Agent LLM Systems Fail?” arXiv preprint arXiv:2503.13657, 2025, Available: https://arxiv.org/abs/2503.136 57 A. Menon, M. Saebo, T. Crosse, S. Gibson, E. Jang, and D. Cruz, “Inherited Goal Drift: Contextual Pressure Can Undermine Agentic Goals,” arXiv preprint arXiv:2603.03258, 2026, Available: https://arxiv.org/abs/2603.03258 Z. Luo et al., “Shadow in the Cache: Unveiling and Mitigating Privacy Risks of KV-cache in LLM Inference,” in Network and Distributed System Security Symposium (NDSS), 2026. doi: 10.14722/ndss.2026.240258. H. Sun, S. Liu, S. Ma, J. Li, M. Xiao, and W. Jiang, “Agent-Assisted Side-Channel Attacks on Non-Prefix KV Cache in RAG,” arXiv preprint arXiv:2606.21842, 2026, Available: https://arxi v.org/abs/2606.21842 A. Kumar et al., “OverThink: Slowdown Attacks on Reasoning LLMs,” arXiv preprint arXiv:2502.02542, 2025, Available: https://arxi v.org/abs/2502.02542 K. Zhou et al., “Beyond Max Tokens: Stealthy Resource Amplification via Tool Calling Chains in LLM Agents,” arXiv preprint arXiv:2601.10955, 2026, Available: https://arxiv.org/abs/2601.1 0955
[82]
[83]
[84]
[85]
[86]
[87]
[88]
[89]
[90]
[91]
[92]
[93]
27
B. Hui, H. Yuan, N. Gong, P. Burlina, and Y. Cao, “PLeak: Prompt Leaking Attacks against Large Language Model Applications,” in ACM SIGSAC Conference on Computer and Communications Security (CCS), 2024. Available: https: //arxiv.org/abs/2405.06823 Google, “AI threats in the wild: The current state of prompt injections on the web,” Google Online Security Blog. [Online]. Available: https://secu rity.googleblog.com/2026/04/ai-threats-in-wildcurrent-state-of.html I. Evtimov, A. Zharmagambetov, A. Grattafiori, C. Guo, and K. Chaudhuri, “WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks,” arXiv preprint arXiv:2504.18575, 2025, Available: https://arxiv.org/abs/2504.18575 T. Kuntz et al., “OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents,” arXiv preprint arXiv:2506.14866, 2025, Available: https: //arxiv.org/abs/2506.14866 “Claude’s memory works everywhere, and you decide what’s in it,” Anthropic. [Online]. Available: https://claude.com/blog/claudes-memory-workseverywhere-and-you-decide-whats-in-it Q. Long, Y. Deng, L. Gan, W. Wang, and S. J. Pan, “Backdoor Attacks on Dense Retrieval via Public and Unintentional Triggers,” arXiv preprint arXiv:2402.13532, 2024, Available: https://arxiv. org/abs/2402.13532 E. Hubinger et al., “Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training,” arXiv preprint arXiv:2401.05566, 2024, Available: https://arxiv.org/abs/2401.05566 Q. Zhan, Z. Liang, Z. Ying, and D. Kang, “InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents,” arXiv preprint arXiv:2403.02691, 2024, Available: https://arxiv.org/abs/2403.02691 NIST National Vulnerability Database, “CVE2026-25253,” NVD — National Vulnerability Database. [Online]. Available: https://nvd.nist .gov/vuln/detail/CVE-2026-25253 NIST National Vulnerability Database, “CVE2026-32922,” NVD — National Vulnerability Database. [Online]. Available: https://nvd.nist .gov/vuln/detail/CVE-2026-32922 K. Chu et al., “Selective KV-Cache Sharing to Mitigate Timing Side-Channels in LLM Inference,” arXiv preprint arXiv:2508.08438, 2025, Available: https://arxiv.org/abs/2508.08438 K. Gao, T. Pang, C. Du, Y. Yang, S.-T. Xia, and M. Lin, “Denial-of-Service Poisoning Attacks against Large Language Models,” arXiv preprint arXiv:2410.10760, 2024, Available: https://arxiv. org/abs/2410.10760
[94]
[95]
[96]
[97]
[98]
[99]
[100]
[101]
[102]
[103]
[104]
[105]
R. Greenblatt, B. Shlegeris, K. Sachan, and F. Roger, “AI Control: Improving Safety Despite Intentional Subversion,” arXiv preprint arXiv:2312.06942, 2023, Available: https://arxi v.org/abs/2312.06942 M. Sharma et al., “Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming,” arXiv preprint arXiv:2501.18837, 2025, Available: https://arxiv. org/abs/2501.18837 H. Cunningham et al., “Constitutional Classifiers++: Efficient Production-Grade Defenses against Universal Jailbreaks,” arXiv preprint arXiv:2601.04603, 2026, Available: https://arxi v.org/abs/2601.04603 Y. Liu, Y. Jia, J. Jia, D. Song, and N. Z. Gong, “DataSentinel: A Game-Theoretic Detection of Prompt Injection Attacks,” arXiv preprint arXiv:2504.11358, 2025, Available: https://arxiv. org/abs/2504.11358 S. Choudhary, D. Anshumaan, N. Palumbo, and S. Jha, “How Not to Detect Prompt Injections with an LLM,” arXiv preprint arXiv:2507.05630, 2025, Available: https://arxiv.org/abs/2507.05630 M. Nasr et al., “The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against LLM Jailbreaks and Prompt Injections,” arXiv preprint arXiv:2510.09023, 2025, Available: https://arxiv.org/abs/2510.09023 B. Baker et al., “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation,” arXiv preprint arXiv:2503.11926, 2025, Available: https://arxiv.org/abs/2503.11926 M. Y. Guan et al., “Monitoring Monitorability,” arXiv preprint arXiv:2512.18311, 2025, Available: https://arxiv.org/abs/2512.18311 OpenAI, “GPT-5.6 System Card,” OpenAI, Jul. 2026. Accessed: Aug. 27, 2026. [Online]. Available: https://deploymentsafety.openai.com/gpt5-6/gpt-5-6.pdf Y.-H. Chen et al., “Reasoning Models Struggle to Control their Chains of Thought,” arXiv preprint arXiv:2603.05706, 2026, Available: https://arxiv. org/abs/2603.05706 S. Chennabasappa et al., “LlamaFirewall: An open source guardrail system for building secure AI agents,” arXiv preprint arXiv:2505.03574, 2025, Available: https://arxiv.org/abs/2505.03574 OpenAI, “How We Monitor Internal Coding Agents for Misalignment,” OpenAI. Accessed: Sep. 07, 2026. [Online]. Available: https://openai.com /index/how-we-monitor-internal-coding-agentsmisalignment/
[106] S. Shiromani and L. Richter, “A False Average: Chain-of-Thought Monitors Collapse Where They Are the Only Defense,” arXiv preprint arXiv:2608.00583, 2026, Available: https://arxi v.org/abs/2608.00583 [107] G. Severi, S. Mirza, B. Bullwinkel, and A. Minnich, “Evading Chain-of-Thought Monitoring Through Model Poisoning,” arXiv preprint arXiv:2608.02820, 2026, Available: https://arxi v.org/abs/2608.02820 [108] T. Korbak et al., “Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety,” arXiv preprint arXiv:2507.11473, 2025, Available: https://arxiv.org/abs/2507.11473 [109] Anthropic, “System Card: Claude Fable 5 & Claude Mythos 5,” Anthropic, Jun. 2026. Available: https://www.anthropic.com/claude-fable-5and-claude-mythos-5-system-card [110] S. Rozenfeld, R. Pankajakshan, I. Zloczower, E. Lenga, G. Gressel, and Y. Mirsky, “GAVEL: Towards Rule-Based Safety Through Activation Monitoring,” arXiv preprint arXiv:2601.19768, 2026, Available: https://arxiv.org/abs/2601.19768 [111] L. Gao et al., “Scaling and Evaluating Sparse Autoencoders,” arXiv preprint arXiv:2406.04093, 2024, Available: https://arxiv.org/abs/2406.040 93 [112] K.-H. Hung, C.-Y. Ko, A. Rawat, I.-H. Chung, W. H. Hsu, and P.-Y. Chen, “Attention Tracker: Detecting Prompt Injection Attacks in LLMs,” arXiv preprint arXiv:2411.00348, 2024, Available: https: //arxiv.org/abs/2411.00348 [113] Z. Zhou et al., “On the Role of Attention Heads in Large Language Model Safety,” arXiv preprint arXiv:2410.13708, 2024, Available: https://arxiv. org/abs/2410.13708 [114] C. Wang et al., “ICON: Indirect Prompt Injection Defense for Agents based on Inference-Time Correction,” arXiv preprint arXiv:2602.20708, 2026, Available: https://arxiv.org/abs/2602.20708 [115] B. Cohen-Wang, Y.-S. Chuang, and A. Madry, “Learning to Attribute with Attention,” arXiv preprint arXiv:2504.13752, 2025, Available: https: //arxiv.org/abs/2504.13752 [116] A. Pan, L. Chen, and J. Steinhardt, “LatentQA: Teaching LLMs to Decode Activations Into Natural Language,” arXiv preprint arXiv:2412.08686, 2024, Available: https://arxiv.org/abs/2412.086 86 [117] OpenAI, “Pacing Model Development in an Era of Cyber-Critical Capabilities,” OpenAI. Accessed: Sep. 07, 2026. [Online]. Available: https://open ai.com/index/pacing-model-development-cybercapabilities/
28
[118] S. Das and F. Fioretto, “NeuroFilter: ActivationBased Guardrails for Privacy-Conscious LLM Agents,” arXiv preprint arXiv:2601.14660, 2026, Available: https://arxiv.org/abs/2601.14660 [119] L. Bailey et al., “Obfuscated Activations Bypass LLM Latent-Space Defenses,” arXiv preprint arXiv:2412.09565, 2024, Available: https://arxiv. org/abs/2412.09565 [120] R. Gupta and E. Jenner, “RL-Obfuscation: Can Language Models Learn to Evade Latent-Space Monitors?” arXiv preprint arXiv:2506.14261, 2025, Available: https://arxiv.org/abs/2506.1 4261 [121] Y. Zheng, Y. Hu, T. Yu, and A. Quinn, “AgentSight: System-Level Observability for AI Agents Using eBPF,” Proceedings of the 4th Workshop on Practical Adoption Challenges of ML for Systems (PACMI), arXiv:2508.02736, 2025, Available: https://arxiv.org/abs/2508.02736 [122] M. Kim et al., “CausalArmor: Efficient Indirect Prompt Injection Guardrails via Causal Attribution,” arXiv preprint arXiv:2602.07918, 2026, Available: https://arxiv.org/abs/2602.07918 [123] J. Kutasov et al., “SHADE-Arena: Evaluating Sabotage and Monitoring in LLM Agents,” arXiv preprint arXiv:2506.15740, 2025, Available: https: //arxiv.org/abs/2506.15740 [124] E. Najt, C. Toft, T. Tracy, F. Roger, and J. Benton, “SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors,” arXiv preprint arXiv:2605.16626, 2026, Available: https://arxiv. org/abs/2605.16626 [125] M. Terekhov et al., “Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols,” arXiv preprint arXiv:2510.09462, 2025, Available: https: //arxiv.org/abs/2510.09462 [126] C. Shi et al., “Lessons from Defending Gemini Against Indirect Prompt Injections,” arXiv preprint arXiv:2505.14534, 2025, Available: https: //arxiv.org/abs/2505.14534 [127] S. Willison, “The Dual LLM Pattern for Building AI Assistants That Can Resist Prompt Injection.” [Online]. Available: https://simonwillison.net/20 23/Apr/25/dual-llm-pattern/ [128] L. Beurer-Kellner et al., “Design Patterns for Securing LLM Agents against Prompt Injections,” arXiv preprint arXiv:2506.08837, 2025, Available: https://arxiv.org/abs/2506.08837 [129] E. Debenedetti et al., “Defeating Prompt Injections by Design,” arXiv preprint arXiv:2503.18813, 2025, Available: https://arxiv.org/abs/2503.188 13 [130] N. Hardy, “KeyKOS Architecture,” ACM SIGOPS Operating Systems Review, 1985.
[131] J. S. Shapiro, J. M. Smith, and D. J. Farber, “EROS: A Fast Capability System,” in ACM SOSP, 1999. doi: 10.1145/319151.319163. [132] G. Klein, K. Elphinstone, and G. Heiser, “seL4: Formal Verification of an OS Kernel,” in ACM SOSP, 2009. doi: 10.1145/1629575.1629596. [133] R. N. M. Watson, J. Anderson, B. Laurie, and K. Kennaway, “Capsicum: Practical Capabilities for UNIX,” in USENIX Security, 2010. [134] Y. Wu, F. Roesner, T. Kohno, N. Zhang, and U. Iqbal, “IsolateGPT: An Execution Isolation Architecture for LLM-Based Agentic Systems,” Network and Distributed System Security Symposium (NDSS); arXiv:2403.04960, 2025, Available: https: //arxiv.org/abs/2403.04960 [135] J. Kim, W. Choi, and B. Lee, “Prompt Flow Integrity to Prevent Privilege Escalation in LLM Agents,” arXiv preprint arXiv:2503.15547, 2025, Available: https://arxiv.org/abs/2503.15547 [136] K. Hines, G. Lopez, M. Hall, F. Zarfati, Y. Zunger, and E. Kiciman, “Defending Against Indirect Prompt Injection Attacks With Spotlighting,” arXiv preprint arXiv:2403.14720, 2024, Available: https://arxiv.org/abs/2403.14720 [137] E. Zverev et al., “ASIDE: Architectural Separation of Instructions and Data in Language Models,” arXiv preprint arXiv:2503.10566, 2025, Available: https://arxiv.org/abs/2503.10566 [138] E. Wallace, K. Xiao, R. Leike, L. Weng, J. Heidecke, and A. Beutel, “The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions,” arXiv preprint arXiv:2404.13208, 2024, Available: https://arxiv.org/abs/2404.13208 [139] T. Wu et al., “Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy,” arXiv preprint arXiv:2410.09102, 2024, Available: https://arxiv.org/abs/2410.09102 [140] S. Kariyappa and G. E. Suh, “Stronger Enforcement of Instruction Hierarchy via Augmented Intermediate Representations,” arXiv preprint arXiv:2505.18907, 2025, Available: https://arxi v.org/abs/2505.18907 [141] S. Chen, J. Piet, C. Sitawarin, and D. Wagner, “StruQ: Defending Against Prompt Injection with Structured Queries,” arXiv preprint arXiv:2402.06363, 2024, Available: https://arxi v.org/abs/2402.06363 [142] S. Chen, A. Zharmagambetov, S. Mahloujifar, K. Chaudhuri, D. Wagner, and C. Guo, “SecAlign: Defending Against Prompt Injection with Preference Optimization,” arXiv preprint arXiv:2410.05451, 2024, Available: https://arxi v.org/abs/2410.05451
29
[143] C. Xiang, T. Wu, Z. Zhong, D. Wagner, D. Chen, and P. Mittal, “Certifiably Robust RAG against Retrieval Corruption,” arXiv preprint arXiv:2405.15556, 2024, Available: https://arxi v.org/abs/2405.15556 [144] Q. Wei et al., “A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memory,” arXiv preprint arXiv:2510.02373, 2025, Available: https: //arxiv.org/abs/2510.02373 [145] C. Ouyang and R. Hou, “MemLineage: LineageGuided Enforcement for LLM Agent Memory,” arXiv preprint arXiv:2605.14421, 2026, Available: https://arxiv.org/abs/2605.14421 [146] M. Kim et al., “LiSA: Lifelong Safety Adaptation via Conservative Policy Induction,” arXiv preprint arXiv:2605.14454, 2026, Available: https://arxiv. org/abs/2605.14454 [147] Apple Security Engineering and Architecture, “Private Cloud Compute: A New Frontier for AI Privacy in the Cloud,” Apple Security Research. [Online]. Available: https://security.apple.com/blog/ private-cloud-compute/ [148] P. Efstathopoulos, M. Krohn, and S. VanDeBogart, “Labels and Event Processes in the Asbestos Operating System,” in ACM SOSP, 2005. doi: 10.1145/1095810.1095813. [149] N. Zeldovich, S. Boyd-Wickizer, E. Kohler, and D. Mazieres, “Making Information Flow Explicit in HiStar,” in USENIX OSDI, 2006. [150] M. Krohn, A. Yip, and M. Brodsky, “Information Flow Control for Standard OS Abstractions,” in ACM SOSP, 2007. doi: 10.1145/1294261.1294293. [151] D. E. Denning, “A Lattice Model of Secure Information Flow,” Communications of the ACM, 1976, doi: 10.1145/360051.360056. [152] M. Costa et al., “Securing AI Agents with Information-Flow Control,” arXiv preprint arXiv:2505.23643, 2025, Available: https://arxiv.org/abs/2505.23643 [153] S. R. Garzon et al., “AI Agents with Decentralized Identifiers and Verifiable Credentials,” arXiv preprint arXiv:2511.02841, 2025, Available: https: //arxiv.org/abs/2511.02841 [154] Anthropic, “Zero Trust for AI Agents,” Anthropic. [Online]. Available: https://claude.com/blog/ze ro-trust-for-ai-agents [155] A. Chan et al., “Visibility into AI Agents,” arXiv preprint arXiv:2401.13138, 2024, Available: https: //arxiv.org/abs/2401.13138 [156] A. Chan et al., “Infrastructure for AI Agents,” arXiv preprint arXiv:2501.10114, 2025, Available: https://arxiv.org/abs/2501.10114 [157] T. Shi et al., “Progent: Securing AI Agents with Privilege Control,” arXiv preprint arXiv:2504.11703, 2025, Available: https://arxiv.org/abs/2504.11703
[158] H. Wang, C. M. Poskitt, and J. Sun, “AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents,” arXiv preprint arXiv:2503.18666, 2025, Available: https://arxi v.org/abs/2503.18666 [159] W. Zhao, Z. Li, P. Zhang, and J. Sun, “ClawGuard: A Runtime Security Framework for ToolAugmented LLM Agents Against Indirect Prompt Injection,” arXiv preprint arXiv:2604.11790, 2026, Available: https://arxiv.org/abs/2604.11790 [160] J. Zhu et al., “MiniScope: A Least Privilege Framework for Authorizing Tool Calling Agents,” arXiv preprint arXiv:2512.11147, 2025, Available: https: //arxiv.org/abs/2512.11147 [161] X. Li et al., “A Vision for Access Control in LLM-based Agent Systems,” arXiv preprint arXiv:2510.11108, 2025, Available: https://arxi v.org/abs/2510.11108 [162] CISA, NSA, and U. NCSC, “Secure Adoption of Agentic AI: Joint Guidance,” 2026. [163] S. G. Patil et al., “GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applications,” arXiv preprint arXiv:2404.06921, 2024, Available: https://arxiv.org/abs/2404.06921 [164] G. Shi et al., “SoK: Trust-Authorization Mismatch in LLM Agent Interactions,” arXiv preprint arXiv:2512.06914, 2025, Available: https://arxiv. org/abs/2512.06914 [165] P. Wang, Y. Li, and Y. Tian, “Reframing LLM Agent Security as an Agent-Human Interaction Problem,” arXiv preprint arXiv:2605.24309, 2026, Available: https://arxiv.org/abs/2605.24309 [166] W3C, “Decentralized Identifiers (DIDs) and Verifiable Credentials,” 2022. [167] SPIFFE Project, “SPIFFE: Secure Production Identity Framework for Everyone,” Cloud Native Computing Foundation. [Online]. Available: http s://spiffe.io/ [168] K. Huang et al., “A Novel Zero-Trust Identity Framework for Agentic AI: Decentralized Authentication and Fine-Grained Access Control,” arXiv preprint arXiv:2505.19301, 2025, Available: https: //arxiv.org/abs/2505.19301 [169] G. Syros, A. Suri, J. Ginesin, C. Nita-Rotaru, and A. Oprea, “SAGA: A Security Architecture for Governing AI Agentic Systems,” arXiv preprint arXiv:2504.21034, 2025, Available: https://arxi v.org/abs/2504.21034 [170] J. Mao et al., “AgentSafe: Safeguarding Large Language Model-based Multi-agent Systems via Hierarchical Data Management,” arXiv preprint arXiv:2503.04392, 2025, Available: https://arxi v.org/abs/2503.04392
30
[171] Z. Ji et al., “Taming Various Privilege Escalation in LLM-Based Agent Systems: A Mandatory Access Control Framework,” arXiv preprint arXiv:2601.11893, 2026, Available: https://arxiv. org/abs/2601.11893 [172] L. Tsai and E. Bagdasarian, “Contextual Agent Security: A Policy for Every Purpose,” arXiv preprint arXiv:2501.17070, 2025, Available: https: //arxiv.org/abs/2501.17070 [173] N. Abaev, D. Klimov, G. Levinov, D. Mimran, Y. Elovici, and A. Shabtai, “AgentGuardian: Learning Access Control Policies to Govern AI Agent Behavior,” arXiv preprint arXiv:2601.10440, 2026, Available: https://arxiv.org/abs/2601.10440 [174] N. Gardner-Challis et al., “When can we trust untrusted monitoring? A safety case sketch across collusion strategies,” arXiv preprint arXiv:2602.20628, 2026, Available: https://arxiv. org/abs/2602.20628 [175] E. Debenedetti, J. Zhang, M. Balunović, L. BeurerKellner, M. Fischer, and F. Tramèr, “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents,” arXiv preprint arXiv:2406.13352, 2024, Available: https://arxiv.org/abs/2406.13352 [176] H. Zhang et al., “Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents,” arXiv preprint arXiv:2410.02644, 2024, Available: https://arxi v.org/abs/2410.02644 [177] Z. Liao et al., “RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments,” arXiv preprint arXiv:2505.21936, 2025, Available: https://arxi v.org/abs/2505.21936 [178] P. Niu et al., “Understanding and Evaluating Claw-like Agent Security Through a ComputerSystems Lens,” arXiv preprint arXiv:2606.30755, 2026, Available: https://arxiv.org/abs/2606.307 55 [179] A. Zharmagambetov, C. Guo, I. Evtimov, M. Pavlova, R. Salakhutdinov, and K. Chaudhuri, “AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents,” arXiv preprint arXiv:2503.09780, 2025, Available: https://arxi v.org/abs/2503.09780 [180] M. Andriushchenko et al., “AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents,” arXiv preprint arXiv:2410.09024, 2024, Available: https://arxiv.org/abs/2410.09024 [181] Z. Zhang et al., “Agent-SafetyBench: Evaluating the Safety of LLM Agents,” arXiv preprint arXiv:2412.14470, 2024, Available: https://arxi v.org/abs/2412.14470
[182] S. Vijayvargiya et al., “OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety,” arXiv preprint arXiv:2507.06134, 2025, Available: https://arxiv.org/abs/2507.06134 [183] Q. Hu et al., “SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces,” arXiv preprint arXiv:2606.01317, 2026, Available: https://arxiv.org/abs/2606.0 1317 [184] J. Ma et al., “ASEval: Automated TrajectoryLevel Security Testing for Autonomous Agents,” arXiv preprint arXiv:2605.22321, 2026, Available: https://arxiv.org/abs/2605.22321 [185] P. Li et al., “AgentCanary: A Security Evaluation Framework for Autonomous AI Agents in Real Executable Environments,” arXiv preprint arXiv:2606.10484, 2026, Available: https://arxi v.org/abs/2606.10484 [186] Y. Ruan et al., “Identifying the Risks of LM Agents with an LM-Emulated Sandbox,” arXiv preprint arXiv:2309.15817, 2023, Available: https: //arxiv.org/abs/2309.15817 [187] Y. Lu et al., “LongSafety: Evaluating LongContext Safety of Large Language Models,” arXiv preprint arXiv:2502.16971, 2025, Available: https: //arxiv.org/abs/2502.16971 [188] S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains,” arXiv preprint arXiv:2406.12045, 2024, Available: https://arxiv. org/abs/2406.12045 [189] A. Khanal, Y. Tao, and J. Zhou, “Beyond pass@1: A Reliability Science Framework for Long-Horizon LLM Agents,” arXiv preprint arXiv:2603.29231, 2026, Available: https://arxiv.org/abs/2603.2 9231 [190] C. Li et al., “ADR: An Agentic Detection System for Enterprise Agentic AI Security,” arXiv preprint arXiv:2605.17380, 2026, Available: https://arxiv. org/abs/2605.17380 [191] N. V. Pandya, A. Labunets, S. Gao, and E. Fernandes, “May I have your Attention? Breaking Fine-Tuning based Prompt Injection Defenses using Architecture-Aware Attacks,” arXiv preprint arXiv:2507.07417, 2025, Available: https://arxiv. org/abs/2507.07417 [192] NIST CAISI, “Technical Blog: Strengthening AI Agent Hijacking Evaluations,” NIST Technical Blog. [Online]. Available: https://www.nist.g ov/news-events/news/2025/01/technical-blogstrengthening-ai-agent-hijacking-evaluations [193] S. Abdelnabi et al., “LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge,” arXiv preprint arXiv:2506.09956, 2025, Available: https://arxiv.org/abs/2506.09956
31
[194] J. Needham, G. Edkins, G. Pimpale, H. Bartsch, and M. Hobbhahn, “Large Language Models Often Know When They Are Being Evaluated,” arXiv preprint arXiv:2505.23836, 2025, Available: https: //arxiv.org/abs/2505.23836 [195] T. van der Weij, F. Hofstätter, O. Jaffe, S. F. Brown, and F. R. Ward, “AI Sandbagging: Language Models can Strategically Underperform on Evaluations,” arXiv preprint arXiv:2406.07358, 2024, Available: https://arxiv.org/abs/2406.0 7358 [196] W. Gurnee et al., “Verbalizable Representations Form a Global Workspace in Language Models,” Transformer Circuits Thread. Accessed: Sep. 09, 2026. [Online]. Available: https://transformercircuits.pub/2026/workspace/index.html [197] H. Li, R. Wen, S. Shi, N. Zhang, Y. Vorobeychik, and C. Xiao, “AgentDyn: Are Your Agent Security Defenses Deployable in Real-World Dynamic Environments?” arXiv preprint arXiv:2602.03117, 2026, Available: https://arxiv.org/abs/2602.031 17
[198] Y. Zheng, Y. Hu, W. Zhang, and A. Quinn, “Towards Agentic OS: An LLM Agent Framework for Linux Schedulers,” arXiv preprint arXiv:2509.01245, 2025, Available: https://arxi v.org/abs/2509.01245 [199] H. Lin, H. Pao, S. Zhan, and H.-T. Zheng, “Model-Native Computing Architecture: Envisioning Future System Architecture Through the Lens of Computer Architecture,” arXiv preprint arXiv:2606.00288, 2026, Available: https://arxiv. org/abs/2606.00288 [200] M. Blaze, J. Feigenbaum, and J. Lacy, “Decentralized Trust Management,” in IEEE Symposium on Security and Privacy, 1996. [201] R. L. Rivest and B. Lampson, “SDSI --- A Simple Distributed Security Infrastructure,” Manuscript, MIT, 1996.
32