PocketAgents: A Manifest-Driven Library of Autonomous Defense Agents Sidnei Barbieri1 , Ágney Lopes Roth Ferraz1 , and Lourenço Alves Pereira Júnior1
arXiv:2605.21694v1 [cs.CR] 20 May 2026
1
Aeronautics Institute of Technology (ITA), São José dos Campos/SP, Brazil
Abstract— Connecting large language models (LLMs) to defensive enforcement requires more than asking a model whether an attack is happening. A defender must decide which model outputs may change the system state, which outputs must be rejected, and how failures should be recorded. We present PocketAgents, a manifest-driven library of autonomous defense agents. Each agent is installed as three data files: a manifest, a prompt, and a runtime context. The shared runtime gives the agent bounded telemetry access and accepts only typed reports whose requested action appears in the manifest. We implemented PocketAgents on top of a cyber arena (Perry), a cyber-deception testbed, and evaluated two agents, Command and Control and Exfiltration, in 18 closed-loop trials of a DarkSide-inspired attack on a small enterprise topology. Thirteen trials produced validated network-block actions and contained the attack; four failed schema validation; one produced a valid no-action decision. The experiments show that a typed boundary makes LLM-driven defense measurable, extensible, and attributable. Keywords— Autonomous defense, LLM agents, cyber experimentation.
I. I NTRODUCTION Enterprise defense has a response bottleneck. Security operations centers (SOCs) process high alert volumes, rely on playbooks that analysts often adapt in practice, and still face incidents in which attackers move faster than manual triage can respond. Prior SOC studies report high falsepositive pressure and mismatches between tools and analyst workflows [1]–[4]. The Equifax breach and the DarkSide campaign against Colonial Pipeline show the operational cost of delayed response [5], [6]. Large language models (LLMs) create a tempting shortcut: let a model inspect telemetry and recommend an action. That shortcut is unsafe if the model can directly invoke enforcement tools or if the system parses free-form prose into network changes. A recommendation such as “block the suspicious host” must become a concrete action, target, and scope before it changes the environment. If the output is malformed, unsupported, or ungrounded in the observed telemetry, the system should reject it and record the reason. PocketAgents studies this boundary as a systems abstraction for autonomous defense. The contribution is not Perry itself; Perry is the cyber-deception testbed we use for closed-loop experiments [7]. The contribution is the agent-library layer on the defender side: agents are installed by writing data files rather than framework code, and their outputs cross a typed enforcement boundary before any action is taken. We use specific terminology throughout: agents investigate and emit Sidnei Barbieri, ORCID 0000-0001-9090-9469, [email protected]; Ágney Lopes Roth Ferraz, ORCID 0009-0009-1202-6447, [email protected]; Lourenço Alves Pereira Júnior, ORCID 0000-0002-9682-0075, [email protected].
structured reports, while policies and manifests constrain what those reports may do. The research idea is a marketplace-style library of defensive agents, indexed by tactical purpose and constrained by a common runtime. The current artifact exercises that idea using Command and Control (C2) and Exfiltration agents, which correspond to two tactics in the MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge) framework [8], and share the same runtime but differ only in their manifest, prompt, and context. A larger library could contain agents for lateral movement, credential abuse, discovery, persistence, or recovery, but the scientific claim in this paper starts with the interface that makes such agents comparable. The 18-trial experiment shows that the interface is executable, that two agents can share it, and that failures become attributable to classes rather than opaque model behavior. The architectural value lies in PocketAgents changing what is being evaluated. A conventional prompt-based experiment asks whether a model can describe an attack. A conventional rule system asks whether a predicate matches. PocketAgents asks whether an autonomous investigation can be packaged as a reusable unit, receive bounded evidence, produce an admissible report, and cause a scoped defensive action whose outcome is visible in the trace. That shift matters because the hard part of LLM-mediated defense is recognizing suspicious behavior and deciding which recognized behavior may safely cross into enforcement. II. S YSTEM M ODEL AND A RCHITECTURE PocketAgents assumes a defender-controlled execution environment. The telemetry source, dispatcher, validation code, and enforcement adapter are trusted components of the defender. The LLM backend is treated as an untrusted reasoning component whose output may be useful but cannot be executed directly. The attacker can generate network activity, trigger alerts, and influence the evidence that appears in telemetry through normal attack behavior. The attacker is not assumed to compromise the dispatcher, modify manifests, alter the validation code, or issue direct commands to the enforcement adapter. This threat model focuses the paper on a single architectural question: can a defensive runtime admit plug-in agents while keeping enforcement behind a deterministic boundary? The present experiment measures contract compliance, containment outcome, and failure attribution under controlled attacks. Prompt injection, memory poisoning, malicious tool descriptions, compromised logs, adversarial manipulation of the LLM provider, and operational false-positive rate are outside the measured scope.
The scope also clarifies the term “agent.” In this paper, an agent is not a natural-language policy nor an arbitrary program with direct access to tools. It is a constrained investigation: a manifest defines what the agent may observe and request, a prompt defines how it investigates, and a context file defines deployment facts. The agent can reason over telemetry and emit a report, but the report becomes an action only if the shared runtime validates it. This distinction is central because a library of agents should make extension easier without compromising enforcement accountability. The context file is part of that accountability. Enterprise traffic is rarely self-explanatory at the packet or flow level. A connection pattern that looks like C2 may be an update service, a monitoring collector, an endpoint-management channel, a telemetry path, or an antivirus coordination server. A bulk transfer may be exfiltration on one host and a scheduled backup on another. PocketAgents, therefore, treats context as a firstclass input, not as prose hidden inside a prompt. The runtime can carry deployment facts such as trusted subnets, host roles, user or session metadata, allowed destinations, and expected services. This is the bridge between an agent library and operational defense: the same tactical agent can run across different environments, while the manifest and context specify what counts as admissible evidence and actions. PocketAgents separates investigation, validation, and enforcement. Investigation is probabilistic and LLM-backed. Validation is deterministic and performed by the dispatcher. Enforcement is scoped and performed by an adapter. Fig. 1 shows the runtime path. A sensor raises a trigger, the dispatcher selects a compatible agent from the registry, injects a bounded telemetry interface, and asks the agent to investigate. The agent returns a typed report. The dispatcher validates that report against the manifest before invoking a network-block adapter or any other action. The agent never calls the enforcement adapter directly. An agent manifest declares the trigger class, action catalog, output schema, required fields, confirmation field, model backend, query budget, and result cap. The prompt gives the investigation procedure. The context file carries environment facts such as trusted subnets, host roles, and allowed telemetry. These files are static inputs to the runtime; they do not contain the evidence for a specific trial. The telemetry slice itself is supplied at execution time through a bounded evidence interface. The boundary checks a report R against the manifest M and the telemetry slice T exposed during the session. Let Σ and A be the report schema and the action catalog declared by M, and let E(T ) be the set of target entities (such as hosts, addresses, and sessions) observable in T . The admission predicate is FM (R, T ) =Fstruct (R, Σ) ∧ [action(R) ∈ A] ∧ [entities(R) ⊆ E(T )].
(1)
The current implementation enforces the first two terms: the report must parse, contain the required typed fields, and request an action in the catalog A. The third term is the grounding check: every target entity in the report must appear in the telemetry returned during the session. In the prototype, grounding is evaluated after the run from recorded query results. This keeps the reported experiment aligned with the code while preserving grounding as part of the architectural contract.
The boundary produces six outcome classes, summarized in Table I. These labels are the mechanism’s output: they tell an operator whether to review evidence, fix a prompt, change a model backend, or inspect enforcement. The table separates classes observed in the present experiments from classes required by the architecture but not yet exercised. “Observed” classes appear in the 18-trial artifact. “Supported” classes are enforced by the current implementation, but did not occur in those trials. “Planned” classes belong to the admission contract but are not yet enforced inside the runtime. This separation prevents design intent from being counted as experimental evidence. PocketAgents is built around four design choices. The first is a data-only installation. A new agent should not require edits to the dispatcher, telemetry adapter, or enforcement adapter when its action shape is already supported. This constraint is stronger than a convenience goal. It is what makes the library auditable: the reviewer can compare two agent directories and see whether a new behavior was introduced through manifest, prompt, and context files or through hidden framework changes. In the current artifact, the C2 and Exfiltration agents satisfy this property for the same network-block action. The second choice is typed reporting instead of direct tool calling. Tool-calling frameworks can restrict which tools a model may call, but a defensive action still needs a security meaning. Blocking an Internet Protocol (IP) endpoint, isolating a host, or restoring a service must carry a target, a reason, and a scope. PocketAgents, therefore, asks the model to emit a report whose fields are named in the manifest. The dispatcher interprets only that report, not the surrounding prose. This is why four Exfiltration failures are counted as rejected reports even though the model produced natural-language analysis. A paragraph may help an analyst, but it is not an admissible enforcement command. The third choice is bounded telemetry access. The agent does not receive the entire environment state or arbitrary shell access. It receives a small interface for the evidence needed by the manifest. This makes the execution trace inspectable. A reviewer can ask which telemetry the agent saw before a block and whether the target appears in those results. The present prototype records this evidence and uses it to compute entity precision after the run. The fourth choice is outcome attribution. A SOC operator needs more than a containment flag. They need to know whether the system rejected malformed output, refused an unsupported action, exhausted a budget, or produced a valid no-action decision. These are different engineering problems. A schema failure can be fixed by prompt and parser work. An unsupported action asks whether the action belongs in the manifest. A no-action decision asks whether the evidence or model reasoning was insufficient. This is the point where PocketAgents differs from a generic “LLM for SOC” framing: the architecture is useful even when the model fails, because the failure is named and localized. III. I NSTANTIATION We instantiated PocketAgents inside Perry’s defender subsystem. Perry supplies emulated hosts, attack execution, telemetry capture, and result traces. PocketAgents adds the agent-library layer: package loading, bounded LLM investigation, typedreport parsing, deterministic validation, and scoped action
Agent library
Trigger
Environment
from detector
state change
Manifest action catalog, schema
Prompt
Shared runtime load
procedure
Dispatcher selects agent binds evidence
evidence
Agent investigation
report
untrusted LLM
Typed boundary (Eq. 1) parse · allowlist · grounding
admit
Action adapter scoped change audit record
Context deployment facts
Telemetry slice
Trace
bounded evidence
outcome class
Fig. 1. PocketAgents runtime. Data-only agents cross a typed boundary before any action is subject to enforcement.
TABLE I
TABLE II
O UTCOME CLASSES PRODUCED BY THE P OCKETAGENTS BOUNDARY.
E VALUATED AGENT CONTRACTS .
Class
Meaning
Status
Typed report passed implemented validation observed and invoked a network block. no action Report was valid but did not confirm the attack. observed schema fail Missing delimiter, malformed JavaScript Ob- observed ject Notation (JSON), wrong type, or missing required field. unsupported action Requested action was absent from the manifest supported catalog. ungrounded Target entity was absent from telemetry re- planned turned during the session. Query or wall-clock budget was reached before supported budget exhaust a valid report. valid block
Field
C2
Exfiltration
Tactical purpose Decision predicate Action target Supporting evidence Admitted action
Command and Control endpoint confirmed network endpoint endpoint convergence network block
Exfiltration transfer confirmed destination endpoint volume and protocol network block
and dispatcher constrain those reports before enforcement. This distinction keeps the contribution in the right category. PocketAgents is not a rule language; it is a runtime boundary that allows autonomous agents to be packaged, constrained, dispatch. The scientific object is this layer and its contract, and evaluated as library members. The two evaluated agents share the same runtime. The not Perry’s experiment engine. C2 agent asks whether hosts converge on a command-andThe instantiation has a narrow boundary with the testbed. control endpoint and reports a confirmed endpoint target. The Perry remains responsible for running the scenario, collecting Exfiltration agent asks whether an anomalous outbound transfer telemetry, and recording outcomes. PocketAgents begins when is in progress and reports a destination, transfer volume, and a defender-side trigger asks whether a bounded investigation protocol. Both agents request the same enforcement action. should run. The runtime loads an agent package, exposes Moving from C2 to Exfil changes only the agent package. a bounded evidence interface, calls the configured LLM This is the operational definition of library membership used backend, parses the final report, and dispatches an action in the paper: a new member of the same action shape can be only after validation. This separation is useful for attribution. installed without modifying the runtime. A failure before the model is called belongs to experiment Table II makes the library contract concrete without exposing orchestration. A malformed report belongs to agent/model prototype internals. Both agents use the same runtime path contract compliance. A validated action that does not contain and request the same enforcement class, but they differ in the the simulated attacker belongs to action or scenario semantics. evidence they must produce. The Exfiltration agent records The paper’s taxonomy follows this boundary. more evidence than is strictly required for admission to action. The evidence interface is the main portability hook. An agent This distinction is useful because an agent can produce richer prompt can ask for facts, but the runtime controls which queries evidence for analyst review without expanding the minimal are available and how many results are returned. In Perry, the interface is backed by experiment telemetry. In a different enforcement predicate. The target architecture also admits a coordinator above the deployment, it could be backed by a Security Information and Event Management (SIEM) system, an Endpoint Detection dispatcher. Its purpose is to reconcile recommendations across and Response (EDR) system, a network-flow database, or a scopes such as global, subnet, user, and session. Enterprise deception platform. The manifest does not need to name those response is rarely flat: a subnet-level block, a user-session backends. It names the action contract and report fields; the quarantine, and a global deny-list entry have different blast runtime maps those contracts to the local environment. This radii and review requirements. The current experiments involve is why we describe PocketAgents as a library layer rather one compatible agent per scenario and host- or network-level than a Perry feature. Perry is the evaluation substrate, while enforcement. This gives a concrete base for the coordinator PocketAgents is the reusable interface between a constrained without making it part of the evaluated mechanism. This scope hierarchy also helps keep the marketplace idea agent investigation and defensive enforcement. The conceptual separation is simple: agents investigate, disciplined. An agent for lateral movement may reason over while constraints admit or reject their outputs. The C2 and host-to-host authentication paths, while an agent for credential Exfiltration modules are agents because they inspect telemetry abuse may reason over user sessions and identity events. They and decide whether to emit a structured report. The manifest need different evidence and may request different target types.
What they can share is the admission pattern: package the investigation, bind it to local context, require a typed report, and admit only actions named by the manifest. PocketAgents evaluates that common pattern on C2 and Exfiltration. The result is small enough to audit yet general enough to explain how additional tactical agents would enter the library without turning each one into a bespoke defender. IV. E VALUATION
TABLE III
O UTCOMES FOR THE 18 EVALUABLE P ERRY TRIALS . Configuration
B/S/N ∆ (s)
C2 + claude-3.5-haiku C2 + gpt-4o-mini C2 + gpt-5.2 Exfiltration + claude-3.5-haiku Exfiltration + gpt-4o-mini Exfiltration + gpt-5.2
3/0/0 2/0/1 3/0/0 2/1/0 0/3/0 3/0/0
46.0 50.0 32.0 75.0 – 34.7
– 7/4 – 7/4 7/4 –
By backend (both agents combined, six trials each) claude-3.5-haiku 5/1/0 gpt-4o-mini 2/3/1 gpt-5.2 6/0/0
57.6 50.0 33.3
83.3% contained 33.3% contained 100% contained
No-block impact
The evaluation uses Perry’s Darkside-EquifaxSmall 13/4/1 45.2 72.2% contained scenario. The attacker emulates scanning, lateral movement, All 18 trials C2 establishment toward a fixed attacker-controlled endpoint, and data exfiltration against a small enterprise topology. The defender runs one PocketAgents agent per trial. The C2 agent either of the investigated web servers; the final impact record is triggered by evidence of command-and-control behavior; shows seven infected hosts, four exfiltrated files, and no block. the Exfiltration agent is triggered by evidence of outbound The artifact package v2.0 contains the 18 canonical trials, data movement. In both cases, the only enforcement action Secure Hash Algorithm 256 (SHA-256) hashes for the final evaluated is Perry’s network-block action. impact records and runtime traces, and scripts to recompute The analysis set contains 18 evaluable trials: three model the outcome distribution. backends crossed with two agents and three trials per configFor each trial π we record the measurement tuple µ(π) = uration. The included backends are claude-3.5-haiku, (o, h, f, τ, η), where o is the boundary outcome, h is the gpt-4o-mini, and gpt-5.2. Another 18 trials are pre- number of infected hosts at termination, f is the number of served separately as application programming interface (API) exfiltrated files, τ is time from attack start to enforcement or configuration errors because the LLM was never reached: two terminal failure, and η is the entity precision for the enforced invalid Anthropic model aliases and one unsupported OpenAI target, measured against the recorded telemetry. In the 13 temperature setting prevented execution. Separating those runs valid block trials, h = 1 and f = 0. In the five nonkeeps provider configuration mistakes out of the agent-quality containing trials, h = 7 and f = 4. The entity precision for measurement while preserving the reproduction trail. applied blocks is η = 1: each enforced IP/port pair appears The experimental unit is a complete Perry run, not an in the trace as the attacker-controlled C2 endpoint. This is a isolated model response. A trial begins when the scenario measurement over the recorded traces; the grounding term in starts, continues through attack execution, alert processing, Eq. 1 indicates that the same check belongs within runtime LLM-backed investigation, boundary validation, and possible admission. enforcement, and ends when Perry records the final state. The latency numbers are useful as experimental traces. Eight This matters because offline prompt evaluation would miss C2 blocks occur in 26–62 seconds after the first prompt; five the contribution that concerns closed-loop enforcement. We Exfiltration blocks occur in 31–76 seconds. These values do inspect the final impact record together with the runtime trace not measure end-to-end SOC latency, because Perry has already that records the agent lifecycle and boundary outcome. raised the trigger by the time the prompt is sent. They measure Table III is the central result. Thirteen of 18 trials produced the agent-runtime segment: bounded evidence access, model validated network-block actions and contained the attack at response, report parsing, validation, and action dispatch. Their the entry host, with zero exfiltrated files. In those trials, the role is to show that the runtime is closed-loop and not offline interval from first LLM prompt to first network block was analysis after the experiment has ended. 26–76 seconds. The five non-containing trials all allowed the The C2 agent is easier than Exfiltration in this experiment. full chain to complete, with seven infected hosts and four Eight of nine C2 trials contained the attack, and the remaining exfiltrated files. The boundary explains those failures: four are trial ended in a valid no-action report. Exfiltration contained schema fail, where the model produced an analysis but five of nine trials and produced all four schema failures. The not the required structured report object; one is no action, difference is informative because the two agents share the same where the report was valid but did not confirm malware. In runtime and action path. C2 has a compact target, the attackerthe table, B/S/N denote validated block, schema failure, and controlled endpoint, and the action target maps directly onto no-action outcomes; ∆ denotes the mean number of seconds that endpoint. Exfiltration asks the model to connect volume, from the first prompt to the first block across block trials. The direction, host role, and protocol before emitting the same upper panel shows per-agent and per-backend counts, while the action. The harder case, therefore, stresses the report contract, lower panel aggregates by backend. This is a systems result, not the enforcement adapter. not a model leaderboard: the architecture reveals whether a Model behavior should also be read through the lens of the backend fails at contract compliance or at the decision itself. boundary, not as a leaderboard. gpt-5.2 produced validated The trace files support the interpretation. In a successful C2 blocks in all six evaluated trials. claude-3.5-haiku trial, the runtime records agent admission, the first prompt, and produced five validated blocks and one schema failure. the network block against the C2 endpoint; the final impact gpt-4o-mini produced two validated blocks, one valid record shows one infected host and no exfiltration. In a failed no-action decision, and three schema failures. The value of Exfiltration trial with gpt-4o-mini, the runtime records this comparison is diagnostic: it indicates whether a backend that the model did not emit the required typed report for meets the enforcement contract, violates the report schema, or
returns a valid decision declining action. A weaker model that produces prose rather than a report is not a near miss from the system’s perspective. It is a rejected enforcement attempt. The execution times clarify what the runtime measured. The mean run time includes scenario execution, attack progress, alert processing, LLM interaction, and enforcement, so it should not be interpreted as pure model latency. In the curated dataset, mean execution time ranges from roughly 224 seconds for Exfiltration with gpt-4o-mini to roughly 949 seconds for C2 with gpt-5.2. The shorter value is not inherently better, as all three Exfiltration gpt-4o-mini trials ended in schema failure and full compromise. For a defensive system, the useful metric is a validated action that arrives before the attacker reaches its objective. This is why the paper reports the outcome class together with infected hosts and exfiltrated files. Two caveats define how the dataset should be read. First, the same scenario is repeated three times per configuration, so the trials measure run-to-run variation in the experiment and model interaction more than diversity in attacker behavior. Second, action success is tied to Perry’s simulated networkblock semantics. Within that scope, the evidence is strong for architectural accountability: the framework reaches the enforcement adapter, applies the action, records the outcome, and assigns non-containment to named boundary classes. V. R EPRODUCIBILITY The public artifact is organized around the paper’s evidence. The canonical dataset contains 18 evaluable trials and excludes the 18 API configuration failures from the quantitative result. The excluded trials are preserved separately because they are useful engineering evidence: they show that benchmark automation can fail before the model receives a prompt. Counting them as model or agent failures would be incorrect; deleting them would hide a real reproduction hazard. The package therefore distinguishes evaluable behavior from runorchestration failure. Each canonical trial directory contains the Perry output, the final impact record, the LLM interaction trace, and a hash entry. The package manifest records the expected files and the SHA-256 digest of the tarball. A reviewer can unpack the package, verify hashes, and rerun the analysis script to recover the 13/4/1 outcome distribution. The model names are the provider backends recorded at execution time; because hosted APIs can change, the artifact’s primary reproducibility target is recomputing outcomes from preserved traces. The source tree also keeps the original Perry implementation separate from the PocketAgents extension so that implementation claims can be audited against the upstream baseline. The artifact is small enough to archive alongside the paper. A SHA-256 digest identifies the dataset version, and the prototype code is packaged as a scoped Perry extension that does not modify the experiment engine. Reviewers can therefore inspect the implementation, verify the dataset, and re-run the analysis without a separate testbed setup. The artifact makes the claims inspectable and repeatable. Every reported number is tied to a run directory, trace, and recomputation script. This matters for autonomous defense because the interesting unit is not an isolated model answer. It is the closed-loop chain from trigger to bounded evidence, report validation, enforcement, and final impact.
TABLE IV
M ECHANISMS DEMONSTRATED BY THE CURRENT ARTIFACT. Mechanism
Evidence in artifact
Why it matters
The Agent library
C2 and Exfiltration share run- New tactical agents can enter time code and differ in three as packages when their action agent files. shape is supported. Typed boundary 13 accepted reports, 4 schema LLM output becomes an acrejections, 1 valid no-action. countable system state before enforcement. Closed-loop action 13 successful trials apply a net- The boundary sits in the runtime work block before impact. path, not in offline analysis. Context separation Manifest, context, prompt, and Similar traffic is judged against evidence are distinct inputs. deployment facts, not raw shape alone. Model sensitivity Outcomes differ across the three The runtime exposes contract evaluated backends. compliance separately from decision behavior. v2.0 package includes hashes Claims are auditable from traces Reproducibility and scripts for recomputation. instead of being reconstructed from prose.
VI. D ISCUSSION The experiments show why the library boundary matters. PocketAgents does not treat an agent as a prompt attached to an enforcement tool. It makes the agent a package whose authority is declared in a manifest, whose evidence comes through a bounded interface, and whose output becomes an action only after validation. In the artifact, this turns C2 and Exfiltration into comparable library members: both share the runtime and action path, but ask different questions of the telemetry. The main systems insight is that defensive action is context-dependent. A flow can resemble C2 traffic because of destination, cadence, or service role, but the same raw shape can also appear in management telemetry, update infrastructure, monitoring agents, or antivirus services. Exfiltration has a similar ambiguity: high-volume outbound transfer is suspicious only relative to host role, protocol, destination, and expected behavior. PocketAgents makes that context explicit by separating the static agent contract from the runtime context and bounded evidence. The action boundary then checks the model’s conclusion against the report shape and the action scope admitted for that agent. The outcome taxonomy turns model behavior into an operational state. A valid block is a traceable enforcement event. A schema failure is a rejected command with a repair path. A no-action report is a valid decision that can be inspected against the evidence. This is the value of the 13/4/1 result: the runtime does not collapse all non-containment into a single failure bucket. It tells the defender where the chain broke. Table IV summarizes what the current artifact demonstrates. The table maps the research idea to evidence: library packaging, typed admission, closed-loop containment, context separation, model sensitivity, and reproducibility. The evaluated agents also show why a marketplace can be organized by tactic without making tactics the enforcement mechanism. MITRE ATT&CK supplies a useful indexing vocabulary: C2 and Exfiltration correspond to different defender questions, evidence needs, and report fields. The enforcement runtime remains independent of that vocabulary. It admits an action because a report satisfies a manifest, not because the agent belongs to a named tactic. This preserves extensibility without turning tactical labels into authority. The evaluated scope is deliberately narrow: two tactics, one
scenario family, and one network-block action. That scope support specialized security tasks when they are embedded in is enough to test the interface that matters here. The artifact constrained workflows [23]–[25]. Cloak studies deception as a shows that agents can be packaged as data, executed in a defensive primitive in a different setting [26]. These papers are shared runtime, rejected deterministically when the report useful for comparison because none of them relies on trusting contract fails, and tied to impact records when enforcement an unconstrained model response as an action. PocketAgents succeeds. Broader traffic mixes, additional topologies, and follows the same systems instinct: the model is useful only action surfaces such as host isolation, session termination, after the surrounding runtime defines what the output means credential revocation, or deception reconfiguration would test and what it may affect. accuracy and breadth. They do not change the central result: autonomous defense becomes easier to evaluate when the agent VIII. C ONCLUSION boundary is explicit, typed, and traceable. PocketAgents packages autonomous defense as a library The broader lesson is that autonomous defense needs a of manifest-driven agents with bounded telemetry access and systems boundary before it needs a larger prompt. Without typed enforcement. Inside Perry, two agents share the same that boundary, a defender sees a model answer and must runtime and differ only in their three data files. Across 18 infer whether it was based on the right evidence, whether the evaluable DarkSide/EquifaxSmall trials, the boundary produced target is admissible, whether the requested action is supported, 13 validated blocks, 4 schema rejections, and 1 valid noand whether the final state improved. With PocketAgents, action decision. The contribution is the accountability this those questions become artifacts: the package states what creates: LLM-mediated defense becomes a set of inspectable the agent may do, the trace states what evidence it saw, outcomes tied to traces, not a binary story about whether a the report states what it requested, and the outcome class model happened to block an attack. states how the runtime handled the request. The experiments provide a concrete demonstration of this claim: two tactical R EFERENCES agents, a shared runtime, 18 complete executions, and a result distribution that distinguishes containment, malformed reports, [1] B. A. Alahmadi, L. Axon, I. Martinovic, “99% false positives: A qualitative study of SOC analysts’ perspectives on security alarms,” in and valid no-action behavior. VII. R ELATED W ORK PocketAgents draws from programmable security, SOC measurement, and LLM-agent safety. SANE, Kinetic, and Precise Security Instrumentation (PSI) showed that explicit controlplane abstractions can make security decisions composable and inspectable [9]–[11]. Our setting differs in that the reasoning component is probabilistic; the deterministic boundary shifts from the reasoning process to the typed output. Provenancebased systems such as UNICORN and NoDoze show the value of grounding security decisions in observed evidence [12], [13]. PocketAgents applies the same discipline at enforcement time by recording the evidence needed to bind action targets to observed telemetry. SOC studies motivate automation while showing why integration matters. Alert fatigue, rule-management work, playbook drift, and tool/analyst mismatch are persistent organizational problems [1]–[4], [14]. Systems such as SOCpilot study how LLMs can support SOC workflows [15]. PocketAgents contributes typed records that can flow into those workflows. LLM-agent work shows why boundaries are necessary. PentestGPT and AutoAttacker show that planning, memory, and tool-use loops improve LLM performance on offensive tasks [16], [17]; related work on multi-host attacks in Perry reaches a similar conclusion [18]. AgentSpec, IsolateGPT, and Progent constrain agents at tool-call, execution, and privilege layers [19]–[21]. PocketAgents places the boundary at the typed-report layer for defensive enforcement. Reliability work argues that single success rates hide deployment-relevant failure modes [22]; our taxonomy is a security-specific instance of that principle. Recent systems also use LLMs as components inside broader security pipelines. CTINexus studies cyber-threatintelligence extraction and fusion, while work on protocol fuzzing and log-patch generation shows that LLMs can
Proceedings of the 31st USENIX Security Symposium (USENIX Security). USENIX Association, 2022, pp. 2783–2800. [2] L. Yang, Z. Chen, C. Wang, Z. Zhang, S. Booma, P. Cao, C. Adam, A. Withers, Z. Kalbarczyk, R. K. Iyer, G. Wang, “True attacks, attack attempts, or benign triggers? an empirical measurement of network alerts in a security operations center,” in Proceedings of the 33rd USENIX Security Symposium (USENIX Security). USENIX Association, 2024, pp. 1525–1542. [3] F. B. Kokulu, A. Soneji, T. Bao, Y. Shoshitaishvili, Z. Zhao, A. Doupé, G.-J. Ahn, “Matched and mismatched SOCs: A qualitative study on security operations center issues,” in Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2019, pp. 1955–1970. [4] D. Schlette, P. Empl, M. Caselli, T. Schreck, G. Pernul, “Do you play it by the books? a study on incident response playbooks,” in Proceedings of the 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024, pp. 3625–3643. [5] Majority Staff Report, 115th Congress, “The equifax data breach,” U.S. House of Representatives, Committee on Oversight and Government Reform, Tech. Rep., 2018. [6] Cybersecurity and Infrastructure Security Agency (CISA) Federal Bureau of Investigation (FBI), “DarkSide ransomware: Best practices for preventing business disruption from ransomware attacks,” U.S. Department of Homeland Security and U.S. Department of Justice, Tech. Rep. AA21-131A, 2021. [7] B. Singer, Y. Saquib, L. Bauer, V. Sekar, “Perry: A high-level framework for accelerating cyber deception experimentation,” in 2025 28th International Symposium on Research in Attacks, Intrusions and Defenses (RAID), 2025, pp. 158–173. [8] B. E. Strom, A. Applebaum, D. P. Miller, K. C. Nickels, A. G. Pennington, C. B. Thomas, “MITRE ATT&CK: Design and philosophy,” The MITRE Corporation, Tech. Rep. MP180360R1, 2020. [9] M. Casado, T. Garfinkel, A. Akella, M. J. Freedman, D. Boneh, N. McKeown, S. Shenker, “SANE: A protection architecture for enterprise networks,” in Proceedings of the 15th USENIX Security Symposium, 2006. [10] H. Kim, J. Reich, A. Gupta, M. Shahbaz, N. Feamster, R. Clark, “Kinetic: Verifiable dynamic network control,” in Proceedings of the 12th USENIX Symposium on Networked Systems Design and Implementation (NSDI), 2015. [11] T. Yu, S. K. Fayaz, M. J. Collier, V. Sekar, S. Seshan, “PSI: Precise security instrumentation for enterprise networks,” in Proceedings of the 24th Annual Network and Distributed System Security Symposium (NDSS), 2017. [12] X. Han, T. Pasquier, A. Bates, J. Mickens, M. Seltzer, “UNICORN: Runtime provenance-based detector for advanced persistent threats,” in Proceedings of the 27th Annual Network and Distributed System Security Symposium (NDSS). Internet Society, 2020.
[13] W. U. Hassan, S. Guo, D. Li, Z. Chen, K. Jee, Z. Li, A. Bates, “NoDoze: Combatting threat alert fatigue with automated provenance triage,” in Proceedings of the 26th Annual Network and Distributed System Security Symposium (NDSS). Internet Society, 2019. [14] M. Vermeer, N. Kadenko, C. Gañán, M. van Eeten, S. Parkin, “Alert alchemy: SOC workflows and decisions in the management of NIDS rules,” in Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM, 2023. [15] S. Barbieri, L. V. d. Meneses, Á. L. Roth Ferraz, L. A. Pereira Júnior, “SOCpilot: Verifying policy compliance for LLM-assisted incident response,” arXiv preprint arXiv:2605.05501, 2026. [16] G. Deng, Y. Liu, V. Mayoral-Vilches, P. Liu, Y. Li, Y. Xu, T. Zhang, Y. Liu, M. Pinzger, S. Rass, “PentestGPT: Evaluating and harnessing large language models for automated penetration testing,” in Proceedings of the 33rd USENIX Security Symposium (USENIX Security), 2024. [17] J. Xu, J. W. Stokes, G. McDonald, X. Bai, D. Marshall, S. Wang, A. Swaminathan, Z. Li, “AUTOATTACKER: A large language model guided system to implement automatic cyber-attacks,” arXiv preprint arXiv:2403.01038, 2024. [18] B. Singer, K. Lucas, L. Adiga, M. Jain, L. Bauer, V. Sekar, “On the feasibility of using LLMs to autonomously execute multi-host network attacks,” arXiv preprint arXiv:2501.16466, 2025. [19] H. Wang, C. M. Poskitt, J. Sun, “AgentSpec: Customizable runtime enforcement for safe and reliable LLM agents,” in Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering (ICSE). ACM, 2026. [20] Y. Wu, F. Roesner, T. Kohno, N. Zhang, U. Iqbal, “IsolateGPT: An execution isolation architecture for LLM-based agentic systems,” in Proceedings of the 32nd Annual Network and Distributed System Security Symposium (NDSS). Internet Society, 2025. [21] T. Shi, J. He, Z. Wang, H. Li, L. Wu, W. Guo, D. Song, “Progent: Securing AI agents with privilege control,” arXiv preprint arXiv:2504.11703, 2025, uC Berkeley. [22] S. Rabanser, S. Kapoor, P. Kirgis, K. Liu, S. Utpala, A. Narayanan, “Towards a science of AI agent reliability,” arXiv preprint arXiv:2602.16666, 2026, princeton University. [23] Y. Cheng, O. Bajaber, S. A. Tsegai, D. Song, P. Gao, “CTINexus: Automatic cyber threat intelligence knowledge graph construction using large language models,” in Proceedings of the 2025 IEEE European Symposium on Security and Privacy (EuroS&P). IEEE, 2025. [24] R. Meng, M. Mirchev, M. Böhme, A. Roychoudhury, “Large language model guided protocol fuzzing,” in Proceedings of the 31st Annual Network and Distributed System Security Symposium (NDSS). Internet Society, 2024. [25] Y. Kim, S. Shin, H. Kim, J. Yoon, “Logs in, patches out: Automated vulnerability repair via tree-of-thought LLM analysis,” in Proceedings of the 34th USENIX Security Symposium (USENIX Security). USENIX Association, 2025. [26] D. Ayzenshteyn, R. Weiss, Y. Mirsky, “Cloak, honey, trap: Proactive defenses against LLM agents,” in Proceedings of the 34th USENIX Security Symposium (USENIX Security). USENIX Association, 2025.