SOCpilot: Verifying Policy Compliance for LLM-Assisted Incident Response Sidnei Barbieri∗ , Leonardo Vaz de Meneses∗ , Ágney Lopes Roth Ferraz∗ and Lourenço Alves Pereira Júnior∗ ∗
arXiv:2605.05501v1 [cs.CR] 6 May 2026
Aeronautics Institute of Technology Email: [email protected], [email protected], [email protected], [email protected]
Abstract—Security operations centers (SOCs) are beginning to use large language models (LLMs) as copilots to draft incidentresponse plans. These plans may include actions that are valid per the catalog but still violate mandatory steps, required ordering, or approval gates before analyst review. SOCpilot makes this compliance question measurable at the plan boundary. It fixes the incident package, action catalog, policy rules, verifier, and public evidence surface, then verifies the copilot’s proposed action trace. We evaluate two LLM providers on 200 real incidents from an anonymized production SOC in a financial-sector case study. We compare their plans to paired analyst-authored baselines from the same security orchestration, automation, and response (SOAR) cases. An identical inline policy text moves the two providers in opposite directions. A deterministic verifier removes 466 noncompliant, approval-gated actions without reducing baseline task recall. Aggregate rates remain stable across 3 reruns of the fixed corpus. The official evidence focuses on approval-gated decisions regarding recovery and containment. The artifact also includes public activation checks for mandatory and ordering repairs. We release the runnable artifact1 so independent reviewers can rederive the public results without access to private incident data.
I. I NTRODUCTION Security operations centers (SOCs) are increasingly using large language models (LLMs) as copilots for incident response. In this paper, a copilot is a non-autonomous proposal component. It reads a canonical incident package and recommends a sequence of catalog actions for analyst review. The copilot does not execute any tools, approve any actions, or alter any hosts. This boundary is important because a plausible recommendation can still be policy-noncompliant. For example, restoring a host before preserving evidence can destroy forensic opportunity. Isolating a production asset without approval can interrupt business operations. Delaying containment can leave an attacker active [1], [2], [3]. The concern is not hypothetical. Field studies of SOCs report that legal, insurance, coordination, and organizational pressures constrain which response actions an organization can safely take [4]. Analysts often diverge from documented playbook steps in systematic ways [5]. Meanwhile, recent work has extended LLM assistance to security tasks such as penetration testing [6], goal-directed web scanning [7], and cybersecurity knowledge assessment [8]. The missingmeasurement question is direct: when an LLM proposes an 1 https://github.com/c2dc/socpilot-artifact
incident-response plan, does the action sequence satisfy the policy rules governing the analyst’s next decision? Current evaluation practice makes it hard to answer that question. A reported “LLM policy-compliance” result can vary depending on the prompt, incident slice, action vocabulary, policy interpretation, provider endpoint, or private postprocessing. Runtime-control systems such as Progent [9] and AgentSpec [10] make constraints explicit for live agents by controlling tool calls. However, copilot deployments expose an earlier boundary: the analyst reads and approves the plan before any tool is invoked. The plan, not the tool call, is therefore the first place where a policy-noncompliant recommendation can do harm. SOCpilot targets that boundary. If the plan is the object the analyst sees, then the plan must be the object the evaluation verifies. SOCpilot (Figure 1) turns this boundary into a fixed, auditable evaluation object. Each run uses one canonical incident package, one action catalog, one typed policy set, and one deterministic verifier path. With the same public inputs, a reviewer can see what the copilot proposed, which policy rule fired, what repair the verifier applied, and which aggregate claim follows. The private SOC data stay behind the release boundary. The public artifact contains the canonical packages, the mapping contract, the verifier, run accounting, and aggregate analyses needed to rederive the reported results. SOCpilot is a sector-agnostic evaluation methodology. The reported experiment is a financial-sector case study. The artifact consumes canonical incident packages and a declared policy surface. Another SOC can replace the released cases with its own anonymized cases and rerun the same verifier. In the case study, external attackers trigger incidents against production assets. Responders operate under explicit constraints, institutional coordination, and a governed playbook [5], [4], [3]. The unit of analysis is an incident-level response plan derived from a canonical SOAR case package. These are paired with analyst-authored workflow traces from the same case. This framing leads to three research questions: RQ1: How frequently do LLM-generated plans violate typed SOC policy rules on real incidents? RQ2: How do these violations compare to paired human baselines from the same incidents?
P ROMPT- ONLY CONTROL
P RIVATE PREPARATION
policy text
raw case export
LLM
anonymize
R ELEASED EVALUATION OBJECT canonical package
catalog + policy
human baseline proposed plan
privacy review
LLM proposal
deterministic verifier
enforced plan + typed trace
compliance is assumed, not enforced RELEASE BOUNDARY Fig. 1. SOCpilot’s declared evaluation object. Private data stops at the release boundary. The released object contains canonical packages, the action catalog and policy, LLM proposals, paired human baselines, and a deterministic verifier that emits enforced plans and typed traces. Together, these components define the surface on which every reported claim is checked.
RQ3: To what extent does deterministic verification reduce can lead to evidence loss, delayed containment, or unsafe violations while preserving plan utility? recovery [1], [2], [3]. Studies of prompt injection, prompt Our central claim is that LLM-assisted incident-response leakage, prompt inversion, and cross-app LLM exploitation planning should verify policy compliance on the proposed plan show that LLM-integrated deployments also surface adverrather than treat compliance as a property of the prompt. One sarial manipulation paths at the serving and integration laydeclared evaluation on 200 real SOC incidents supports this ers [12], [13], [14], [15], [16]. Wu et al. (IsolateGPT), for example, demonstrate that execution isolation between LLMclaim. The first observation is provider fragility. Under one prompt backed apps is necessary to prevent cross-app data leakage and lineage and one shared incident package, two production LLM instruction manipulation [16]. Our focus is on pre-execution providers produce action-level violation rates that diverge. We plan compliance rather than autonomous tool execution or report this gap with Cohen’s h, an effect-size measure for the serving-layer defense. That focus motivates a typed policy difference between two proportions: the same inline policy text layer that constrains proposed actions before execution rather produces a large adverse shift for claude-sonnet-4-6 than relying on prompt formulation for safety. The threat model is the plan-review boundary before execu(h = 1.12) and only a small shift for gpt-5.2 (|h| ≈ 0.14) [11]. The second observation is enforcement stability at tion: the assistant neither executes tools nor alters hosts nor the action boundary. The verifier operates only on the explicit directly accesses approval systems. The main risk is that a action sequence and removes 466 non-compliant approval- non-compliant plan may appear plausible during review while gated actions while preserving paired task recall against violating evidence-preservation or approval constraints. Acanalyst-authored baseline plans for the same incidents. The cordingly, the explicit action sequence, rather than the hidden third observation is protocol stability: across 3 reruns of the model state, is the security-relevant unit. We therefore measure fixed corpus, the violation rate stays within 0.8700–0.8750, the whether governance-relevant steps are made explicit in the enforcement-modification rate within 0.5487–0.5613, and no plan, including typed prerequisites required by operational practice [5], [4]. run loses any task coverage relative to its raw proposal. Because the official provider’s evidence concentrates on III. P OLICY M ODEL AND V ERIFIER approval-gated rules, the artifact separately exposes public activation checks for mandatory and ordering repairs. This To make policy compliance measurable, we explicitly model separation makes scope explicit and provides a stable reference policy enforcement. The verifier is a deterministic, typed repair point for later comparisons: replace the prompt, catalog, map- engine over action sequences that encodes action-level policy ping contract, rule file, provider, or verifier, and re-evaluate invariants for proposed plans. Given the incident context and against a declared object rather than a moving target. Figure 1 a proposed plan, the verifier produces an enforced plan with summarizes the resulting evaluation contract. structured violation traces. Throughout the paper, a policy set denotes the complete declared rule collection for the reported II. T HREAT M ODEL evaluation, and a rule refers to a single entry in that set. LLM assistance introduces a pre-execution risk: the model Policy is represented by three rule families: mandatory, may recommend actions that conflict with policy, such prohibit_before, and require_approval. In the curas restoring a host before forensic analysis or suggesting rent implementation, the rule scope is at the incident and action approval-gated actions without authorization. Such violations levels; role and multi-entity constraints are not included. The
2
global policy surface is defined by the declared action catalog TABLE I D ECLARED ACTION - POLICY SURFACE FOR THE OFFICIAL EVALUATION . and typed rule set, whereas observed rule activation depends on how these artifacts intersect with the fixed corpus. Approval Paired The reported rules encode SOC governance points where Action baseline Rule treatment gate policy treatment is operationally consequential. R1 captures Collect forensics no yes R2 declared; inactive mandatory containment for a high-confidence reverse-shell Isolate host yes yes R1 declared; R4 fires pattern, not generic command execution, to avoid treating Reset admin no yes No policy rule ambiguous administrative commands as containment failures. credentials egress no no No rule R2 encodes evidence preservation before recovery. R3 and R4 Block address encode approval control for recovery and containment actions Restore host yes no R2 declared; R3 fires that can change system state, interrupt business operations, or destroy forensic opportunity. These choices reflect the incident-response and playbook literature’s emphasis on evi- under two applicable rules: R2 requires forensic collection dence preservation, organizational coordination, and governed before restoration, and R3 marks restoration as approvalrecovery paths in real SOC work [1], [5], [4], [3]. The gated. Pass 2 inserts collect_forensics and records official corpus exercises approval governance more strongly a prohibit_before violation; Pass 3 applies the conthan mandatory or ordering repair, since R1 and R2 require figured approval treatment to host restoration, records telemetry and sequencing patterns absent from the released approval_required, and, under the primary reported slice. This activation pattern is a corpus property; public mode, removes the restoration action. The enforced plan activation checks exercise the same verifier semantics over a retains only the forensics step, and the verifier emits richer candidate rule surface. two typed traces, R2/order_violation (insertion of Operationally, each rule asks one well-formed question collect_forensics) and R3/approval_required about the proposed action sequence. A mandatory rule asks (removal of restore_host), at the same level of evidence whether an action must be present when telemetry matches a recorded for every official run. declared scope. A prohibit_before rule asks whether a The procedure is finite, bounded, and deterministic under prerequisite action appears before a state-changing action. A fixed inputs. Four implementation properties are checked by require_approval rule asks whether an approval-gated design: termination over finite action sequences, bounded action has explicit approval evidence in the public canonical repair work under the fixed rule set, idempotence after enpackage. The verifier answers these questions by minimal forcement, and deterministic arbitration through a fixed pass repair: it inserts missing mandatory actions, inserts missing order. Mandatory and approval-gate rules are applied once prerequisites before constrained actions, and removes unap- per pass, whereas ordering constraints are repaired to a fixed proved approval-gated actions in the official mode. The alter- point after fail-fast checks reject self-dependencies and cycles. native approval treatment, defer_to_human_approval, As a consistency check on the released policy, reapplying is reported as a sensitivity path. Each repair emits a typed enforcement to already-enforced plans produces no further violation record. The verifier deterministically transforms the modifications within the active rule subset (R3, R4). This proposed action list into the policy-treated sequence used for places the reported system in an intentionally narrow and measurement. auditable region of the design space: deterministic repair over Table I surfaces the declared action vocabulary and typed action traces rather than global satisfiability solving over the rule treatment attached to each action. The ac- a richer workflow language. tion catalog contains five actions spanning forensics, IV. V ERIFICATION A RCHITECTURE containment, credential reset, network blocking, and reA system-level question remains beyond the policy framecovery. R1 uses a narrow reverse-shell scope predicate (event_type_contains=command_execution, work: where to place the verifier relative to private incident command_contains=[bash -i,/dev/tcp/]); the re- handling and the public evaluation path. Figure 1 depicts that leased official corpus contains no matching telemetry, so R1 flow. From left to right, raw exports and anonymization remain contributes declared verifier semantics but no provider-run private, canonical packages cross the release boundary, and both LLM proposals and paired human baselines follow the violations. The verifier expects a structured list of action identifiers. In same public verifier path. The result is a single enforcement LLM mode, the adapter requires JavaScript Object Notation layer applied to two proposal sources, with the release bound(JSON) with the recommended_actions field. Out-of- ary positioned after anonymization rather than after model catalog actions are filtered before policy enforcement and execution. The verifier is implemented as a rule engine, not as another logged in the artifact outputs. This contract constrains syntax at the verifier boundary, not semantic appropriateness. Catalog- LLM. Three designs were considered. Prompt-only guidance valid but operationally mismatched actions remain part of the keeps compliance in model and avoids a separate enforcement layer, but produces no repeatable audit trail and is sensitive measured policy and utility burden. Consider an input plan that contains only restore_host to prompt drift across provider versions or model updates. An
3
LLM judge introduces a structured verdict and a separate audit- Response (EDR) alerts into a Security Orchestration, Auing model, but replaces deterministic enforcement with a sec- tomation, and Response (SOAR) case-management platform. ond, non-deterministic decision layer, complicating cross-run This platform embeds playbook logic inside incident records, comparisons and the audit of the verdict itself. Deterministic so exports contain both detection evidence and workflow rule enforcement keeps the enforcement decision outside the traces. In practice, incident-specific artifacts provide telemetry, model, produces a typed violation trace for every run, and is in- case metadata, and mapped human-action traces, while global sensitive to provider drift under fixed policy artifacts. We chose artifacts provide the action vocabulary, policy semantics, and the deterministic verifier for these repeatability and auditability the mapping contract used to interpret those traces across the properties, and for the operational simplicity it provides in a corpus. declared evaluation state. Solver-backed enforcement remains Each raw export is a closed incident case authored by a a relevant extension for richer policy semantics. This approach human analyst within the SOAR platform: every selected case aligns with the monitor-based enforcement tradition in security in the corpus reached phase_id = "Complete" with at policy research, where deterministic monitors serve as the final least one analyst-executed task on record. The export cardecision layer [17], [18], [19], [20]. ries three workflow signals: playbooks and action menus The system-boundary decision is central to the contribution. (template references), extracted_tasks (analyst-executed In our setting, the assistant does not execute tools directly, work units recorded by the platform), and note/telemetry so the reviewed action sequence serves as the control point fields (incident evidence). The paired human baseline is reconshared among providers, analysts, and the public artifact. structed deterministically: each extracted_task is passed Deterministic enforcement makes that boundary versionable through the released task-to-action mapping contract, which and auditable even as provider behavior or prompt wording projects it onto the same canonical action catalog the LLM is constrained to use. The LLM arm and the human arm enter changes upstream. The private/public split is determined by data governance the verifier under one shared action vocabulary derived from requirements. In the case-study SOC, raw incident exports the same incident package; they differ only in how the action cannot leave the institution, in accordance with formal process sequence was authored (LLM proposal vs. recorded analyst and legal-governance constraints for incident handling [5], execution). playbooks and platform action menus are not [4]. The anonymizer is therefore the sole bridge between used to construct the baseline; they appear in the export only private data and publishable artifacts. Placing the boundary as contextual metadata. Mapping completeness is validated by after anonymization ensures that the verification and evalu- parseability, coverage, and unmatched-task reporting, not by ation pipeline remains public and artifact-evaluable, without enforcing a minimum task count. The fixed corpus consists of 200 real incidents selected exposing operational data. from a production SOC export under three fixed inclusion V. C ORPUS AND R ELEASE B OUNDARY criteria. The unit of analysis is a single closed incident case Raw exports and irreversible mappings remain local. Public in the SOAR case management platform. Each selected case artifacts are generated only after anonymization and privacy had to be closed, expose structured extracted_tasks checks. The released artifacts are authored policy inputs, that support deterministic task-to-action mapping, and prenot verbatim copies of incident traces. Three release gates serve cross-category diversity rather than collapsing into a make the corpus usable for public review: privacy scanning, single alert type. This is a purposive measurement corpus, parseability checks, and deterministic task-to-action mapping. not a prevalence sample of all SOC alerts: the design priThe full corpus is audited for coverage diagnostics and oritizes per-incident measurement depth and cross-category candidate gaps. In the current 200-incident audit, this process diversity over prevalence estimation. The released slice spans yielded no new action-catalog candidates and one corpus-wide 10 incident categories (access=7, authentication=4, exploit=19, approval gap (require_approval(isolate_host)), falcon detection method=4, malware=76, policy=3, potenwhich was added to the active policy only after manual tial exploit=4, recon=2, suspicious activity=73, system=8) review. The public package contains canonical incident pack- and exhibits operationally realistic severity variation across the ages, active global artifacts, the protocol manifest, non-private low-through-critical range observed in the source SOC. analysis outputs, and the code, scripts, and runbooks needed Each incident contributes an instrumented case package: to rerun integrity checks, dataset audits, and public assessment telemetry, case metadata, mapped human actions, and the evirechecks. It excludes raw exports, private-stage logs, and dence needed to audit conversion quality. This package enables irreversible local mappings. Each per-incident public object the public verification path without exposing raw operational contains telemetry (detection events, labels, and indicators), data. The corpus should therefore be read as 200 instrumented anonymized case metadata, deterministically mapped human evaluation cases from one anonymized institution, not as a actions, and quality traces that enable auditing of the conver- claim about that institution’s full incident distribution or actionsion. space coverage. Each case also incurs paired human-baseline The corpus and canonicalization pipeline originates in a construction, independent LLM plan generation across mulproduction SOC that ingests correlated Security Information tiple model and prompt-arm combinations, deterministic poland Event Management (SIEM) and Endpoint Detection and icy verification, and multi-metric evaluation. This depth-over-
4
breadth tradeoff is consistent with controlled experimental The design separates proposal variability from enforcement designs in security measurement [21]. variability. Differences across models, arms, and reruns are The abstraction is intentional. Playbook studies show that part of the measured phenomenon and are reported as such, operational response procedures combine generic technical whereas enforcement variability is expected to collapse to steps with organization-specific context, escalation paths, zero under fixed inputs and policy artifacts. Rerun stability and local constraints, whereas incident-response field studies reflects the declared evaluation object rather than the universal demonstrate that legal, coordination, and workflow pressures determinism of providers. In the language of applied replicabilshape what can safely be done in a case [5], [4], [3]. The ity, allowed variability belongs to the hypothesis, while fixed canonical package preserves the evidence needed to evaluate elements define the protocol [21]. one incident-level plan under a declared action contract, without inferring unshared context from guessed metadata. That TABLE II D ECLARED EVALUATION OBJECT FOR THE OFFICIAL STUDY. discipline keeps the released corpus aligned with the public claim surface rather than silently treating the private workflow Component Frozen value context as observable. 200 canonical incidents Raw incident exports are processed outside the public repos- Public corpus 200 incidents itory by a local private conversion stage that tokenizes sensitive Official slice Model set 2 providers fields, emits a canonical package, and records conversion Prompt arms 2 arms metadata. Before any public artifact is released, the pipeline re- LLM trajectories 800 planned; 800 completed 200 paired places email addresses, Internet Protocol (IP) addresses, phone Human baselines 3 reruns numbers, hostnames, and user identifiers. Anonymization fol- Stability reruns Active rule slice approval gates (R3, R4) lows a dual-review protocol: the SOC operator performs initial sanitization and review in the operational environment, and the Table II clarifies the measurement object, since the paper research team independently reprocesses the exported records evaluates a declared evaluation state rather than a free-form through automated anonymization with personally identifiassistant deployment: the incident slice, global artifacts, model able information (PII) scanning, forbidden-term checks, and set, and active comparison arms are all fixed before any manual verification. This two-person, two-toolchain process outcome is counted. reduces the risk of residual identifiers surviving either stage In the current evaluation slice, the exercised rules are alone and makes the public release possible without exposing the approval-gated actions R3 and R4, while R1 and R2 raw institution-internal incident data. remain defined but inactive. Results should therefore be read The primary risks are privacy leakage through logs, prompts, as measurements over approval-governed behavior under the or snapshots; run contamination from stale artifacts; and declared protocol, not over the full typed rule surface. non-deterministic preprocessing that could weaken traceability Two reporting rules follow from this freeze. First, topand public verification. Mitigations include the dual-review line violation rates carry meaning only in the context of protocol described above, PII and forbidden-term scanning, the actually exercised rule slice and its repair distribution. A strict dataset audit, release-boundary checks, and conversion low rate under approval-only activation is not equivalent to manifests with quality metrics. The same release boundary a low rate under a richer policy surface that also exercises also defines the public evidence surface for the paper: canonmandatory insertion or prohibit_before repair. Secical incident packages for the active evaluation state, the ond, paired-baseline deltas remain comparable only as long as declared action catalog and typed policy set, the task-tothe mapping contract, action catalog, and verifier semantics are action mapping contract used to derive paired human baselines, preserved. In measurement terms, these conditions define the the manifest, shipped non-private analysis outputs, the public run-accounting manifest for the reported official evaluation, hypothesis boundary, not ancillary implementation detail [21]. Primary outcomes are run-level violation rate (fraction of and the code and runbooks required to validate the released runs containing any policy violation), hard-violation burden boundary. This declared surface is what artifact evaluators and (hard violations per run, measuring within-run edit depth), future benchmark authors should treat as the checkable public and enforcement-modification rate (fraction of runs where the object of the study. verifier modifies the proposed plan, estimating safety-layer VI. E VALUATION D ESIGN intervention burden). Secondary outcomes are task coverage (recall against the paired human baseline after enforcement) Given the fixed corpus and global artifacts described above, and ∆Jaccard (change in action-set overlap after enforcement; the evaluation unit consists of one incident, one model, one positive values indicate movement toward the paired human prompt arm, and one repeat; each incident also contributes baseline). one paired human baseline, evaluated using the same verifier and policy set. The active evaluation combines 200 incidents, 2 providers, 2 prompt arms, and one repeat, yielding 800 A. Provider Execution For the active evaluation, LLM generation is minimally planned LLM trajectories and 200 paired baselines. Table II constrained. Policy enforcement is applied post-generation consolidates the declared object end-to-end.
5
so that measurements capture raw policy-compliance risk in B. Paired Baseline and Endpoints proposed plans, rather than the effect of upstream filters. Human incident actions define the paired human baseBoth arms share the same copilot contract: the model is a line for comparing LLM outputs on the same incidents. In non-autonomous proposal component that returns a bounded the private stage, baseline actions are derived from SOAR action trace for analyst review, not an executable procedure extracted_tasks via a deterministic task-to-action mapor an approval authority. The user message contains incident per with hash-tracked rules, producing ordered action idenmetadata, a bounded telemetry sample, the action catalog with tifiers in the public canonical package. This yields a deterapproval and reversibility metadata, an operational-rules block, ministic workflow-level reference for overlap, burden, and and the required recommended_actions JSON schema. compliance comparisons without claiming a full host-level The zero-arm leaves the operational-rules block empty; the execution trace. Thus, the paired human baseline is a mapped policy-arm fills it with one line per active rule, including workflow reference under the same action contract, not a rule_id, family, target action, severity, a prerequisite when ground-truth execution trace. The action catalog, typed policy applicable, repair operator, and a concise policy rationale. No set, and task-to-action mapping rules remain authored inputs provider receives a different policy surface. The only provider- before execution; after freeze, canonicalization checks, LLM specific formatting occurs at the application programming plan generation, deterministic verification, metric computation, interface (API) level: OpenAI-compatible endpoints receive and artifact rendering are automated. the system instruction as a chat message, whereas Anthropic Two truth anchors are used: the paired human baseline receives the identical system text in the provider’s system under the fixed-mapping contract and the verifier’s decisions field. under the reported policy rules and action catalog. The primary endpoint is compliance under the declared policy, Recommended actions must resolve cleanly to catalog ac- not host-level outcome truth or proof of policy optimality. tion identifiers via the JSON output contract. Suggestions The primary enforcement mode for approval-gated actions that do not resolve, such as synonyms, composite labels, or is remove. Incident-scoped approval evidence can be used obfuscated names, are logged as out-of-catalog and excluded when available, but in this corpus, approval annotations for from enforcement inputs rather than being coerced into un- mapped human actions are drawn from the task-to-action certain semantic matches. In the official execution lineage, contract rather than from host-level approval logs. Therefore, per-run prompt messages and prompt hashes are stored. The the endpoint reflects compliance in the proxy-approval context, public artifact exposes the arm logic, model registry, mapping and defer_to_human_approval remains available as an contract, arm-to-prompt mapping, and shared prompt template explicit sensitivity path, not the reported primary endpoint. needed to audit the setup without rerunning provider calls. C. Metrics and Statistics The official cross-provider evaluation runs on 200 incidents under identical settings. The evaluation-facing model pair is gpt-5.2 and claude-sonnet-4-6. All official runs use deterministic decoding (temperature=0.0, max_tokens=512). Provider-declared model identifiers are recorded verbatim in run manifests and tables, so the paper reports the exact API labels used at execution time rather than retrospectively normalized names. The provider application programming interfaces do not expose fixed-seed control, so cross-run comparability is anchored by the declared evaluation state, manifests, and full run capture rather than by seed locking. To bound remaining serving-layer variance, the full fixed 200-incident corpus was rerun 3 times under the same protocol, and the resulting stability range is reported in Section VII.
Violation outcomes are classified by policy-rule type and severity. Paired deltas in run-level violation rate and hardviolation burden (hard violations per run) are computed between LLM outputs and paired human baselines per incident. The verifier effect is measured as the reduction in hard-rule risk versus action preservation. Statistical reporting uses Wilson confidence intervals [22], paired proportion tests, McNemar for matched incidents [23], Holm correction for multiple comparisons [24], and Cohen’s h as a descriptive magnitude summary for marginal rate separation [11], following replicability-oriented reporting practices in modern security experimentation [21]. Beyond the binary run-level endpoint, per-run violation counts, severity counts, violation types, edit operations (insert, remove, reorder, defer), and action-overlap deltas versus paired human baselines are recorded, ensuring aggregation does not hide within-run edit burden. Inferential statistics are limited to contrasts among observed model/arm cells in the declared evaluation state, not to population-level claims across SOCs or policy spaces. The declared manifest fixes the policy rules, mapping rules, prompt templates, model identifiers, and primary outcomes; any changes must be reported as separate evaluation objects. To ensure traceability, mapping-rule Secure Hash Algorithm 256-bit (SHA-256) values are recorded in source manifests, prompt/input hashes are stored, and the paper’s tables and figures are rendered from the
Hard violations are rule violations labeled hard in the policy set. In this study, these correspond to operationally unsafe recommendations such as missing mandatory containment steps, violating ordering constraints, or proposing approvalgated actions without approval context. The hard-violation rate is the fraction of runs that have at least one hard violation under the proposed plan. Because all active rules are labeled hard, the binary hard-violation rate coincides with the binary run-level violation rate, and evaluation tables also report hard violations per run as the count-normalized burden metric.
6
maintained analysis bundle and aggregate summaries shipped in the public artifact. Execution begins only after privacy, parseability, mapping quality, and manifest checks pass. The active 200-incident configuration contains one LLM trajectory for each incident, provider, and prompt arm, plus one paired human baseline per incident. This design gives 800 planned LLM trajectories and 200 paired baselines. The realized evaluation completed 800 LLM trajectories, with 0 execution failures. Run accounting is preserved in the manifests, but it does not carry the compliance claim.
The pooled screening view is reported only as a protocollevel check, since the two providers move in opposite directions under policy prompting. Across both direct-provider models, the pooled arm rates are 0.4500 for llm_zero and 0.6700 for llm_policy_prompt, while the paired human baseline has no violating runs (0/200; 95% confidence interval [0, 0.0188]). These pooled counts provide context but do not constitute a separate substantive claim, as the providerlevel decomposition below drives the main interpretation. Two aggregate signals remain stable: violations are concentrated in approval-gated rules, and the stored plans contain no outof-catalog actions under the reported output contract. This rules out syntactic catalog escapes in the stored plans, but not catalog-valid actions that may still misalign with the paired baseline.
VII. R ESULTS
The released corpus of 200 real incidents supports end-toend public audit and reruns at the released boundary. The main finding of the official evaluation is provider-specific within the approval-governed slice: under identical protocol B. Provider-Level Compliance (RQ1) conditions, prompt-level policy text improves compliance for We then report the official 200-incident evaluation. Under gpt-5.2 and degrades it for claude-sonnet-4-6, while the fixed policy surface, the policy arm produces a borderlinethe resulting burden remains concentrated in approval-gated detectable improvement for gpt-5.2 (Cohen’s |h| ≈ 0.14) actions within the approved catalog. The evidence is therefore and a large degradation for claude-sonnet-4-6 (h = strongest for the approval-governed slice, where the observed 1.12). Across providers, all observed official violations fall in verifier action is removal, not insertion, reordering, or deferral. approval-gated rules, while deterministic enforcement reduces We begin with corpus release checks, then localize the main risk without reducing task coverage. The official run attempted findings across provider-level rates, rule concentration, utility, 800 LLM trajectories and completed 800 (0 preflight failures; and stability. 0 execution failures). The rule semantics are stated once in Table I. The ofA. Corpus Release Checks ficial evaluation exercises only the approval-centered slice: Corpus release quality is defined by the released conversion R1 and R2 remain inactive, and observed enforcement is gates rather than by workflow completeness in the private removal rather than insertion, reordering, or deferral. Figure 2 environment. Across all 200 incidents, privacy and parseability shows provider-level run-level violation rates (lower is better). checks passed, task extraction remained valid, and task-to- The provider-level effect of policy prompting is asymmetric action mapping achieved complete released-surface coverage rather than a small fluctuation around a common trend: the (1147/1147) with zero unmatched tasks, zero ambiguous same inline policy text increases violation prevalence for matches, and zero audited privacy issues. Under the declared claude-sonnet-4-6 while decreasing it for gpt-5.2. protocol, these gates establish that the public corpus is fit for Table III and Table IV report per-cell rates and paired paired evaluation and reproducible reruns: the measurement contrasts (n = 200 per model/arm cell). All six paired object is well-formed, auditable, and free from conversion- contrasts survive Holm correction, but effect sizes vary widely. induced distortion at the released boundary. The three contrasts involving claude-sonnet-4-6/policy The same release evidence also bounds the claim. Mapper and the cross-provider zero-arm contrast (claude—zero vs. coverage over the full catalog is 0.6000 (3/5 catalog gpt-5.2—zero) carry the largest separations. The remaining actions), reflecting human-task distribution rather than full two contrasts (claude—zero vs. gpt-5.2—policy and the action-space coverage. Paired-baseline support concentrates within-gpt-5.2 arm contrast) survive correction with effect in three actions: collect_forensics carries the deepest sizes near |h| ≈ 0.14, a magnitude small enough that the withinsupport (200 cases, 765 matches, of which 0.7477 of unique GPT effect should be read as borderline-detectable rather than matches are single-keyword); isolate_host (191/191) substantively important. The figures and tables below quantify and reset_admin_credentials (191/191) carry these claims. stronger multi-keyword support; and block_egress_ip, restore_host have no paired support in the reported slice. C. Rule Concentration and Enforcement (RQ2, RQ3) Approval-proxy support is similarly narrow: isolate_host Together, these results separate proposal-level variabilis covered in the mapping contract, whereas restore_host ity from enforcement-level stability. Figure 2 captures the remains approval-gated without paired baseline support. The provider split in violation prevalence, while the outcome tables released artifact therefore supports a specific claim about show that deterministic enforcement improves precision in auditable policy enforcement over a fixed action interface, every model/arm cell and preserves task coverage. Rule control not a claim that the released slice exhausts the institution’s follows directly from Table I: in the official provider run, operational action space. approval rules R3 and R4 account for all observed violations,
7
Claude / zero
D. Rerun Stability
0.36 ≈0
Three observations summarize the official evaluation. First, the private-to-canonical conversion passed the parseability and ≈0 privacy gates for all 200 incidents, and the task-to-action 0.54 GPT / zero ≈ 0 mapping achieved 1147/1147 with weighted coverage 1.0000. Across the official evaluation, run-level violation rates were 0.47 GPT / policy 0.3600 and 0.8700 for claude-sonnet-4-6 (zero vs. pol≈0 icy) and 0.5400 and 0.4700 for gpt-5.2. In the declared 0 0.2 0.4 0.6 0.8 1.0 evaluation state, prompt-level policy text does not uniformly violation rate (lower is better) improve compliance across providers. raw proposal after verifier Second, observed violations were concentrated entirely in approval-gated rules: R3 produced 431 violations and R4 Fig. 2. Provider-level violation rates before and after deterministic verification. produced 35, with no out-of-catalog actions under the reported Raw proposals diverge under identical policy text; post-verifier rates collapse output contract. Deterministic enforcement modified 0.5600 toward zero because the official violations are approval-gated. of LLM runs and removed 466 approval-gated actions; no run lost task coverage relative to its raw proposal, and the average ∆Jaccard after enforcement remained positive in all while the mandatory and prohibit_before rules never four model/arm cells. The dominant observed failure mode is activate. non-compliant action selection within the approved catalog, The concentration is specified by the protocol rather than not catalog escape. Third, action disagreement with the paired human being incidental. In the reported slice, approval rules bind to catalog actions proposed by both providers, whereas R1 and baseline was directional: LLM-only actions were domiR2 require conditions absent from the released cases. The nated by restore_host (431) and block_egress_ip observed burden results from this intersection: the incident (90), while baseline-only actions were dominated by mix activates approval-gated actions in the approved catalog, reset_admin_credentials (515). Across 3 completed while mandatory and prohibit_before never receive repeated summaries of the fixed 200-incident corpus, the aggregate picture remained stable: the incident-violation rate the telemetry or sequencing evidence needed to fire. The tables anchor these visual patterns to exact per-cell stayed within 0.8700–0.8750, the enforcement-modification rates and paired contrasts. Execution diagnostics reveal the rate within 0.5487–0.5613, no run lost task coverage in any operational decomposition of the repair burden: most removals rerun, R3 stayed within 422–435 violations, and R4 held fixed are restore_host approvals (R3), with a smaller, stable at 35. Across providers and reruns, the same pattern holds: isolate_host approval component (R4). Across the four provider-level policy prompting and action-level policy adherprovider/arm cells, all 800 planned LLM trajectories com- ence diverge under the declared protocol, and deterministic pleted, and the stored official plans contain no out-of-catalog verification remains the last stable pre-execution control point over governed actions in the reported evaluation. action identifiers. The discussion below interprets this as a systems result: Table VI decomposes overlap into raw and enforced SOCpilot makes plan-level policy compliance measurable at precision, showing that deterministic enforcement conthe action boundary and shows that deterministic enforcement sistently improves precision, while Table III confirms is the stable control surface under the reported provider task coverage holds. The remaining utility gap is direcvariation. tional rather than diffuse. LLM-only actions concentrate VIII. D ISCUSSION in restore_host (431) and block_egress_ip (90), with smaller contributions from isolate_host (35) and SOCpilot changes the deployment question from “Did the reset_admin_credentials (14); baseline-only actions prompt mention the policy?” to “Does the reviewed action concentrate in reset_admin_credentials (515) and trace satisfy the policy?” The evaluation shows why this isolate_host (43). change matters. Under one declared prompt lineage and one The action-level anatomy explains why the approval slice is shared incident package, prompt-level policy text did not operationally important despite its narrow rule coverage. The yield stable action-level compliance across providers, whereas repair burden is driven primarily by recovery: R3 accounts deterministic verification of typed action sequences did. The for 431 of the 466 removed approval-gated actions, whereas implication for system design is that the reviewed action trace, R4 accounts for 35. Enforcement improves overlap mainly not the prompt, is the enforceable boundary. by removing excess approval-gated recovery proposals, with The official provider evidence is concentrated in approvalcontainment contributing a smaller but still policy-relevant governed actions (RQ1, RQ3). In the fixed corpus, R1 depends share. The block_egress_ip disagreement affects utility on a high-confidence reverse-shell signature absent from the but not policy violation under the current rule set, since the released incident slice, while R2 requires restoration-beforeaction is catalog-valid and not approval-gated in this freeze. forensics patterns not observed in the paired baselines or Claude / policy
0.87
8
TABLE III O FFICIAL EVALUATION OUTCOMES BY MODEL AND ARM . T HE SHADED ROW IS THE PAIRED HUMAN BASELINE ; BOLD MARKS THE HIGHEST VIOLATION RATE .
Provider / arm
Runs
Violation rate
95% Wilson interval
Hard violations/run
Verifier edit rate
Task coverage
∆Jaccard
Claude / zero Claude / policy GPT / zero GPT / policy Human baseline
200 200 200 200 200
0.3600 0.8700 0.5400 0.4700 0
[0.2967, 0.4286] [0.8163, 0.9097] [0.4708, 0.6077] [0.4020, 0.5391] [0, 0.0188]
0.3650 0.9100 0.5650 0.4900 0
0.3600 0.8700 0.5400 0.4700 0
0.7567 0.7634 0.8100 0.7400 1.0000
0.0678 0.1544 0.1092 0.0917 0
TABLE IV O FFICIAL PAIRED CONTRASTS BY MODEL AND ARM . H OLM - ADJUSTED M C N EMAR TESTS AND C OHEN ’ S h SUMMARIZE MATCHED RATE SEPARATION . Contrast
n
∆ rate
McNemar p
Holm p
Cohen’s h
Claude policy vs Claude zero Claude policy vs GPT policy Claude policy vs GPT zero Claude zero vs GPT policy Claude zero vs GPT zero GPT policy vs GPT zero
200 200 200 200 200 200
+0.5100 +0.4000 +0.3300 −0.1100 −0.1800 −0.0700
<0.0001 <0.0001 <0.0001 0.0003 <0.0001 0.0056
<0.0001 <0.0001 <0.0001 0.0006 <0.0001 0.0056
+1.1169 +0.8931 +0.7530 −0.2238 −0.3639 −0.1401
pressure is highest. Within the evaluated protocol, SOCpilot is a bounded proposal engine plus a deterministic policy layer. The copilot proposes catalog actions; the verifier enforces typed approvals and ordering constraints before analyst review; the public/private split defines exactly what outside auditors can rerun without crossing the institutional boundary or rerunning provider calls. Two public checks keep richer semantics separate from the official estimate. First, a richer-policy activation check evaluates a candidate surface over the released corpus. It scans 200 incidents, defines 14 candidate actions and 18 candidate rules, and exercises all supported repair modes: insert, insert_before, remove, and defer_to_human_approval. The ordering probe identifies three deterministic repairs in a single chain. This activation evidence, summarized in Table V, shows that richer mandatory and ordering behavior is checkable and not pooled into the official provider estimate. Second, the public-metric approval sensitivity check treats the 466 removed approval-gated actions as deferred rather than suppressed. Deferral preserves task coverage across all models/arm cells but lowers precision and Jaccard scores relative to removal, as expected when proposed actions are retained for analyst approval.
official LLM trajectories. This coverage result is part of the measurement: the paper reports which rule families generated provider evidence, and the artifact separately exercises richer repair modes outside the official estimate. The released artifact matters because it fixes what may vary across reruns and what must remain invariant when later studies claim improvement. The main lesson from the data is that non-compliant action selection happens within the approved catalog. Out-of-catalog actions remain at zero under the reported output contract, yet provider behavior still diverges under inline policy text. The opposite prompting effects are informative: policy text can shift proposal distributions, but it does not guarantee action-level compliance. A degraded provider cell under inline policy text demonstrates that prompt-level policy exposure and action-level policy compliance are distinct. Conversely, zero out-of-catalog actions establishes syntax control, not operational adequacy. The measured burden remains inside the approved catalog, where governance-sensitive choices, such as restoration and containment, still require typed control before execution. The protocol supports a concrete systems claim. Promptlevel policy text can shift proposal distributions, but the reviewed action trace is the enforceable boundary. The verifier is not an accessory to generation or a cleanup stage; it is the pre-execution control point once proposal variability reaches the action trace. Operationally, approval concentration is central. Incidentresponse playbooks and studies of real SOC practice emphasize that containment and recovery steps are shaped by organizational coordination, approval paths, and external constraints, not just by detection evidence [5], [4], [3]. Actions such as restoring a host or isolating it prematurely can alter the system state, disrupt business workflows, or compromise forensic evidence. The activated slice sits where governance
TABLE V P OLICY COVERAGE SEPARATES THE REPORTED PROVIDER ESTIMATE FROM RICHER - RULE ACTIVATION CHECKS . Surface Official evaluation Richer-rule check
Actions Rules Activated 5 14
4 18
Role
R3, R4 provider estimate 17 candidate rules activation check
The verifier is also a systems component in a narrower sense than generic agent-safety layers. The monitor-based enforcement tradition establishes that policies over event se-
9
quences can be enforced by automata that suppress or edit expose seed control. What is stable is what the protocol fixes. executions [17], [18]; Progent and AgentSpec instantiate this Releasing the catalog, policy file, mapping rules, run manifest, idea for LLM agents by intercepting tool calls and evaluating and paired-baseline bundle enables a reader to distinguish declarative constraints before execution [9], [10]. SOCpilot a stronger system from a weaker policy surface in future shares the architectural instinct of enforcement outside the comparisons. generative core, but applies it to a different governed object: IX. T HREATS TO VALIDITY the reviewed action plan submitted to an analyst for deciTask-to-action mapping is the primary construct validity sion support. The plan-level action sequence is the surface shared across providers, paired against a human baseline, and surface, as mapping errors can bias downstream overlap and releasable for public audit. That surface is why the action burden estimates. The paired human baseline is derived from boundary, rather than the tool-call boundary, is the stable analyst-authored SOAR workflow records, not newly elicited study responses. In the released audit, coverage is complete: measurement point under the declared protocol. The paired human baseline is a compliance anchor. It there are 0 unmatched tasks and 0 ambiguous ties across ties the same incidents to the same mapping contract and 1147 mapped tasks. The remaining mapping evidence is lexical catalog, so omission, overlap, and disagreement can be read rather than tie-based: 579/1147 unique matches rely on singleas workflow-level compliance phenomena. The substantive keyword support. This dependence is localized: it concentrates comparison is of proposal paths evaluated against a paired in collect_forensics, while isolate_host relies entirely on multi-keyword support in the released manifests. analyst-authored baseline under a single verifier. The same method admits richer declared evaluation objects The paired human baseline is therefore a workflow-level that activate additional rule families, expand the approved compliance reference grounded in recorded analyst work. action space, and preserve a public evidence path that outside The artifact keeps the mapping surface inspectable through auditors can rerun without private access. Examples include per-incident mapping-support manifests, a stratified mappingmandatory insertion, ordering repair, approval deferral, and inspection packet, and the global assessment bundle, without multi-step dependencies, such as forensic collection before redistributing private task text or requiring new humans to restoration and before reconnection. The method also supports construct the baseline. Utility metrics are most informative jointly. A zero taskcross-institution and cross-sector case studies: another SOC can replace the canonical cases, declare its own rule subset, coverage drop means deterministic enforcement did not reand reuse the verifier and the paired baseline. The reporting move baseline-covered steps beyond those the raw LLM rule follows from the method: changed inputs, policy surface, had already omitted. Table VI shows that enforcement imor verifier behavior define a new evaluation object rather than proves precision in all model/arm cells. The remaining gap a silent extension of the current estimate. In the language of is directional rather than diffuse: extra recovery and egressapplied replicability, later studies should widen the bounds of block actions dominate disagreement with the paired huthe same hypothesis rather than quietly replace it with a new man baseline. Zero out-of-catalog rate establishes syntactic contract compliance, and the policy burden measured here one [21]. For this reason, the current paper reports a single, well- is catalog-valid but governance-sensitive action choice. The scoped result rather than a generic benchmark leaderboard. It alternative defer_to_human_approval mode remains shows that, under the released protocol, plan-level enforcement implemented, whereas the official study reports remove as is the stable control surface even when prompt-level guidance the primary mode because it yields the cleanest repair trace and is not. Scenario diversity strengthens the research program the least workflow ambiguity in the public artifact. The public when introduced as an explicitly richer evaluation object, sensitivity report shows the expected trade-off: deferral preserves coverage while also allowing more LLM-only actions rather than as a silent change pooled into a single estimate. The reported evaluation changes how plan-level compliance for analyst review. The official provider runs exercise removal, should be stated. Prompt-level policy text and action-level and the artifact separately exercises insertion, ordering, and policy compliance are different properties: under one declared deferral through public checks. prompt lineage, the same inline policy block changed Claude’s Approval evidence is interpreted at the public evidence run-level violation rate from 0.3600 to 0.8700, whereas it boundary. The released package shows that an action is moved GPT only from 0.5400 to 0.4700. Reporting policy- approval-gated under the declared catalog and policy set; it compliance numbers without separating these layers risks does not expose institution-internal approval logs or latent attributing behavior to “the LLM” that depends on a specific supervisor context. Some historically executed actions may provider, prompt, and policy-text version. therefore have been acceptable in a private context that the The evaluation also shows why the declared object matters. public artifact cannot reveal. The reported evaluation chooses The deterministic verifier removed 466 approval-gated actions the externally auditable reading by design: without explicit across the official run without any run losing task coverage approval context in the public evidence surface, the verifier relative to its paired analyst-authored baseline. Across 3 re- treats the action as non-compliant. This preserves auditability runs, the same fixed corpus, catalog, and rule file produced and prevents the public claim from inheriting private assumprates within 0.8700–0.8750, even though the providers do not tions that outside readers cannot inspect.
10
A second construct validity surface is the abstraction loss be- boundary rather than attributing it to hidden mechanisms in the tween institution-internal playbooks and the released canonical model. Provider-side guardrails, instruction hierarchy handling, packages. Prior work on incident-response playbooks shows refusal style, and latent planning preferences may all conthat real response procedures are shaped by organizational tribute. The contribution is behavioral and operational: under actors, negotiation, and local context beyond what a single a single declared prompt lineage and a shared verifier path, incident record exposes [5], [4]. The public artifact omits that policy text alone did not yield stable action-level compliance private context so that the released evidence surface remains across providers. auditable. The paper measures compliance under the exported Rule frequencies and disagreement patterns depend jointly plan/evidence surface: exactly the object that reviewers can on the incident mix, the exposed action surface, and the inspect and future systems can reuse. approval evidence available in the released package. A corpus A third validity surface is protocol-surface compression. with more restoration-heavy cases, richer host-level approval The declared action catalog contains five actions, and the traces, or broader mandatory-rule activation would define a empirically exercised provider slice is approval treatment over different estimate. The present result is an auditable slice restore_host and isolate_host. This is the declared with declared boundaries, which is precisely what enables measurement object and the source of the paper’s official comparison and extension. Comparison must therefore be declared rather than silent: estimates. Richer mandatory chains, denser ordering dependencies, and a broader recovery vocabulary belong in the any change to the incident slice, mapping contract, policy file, next declared evaluation object, where lower burden can be prompt template, provider endpoint, or model identifier creates attributed only after the corpus, catalog, and rule surface are a separate evaluation object rather than a direct extension of the current estimates. fixed. X. R ELATED W ORK
TABLE VI U TILITY DECOMPOSITION BY MODEL AND ARM . P RECISION RISES AFTER DETERMINISTIC ENFORCEMENT IN ALL FOUR CELLS .
Model / arm
Raw precision
Enforced precision
∆Jaccard
Claude / zero Claude / policy GPT / zero GPT / policy
0.8268 0.6673 0.8071 0.8108
0.9454 0.9271 0.9792 0.9733
0.0678 0.1544 0.1092 0.0917
The asset_criticality is currently a proxy for incident severity copied during canonicalization, so severityand criticality-stratified summaries are contextual rather than explanatory. The verifier measures adherence to the reported policy set. Inconsistent rule sets (self-dependencies and ordering cycles) fail fast and are escalated rather than silently repaired. Under fixed rules and a fixed action catalog, the verifier acts as a deterministic finite repair transducer over action sequences. Our claim is therefore bounded and traceable: policy treatment over one declared surface, not a generalpurpose policy reasoner. External validity is anchored to the provider state and institutional setting. In the official evaluation, the two providers respond in opposite directions to policy prompting. Table IV shows the within-model Claude contrast surviving Holm correction with large marginal separation (Cohen’s h = 1.12), whereas the within-model GPT contrast also survives correction but at h = −0.14, an effect size small enough to be read as borderline-detectable rather than substantively important. The result is asymmetric and useful: policy text worsens Claude in the reported evaluation, while its effect on GPT is too small to be operationally meaningful. The lesson is architectural rather than provider-specific: policy-aware text is not the same as policy-conformant action selection. The design measures provider divergence at the action
11
Prior work addresses related aspects of this problem across four research lines, each of which stops at a boundary that SOCpilot crosses: workflow characterization, event-level detection, model capability assessment, and runtime agent control. No adjacent work combines a real SOC corpus, typed policy treatment, a paired human baseline, and a fixed public verification surface. SOC operations and governance. Interviews and field-study work examine SOC organizational factors. Kokulu et al. study how institutional alignment between security and business units shapes incident-response processes through qualitative interviews across SOCs of varying maturity [1], whereas Jones et al. document the operational pressures on endpoint management introduced by the COVID-19 work-from-home (WFH) transition [25]. At the alert layer, Vermeer et al. (AsiaCCS 2022) track network intrusion detection system (NIDS) ruleset evolution longitudinally, showing that a small fraction of rules accounts for the majority of alerts [26], and Yang et al. measure the distribution of true attacks, attack attempts, and benign triggers in four years of production traffic [27]. Vermeer et al. (CCS 2023) characterize the decision workflows by which SOC analysts triage alerts into actionable incidents [3]. Woods et al. further demonstrate that legal, insurance, and coordination pressures imposed by external stakeholders constrain which incident-response actions an organization can safely take [4]. Closest in spirit to SOCpilot, Schlette et al. measure the gap between documented playbook steps and the actions analysts actually select across a multi-SOC study [5]. However, none of these works provides a mechanism to enforce or verify compliance, nor do they produce a quantitative measurement of plan-level policy adherence against a paired human baseline under a fixed public evidence surface. Provenance, detection, and reconstruction. Event-level reasoning work addresses a different layer. Altinisik et al. and
Zhang et al. apply graph representation learning and task- operational software-defined networking (SDN) and networkguided segmentation, respectively, to search and detect ad- policy systems [19], [20]. Collaborative Automated Course vanced persistent threat (APT) behavior within system-audit- of Action Operations (CACAO) standardizes cybersecurity log provenance graphs [28], [29]. Dong et al. and Aly et al. playbooks as structured workflow objects with typed branching reconstruct attack stories from endpoint detection records [30], and parallelism [39]. SOCpilot applies the monitor intuition to [31]. These works answer what happened in an incident, not a new governed object: pre-execution incident-response plans whether the subsequent response plan satisfies organizational over a declared action catalog. The same typed verifier path policy constraints. is applied to both LLM proposals and paired human baselines, LLM assistance in cybersecurity tasks. Deng et al. build a with all results anchored to a fixed, publicly verifiable evidence penetration-testing agent and evaluate it on structured ex- surface. ploitation tasks [6]; Stafeev et al. extend the goal-directed XI. C ONCLUSION agent paradigm to task-driven web-application scanning [7]; Deng et al. (RACONTEUR) evaluate LLMs on securitySOCpilot establishes that policy compliance in LLMreport generation [32]; Wang et al. systematically benchmark assisted incident-response planning is a property of the proLLM security knowledge and multi-step reasoning against posed action trace, not of the prompt that generated it. Across a human analyst baselines on a domain-specific question set [8]. fixed evaluation object, identical policy text induces divergent CyberSecEval [33] provides suite-level capability assessment provider behavior, showing that policy-aware generation does at the prompt and code levels. All of these works measure not imply policy-compliant action selection. By moving enwhat a model can produce in an open-task setting; none forcement to a deterministic layer over typed action sequences, measure whether the proposed action sequence satisfies a SOCpilot turns compliance into a verifiable property at the typed, institution-specific policy when applied to real incidents plan-review boundary, independent of model internals, prompt using a fixed action catalog. formulation, or provider-specific behavior. This reframes how LLM serving-layer risk and runtime control. Measurement compliance claims should be made and evaluated: not as work on prompt leakage, prompt inversion, plugin trojaning, attributes of model outputs in isolation, but as properties and cross-app isolation identifies attack surfaces in LLMof a declared, auditable object that binds incident context, integrated deployments [13], [14], [15], [16]. On the defensive action vocabulary, policy semantics, and enforcement. For side, Shi et al. (Progent) introduce a programmable privilegeLLM-assisted security operations, the relevant contribution is control layer that intercepts tool calls before execution and whether the plan an analyst is about to approve satisfies the enforces security policies specified in a domain-specific langoverning rules under a fixed, inspectable protocol. guage [9], and Wang et al. (AgentSpec) provide a declarative constraint language for bounding autonomous agent E THICAL C ONSIDERATIONS behavior across planning steps [10]. Recent and concurrent This study uses real incident records from an anonymized systems broaden this design space: ToolGuard-style guard production SOC in a financial-sector case study. The primary generation compiles business-policy documents into tool-level ethical risks are privacy leakage, unintended disclosure of opchecks [34], Policy Compiler for Secure Agentic Systems erational procedures, and overclaiming beyond evidence from (PCAS) compiles authorization policies into an instrumented a single institution and a single declared policy/action space. agentic system with a reference monitor over dependency These risks are mitigated by enforcing a strict private/public graphs [35], VeriGuard synthesizes and formally verifies policy boundary: raw exports remain institution-internal, anonymizacode before runtime monitoring [36], ShieldAgent reasons tion precedes any public release, and the released artifact over action trajectories with verifiable policy circuits [37], contains only canonical packages and non-private analysis and LogiSafetyGen evaluates implicit regulatory compliance outputs. Anonymization followed a dual-review process, with in tool-invocation traces [38]. These systems reinforce the operational review before export and an independent researchsame architectural lesson: policy treatment should sit outside side rerun with PII scanning, forbidden-term checks, and model text alone, and they govern live agents, tool calls, manual verification. The study evaluates only pre-execution generated guard code, or synthetic tool-use traces. SOCpilot’s verifier governs a different object: the reviewed incident- plan compliance: the LLM does not execute tools; the verifier response plan, submitted by an assistant that does not execute acts before any response action is taken; and the paper limits tools directly. The action sequence is the only surface shared its strongest claims to the current approval-gated rule slice across providers, paired human baseline, and public artifact, rather than to universal cross-institutional safety. so enforcement operates pre-execution at the plan boundary R EFERENCES rather than at the tool-call boundary during live execution. Monitor-based enforcement. Schneider and Ligatti et al. es- [1] F. B. Kokulu, A. Soneji, T. Bao, Y. Shoshitaishvili, Z. Zhao, A. Doupe, tablish the theoretical foundation for safety properties enand G.-J. Ahn, “Matched and mismatched socs: A qualitative study on security operations center issues,” in Proceedings of the ACM SIGSAC forceable by automata-based monitors that suppress or edit Conference on Computer and Communications Security. New York, executions [17], [18]. Kim et al. (Kinetic) and Yu et al. (PreNY, USA: Association for Computing Machinery, 2019, pp. 1955–1970. cise Security Instrumentation, PSI) instantiate these ideas in [Online]. Available: https://doi.org/10.1145/3319535.3354239
12
[2] B. A. AlAhmadi, L. Axon, and I. Martinovic, “99% false positives: A qualitative study of soc analysts’ perspectives on security alarms,” in USENIX Security Symposium. Berkeley, CA, USA: USENIX Association, 2022, pp. 2783–2800. [Online]. Available: https: //www.usenix.org/conference/usenixsecurity22/presentation/alahmadi [3] M. Vermeer, N. Kadenko, M. van Eeten, C. Ganan, and S. Parkin, “Alert alchemy: Soc workflows and decisions in the management of nids rules,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. New York, NY, USA: Association for Computing Machinery, 2023, pp. 2770–2784. [Online]. Available: https://doi.org/10.1145/3576915.3616581 [4] D. W. Woods, R. Böhme, J. Wolff, and D. Schwarcz, “Lessons lost: Incident response in the age of cyber insurance and breach attorneys,” in USENIX Security Symposium. Berkeley, CA, USA: USENIX Association, 2023, pp. 2259–2273. [Online]. Available: https://www.usenix.org/conference/usenixsecurity23/presentation/woods [5] D. Schlette, P. Empl, M. Caselli, T. Schreck, and G. Pernul, “Do you play it by the books? a study on incident response playbooks and influencing factors,” in IEEE Symposium on Security and Privacy (S&P). Los Alamitos, CA, USA: IEEE, 2024, pp. 3625–3643. [Online]. Available: https://doi.org/10.1109/SP54263.2024.00060 [6] G. Deng, Y. Liu, V. M. Vilches, P. Liu, Y. Li, Y. Xu, M. Pinzger, S. Rass, T. Zhang, and Y. Liu, “Pentestgpt: Evaluating and harnessing large language models for automated penetration testing,” in USENIX Security Symposium. Berkeley, CA, USA: USENIX Association, 2024. [Online]. Available: https : / / www. usenix . org / conference/usenixsecurity24/presentation/deng [7] A. Stafeev, T. Recktenwald, G. D. Stefano, S. Khodayari, and G. Pellegrino, “Yurascanner: Leveraging llms for task-driven web app scanning,” in Proceedings of the Network and Distributed System Security Symposium (NDSS). Reston, VA, USA: The Internet Society, 2025, pp. 1–16. [Online]. Available: https://www.ndss- symposium. org/ndss- paper/yurascanner- leveraging- llms- for- task- driven- web- appscanning/ [8] D. Wang, G. Zhou, X. Li, Y. Bai, L. Chen, T. Qin, J. Sun, and D. Li, “The digital cybersecurity expert: How far have we come?” in IEEE Symposium on Security and Privacy (S&P). Los Alamitos, CA, USA: IEEE, 2025, pp. 3273–3290. [Online]. Available: https://doi.org/10.1109/SP61157.2025.00198 [9] T. Shi, J. He, Z. Wang, H. Li, L. Wu, W. Guo, and D. Song, “Progent: Programmable privilege control for llm agents,” arXiv preprint arXiv:2504.11703, 2025. [10] H. Wang, C. M. Poskitt, and J. Sun, “Agentspec: Customizable runtime enforcement for safe and reliable llm agents,” 2025. [Online]. Available: https://arxiv.org/abs/2503.18666 [11] J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Hillsdale, NJ: Lawrence Erlbaum Associates, 1988. [12] K. Greshake, S. Abdelnabi, S. Mishra, C. Endres, T. Holz, and M. Fritz, “Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection,” in Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security (AISec). New York, NY, USA: Association for Computing Machinery, 2023, pp. 79–90. [Online]. Available: https://doi.org/10.1145/3605764.3623985 [13] G. Wu, Z. Zhang, Y. Zhang, W. Wang, J. Niu, Y. Wu, and Y. Zhang, “I know what you asked: Prompt leakage via kvcache sharing in multi-tenant llm serving,” in Proceedings of the Network and Distributed System Security Symposium (NDSS). Reston, VA, USA: The Internet Society, 2025. [Online]. Available: https: //www.ndss-symposium.org/wp-content/uploads/2025-1772-paper.pdf [14] W. Qu, Y. Zhou, Y. Wu, T. Xiao, B. Yuan, Y. Li, and J. Zhang, “Prompt inversion attack against collaborative inference of large language models,” in IEEE Symposium on Security and Privacy (S&P). Los Alamitos, CA, USA: IEEE, 2025, pp. 1695–1712. [Online]. Available: https://doi.org/10.1109/SP61157.2025.00160 [15] T. Dong, M. Xue, G. Chen, R. Holland, Y. Meng, S. Li, Z. Liu, and H. Zhu, “The philosopher’s stone: Trojaning plugins of large language models,” in Proceedings of the Network and Distributed System Security Symposium (NDSS). Reston, VA, USA: The Internet Society, 2025. [Online]. Available: https://www.ndss- symposium.org/ndss- paper/thephilosophers-stone-trojaning-plugins-of-large-language-models/ [16] Y. Wu, F. Roesner, T. Kohno, N. Zhang, and U. Iqbal, “Isolategpt: An execution isolation architecture for llm-based agentic systems,” in Proceedings of the Network and Distributed System Security Symposium (NDSS). Reston, VA, USA: The Internet Society, 2025, pp. 1–20.
13
[Online]. Available: https : / / www. ndss - symposium . org / wp - content / uploads/2025-1131-paper.pdf [17] F. B. Schneider, “Enforceable security policies,” ACM Transactions on Information and System Security (TISSEC), vol. 3, no. 1, pp. 30–50, 2000. [Online]. Available: https://doi.org/10.1145/353323.353382 [18] J. Ligatti, L. Bauer, and D. Walker, “Edit automata: Enforcement mechanisms for run-time security policies,” International Journal of Information Security, vol. 4, no. 1, pp. 2–16, 2005. [Online]. Available: https://doi.org/10.1007/s10207-004-0046-8 [19] H. Kim, J. Reich, A. Gupta, M. Shahbaz, N. Feamster, and R. Clark, “Kinetic: Verifiable dynamic network control,” in USENIX Symposium on Networked Systems Design and Implementation (NSDI). Berkeley, CA, USA: USENIX Association, 2015, pp. 59–72. [Online]. Available: https://www.usenix.org/conference/nsdi15/technical-sessions/ presentation/kim [20] T. Yu, S. K. Fayaz, M. Collins, V. Sekar, and S. Seshan, “Psi: Precise security instrumentation for enterprise networks,” in Proceedings of the Network and Distributed System Security Symposium (NDSS). Reston, VA, USA: The Internet Society, 2017, pp. 1–15. [Online]. Available: https://www.ndss-symposium.org/ndss2017/ndss-2017-programme/psiprecise-security-instrumentation-enterprise-networks/ [21] D. Olszewski, T. Tucker, K. R. B. Butler, and P. Traynor, “Sok: Towards a unified approach to applied replicability for computer security,” in USENIX Security Symposium. Berkeley, CA, USA: USENIX Association, 2025, pp. 469–488. [Online]. Available: https: //www.usenix.org/conference/usenixsecurity25/presentation/olszewski [22] E. B. Wilson, “Probable inference, the law of succession, and statistical inference,” Journal of the American Statistical Association, vol. 22, no. 158, pp. 209–212, 1927. [Online]. Available: https : //doi.org/10.1080/01621459.1927.10502953 [23] Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947. [Online]. Available: https://doi.org/10.1007/BF02295996 [24] S. Holm, “A simple sequentially rejective multiple test procedure,” Scandinavian Journal of Statistics, vol. 6, no. 2, pp. 65–70, 1979. [Online]. Available: https://www.jstor.org/stable/4615733 [25] K. R. Jones, D. A. Brucker-Hahn, B. Fidler, and A. G. Bardas, “Work-from-home and covid-19: Trajectories of endpoint security management in a security operations center,” in USENIX Security Symposium. Berkeley, CA, USA: USENIX Association, 2023, pp. 2293–2310. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity23/presentation/jones [26] M. Vermeer, M. van Eeten, and C. Gañán, “Ruling the rules: Quantifying the evolution of rulesets, alerts and incidents in network intrusion detection,” in Proceedings of the ACM Asia Conference on Computer and Communications Security (AsiaCCS). New York, NY, USA: Association for Computing Machinery, 2022, pp. 799–814. [Online]. Available: https://doi.org/10.1145/3488932.3517412 [27] L. Yang, Z. Chen, C. Wang, Z. Zhang, S. Booma, P. Cao, C. Adam, A. Withers, Z. Kalbarczyk, R. K. Iyer, and G. Wang, “True attacks, attack attempts, or benign triggers? an empirical measurement of network alerts in a security operations center,” in USENIX Security Symposium. Berkeley, CA, USA: USENIX Association, 2024, pp. 1525–1542. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity24/presentation/yang-limin [28] E. Altinisik, F. Deniz, and H. T. Sencar, “Provg-searcher: A graph representation learning approach for efficient provenance graph search,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. New York, NY, USA: Association for Computing Machinery, 2023, pp. 2247–2261. [Online]. Available: https://doi.org/10.1145/3576915.3623187 [29] B. Zhang, Y. Gao, C. Yu, B. Kuang, Z. Zhang, H. Kim, and A. Fu, “Tapas: An efficient online apt detection with task-guided process provenance graph segmentation and analysis,” in USENIX Security Symposium. Berkeley, CA, USA: USENIX Association, 2025, pp. 607–624. [Online]. Available: https://www.usenix.org/conference/ usenixsecurity25/presentation/zhang-bo-tapas [30] F. Dong, S. Li, P. Jiang, D. Li, H. Wang, L. Huang, X. Xiao, J. Chen, X. Luo, Y. Guo, and X. Chen, “Are we there yet? an industrial viewpoint on provenance-based endpoint detection and response tools,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. New York, NY, USA: Association for Computing Machinery, 2023, pp. 2396–2410. [Online]. Available: https://doi.org/10.1145/3576915.3616580
[31] A. Aly, E. Mansour, and A. M. Youssef, “Ocr-apt: Reconstructing apt stories from audit logs using subgraph anomaly detection and llms,” in Proceedings of the ACM SIGSAC Conference on Computer and Communications Security. New York, NY, USA: Association for Computing Machinery, 2025, pp. 261–275. [Online]. Available: https://doi.org/10.1145/3719027.3765219 [32] J. Deng, X. Li, Y. Chen, Y. Bai, H. Weng, Y. Liu, T. Wei, and W. Xu, “Raconteur: A knowledgeable, insightful, and portable llm-powered shell command explainer,” in Proceedings of the Network and Distributed System Security Symposium (NDSS). Reston, VA, USA: The Internet Society, 2025, pp. 1–18. [Online]. Available: https: //www.ndss- symposium.org/ndss- paper/raconteur- a- knowledgeableinsightful-and-portable-llm-powered-shell-command-explainer/ [33] M. Bhatt, S. Chennabasappa, Y. Li, C. Nikolaidis, D. Song, S. Wan, F. Ahmad, C. Aschermann, Y. Chen, D. Kapil, D. Molnar, S. Whitman, and J. Saxe, “Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models,” 2024. [Online]. Available: https://arxiv.org/abs/2404.13161 [34] N. Zwerdling, D. Boaz, D. Amid, E. Rabinovich, A. Anaby-Tavor, and G. Uziel, “Towards enforcing company policy adherence in agentic workflows,” in Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track, 2025. [Online]. Available: https://aclanthology.org/2025.emnlp-industry.41/ [35] N. Palumbo, S. Choudhary, J. Choi, P. Chalasani, and S. Jha, “Policy compiler for secure agentic systems,” 2026. [Online]. Available: https://arxiv.org/abs/2602.16708 [36] L. Miculicich, M. Parmar, H. Palangi, K. D. Dvijotham, M. Montanari, T. Pfister, and L. T. Le, “Veriguard: Enhancing llm agent safety via verified code generation,” 2025. [Online]. Available: https : //arxiv.org/abs/2510.05156 [37] Z. Chen, M. Kang, and B. Li, “Shieldagent: Shielding agents via verifiable safety policy reasoning,” 2025. [Online]. Available: https://arxiv.org/abs/2503.22738 [38] D. Song, Y. Huang, B. Chen, T. Cong, R. Goebel, L. Ma, and F. Khomh, “Evaluating implicit regulatory compliance in llm tool invocation via logic-guided synthesis,” 2026. [Online]. Available: https://arxiv.org/abs/2601.08196 [39] B. Jordan and A. Thomson, “Cacao security playbooks version 2.0,” OASIS Committee Specification 01, Nov. 2023, oASIS Open. [Online]. Available: https://docs.oasis- open.org/cacao/security- playbooks/v2.0/ cs01/security-playbooks-v2.0-cs01.html
A PPENDIX O PEN S CIENCE
modes allowed by the verifier must be fixed before a lower violation rate can be read as a comparable improvement. The paired human baseline is included for the same reason: a verifier-only release could show correction under one rule set, but it could not show whether surviving plans align with recorded workflow patterns from the same incidents. Claim Surface and Verification Workflow The public bundle anchors three claim families. Corpus release and boundary claims are supported by canonical incident packages and declared-input manifests. Violation, enforcement, utility, execution accounting, and paired contrasts are supported by the official metrics, aggregate summaries, public run accounting, and paired-test outputs. Baseline validity and surface fragility are supported by mapping-support manifests and the global-artifact assessment. Provider calls, raw provider payloads, host-level ground truth, and behavior under new prompts, models, or incident slices are out of scope. Detailed file inventories, schemas, and reproduction commands belong in the artifact README, not in the paper. Five interpretation rules govern claims made against the declared evaluation state. A new provider result under the same incident slice, mapping contract, policy set, action catalog, and public audit surface is directly comparable within the current evaluation object. A change to any of those elements creates a separate evaluation object rather than a silent improvement over the current one. The paired human baseline is a second auditable compliance reference under the same slice and mapping contract; it is not a universal optimum or an abstract provider-versus-human ranking. The verifier is the last pre-execution control point over policy treatment, not a substitute for analyst reasoning or a general planner. Finally, public validation of the shipped non-private analysis bundle constitutes the external audit of the paper’s strongest public claim, whereas host-level outcome truth, institution-internal lineage, and provider reruns lie outside the current public evidence surface. Public reproduction is documented in the artifact README. The release workflow confirms the packaged files and their hashes, regenerates public assessment outputs, regenerates the paper result assets, and writes a reproduction report. It does not rerun LLM calls, but it reproduces the public analyses and audits the evidence surface on which the main claims rest.
The paper’s main claims are carried in the body. This appendix is supplementary: it states the public audit question, records the interpretation rules for the declared evaluation state, and preserves secondary quantitative views that would interrupt the main argument. The artifact supports one external audit question: Can an independent evaluator confirm that the released inputs, manifests, public run accounting, aggregate outputs, and public rechecks substantiate the reported claims under the declared evaluation state? This is narrower than a full institutional replay, but it is U SE OF G ENERATIVE A RTIFICIAL I NTELLIGENCE T OOLS the right question for a release that excludes raw case lineage, private approval context, and provider reruns. The authors used Grammarly and ChatGPT for grammar Within that boundary, the bundle exposes the action vo- checking, limited editorial revision, and minor formatting cabulary, typed policy file, mapping-support evidence, official assistance for figures and illustrations. All scientific claims, analysis summaries, public run accounting, paired contrasts, analyses, experiments, visual representations, and conclusions and human-baseline analysis needed to rederive the public were produced and verified by the authors. results. The release prioritizes transparent re-analysis of the declared study before any attempt to widen the hypothesis to Supplementary Quantitative Detail The appendix keeps only quantitative views that support new providers, policies, or incident slices [21]. The same discipline matters for future benchmark design. audit decisions, but would interrupt the main argument in Provider names, prompt arms, the declared action catalog, the the body. Table VII preserves the per-cell repair burden and active rule slice, the paired-baseline contract, and the repair execution accounting behind the rule-concentration discussion.
14
Table VIII preserves the public sensitivity view for approval deferral. Table IX preserves the pooled across-provider screening view; the body focuses on provider-resolved rates because the two providers move in opposite directions under policy prompting. Table X preserves the exact rule-level treatment rates behind the approval-slice discussion. These appendixonly objects extend numeric traceability without changing the interpretation established in the main text. TABLE VII E XECUTION AND RULE - BURDEN DIAGNOSTICS BY PROVIDER CELL .
Model / arm
Complete
R3
R4
Out-ofcatalog
Failures
Claude / policy Claude / zero GPT / policy GPT / zero
200/200 200/200 200/200 200/200
173 65 89 104
9 8 9 9
0 0 0 0
0 0 0 0
TABLE VIII A PPROVAL DEFERRAL SENSITIVITY FROM PUBLIC METRICS .
Model / arm
Deferred
Precision remove/defer
Jaccard remove/defer
Coverage loss
Claude / policy Claude / zero GPT / policy GPT / zero
182 73 98 113
0.927/0.667 0.945/0.827 0.973/0.811 0.979/0.807
0.705/0.550 0.725/0.657 0.722/0.630 0.791/0.682
none none none none
TABLE IX P OOLED OFFICIAL EVALUATION BY ARM . P ROVIDER - RESOLVED RATES APPEAR IN F IGURE 2. 95% interval
Arm
Runs
Hard count
Violation rate
LLM zero LLM policy Human baseline
400 400 200
186 280 0
0.4500 0.4020–0.4990 0.6700 0.6225–0.7143 0 0–0.0188
Verifier edit rate 0.4500 0.6700 0
TABLE X O FFICIAL EVALUATION RULE - LEVEL TREATMENT COMPARISON ( ZERO VS . POLICY PROMPT ). B OLD MARKS THE DOMINANT RULE - FAMILY EFFECT. Model/rule
Zero
Policy
∆
Runs
claude / R3 claude / R4 GPT / R3 GPT / R4
0.3250 0.0400 0.5200 0.0450
0.8650 0.0450 0.4450 0.0450
+0.5400 +0.0050 −0.0750 0
200 200 200 200
15