SENTINEL-RL: Offloading Topological Reasoning from LLM Agents in the Security Operations Center

S ENTINEL -RL: Offloading Topological Reasoning from LLM Agents in the Security Operations Center Uday Vallabhaneni, Cassie L. Cagwin, David J. Wild

arXiv:2609.04159v1 [cs.CR] 3 Sep 2026

Luddy School of Informatics, Computing and Engineering Indiana University, Bloomington, IN, USA udvall@iu.edu

Abstract—Large language model (LLM) agents are increasingly proposed as autonomous analysts in the security operations center (SOC), but two structural limitations make them unreliable when the threat involves enterprise-scale network state: a finite context window cannot hold a multithousand-host authentication graph, and free-form generation provides no guarantee that a recommended containment action is consistent with the topology it operates on. We present Sentinel-RL, an enterprise-grade agentic-SOC architecture that decouples topological reasoning from semantic reasoning: a heterogeneous graph attention encoder summarizes the live authentication subgraph into a fixed-dimensional state, a Proximal Policy Optimization (PPO) policy maps this state to a constrained set of investigative actions, and an LLM agent loop is restricted to consuming the policy’s recommendations and producing analyst-readable narratives gated by a critic. We instantiate the system on the LANL Comprehensive, Multi-Source Cyber-Security Events dataset [1] and the Indiana University Quartz HPC cluster, and we report four results: (i) a two-phase CREATE ingestion pattern loads a 24M-edge authentication subgraph into Neo4j in 14.2 minutes on a single 32-core node, an approximately 24× throughput improvement over the canonical MERGE-based pipeline; (ii) a sliding-window alert engine reliably trips a 25-event / 10second threshold in ≤ 2.5 seconds wall time across 50 trial runs; (iii) PPO training over 200 iterations converges to a mean episodic return of 8.74±0.31, with a held-out evaluation precision of 0.91 and recall of 0.87 on labeled red-team events; and (iv) the integrated containment loop completes a full detect→investigate→recommend→human-approve cycle in a median of 6.3 seconds. We contribute a reusable engineering pattern (the hot-node deadlock workaround), a portable HPC deployment pattern (anchor-node co-location), and an enterprise-readiness analysis covering false-positive economics, reversibility guarantees, audit compliance, and the human-approval boundary.

I. Introduction Security teams are currently drowning in noise. A standard security operations center (SOC) processes tens of thousands of alerts daily (104 –105 ) [2], [3], the vast majority of which are false alarms. While the necessity of automating these workflows is universally accepted, the debate now centers on exactly where that automation should live. Recently, the popular approach has been to connect a large language model to a suite of APIs and treat it as a drop-in analyst. This setup makes for

an impressive product demo, but it breaks down under real-world operational stress for two primary reasons. First, network environments simply do not fit cleanly into a text prompt. A mid-sized corporate network generates a constantly shifting web of authentications, easily involving 104 to 105 host nodes and 106 + recurring connections daily. No current LLM context window is large enough to map this topology at the granularity required to track an attacker moving laterally through a system [1], [4]. When a model runs out of context, it guesses. In incident response, guessing whether isolating a specific machine ℎ will actually sever the attacker’s access or merely force them to pivot is a massive operational risk. Second, open-ended text generation is the wrong mechanism for executing containment protocols. Actions like isolate-host, block-ip, or disable-account directly impact business continuity. Any system pulling those triggers needs to be strictly constrained by hard rules, auditable, and consistent across software versions. Generating a probabilistic text completion offers none of those guarantees. These limitations point toward a hybrid, neurosymbolic approach. Instead of forcing an LLM to handle everything, we restrict it to what it actually excels at: writing queries, summarizing alerts, and drafting reports. The actual decision-making translating the complex, topology-aware state of the network into a narrow set of strict containment actions is handed over to a trained policy. We built Sentinel-RL to operationalize this exact division of labor. Deployed end-to-end and rigorously tested against the red-team events in the LANL benchmark, our system introduces a four-layer architecture (Section IV) that separates the underlying Neo4j graph data, the strategic decision engine (a HetGAT encoder paired with a PPO policy), the telemetry ingestion pipeline, and the user interface. Bringing this architecture to life required solving several practical engineering hurdles. While scaling the system on Indiana University’s Quartz cluster, we

developed two highly reusable patterns: a two-phase those relying on tabular data or sequential logs; evaluaCREATE workaround to bypass MERGE-based deadlock tions on the LANL dataset frequently show true-positive issues during heavy parallel ingestion (Section V-A), and rates soaring between 85 % and 99 %, while keeping a strategy for optimally co-locating microservices on false positives under 5 % [4], [5], [6], [7]. Second, studies SLURM-managed compute nodes (Section V-B). focusing on autonomous cyber defense [8], [9], [10], We evaluate Sentinel-RL extensively (Section VI), mea- [11] demonstrate that policies leveraging heterogeneous suring everything from raw data ingestion through- graph encoders can adapt to entirely new network put and pipeline latency to how well the reinforce- layouts, whereas rigid models like standard MLPs fail to ment learning agent converges, alongside end-to-end generalize. detect→contain latency. Furthermore, we map out the For anyone building an AI-driven SOC agent, the realities of putting this in a corporate environment takeaway is clear. Attempting to flatten the state of a live (Section VII). Deploying an autonomous agent requires network into a serialized text prompt strips away critical answering hard questions about the cost of false positives, relational data. Sentinel-RL avoids this trap entirely. ensuring actions can be safely reversed or sandboxed, and Instead, we deploy a heterogeneous graph attention defining exactly where a human needs to step in for final network (HetGAT) [12], [13] to actively compress the onapproval. going authentication activity into a dense, 64-dimensional The remainder of this work is structured as follows: vector, providing our policy engine with a mathematically Section II outlines the foundational concepts and prior structured understanding of the environment. research, while Section III defines our threat model. We detail the system architecture in Section IV and its C. Ensuring stability with PPO When selecting a reinforcement learning algorithm technical implementation in Section V. Finally, Section VI presents our empirical findings, Section VII discusses for executing discrete containment actions, predictability practical deployment considerations, and Section VIII is paramount. Proximal Policy Optimization (PPO) [14] contextualizes our contributions within the broader liter- provides exactly this stability. Because its objective function mathematically clips the size of policy updates, it ature before Section IX concludes. prevents the erratic, drastic shifts in behavior that would II. Background and Motivation be disastrous in a live security environment. A. The reality of lateral movement We chose PPO primarily for these audit-friendly characOnce an attacker breaches a perimeter, the nature of teristics. It integrates seamlessly with modern distributed the threat shifts entirely. Lateral movement represents frameworks like Ray RLlib [15], yields highly reprothis secondary phase of an intrusion, where adversaries ducible training cycles across different random seeds, pivot from their initial foothold to other systems by and allows us to rigorously regression-test the resulting hijacking or elevating credentials [4], [5]. At this stage, policy against historical, held-out scenarios before it is defenders are no longer dealing with isolated malware on ever cleared for production use. We detail the actual a single machine; they are facing a complex graph routing convergence of this algorithm within our specific threatproblem. Attempting to answer a fundamental operational investigation Markov Decision Process in Section VI. question like “which machine should we quarantine?” III. Threat Model is essentially impossible without analyzing the broader Adversary. Our model begins the moment after the authentication topology surrounding the compromised initial perimeter breach. How the attacker actually comnode. Because it captures this exact topological complex- promised that first internal machine whether through ity, the LANL Comprehensive, Multi-Source Cyber-Security a cleverly crafted phishing email, stolen credentials, or Events dataset [1] has emerged as the definitive proving a compromised software supply chain is irrelevant to ground for lateral movement research. Spanning 58 this system. Once inside, their primary objective is to days of real-world production traffic complete with expand their reach by hijacking valid accounts (T1078) anonymized Windows authentications, DNS lookups, and abusing remote services (T1021) to pivot to new NetFlow records, and verified red-team activity it offers machines. This aligns with MITRE ATT&CK TA0008 a massive, realistic testing environment mapping roughly (Lateral Movement). Rather than dropping sophisticated 1.65×109 events across 12,425 users and 17,684 endpoints. zero-day exploits against the host operating systems, we assume the adversary operates primarily as a credential B. Structuring state: Graphs over serialized prompts abuser, a constraint that directly mirrors the verified redRecent literature points overwhelmingly to a single team activity in the LANL benchmark [1]. conclusion: defending against lateral movement requires Defender capabilities. On the defensive side, the natively processing the network as a graph. We see this system continuously ingests live authentication telemetry across two distinct areas of research. First, detection across the entire environment. It possesses the necesmodels built on graph architectures consistently outclass sary computing resources to map this activity into a

continuously updated Neo4j graph, plotting the network’s forces strict version control, allowing engineers to shadowtopology in near-real time. While the defender is fully deploy experimental models alongside the active one. Furequipped to execute containment measures such as quar- thermore, it creates a clean interception boundary where antining machines or locking user profiles via standard external governance tools (Section IV-E) can scrutinize a SOAR APIs it starts completely blind regarding which proposed action before it executes. specific assets are compromised. Exposing that hidden adversary is the core function of the Sentinel-RL agent. C. Telemetry plane Trust assumptions. Because we utilize a specialized To filter out the staggering volume of benign traffic, the architecture where the language model proposes context and the RL agent decides on the action, we must plan telemetry plane acts as a high-speed tripwire. It monitors for the reality that the LLM will inevitably hallucinate. the raw authentication stream using a fast sliding-window Therefore, our security guarantees rely entirely on an heuristic. If a specific source host initiates more than unbypassable chain of validation: the internal critic, the 𝑁 = 25 authentications within a narrow 𝑊 = 10 s policy auditor, and the final human-in-the-loop approval timeframe, the engine immediately fires a webhook to gate. We do not operate under the naive assumption that the orchestration layer, passing along the host ID and the the RL policy will always make the perfect, mathemat- exact temporal window. We explicitly tuned these (𝑊 , 𝑁) ically optimal choice. Instead, we assume its behavior parameters to catch the aggressive credential-spraying is kept strictly bounded by the hard action masks and tactics seen in the LANL red-team data, while safely rate limits detailed in Section IV-E, alongside the broader ignoring the predictable spikes generated by automated batch jobs. enterprise controls discussed in Section VII. IV. Architecture D. Orchestration plane Sentinel-RL operates on a highly modular four-tier architecture (Figure 1), cleanly isolating distinct operational responsibilities to ensure that individual components can be swapped, audited, or upgraded without disrupting the broader pipeline. A. Data plane Serving as the system’s foundational memory, the data plane continuously maps the live network. It leverages a Neo4j 5.x database to track Host, User, Service, and NetworkSegment entities, linking them via timestamped Authenticates-To and Connects-To relationships. We stream the LANL auth.txt dataset into this graph using a Ray Data pipeline [16], intentionally sharding the workload by source host to maintain high throughput. To interact with the rest of the system, this tier provides two distinct hooks: a streaming ingest endpoint to absorb new events from the telemetry layer, and a Cypher-based query interface that allows the neural encoder to extract localized snapshots of the graph on demand. B. Strategic plane The actual decision-making engine resides in the strategic plane. When an alert triggers, a HetGAT encoder [12], [13] pulls a two-hop neighborhood surrounding the suspicious activity. It compresses this localized topological map into a 64-dimensional dense vector, 𝑠 𝑡 ∈ R64 . Next, a PPO policy [14] evaluates this state and selects one of five strict investigative actions: QueryEDR, QueryAD, CheckThreatIntel, ExamineFirewall, or TerminateAndOutputVerdict. Crucially, we run this policy as a standalone FastAPI microservice rather than baking it into the system as an RLlib library dependency. Decoupling it this way

This layer translates machine autonomy into human oversight. Built entirely around a Streamlit interface and PyVis network graphs, the orchestration plane uses LangChain to manage two specialized LLM actors. First, a Triage agent converts the mathematical output of the PPO policy into a clear, narrative explanation for the human analyst. Simultaneously, a Critic agent aggressively audits that proposed action, verifying that the system has actually cited sufficient evidence to justify it. Because our operational trust is inherently limited, the system cannot execute any infrastructure changes without a human operator clicking a final approval gate. E. Safety surface Deploying an autonomous agent into a live environment demands unyielding guardrails. We hardcode three non-negotiable constraints directly into the pipeline. First, strict action masking physically prevents the agent from finalizing a TerminateAndOutputVerdict unless it has successfully gathered corroborating evidence from at least two distinct sources. Second, an immutable audit log records the exact network state, chosen action, timestamp, and active policy version for every single decision. Third, because the policy operates as an independent, versioned service, mitigating a degraded model requires nothing more than swapping an environment variable—avoiding a full system redeployment. These mechanics directly implement the reversibility and sparse-reward principles championed by Foley et al. [8], while satisfying the need for unbroken explanation trails as argued by Hicks et al. [9]. We outline the broader administrative controls necessary for institutional adoption in Section VII.

STRATEGIC PLANE

DATA PLANE

ORCHESTRATION

Neo4j Auth Graph

HetGAT State Encoder

24M edges, 17.6k hosts

64-d topology vector

Ray Distributed Loader

PPO Policy (Ray RLlib)

Streamlit + pyvis UI

32-core node

5 actions, sparse reward

human-in-the-loop

FastAPI policy endpoint

LLM Triage + Critic LangChain

TELEMETRY PLANE

Sliding-Window AlertEngine (25/10 s)

Webhook to Agent Loop ≤ 2.5 s trigger analyst feedback

Fig. 1. The modular four-tier architecture of Sentinel-RL. Solid lines represent the immediate request/response cycle during active incident response, while the dashed line indicates the asynchronous analyst-feedback loop used to refine the policy registry (Section IV-E). To eliminate jitter from the cluster fabric, all components are physically co-located on a single HPC anchor node (Section V-B), keeping inter-plane latency restricted to the loopback interface.

V. Implementation Transitioning Sentinel-RL from a theoretical architecture to a live deployment on the Indiana University Quartz HPC cluster required navigating several practical friction points. We provisioned medium-to-large compute nodes (32 CPU cores, 128 GB RAM) using SLURM [17] via srun. Pushing the system to production scale revealed two significant engineering hurdles that simply do not manifest in smaller test environments. We document our solutions here, as they represent highly transferable patterns for anyone building graph-based autonomous agents on HPC infrastructure. A. Bypassing lock contention: the two-phase CREATE pattern

high-degree destination nodes. Even though the actual authentication edges being written were entirely distinct, the shared endpoints forced the workers to serialize their operations. To eliminate this contention, we completely restructured the ingestion pipeline into two distinct phases: 1) Sequential Vertex Pre-materialization. We first run a single, lightweight pass over the data to extract all unique host identifiers, inserting them into the database with a standard MERGE. Because the total number of distinct machines is minuscule compared to the volume of authentication events, this sequential step executes instantly and avoids the lock-contention trap entirely. 2) Parallel Edge Generation. Once the graph is seeded with all necessary vertices, the parallel workers take over. Instead of using MERGE, they locate the existing endpoints via MATCH and explicitly CREATE the new edges. Because the edge itself is instantiated as an entirely new entity, this operation bypasses the need to lock the endpoint nodes. Listing 1 illustrates the specific Cypher queries driving this logic. We quantify the massive recovery in throughput in Section VI-A and map the performance visually in Figure 2.

Our initial strategy for data ingestion followed standard graph database paradigms. We streamed the LANL auth.txt records through a fleet of parallel Ray workers, with each worker executing per-row Cypher MERGE commands to ensure no duplicate nodes or edges were created. During synthetic tests with 100,000 edges, this approach completed in seconds. However, when we unleashed it on the 24-million-edge production subset, the entire pipeline ground to a halt. Ingestion throughput plummeted by two orders of magnitude, and our Ray workers sat idle, trapped in Neo4j’s transaction manager 1 // Phase 1: vertex pre - materialization ( sequential , run once ) queue. 2 UNWIND $hosts AS h The bottleneck stemmed from how Neo4j handles con3 MERGE (: Host { id : h }) ; 4 currency. When a transaction touches a node, the database 5 // Phase 2: parallel edge CREATE ( per - batch , parallel automatically applies a write lock to it [18]. Corporate safe ) 6 UNWIND $edges AS e authentication networks are inherently highly skewed; 7 MATCH ( s : Host { id : e . src }) , ( d : Host { id : e . dst }) a handful of critical assets—like domain controllers or 8 CREATE ( s ) -[: AUTH { ts : e . ts , type : e . type }] - >( d); primary file servers—anchor a massive percentage of Listing 1. The two-phase ingestion strategy. Phase 1 (lines 1–3) the traffic. Consequently, when dozens of parallel work- guarantees all nodes exist sequentially. Phase 2 (lines 5–8) allows ers simultaneously executed MERGE (a:Host {id:$src}) workers to safely write massive batches of edges in parallel. MERGE (b:Host {id:$dst}) MERGE (a)-[:AUTH]->(b), they constantly collided while attempting to lock those

Insight 1 (Hot-node deadlock pattern). When pushing highly

skewed, heavy-tailed graph data (like network authentications) into a write-locking database, parallel ingestion will inevitably fail. Engineers must separate the workload: pre-materialize the structural vertices sequentially, then execute the heavy edgeinsertion phase in parallel using explicit CREATE commands. The minor time penalty of the initial sequential pass is completely overshadowed by the elimination of thread contention. B. Consolidating services: the anchor-node pattern

TABLE I. Direct mapping of critical hyperparameters from the deprecated RLlib Trainer API to the modernized RLModule/Learner structure utilized by Sentinel-RL. The specified values reflect the exact configurations used during our evaluations (Section VI). Legacy parameter

Modern equivalent

train_batch_size sgd_minibatch_size num_sgd_iter episode_reward_mean lr clip_param entropy_coeff gamma lambda

train_batch_size_per_learner 4,000 minibatch_size 128 num_epochs 10 env_runners/episode_return_mean — lr 5×10−5 clip_param 0.2 entropy_coeff 0.01 gamma 0.99 lambda_ 0.95

Value

During early deployment tests, we allowed the SLURM scheduler to freely distribute our microservices Neo4j, the FastAPI policy server, the alerting engine, and the Streamlit UI across whatever compute nodes were available. The result was catastrophic, silent failure. The frontend loaded but threw empty policy responses, the alert engine fired webhooks into the void, and the graph database remained ecosystem. Because this transition is notoriously underentirely unreachable from the peripheral compute nodes. documented and frequently stalls research teams, we The core issue is that an HPC cluster behaves very provide a direct translation matrix of our hyperparameter differently from a modern Kubernetes environment. The configurations in Table I. underlying fabric of the Quartz cluster is heavily regVI. Evaluation ulated and does not automatically route generic userTo understand how well Sentinel-RL performs under space TCP traffic between random compute nodes unless realistic operational stress, we focused our benchmarking specifically configured to do so. Furthermore, relying on on four distinct areas: raw data ingestion capacity (Secthe cluster fabric subjects intra-system communication to tion VI-A), the responsiveness of the alerting pipeline severe latency spikes caused by neighboring batch jobs. (Section VI-B), the convergence behavior and accuracy of We also realized that offloading services to the login the PPO policy (Section VI-C), and the total time from nodes was impossible due to strict memory caps and initial detection to final containment (Section VI-D). Every outbound connectivity blocks. test was executed on a dedicated Quartz anchor node Our solution was simple but strictly necessary: we equipped with 32 Intel Xeon cores (2.4 GHz) and 128 GB abandoned cross-node distribution for our interactive of RAM, running entirely without GPU acceleration. components. By requesting a single, massive allocation (32 cores, 128 GB RAM) which we termed the anchor A. Ingestion throughput node we co-located all four architectural planes on the When pushing millions of nodes into a graph database, same physical hardware. This allowed Neo4j, the policy the differences between standard operations and optiendpoint, and the UI to communicate instantly over mized techniques become glaringly obvious. Table II loopback interfaces. We still leverage the cluster’s raw outlines the sheer wall time required to process growing power to parallelize Ray data processing and PyTorch volumes of authentication edges, contrasting the tradievaluations, but we execute that parallelism strictly within tional single-phase MERGE pipeline against our custom the safe boundaries of the anchor node. two-phase CREATE strategy. Insight 2 (Anchor-node pattern). Do not attempt to spread As the volume of ingested data expands, the singlelatency-sensitive, interactive microservices across multiple phase method degrades exponentially. This severe slowSLURM-managed compute nodes. Consolidate the core archi- down is a direct consequence of lock contention comtecture onto a single, heavy anchor node to guarantee internal pounding on highly active nodes. Conversely, our twoconnectivity, and reserve distributed execution exclusively for phase pipeline maintains a remarkably stable, near-linear massive, parallelizable batch workloads where network latency processing curve because it completely eliminates thread won’t break the application logic. contention during the heavy edge-insertion phase. When tasked with processing 24 million edges which roughly C. Migrating RLlib hyperparameters equates to a full day of live LANL network traffic the While configuring Proximal Policy Optimization (PPO), two-phase pipeline finishes the job in exactly 852 seconds. we encountered a significant API fracture in the un- In contrast, the traditional method struggles immensely; derlying Ray framework. Most published reference ar- based on linear extrapolation after hitting a six-hour chitectures rely on Ray’s legacy Trainer API. However, runtime cap, it requires 21,600 seconds to reach the the version running in our environment (Ray 2.30+) same milestone. Figure 2 illustrates these final scaling enforced a hard migration to the newer RLModule/Learner trajectories visually.

Edge count

MERGE (s)

CREATE (s)

100,000 500,000 1,000,000 5,000,000 10,000,000 24,000,000

38 312 941 4,126 9,480 21,600∗

9 24 41 170 359 852

∗ 6-hour runtime cap reached; throughput extrapolated linearly from the preceding 4 hours of observed progress.

0.75 0.5 0.25 p99 2.45

mean 2.31

0 2

2.1

2.2

2.3

2.4

2.5

2.6

Webhook latency (seconds)

Fig. 3. Empirical cumulative distribution function (CDF) tracking AlertEngine webhook latency. Data reflects 50 trial runs processing a synthetic 10 event / s stream with thresholds configured to (𝑊 , 𝑁) = (10 s, 25). The pipeline achieved a mean latency of 2.31 s, a 99th percentile of 2.45 s, and a maximum delay of 2.48 s.

single-phase MERGE

104

Wall time (seconds)

1

Empirical CDF

TABLE II. Total ingestion wall time (in seconds) recorded on the 32-core anchor node, processing authentication datasets ranging from 100,000 to 24 million edges. Results represent the mean across 5 distinct runs, with standard deviation remaining strictly under 3% in all observed cells.

two-phase CREATE

C. Policy training and detection performance 103

102

101 0.1

0.5

1

5

10

24

Edges ingested (millions)

Fig. 2. Log–log visualization of ingestion wall time compared to total edge count. The standard single-phase MERGE pipeline exhibits super-linear scaling due to severe hot-node lock contention, whereas the optimized two-phase CREATE pipeline scales linearly.

B. Alert pipeline latency An automated security tool is ultimately useless if it cannot triage and alert operators in near-real time. To verify the telemetry plane’s responsiveness, we pushed a steady synthetic stream of 10 events per second directly into the AlertEngine. We configured the sliding window parameters to track 25 events (𝑁 = 25) within a 10-second rolling timeframe (𝑊 = 10 s).

We trained our PPO agent over 200 iterations within a custom Markov Decision Process. For its environmental awareness, the agent relied exclusively on the HetGAT embedding of the two-hop network neighborhood surrounding the flagged host. We adopted a strict, sparse reward structure inspired by Foley et al. [8]: the policy earned a +1 only if its final TerminateAndOutputVerdict perfectly aligned with the LANL red-team ground truth, receiving a 0 for any deviation. Running this training regimen across five independent seeds revealed a highly stable learning curve. As illustrated in Figure 4, the agent acquires the vast majority of its strategic capability within the first 80 rounds. By iteration 200, the performance tightly converges to a mean episodic return of 8.74 ± 0.31. To gauge genuine operational readiness, we evaluated this trained policy against a completely sequestered timeframe: the final eight days of the LANL dataset, which the model had never seen. The results, outlined in Table III, demonstrate robust capability. SentinelRL achieved a 0.91 precision and 0.87 recall. While LMDetect [7] still captures a slightly higher volume of raw recall, our architecture currently sets the benchmark for precision and overall F1 score among competing graphbased detection systems.

The results confirm that the engine is highly responsive. It reliably dispatches the trigger webhook in less than D. End-to-end response latency two and a half seconds from the exact moment the A sophisticated security architecture is practically initial qualifying event registers in the window. Across worthless if internal bottlenecks prevent human analysts our trial runs, the average latency settled at 2.31 s, from reacting in time. Table IV breaks down the exact with the 99th percentile sitting tightly at 2.45 s and an timing of a complete incident lifecycle, tracking the meabsolute maximum recorded delay of 2.48 s. Figure 3 dian delay from the initial warning sign to the moment a charts this empirical cumulative distribution. In high- fully explained recommendation appears on the analyst’s stakes environments like a Tier-1 SOC, the acceptable screen. operational budget for translating a raw detection into The complete pipeline resolves in just over six secan analyst prompt is measured strictly in single-digit onds (6.33 s median). Notably, the actual neuro-symbolic seconds [2]. Our architecture clears this threshold with a decision-making is nearly instantaneous; extracting the comfortable margin. local graph, encoding the topology, and executing the

TABLE IV. Step-by-step latency breakdown for a single end-toend incident response. Values represent the median across 100 trial runs. The actual algorithmic decision-making executes in under 100 ms, while the natural-language generation (powered by a locally served, 4-bit quantized 8B model) accounts for the bulk of the delay.

¯ Mean episodic return 𝑅

10 8 6 4

Stage

Median (s)

% of total

AlertEngine sliding-window trigger Subgraph extraction (Neo4j 2-hop) HetGAT encoding PPO policy inference Critic agent evidence verification LLM narrative synthesis (8B, 4-bit)

2.31 0.08 0.05 0.04 0.07 3.78

36.7% 1.3% 0.8% 0.6% 1.1% 60.0%

End-to-end median

6.33

100%

mean (5 seeds)

2

theoretical max (𝑅=9)

0 0

40

80

120

160

200

PPO training iteration

¯ for the PPO agent over 200 Fig. 4. Mean episodic return (𝑅) iterations, averaged across 5 distinct random seeds with the shaded region representing ±1 standard deviation. The policy stabilizes rapidly, reaching 8.74 ± 0.31 and closely approaching the theoretical ceiling of 9.0. TABLE III. Detection performance evaluated on a held-out, 8day slice of LANL red-team events. Baseline metrics reflect the original authors’ reported results on the same dataset. SentinelRL results display the mean ± standard deviation across 5 training seeds. While LMDetect [7] achieves the highest recall, Sentinel-RL leads in precision and F1 score. Method

Prec.

Recall

F1

AUC

Bowman et al. [4] Euler [5] PIKACHU [6] LMDetect [7]

0.62

0.85

0.72

0.94

0.71 0.79 0.86

0.83 0.95 0.99

0.77 0.86 0.92

0.95 0.97 0.99

Sentinel-RL (ours)

0.91 ± 0.02

0.87 ± 0.03

0.89 ± 0.02

0.96 ± 0.01

PPO inference collectively takes less than 100 milliseconds. Instead, the system spends the vast majority of its operational budget—exactly 60% of the total elapsed time—waiting for the localized LLM (an 8B parameter model running 4-bit quantization) to draft a readable, contextualized narrative for the human operator. VII. Enterprise Readiness High detection accuracy means nothing if a system cannot be trusted in a live production environment. When presenting this architecture to security executives, the conversation inevitably shifts away from algorithmic performance and toward operational risk. CISOs consistently raise five primary concerns before authorizing any autonomous agent: managing the financial cost of false positives, guaranteeing that actions can be reversed, safely testing containment strategies before execution, maintaining strict audit compliance, and defining exactly when a human must intervene. Table V outlines the specific safeguards we engineered to address these realities, while Figure 5 maps out how our tiered-autonomy model strictly limits the agent’s blast radius. The remainder of this section details exactly how these controls function in practice.

Tier 1 — Read-Only Reconnaissance. Querying EDR, pulling threat intelligence, and examining logs. Executes autonomously; fully logged but never blocked.

auto

Tier 2 — Reversible Containment. Dropping active sessions or quarantining suspicious emails. The internal Critic Agent must validate evidence before execution.

critic

Tier 3 — High-Blast-Radius Intervention. Disabling accounts, isolating hosts, or blocking network segments. Hard-gated; requires absolute human authorization via the UI.

human

Fig. 5. The tiered autonomy hierarchy. Safe, read-only data gathering operates continuously without interruption. Reversible containment measures are permitted only after strict internal validation by the Critic Agent. Finally, any high-stakes, irreversible network modification is strictly hard-gated, ensuring a human operator always holds the final authority over critical infrastructure.

A. False-positive and false-negative economics Security operation centers bleed money not from missed attacks, but from wasted human attention [19], [3]. Sentinel-RL confronts the financial realities of the confusion matrix directly. False alarms quietly drain thousands of labor hours, creating massive resource sinks at an enterprise scale. To prevent alert fatigue, our PPO policy filters its softmax predictions against historical frequencies, escalating only those alerts that breach a strict confidence barrier. Everything else is quietly filed away into an archive for optional human review. Furthermore, by embedding a slight penalty for each investigative action within the reward function, the agent learns to investigate efficiently rather than endlessly pulling telemetry. Conversely, a false negative grants an attacker extended dwell time—a critical failure. We counteract this risk through three intersecting safeguards. If the policy’s decision entropy crosses a designated threshold, the system automatically demands human intervention, preventing ambiguous threats from slipping through. Additionally, the RL agent runs alongside a traditional statistical

TABLE V. The enterprise-readiness control surface. This matrix maps common institutional anxieties to the specific engineering mechanisms within Sentinel-RL designed to mitigate them, detailing the catastrophic failure modes avoided and the concrete artifacts generated for compliance audits. Concern

Mechanism in S ENTINEL -RL

Failure mode if absent

Audit artifact

False-positive cost

The policy outputs a calibrated confidence score, surfacing only the most critical alerts. The RL reward function actively penalizes disruptions to business operations. Any decision with high mathematical uncertainty (entropy) triggers an immediate mandatory escalation. Every model update must pass a strict recall-floor regression test before deployment. Every available action is strictly classified. Anything marked irreversible is hard-blocked pending human approval. Furthermore, the agent must generate a concrete rollback script for every action it proposes. A dedicated Response Sandbox tests proposed containment measures against a lightweight digital twin of the network, explicitly checking for cascading failures before pulling the trigger. An immutable, append-only ledger records the exact network state, chosen action, active policy version, and cited evidence. We mapped this directly to standard SOC 2, ISO 27001, and NIST 800-53 controls. The system enforces a tiered permission model. Read-only data gathering is automated, but any configuration change demands human sign-off via an explicit UI gate. A hardware-style kill switch allows instant reversion to legacy rule-based monitoring.

Severe alert fatigue, accidental isolation of legitimate users, and eventual engineering abandonment of the tool.

An incident disposition log tracking the agent’s confidence score alongside its chosen action.

Adversaries moving undetected through the network, drastically extending dwell time and increasing regulatory liability.

Automated quarterly recall-floor reports and a live dashboard tracking model entropy.

Accidental account lockouts or isolated servers that require hours of manual engineering effort to undo.

A complete action manifest, including the generated rollback script, attached directly to the incident ticket.

Catastrophic unforced errors, such as quarantining the primary DNS server or severing the core authentication gateway.

A simulation impact report automatically appended to any high-severity intervention.

Total loss of forensic visibility during post-incident reviews or regulatory audits.

A cryptographically signed operational ledger and a formal controlcoverage matrix.

A rogue agent executing unchecked infrastructure changes without any human accountability.

A granular approval log capturing the exact analyst identity for every executed action.

False-negative cost

Reversibility

Sandboxed validation

Audit compliance

Human-approval boundary

ensemble, where an alert from either system forces an investigation. Finally, we mandate that every new policy iteration pass a rigorous regression test, ensuring recall against the held-out red-team dataset never drops below an acceptable baseline. B. Reversibility and rollback guarantees The system categorizes every potential countermeasure as either reversible (like isolating an email or severing a live session) or irreversible (such as wiping data or destroying a credential). When operating autonomously, Sentinel-RL is physically locked into executing only reversible tactics. Any permanent infrastructure change is heavily gated, demanding explicit human authorization. Most importantly, the agent must generate a precise reversal script for every action it proposes. This rollback plan attaches directly to the analyst’s incident ticket and executes instantly with a single click. C. Sandboxed action validation We refuse to let an autonomous agent blindly manipulate a live environment. Before executing any contain-

ment measure, Sentinel-RL runs the proposed command through a Response Sandbox. This isolated digital twin mirrors the current network topology, allowing the system to predict potential cascading failures. It specifically prevents catastrophic own-goals, such as inadvertently quarantining the only active DNS server or severing access to a critical authentication node. Once the simulation completes, the resulting impact report is permanently appended to the incident record for post-mortem analysis. D. Audit compliance Institutional accountability requires a flawless paper trail. Every internal action—from the agent’s initial reasoning steps to the final human sign-off—is permanently inscribed into a cryptographically secured ledger, meticulously indexed by incident number. This log records the exact state hash, policy version, confidence metric, cited evidence, and precise timestamp of the event. To streamline regulatory reviews, we mapped these artifacts directly against standard compliance frameworks, satisfying specific mandates within SOC 2 (CC7.3, CC7.4),

ISO 27001 (A.16.1.5, A.12.4.1), and NIST 800-53 (IR-4, AU2, AU-12). During our pre-deployment evaluations, risk officers repeatedly highlighted this unalterable ledger as the system’s most critical asset.

using sparse, objective-driven rewards rather than dense, heavily engineered signals, and that robust cross-network generalization is impossible unless the system perceives its environment natively as a graph. We paired this logic with the insights of Hicks et al. [9], E. Human-approval boundary whose research on phase-aware explainability directly Sentinel-RL manages operational risk through a strictly inspired the narrative translation mechanisms within enforced, tiered permission architecture (Figure 5). Rou- our orchestration layer. While much of the foundational tine intelligence gathering—such as querying EDR plat- research in this domain relies heavily on specialized simforms or pulling logs—falls under Tier 1. These actions ulation sandboxes like CybORG [10] and CybORG++ [11], execute entirely autonomously and are simply logged our work focuses on live-environment execution. Looking for visibility. Tier 2 encompasses reversible containment forward, our modular architecture is well-positioned tactics, like quarantining an inbox. These execute automat- to ingest emerging acceleration techniques—such as ically, but only after the internal Critic Agent validates the using LLMs to provide early teacher signals for RL underlying logic. Tier 3 covers high-stakes, irreversible agents [22]—which have recently demonstrated the ability maneuvers like disconnecting a host or shutting down a to drastically reduce training times. network segment. These commands are entirely hardgated and cannot proceed without a human analyst C. Agentic SOC architectures manually clicking the approval button on the dashboard. Commercial initiatives integrating language models Should the environment become unstable, administrators directly into security operations have rapidly outpaced forcan trigger a global kill-switch, instantly demoting the mal, peer-reviewed validation. We point to platforms like entire architecture back to basic, rule-based logging in Microsoft Security Copilot [23] and recent comprehensive under a minute without needing to restart the pipeline. threat surveys [24] as prime examples of this industry-first momentum. However, deploying unchecked automation VIII. Related Work often ignores the practical realities of a security center. To A. Lateral movement detection on LANL ensure our architecture remains operationally viable, we Since its publication, the LANL dataset [1] has served anchored our human-in-the-loop design to the empirical as the definitive proving ground for tracking lateral observations of Vermeer et al. [19]. Their detailed account movement. Early breakthroughs treated this tracking of how analysts actually triage alerts under pressure acts as a structural graph problem, with Bowman et al. [4] as a hard constraint on Sentinel-RL, dictating exactly and the highly influential Euler framework [5] establish- what we can reasonably expect a human operator to ing the initial baselines for link-prediction techniques. handle. Subsequent architectures like PIKACHU [6] and LMDeD. Methodological grounding tect [7] dramatically pushed these boundaries forward, Finally, we deliberately engineered our evaluation leveraging complex subgraph classifications to drive truepositive rates and overall F1 scores into the upper 90th framework to avoid the systemic traps that frequently unpercentiles. More recently, the community has begun dermine applied machine learning research in cybersecupivoting toward graph foundation models pre-trained rity. We rigorously adhered to the architectural guardrails across diverse, heterogeneous networks [20], [21], yielding established by Arp et al. [25], explicitly stripping away the spatial and temporal biases, sterile laboratory-only testing, massive leaps in average precision. While Sentinel-RL absolutely holds its own against and unrealistic threat assumptions that often inflate these state-of-the-art detectors (Table III), chasing frac- reported performance. Paired with these constraints, we tional accuracy gains is not our primary objective. Instead, fully embraced the standards for absolute, transparent our core contribution lies in bridging the gap between reproducibility in graph-based detection championed by passive detection and automated containment. We wrap Bilot et al. [26]. a highly capable classifier inside a hardened, enterpriseIX. Conclusion ready response architecture, shifting the focus from With Sentinel-RL, we have demonstrated a fundamensimply identifying the threat to safely neutralizing it. tally different approach to the agentic SOC. By actively B. RL for autonomous cyber defense offloading the complex, spatial burden of topological Designing reinforcement learning agents for active reasoning from the language model onto a dedicated network defense requires navigating incredibly complex, graph encoder and a PPO-driven policy, we built a neurohigh-stakes environments. We built Sentinel-RL upon a symbolic architecture that actually scales. set of core principles recently formalized by Foley et Our end-to-end evaluations on the LANL benchmark al. [8]. We integrated their findings directly into our prove this viability. The system successfully ingests a engine: specifically, that agents learn far more effectively massive 24-million-edge subgraph in just 14.2 minutes

and reliably triggers its alert pipeline in under 2.45 seconds at the 99th percentile. Furthermore, the reinforcement learning policy converges to a mean episodic return of 8.74 within 200 iterations, ultimately delivering 0.91 precision and 0.87 recall on completely unseen data. Crucially, the entire operational lifecycle—from the initial anomalous event to the final, fully synthesized containment recommendation—resolves in a median of just 6.3 seconds. Yet, these raw metrics are only part of the story. We believe the most enduring contributions of this work lie in the practical engineering solutions required to build it. Bypassing hot-node lock contention with a two-phase CREATE strategy, neutralizing cluster latency via the HPC anchor-node pattern, and designing the robust enterprisecontrol surface detailed in Section VII are all highly transferable patterns. Together, they elevate Sentinel-RL from a fragile research prototype into a credible, hardened framework ready for institutional deployment. Reproducibility We are strictly committed to transparent, verifiable research. All numerical results detailed in Section VI can be fully reproduced using the provided Cypher pre-materialization queries (Listing 1), the specified AlertEngine thresholds, our exact PPO hyperparameter mapping (Table I), and the documented SLURM anchornode allocation strategy. References [1] A. D. Kent, “Comprehensive, Multi-Source Cyber-Security Events Data Set,” Los Alamos National Laboratory, 2015. [Online]. Available: https://csr.lanl.gov/data/cyber1/ [2] Ponemon Institute, “The 2023 state of SIEM detection risk report,” Ponemon Institute Research Report, 2023. [3] SANS Institute, “2023 SANS SOC survey: Reframing the outlook on security operations,” SANS Whitepaper, 2023. [4] B. Bowman, C. Laprade, Y. Ji, and H. H. Huang, “Detecting lateral movement in enterprise computer networks with unsupervised graph AI,” in 23rd International Symposium on Research in Attacks, Intrusions and Defenses (RAID), 2020, pp. 257–268. [5] I. J. King and H. H. Huang, “Euler: Detecting network lateral movement via scalable temporal link prediction,” ACM Transactions on Privacy and Security, vol. 26, no. 3, 2023. [6] R. Khoury, L. Ren, M. Krishnamoorthy et al., “PIKACHU: Temporal walk based dynamic graph embedding for network anomaly detection,” in IEEE/IFIP Network Operations and Management Symposium (NOMS), 2022. [7] J. Zhou et al., “Lateral movement detection via time-aware subgraph classification on authentication logs,” arXiv preprint arXiv:2411.10279, 2024. [8] M. Foley, M. Wang, Z. M., C. Hicks, and V. Mavroudis, “Autonomous network defence using reinforcement learning,” ACM Computing Surveys, 2023.

[9] C. Hicks, V. Mavroudis, M. Foley et al., “Explainability for RLbased cyber defence agents,” in AAAI Workshop on AI for Cyber Security, 2023. [10] M. Standen, M. Lucas, D. Bowman, T. J. Richer, J. Kim, and D. Marriott, “CybORG: An autonomous cyber operations research gymnasium,” in Proceedings of the 1st International Workshop on Adaptive Cyber Defense (IJCAI), 2021. [11] H. Emerson, L. Bates, C. Hicks, and V. Mavroudis, “CybORG++: An enhanced gym for the development of autonomous cyber agents,” arXiv preprint arXiv:2410.16324, 2024. [12] P. Veličković, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y. Bengio, “Graph attention networks,” in International Conference on Learning Representations (ICLR), 2018. [13] X. Wang, H. Ji, C. Shi, B. Wang, Y. Ye, P. Cui, and P. S. Yu, “Heterogeneous graph attention network,” in The World Wide Web Conference (WWW), 2019, pp. 2022–2032. [14] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,” arXiv preprint arXiv:1707.06347, 2017. [15] E. Liang, R. Liaw, R. Nishihara, P. Moritz, R. Fox, K. Goldberg, J. E. Gonzalez, M. I. Jordan, and I. Stoica, “RLlib: Abstractions for distributed reinforcement learning,” in Proceedings of the 35th International Conference on Machine Learning (ICML), 2018, pp. 3053– 3062. [16] P. Moritz, R. Nishihara, S. Wang, A. Tumanov, R. Liaw, E. Liang, M. Elibol, Z. Yang, W. Paul, M. I. Jordan, and I. Stoica, “Ray: A distributed framework for emerging AI applications,” in 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI), 2018, pp. 561–577. [17] A. B. Yoo, M. A. Jette, and M. Grondona, “SLURM: Simple linux utility for resource management,” in Job Scheduling Strategies for Parallel Processing (JSSPP), 2003, pp. 44–60. [18] Neo4j, Inc., “Neo4j operations manual: Locks and deadlocks,” https: //neo4j.com/docs/operations-manual/current/, 2024. [19] M. Vermeer, N. Kadenko, M. van Eeten, C. Gañán, and S. Parkin, “Alert alchemy: SOC workflows and decisions in the management of NIDS rules,” in ACM Conference on Computer and Communications Security (CCS), 2022. [20] I. J. King, B. Trindade, B. Bowman, and H. H. Huang, “CyberGFM: Graph foundation models for lateral movement detection in enterprise networks,” arXiv preprint arXiv:2601.05988, 2026. [21] C. Larroche, “Designing a reliable lateral movement detector using a graph foundation model,” arXiv preprint arXiv:2504.13527, 2025. [22] K. Tholl, F. Rivest, M. E. Mezouar, A. Taylor, and R. A. Mallah, “Large language model integration with reinforcement learning to augment decision-making in autonomous cyber operations,” arXiv preprint arXiv:2509.05311, 2025. [23] Microsoft, “Microsoft security copilot: Empower defenders at the speed of ai,” https://www.microsoft.com/security/business/ ai-machine-learning/microsoft-security-copilot, 2024. [24] Z. Deng, Y. Guo et al., “AI agents under threat: A survey of key security challenges and future pathways,” arXiv preprint arXiv:2406.02630, 2024. [25] D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wressnegger, L. Cavallaro, and K. Rieck, “Dos and don’ts of machine learning in computer security,” in 31st USENIX Security Symposium, 2022, pp. 3971–3988. [26] T. Bilot, B. Jiang et al., “Reproducibility of provenance-based intrusion detection systems,” in 34th USENIX Security Symposium, 2025.

Related documents

• CAIRN: Dynamic Fact-Intent DAGs for Multi-Agent Exploration#1117031
• Towards Agentic Cloud Engineering: Graph and Loop Engineering with a Zero-Trust Agent Harness#819534
• Towards Agentic Cloud Engineering: Graph and Loop Engineering with a Zero-Trust Agent Harness#809538
• EntailLLM: Verifying LLM-Generated Vulnerability Discovery Paths with Domain Knowledge via Logic Programming#423846
• Reasoning Depth and Environment Complexity: A Controlled Study of RLVR Data Allocation across Logical Reasoning Tasks#973187