Learning Red Agent Policy from Observations for Neurosymbolic Autonomous Cyber Agents

Learning Red Agent Policy from Observations for Neurosymbolic Autonomous Cyber Agents Ankita Samaddar, Sandeep Neema, Daniel Balasubramanian, Xenofon Koutsoukos

arXiv:2606.18223v1 [cs.CR] 16 Jun 2026

Departmaent of Computer Science Vanderbilt University Nashville, TN {ankita.samaddar, sandeep.neema, daniel.a.balasubramanian, xenofon.koutsoukos}@vanderbilt.edu

Abstract—With sophisticated cyber-attacks becoming increasingly prevalent, modern networks require intelligent autonomous cyber-defense agents trained via Reinforcement Learning (RL). These agents employ neurosymbolic approaches such as behavior trees with learning-enabled components (LECs) to learn, reason, adapt, and implement security rules while maintaining critical operations. However, these autonomous networks are partially observable systems, i.e., the cyber-attacker’s (red agent’s) actions are not observable, making it difficult for the defender to predict red actions, learn red policies, or assess the attacker’s intrusion levels. To address this, we propose a Policy Learning Technique using imitation learning to learn policies for partially observable RL agents with discrete states and discrete actions. We apply this technique in an autonomous cyber environment to predict red agent’s actions from network observations and defender actions. Integrated with a neurosymbolic cyber-defense agent, our method effectively handles different red policies and achieves high prediction accuracy across diverse simulated scenarios.

I. I NTRODUCTION Modern cyber applications take the advantage of autonomous networks to automate workflows by deploying agents that independently learn and enforce security rules to defend against cyber-attacks [11]. Achieving this defense necessitates continuous system monitoring, early breach detection, and timely selection of appropriate countermeasures to contain the breaches, while ensuring uninterrupted operational workflows. Conventional security standards, however, often prove inadequate against sophisticated adversarial strategies. To address this limitation, a hybrid approach is needed that combines security standards with learning enabled components (LECs). These LECs are often reinforcement learning (RL) based function approximators enabling the cyber-defense agents to adapt dynamically and take optimal actions against evolving threats. In autonomous cyber environments, the network evolves from one state to another under the combined influence of red (attacker) and blue (defender) actions. However, the environment is partially observable, i.e., the defender cannot directly observe the red agent’s actions; instead it perceives only partial state transitions and indirect outcomes of attacks, making it difficult to reconstruct the true dynamics of the system. This partial observability poses challenges in predicting attacker’s (red) actions, and thereby, inferring attacker’s (red agent’s) policy. These challenges significantly limit the defender from

designing effective autonomous defense strategies. To overcome this limitation, we propose solutions to learn red agent policies and predict red actions at runtime. A few notable works in the literature leveraged imitation learning to learn latent policies from observations in classical control environments [4], [6], [8] and in internet video [18]. However, none of these works are directly applicable to autonomous network environments. This is because the states in autonomous networks are typically discrete in nature and contain a snapshot of different ongoing network activity on each host/server. Moreover, the state transitions in autonomous networks are dependent on both red agent (attacker’s) actions and blue agent (defender’s) actions in each timestep. Thus, we propose a Policy Learning Technique that learns red agent policy and predicts red actions at runtime from network observations and blue (defender) actions. Predicting red actions at runtime enables early detection of adversarial movements and estimation of attacker’s intrusion levels over time before deep network compromise. Reinforcement learning (RL) is widely used for autonomous cyber-defense, however, RL-based policies face challenges in scalability, interpretability, and adaptability in complex networks. Neurosymbolic methods address these limitations by combining learning with structured, interpretable reasoning. Recent works proposed behavior trees (BTs) [5], [12] augmented with learning-enabled components (LECs), known as Evolving Behavior Trees (EBTs), for autonomous cyber defense [15], [17]. EBTs are modular and hierarchical, enabling them to encode explicit subtasks, capture control flows, and incorporate adaptive LECs for decision-making under uncertainty. This design not only ensures the agent’s ability to respond to diverse and dynamic attack patterns, but also ensures that the resulting models generalize effectively across simulation and real-world deployment. In this work, we extend the EBT proposed in [17] by incorporating new behaviors to learn red agent policies and predict red actions at runtime. To train and evaluate our approach, we use the CybORG CAGE Challenge 2 environment [19], [20], which leverages tactics and techniques from MITRE ATT&CK [2] to create realistic cybersecurity environments for training red and blue agents. Thus, the main contributions of our work are as follows. 1) We propose a Policy Learning Technique that learns red agent (attacker) policies and predicts red (attacker)

actions from observations and blue (defender) actions in a partially observable RL-based autonomous cyberdefense environment with discrete states and discrete actions. 2) We integrate our proposed technique with a neurosymbolic behavior tree-based autonomous cyber-defense agent and demonstrate its ability to predict red actions in autonomous networks at runtime. 3) We evaluate our proposed technique using CybORG CAGE Challenge 2 [19], [20], a realistic cyber network simulation environment with tactics and techniques from MITRE ATT&CK [2]. Our approach demonstrates high accuracy in predicting the red actions under different adversarial strategies and under dynamic adversarial strategy switching. II. AUTONOMOUS C YBER -D EFENSE E NVIRONMENT The MITRE ATT&CK knowledge base captures real-world adversarial techniques and provides a structured model of attacker behavior across domains such as Enterprise, Mobile, and Industrial Control Systems [2]. It systematically organizes adversarial activity into: Tactics, Techniques, Sub-techniques and Procedures. The CybORG CAGE Challenge 2 interface, based on the MITRE ATT&CK for Enterprise networks [10], is designed to evaluate autonomous cyber-defense agents. The network consists of three subnets: Subnet 1 with five non-critical user hosts, Subnet 2 with three enterprise servers controlling the hosts in Subnet 1 and a host defending the network, and, Subnet 3 with three operational hosts and a critical operational server ensuring proper functioning of the network. Each scenario run or episode in CybORG runs for fixed timesteps where red (attacker) and blue (defender) agents select actions from their respective action spaces. Each red action in CybORG is derived from the MITRE ATT&CK knowledge base [19]. A description of the red actions is given in Table I. The red agent begins with an access to a user host and progresses through reconnaissance, exploitation, and privilege escalation to reach the enterprise servers and then the operational server, aiming to disrupt services via impact action [19]. TABLE I D ESCRIPTION OF THE RED ACTIONS [2]. Action Discover Remote Systems Discover Network Services Exploit Network Escalate Impact

Purpose ATT&CK Technique T1018; Discovers new hosts/IP addresses in the network through active scanning. ATT&CK Technique T1046; Discovers responsive services on a selected host by initiating a connection with that host. ATT&CK Technique T1210; Attempts to exploit a specified service on a remote system. ATT&CK Tactic TA0004; Escalates the agent’s privilege on the host ATT&CK Technique T1489; Disrupts the performance of the network and fulfils the attacker’s objective of denying the operational service.

The blue agent mitigates threats using actions such as monitor, analyse, deploy decoys, remove threats, and restore [10].

The environment includes two types of attacker strategies: M eander that explores and exploits the network one subnet at a time, seeking and gaining privileged access on all the hosts in a subnet before moving on to the next one, eventually reaching the Operational Server, and B line that attempts to reach the Operational Server directly using prior knowledge of the network layout. Potteiger et al. introduced a third red agent strategy, RedSwitch, that instantiates a red agent using M eander and switches to B line after a random number of timesteps [15]. III. AUTONOMOUS C YBER -D EFENSE AGENT USING E VOLVING B EHAVIOR T REES Neurosymbolic AI has emerged as a promising paradigm for autonomous cyber-defense, combining learning with structured reasoning to handle sophisticated attacks. Behavior Trees (BTs) serve as interpretable control structures that select defensive actions, and when augmented with learning-enabled components (LECs), they enable adaptive, high-level decisionmaking in dynamic environments. Their modularity supports easy integration of new behaviors for evolving threats. Prior work introduced Evolving Behavior Trees (EBTs), integrating BTs with LECs for adaptive defense [15], and extended them for runtime monitoring to detect anomalies and out-ofdistribution states in cyber-defense systems [17]. A Behavior Tree (BT) executes in discrete timesteps called ticks. At each tick, traversal starts from the root in a depthfirst manner, with each node returning one of three statuses: Running (execution ongoing), Success (goal achieved), or F ailure (goal not met). Nodes are of two types: Control behaviors and Execution behaviors. Control behaviors define execution flow: Sequence (succeeds if all children succeed), F allback (succeeds if any child succeeds), and P arallel (executes children simultaneously with configurable success conditions). Execution behaviors are leaf nodes representing Conditions (checks) or Actions (tasks). During each tick, leaf node statuses propagate upward recursively, determining the overall outcome at the root. Fig. 1 presents the EBT from [17], comprising six Action behaviors: SelectStrategy! (chooses defense strategy), GetSaf eAction! (restores system to a safe state), GetM etaAction! (selects a defensive behavior), GetAnalysisAction! (monitors and analyzes the network), GetDetectorAction! (deploys detectors), and GetM itigateAction! (removes threats or restores hosts). The EBT also includes five Condition behaviors to ensure correct strategy selection, maintain operation within the training distribution, and validate execution of defensive actions. These behaviors map directly to the CybORG CAGE Challenge 2 environment: strategy selection aligns with controller policies, GetSaf eAction! maps to Restore, GetAnalysisAction! to Monitor/Analyze, GetDetectorAction! to DeployDecoy, and GetM itigateAction! to Remove/Restore, enabling effective mitigation of adversarial activity.

Fig. 1. Evolving Behavior Tree (EBT) for Autonomous Cyber-Defense with online safety monitoring [17].

IV. L EARNING FROM O BSERVATIONS Autonomous networks are partially observable, as exemplified by CybORG CAGE Challenge 2 environment, which models real-world scenarios using the MITRE ATT&CK. In such environment, the defender has no direct knowledge of attacker policies or actions and must infer system states by monitoring and analyzing hosts and network activity at each timestep. A. Problem Formulation Our network can be represented as N = {U ∪ E ∪ O}, where U = {u1 , u2 , . . . , un1 } denotes the user subnet with n1 users, E = {e1 , e2 , . . . , en2 } denotes the enterprise subnet with n2 enterprise servers and hosts, and O = {o1 , o2 , . . . , on3 } denotes the operational subnet with n3 operational servers and hosts. There is a red agent (attacker) in the network with a set of k possible red actions RA = {r1 , r2 , . . . , rk } derived from tactics and techniques of MITRE ATT&CK [2], [19], [20]. To defend the network, there is a blue agent (defender) residing in the network with a possible set of m blue actions BA = {b1 , b2 , . . . , bm } [19], [20]. Depending on the target on which red and blue actions are executed, we define red and blue action spaces as follows. Definition 1: Red Action Space: The red agent is associated with a red action space Ar that is a combination of red action, ri ∈ RA, and the host or subnet, x, on which the action is executed. Ar : ri × x, where ri ∈ RA and (x = U ) ∨ (x = E) ∨ (x = O) ∨ (x ∈ U) ∨ (x ∈ E) ∨ (x ∈ O). Definition 2: Blue Action Space: To mitigate red actions, the blue agent is associated with a blue action space Ab that is a combination of blue action, bi ∈ BA, and the host or server, x, on which the action is executed. Ab : bi × x, where bi ∈ BA and (x ∈ U ) ∨ (x ∈ E) ∨ (x ∈ O). Our network can be represented as a discrete-time Partially Observable Markov Decision Process (POMDP), M := (S, A, O, T, R) [16]. Here, S denotes the finite set of states, A = {Ar , Ab } denotes the set of joint action spaces of the red and blue agents. At timestep t − 1, the red and blue agents perform the joint action ⃗at−1 = ⟨rat−1 , bat−1 ⟩ from Ar and Ab , respectively, resulting in the network state to transition from st−1 to st . O denotes the finite set of observations for the red and blue agents. At timestep t− 1, the

Fig. 2. Three stages of learning Red Agent Policy π ′ from blue observations and blue action.

joint observation ⃗ot−1 = ⟨ort−1 , obt−1 ⟩ denotes the observations available to the red and blue agents, respectively. T denotes the transition probabilities between states, where T (s′ , ⃗o | s, ⃗a) denotes the probability to transition to state s′ producing the joint observation ⃗o given the current state s and joint action ⃗a. R = {Rr , Rb } is the reward function for the red and blue agents, respectively. Note that our network is partially observable, i.e., the red observations and red actions are not observable to the blue agent. The main objective of the blue agent is to select a blue action timestep such that the Pt→∞at each b cumulative blue rewards t=1 Rt−1 maximize over time. Although the RL-based defender (blue agent) executes actions at each timestep by scanning and analyzing the network states, it lacks knowledge of the red agent’s policy or red actions. Therefore, our primary objective is to infer the red agent’s policy and predict red actions using the blue observations and the blue action at each timestep. Problem Statement: Given a network N with user subnet, enterprise subnet and operational subnet, a neurosymbolic autonomous cyber-defense agent (blue agent) trained with a RL policy π against a given cyber-attacker (red agent), our objective is to learn the red agent’s policy π ′ and predict red action rat−1 at any timestep t − 1 from blue observations in two consecutive timesteps, obt−1 and obt , and blue action bat−1 executed at timestep t − 1.

B. Learning Red Agent Policy We propose a Policy Learning Technique to learn a red agent policy π ′ given blue agent policy π. Our proposed technique, at each timestep t, takes the previous and current blue observations (obt−1 and obt ), and the previous blue action (bat−1 ) at timestep t − 1 as inputs, and predicts the red action (rat−1 ) executed at t − 1. Inferring red agent’s policies under partial observability is challenging due to complex network dynamics. While prior work such as LAPO learns latent actions from video using imitation learning [18], it is not directly applicable to our network as the blue observations in our network are discrete binary vectors, and each state transition depends on both observable blue action and non-observable red action. Algorithm 1 highlights the key steps of our proposed Policy Learning Technique which consists of three phases: 1. Data Generation Phase: For a given blue agent policy π, we collect data, i.e., the blue observation (obt ), blue action (bat ), and the blue observation transitions, obt−1 , bat−1 → obt , at each timestep by interacting with our system to generate training dataset D for T timesteps, (T is a large number), over N episodes (line 4 of Algorithm 1). Additionally, we generate a red action labeled dataset DR from the system for transitions in D (line 5 of Algorithm 1). Note that these red action labels are not observable to the blue agent in autonomous networks as well as our system. We tweak our system to generate these labels only for training purpose. We generate dataset DˆR by pre-processing the action labels in DR and splitting each red action label rat−1 into red action name, ri ∈ RA, host name, y ∈ N , and subnet z, where z can be either U, E, or O. 2. Policy Learning Phase: Learning any red agent policy π ′ involves the following steps as discussed below (see Fig. 2). I. Encoding: For each transition (obt−1 , bat−1 → obt ) in D, the blue action (bat−1 ) is decomposed into its action type bi ∈ BA and target host x ∈ N . These components are then one-hot encoded w.r.t. the blue action set BA and the set of hosts N , producing encoded blue action bˆi and encoded host x̂. II. Learning latent red actions: This stage jointly trains two dynamics models to learn latent red actions (lt−1 ) (lines 1012 of Algorithm 1). The first is an inverse-dynamics model (IDM1 ) which estimates PIDM1 (lt−1 | obt−1 , obt , bˆi , x̂). The second is a forward-dynamics model (F DM1 ) which predicts the current blue observation as PF DM1 (oebt | obt−1 , bˆi , x̂, lt−1 ). For each transition (obt−1 , bat−1 → obt ) in D, the IDM1 takes obt−1 , obt , and (bˆi , x̂) as input to infer the latent red action lt−1 . The F DM1 then uses obt−1 , (bˆi , x̂) and lt−1 to predict the current observation o˜bt . Both the models are trained jointly via stochastic gradient descent to minimize the observation error |o˜bt − obt |, thereby learning meaningful latent red actions. Since the blue observations in our networks are discrete binary vectors, we set a threshold to categorize the predicted observation vectors in binary. III. Training to learn Red Agent Policy π ′ : This stage trains a second inverse dynamics model IDM2 following b b ˆ PIDM2 (lg t−1 | ot−1 , ot , bi , x̂) to imitate the latent red actions learned by IDM1 (lines 13–14 of Algorithm 1). Trained

Algorithm 1: Policy Learning Technique (π) Data Generation Phase 1 Assign D, DR ,DˆR to {}; 2 for e = 1, 2, . . . , N episodes do 3 for t = 1, 2, . . . , T timesteps do 4 Collect transitions (obt−1 , bat−1 → obt ) from the system for blue agent policy π to generate D; 5 Label rat−1 from the system for transitions in D to generate DR ; 6 Pre-process rat−1 ∈ DR to generate DˆR ; Policy Learning Phase // Encoding b b 7 for each transition (ot−1 , bat−1 → ot ) ∈ D do 8 Split bat−1 into bi and x | bi ∈ BA, x ∈ N ; 9 Encode(bi ) → bˆi , Encode(x) → x̂; // Learn latent red actions Model IDM1 : PIDM1 (lt−1 | obt−1 , obt , bˆi , x̂); e | ob , bˆ , x̂, l ); 11 Model F DM1 : PF DM1 (ob t−1 t t−1 i 12 Train IDM1 and F DM1 jointly over D to learn lt−1 ; // Train red agent policy π ′ b b ˆ 13 Model IDM2 : PIDM (lg t−1 | ot−1 , ot , bi , x̂); ′ 14 Train IDM2 over D to learn policy π ; // Map latent red actions to red actions 15 Model F DM2 : PF DM2 (rei , y e, ze | lg t−1 ); ˆ 16 Train F DM2 over DR ; 17 Decode (rei , y e, ze) → r^ at−1 ; Action Prediction Phase ′ ′ ′ ′ 18 Select πi from {π1 , π2 , . . . , πn } corresponding to blue agent policy πi ; ′ b b ^ at−1 ; 19 πi (ot−1 , ot , bat−1 ) → r 20 return r ^ at−1 ; 10

via stochastic gradient descent, IDM2 learns policy π ′ by imitating latent red actions, minimizing the loss |lg t−1 − lt−1 |. IV. Mapping latent red actions to actual red actions in Ar : The predicted latent red actions are continuous vectors. To map them to discrete red actions in Ar , we train a forward dynamics model F DM2 defined as PF DM2 (rei , ye, ze | lg t−1 ) using the action-labeled preprocessed dataset DˆR (lines 15-16 of Algorithm 1). F DM2 learns to map each latent red action to its corresponding discrete action components. The outputs of F DM2 are continuous vectors, which are thresholded to obtain binary one-hot encodings. Each predicted red action comprises three components: the action name rei from RA, the target host ye and the subnet ze. A decoder is used to segment and map these outputs into the final predicted red action, r^ at−1 ∈ Ar (line 17 of Algorithm 1). Note that, all dynamics models across the training stages are implemented as deep neural networks (DNNs), with different model specifications for each stage. 3. Prediction Phase: Given a set of n trained blue agent

policies, {π1 , π2 , . . . , πn }, we learn a corresponding set of red agent policies, {π1′ , π2′ , . . . , πn′ }, offline using the proposed framework. At runtime, the system selects the appropriate red policy πi′ based on the active blue policy πi . At each timestep t, the selected policy takes previous blue observation obt−1 , current blue observation obt , and previous blue action bat−1 as input to predict the red action executed at timestep t − 1. V. I NCORPORATION OF R ED ACTION P REDICTION B EHAVIOR IN THE E VOLVING B EHAVIOR T REE We extend the EBT proposed in [17] by incorporating red action prediction behaviors and learning-enabled components (LECs) to enable red action prediction at every tick for different red agent policies. As shown in Fig. 3, we introduce one Condition and three Action behaviors (marked in red). The condition N umObs < 2? ensures availability of at least one prior observation (obt−1 ) before prediction. Leveraging the SelectStrategy! behavior [15], the defender identifies the active blue policy based on adversarial behavior and accordingly selects the learned red policy. The Action behaviors, Red Action in M eander! and Red Action in B line!, predict red actions under B line and M eander respectively. If no learned red policy exists, P olicyN otF ound! is triggered to log state transitions for offline analysis. VI. E XPERIMENTS AND E VALUATION We evaluate our proposed Policy Learning Technique in the CybORG CAGE Challenge 2 environment. To enable runtime prediction of red actions, we extend the EBT from [17] by integrating learned red agent policies and red action prediction behaviors. We evaluate our approach across two distinct red agent strategies and under dynamic red strategy switching. A. Experimental Setup For evaluation, we develop a software architecture extended from [17] that utilizes PyTrees and executes the Blue EBT agent, Runtime Monitoring, and the learned red policies on CybORG CAGE Challenge 2. Communications between the simulator and the EBT occurs via a blackboard interface to prevent data leakages. We perform experiments on a Linux machine (2.1 GHz Intel Xeon, 32 GB RAM). We train our models to learn two red agent policies by generating dataset D, consisting of 10, 000 transitions per strategy (100 episodes × 100 timesteps). From D, we derive the action-labeled dataset DR and processed it to generate DˆR for model training. We employ a three-stage dynamics model (Table II) optimized via stochastic gradient descent, utilizing Cross Entropy for Stages 1 and 3, and Mean Absolute Error for Stage 2. Blue actions are decomposed into action name and host, and encoded using one-hot encoding. In Stage 1, the inverse dynamics model (IDM1 ) takes inputs of size 130, derived from two consecutive binary state vectors (size 52 each), 13 blue actions, and 13 hosts. The forward model (F DM1 ) input size varies with latent action dimensions, which are tuned between 10-50; optimal sizes are 18 (B line) and 29 (M eander), reflecting higher variability in Meander behavior.

Stage 2 uses IDM2 with the same input size (130) to imitate latent actions. In Stage 3, F DM2 maps latent actions to discrete outputs, with input sizes 18 and 29 for B line and M eander, respectively. Training uses mini-batches of size 10 (with replacement) for 1000 epochs at a learning rate of 0.01. Outputs are thresholded to obtain binary representations. The final output is a 21-bit vector: 5 bits for action type, 3 for subnet, and 13 for host. A decoder maps these into executable red actions. These trained models are integrated as LECs within the EBT, enabling runtime prediction of red actions using consecutive blue observations and encoded blue actions. The link to our codebase is available at [1]. B. Evaluation and Results We evaluate our models under two adversarial strategies, B line and M eander, and further assess generalization under the RedSwitch strategy without additional training. For each policy, we conduct 10 test runs, each test run comprising 10 episodes of 100 timesteps each. Performance is measured using Prediction Accuracy (the ratio of correct predictions to total predictions) across predicted observations, predicted latent red actions, and predicted red actions. 1) Evaluation using Prediction Accuracy: Table III reports the average prediction accuracy across the three stages over 10 test runs. The highest accuracy for both blue observation and red action prediction is observed under B line due to its more deterministic and goal-oriented behavior as compared to higher variability under M eander and RedSwitch strategies. Table IV presents bit-level blue observation prediction errors (in percentage) in Stage 1 under B line, M eander and RedSwitch strategies. Errors are minimal under B line (upto 2 bit errors), while M eander and RedSwitch exhibit higher errors (upto 5 bit errors). Although M eander shows a higher percentage of 1-bit prediction errors than RedSwitch, it shows less percentage of n bit errors (n > 1). This is because, under red strategy switching, our network transitions to new states over some timesteps which are not in the training distribution. From Table III, it is evident that the latent red actions in Stage 2 are predicted with a low error bound (≤ 0.1) across all cases. In Stage 3, the red action prediction accuracy (at host and subnet levels) is highest for B line, reflecting its deterministic nature, while red action prediction is less for M eander due to higher randomness. Under RedSwitch, the red action prediction accuracy (host level) remains low for timesteps during which the red agent follows M eander strategy. However, on switching to B line, prediction accuracy increases which increases the overall prediction accuracy over an episode as compared to M eander. Also, across all three strategies, subnet-level red action prediction accuracy consistently exceeds host-level accuracy. This is due to the inherent randomness in red agents’ selection of target hosts, which makes fine-grained (host-level) prediction more challenging. 2) Evaluation using Intrusion Level Measure: The red actions predicted by our proposed technique can be further analyzed to assess the intrusion level of different red agents

Fig. 3. EBT in [17] with newly added behaviors (nodes marked in red) to predict red actions at runtime. TABLE II M ODEL SPECIFICATIONS OF THE THREE STAGE DYNAMICS MODELS FOR TRAINING DIFFERENT RED STRATEGIES . Stage Stage 1 (Learn (latent red actions) Stage 2 (Train red policy) Stage 3 (Predict red actions)

Model IDM1

Input obt−1 , obt , bat−1

F DM1

obt−1 , bat−1 , lt−1

IDM2

obt−1 , obt , bat−1

F DM2

lg t−1

Architecture DNN with 3 hidden layers each of size 150, and ReLU activation in each layer DNN with 3 hidden layers each of size 120, ReLU activation in each layer and Softplus in the last layer DNN with 5 hidden layers each of size 150, and ReLU activation in each layer DNN with 6 hidden layers each of size 15, ReLU activation function in each layer and Softplus in the last layer

TABLE III P REDICTION ACCURACY OF OUR MODELS UNDER DIFFERENT RED STRATEGIES . Red Policy

Stage 1 99.93%

B line

94.96%

M eander RedSwitch

95.03%

Stage 2 All latent red actions are learnt with an error bound of 0.1

Stage 3 95.28% (host) 99.82% (subnet) 77.71% (host) 97.49% (subnet) 83.83% (host) 95.85% (subnet)

TABLE IV B LUE O BSERVATION P REDICTION E RROR UNDER D IFFERENT R ED S TRATEGIES . Red Policy B line M eander RedSwitch

1-bit 0.02% 4.48% 2.28%

Blue Observation Prediction Error 2-bit 3-bit 4-bit 5-bit 0.04% 0% 0% 0% 0.27% 0.11% 0.07% 0.07% 1.51% 0.64% 0.49% 0.03%

  0 Ii = 1   10

Output lt−1 (intermediate) o˜b t

lg t−1 ra ^ t−1

if i is in No Compromise, if i is in User-Level Compromise, if i is in Privilege-Level Compromise

P|N | The total network intrusion at each timestep is i=1 Ii . Table V shows predicted red actions, their target host/subnet and red action frequencies under Privilege-Level Compromise out of 100 episodes each with 100 timesteps. From Table V, our network undergoes Privilege-Level Compromise for 82 timesteps under M eander, 903 timesteps under B line and 724 timesteps under RedSwitch. These results indicate that our blue EBT agent is significantly more effective in mitigating attacks under M eander strategy as compared to B line and RedSwitch. By analyzing the blue observations, predicted red actions and network parameters in these timesteps, we can identify the weaknesses in our network and refine our mitigation actions accordingly. VII. R ELATED W ORKS

in the network. This analysis further enables: 1) Evaluation of blue agent’s effectiveness and refinement of mitigation strategies if needed, and 2) Vulnerability analysis to identify weaknesses exposed by the red agent activity. In CybORG CAGE Challenge 2, the blue observations are represented as 52-bit binary vectors, with four bits per host/server. Two of these bits encode compromise levels: ‘00’ (No Compromise), ‘01’ (User-Level Compromise), ‘11’ (Privilege-Level Compromise). We quantify intrusion by assigning an intrusion score Ii to each host/server i, (i ∈ N ).

The cybersecurity landscape is rapidly evolving as attacks become more sophisticated. The MITRE ATT&CK knowledge base provides a foundation for modeling adversarial behavior [2], while the CybORG CAGE Challenge 2 serves as a widely used environment for training and evaluating RL-based defense agents [10]. Prior work has explored hierarchical RL [7], ensemble methods [20], curriculum learning [13], and reward shaping [3] to address sparse, penalty-driven feedback in cyber environments. Parallel efforts in neurosymbolic AI combine neural learning with symbolic reasoning for robust and interpretable

TABLE V P REDICTED R ED ACTIONS WITH TARGET HOST / SUBNET AND FREQUENCIES UNDER P RIVILEGE -L EVEL C OMPROMISE FOR D IFFERENT R ED S TRATEGIES .

Red Policy B line

M eander RedSwitch

Predicted Red Actions with target host/subnet and their frequencies DiscoverNetworkServices DiscoverRemoteSystems ExploitRemoteService PrivilegeEscalate Enterprise0: 203, Enterprise2: 16, Enterprise0: 16, User3: 190, Enterprise0: 257, User3: 180, User2: 1, Op Server0: 3 Enterprise1: 2 Enterprise2: 16 Enterprise0: 16, Enterprise1: 3, Enterprise2: 3 Enterprise1: 7 User2: 2, User1: 1, User4: 1, User3: 47, User4: 1, User3: 1, Enterprise0: 1 Enterprise1: 21 User3: 3, Enterprise0: 143, Enterprise0: 18, User3: 132, Enterprise0: 187, User3: 171, User4: 2, User2: 2, Enterprise1: 5, Enterprise2: 17, Enterprise1: 1 Enterprise2: 16, Enterprise1: 1 Enterprise0: 18, Enterprise2: 4 Op Server0: 4

cyber defense [9]. Behavior Trees (BTs), in particular, enable modular and interpretable policy design, often integrated with RL or HRL [12]. Extensions such as Evolving Behavior Trees (EBTs) [15] and runtime monitoring frameworks [17] improve adaptability and safety in partially observable settings. However, these approaches do not explicitly model or predict attacker (red agent) policies. In autonomous network defense, partial observability remains a key challenge, as defenders must act with limited visibility while attacker actions remain hidden. Although imitation learning has been used to infer latent policies in other domains [4], [6], [8], [14], [18], none of these works directly apply to discrete, partially observable cyber environments. VIII. C ONCLUSION AND F UTURE W ORKS Autonomous networks increasingly rely on RL-based neurosymbolic agents for cyber defense, but partial observability limits a defender’s ability to infer attacker behavior. To address this, we propose a Policy Learning Technique that uses imitation learning to infer red agent policies and predict red actions at runtime from consecutive observations and blue actions. Beyond cyber defense, our proposed technique can also be adopted to learn a policy for any partially observable RL-based agent with discrete states and discrete actions. We integrate this technique into a behavior tree-based framework, enabling proactive defense and improved intrusion assessment. Evaluations in the CybORG CAGE Challenge 2 environment across multiple adversarial strategies show high prediction accuracy and effective intrusion-level estimation. However, as this environment is simulated (based on the MITRE ATT&CK model), real-world networks may exhibit more complex dynamics. Future work will focus on validating the approach in realistic emulated environments and leveraging predicted red actions to train more adaptive and robust blue agents. IX. ACKNOWLEDGMENT This material is based on research sponsored by DARPA under agreement number HR001124C0425. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of DARPA or the U.S. Government.

R EFERENCES [1] Codebase. https://github.com/anki2911/Learning Red Policy. [2] MITRE ATT&CK. https://attack.mitre.org/. [3] Elizabeth Bates, Vasilios Mavroudis, and Chris Hicks. Reward Shaping for Happier Autonomous Cyber Security Agents. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security, AISec, 2023. [4] Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Jianhao Wang, Alex Yuan Gao, Wenzhe Li, Liang Bin, Chelsea Finn, and Chongjie Zhang. LAPO: Latent-Variable AdvantageWeighted Policy Optimization for Offline Reinforcement Learning. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35. Curran Associates, Inc., 2022. [5] Michele Colledanchise and Petter Ögren. Behavior Trees in Robotics and AI. 2018. [6] Ashley Edwards, Himanshu Sahni, Yannick Schroecker, and Charles Isbell. Imitating Latent Policies from Observation. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. PMLR, 2019. [7] Myles Foley, Chris Hicks, Kate Highnam, and Vasilios Mavroudis. Autonomous Network Defence using Reinforcement Learning. In Proceedings of the 2022 ACM on Asia Conference on Computer and Communications Security, ASIA CCS 2022. [8] Dylan J. Foster, Adam Block, and Dipendra Misra. Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning. In Advances in Neural Information Processing Systems, volume 37, 2024. [9] Brian Jalaian and Nathaniel D. Bastian. Neurosymbolic AI in Cybersecurity: Bridging Pattern Recognition and Symbolic Reasoning. In MILCOM 2023 - 2023 IEEE Military Communications Conference (MILCOM), 2023. [10] Mitchell Kiely, David Bowman, Maxwell Standen, and Christopher Moir. On autonomous agents in a cyber defence environment. ArXiv, 2023. [11] Alexander Kott, Paul Théron, Martin Drašar, Edlira Dushku, Benoı̂t LeBlanc, Paul Losiewicz, Alessandro Guarino, Luigi Mancini, Agostino Panico, Mauno Pihelgas, Krzysztof Rzadca, and Fabio De Gaspari. Autonomous intelligent cyber-defense agent (aica) reference architecture. release 2.0, 2023. [12] Lei Li, Lei Wang, Yuanzhi Li, and Jie Sheng. Mixed Deep Reinforcement Learningbehavior Tree for Intelligent Agents Design. In International Conference on Agents and Artificial Intelligence, 2021. [13] Andres Molina-Markham, Cory Miniter, Becky Powell, and Ahmad Ridley. Network Environment Design for Autonomous Cyberdefense. arXiv e-prints, 2021. [14] Malte Mosbach, Jan Niklas Ewertz, Angel Villar-Corrales, and Sven Behnke. Learning Object-centric Latent Dynamics for Reinforcement Learning from Pixels. In The Thirteenth International Conference on Learning Representations (ICLR), 2025. [15] Nicholas Potteiger, Ankita Samaddar, Hunter Bergstrom, and Xenofon Koutsoukos. Designing Robust Cyber-Defense Agents with Evolving Behavior Trees. In 2024 International Conference on Assured Autonomy (ICAA). IEEE, 2024. [16] Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. John Wiley & Sons, 2014. [17] Ankita Samaddar, Nicholas Potteiger, and Xenofon Koutsoukos. Out-ofDistribution Detection for Neurosymbolic Autonomous Cyber Agents. In 2025 IEEE 4th International Conference on AI in Cybersecurity (ICAIC), 2025. [18] Dominik Schmidt and Minqi Jiang. Learning to act without actions. In The Twelfth International Conference on Learning Representations (ICLR), 2024. [19] Maxwell Standen, Martin Lucas, David Bowman, Toby J. Richer, Junae Kim, and Damian Marriott. CybORG: A Gym for the Development of Autonomous Cyber Agents. In IJCAI-21 1st International Workshop on Adaptive Cyber Defense., 2021. [20] Melody Wolk, Andy Applebaum, Camron Dennler, Patrick Dwyer, Marina Moskowitz, Harold Nguyen, Nicole Nichols, Nicole Park, Paul Rachwalski, Frank Rau, , and Adrian Webster. Beyond CAGE: Investigating Generalization of Learned Autonomous Network Defense Policies. In NeurIPS, 2022.