arXiv CSopen access

Safety-Contract Graph Multi-Agent Reinforcement Learning for Autonomous Network Security Response

Safety-Contract Graph Multi-Agent Reinforcement Learning for Autonomous Network Security Response Jose Luis Lima de Jesus Silvaa,b a Oxaala Tecnologias, Rua Dinah Silveira de Queirós, 06, Quinta do Candeal, Horto Florestal, Salvador, 40296-160, Bahia, Brazil

arXiv:2606.13832v1 [cs.MA] 11 Jun 2026

b Universidade Federal da Bahia, Instituto de Geociências, Rua Barão de Jeremoabo, s/n, Ondina, Salvador, Bahia, 40170-115, Brazil

ARTICLE INFO

ABSTRACT

Keywords: Multi-agent reinforcement learning Constrained Markov decision process Graph attention network Network security Expert system Decision support Operational safety

Autonomous network-security response systems promise to reduce Security Operations Centre (SOC) reaction latency, but reward-only multi-agent reinforcement learning (MARL) can improve security reward while remaining non-deployable. We present a safety-contract graph MARL framework and instantiate it as ACD3 -GAT (Adaptive Constrained Counterfactual Decisioning with a Graph Attention Network encoder), an architecture that separates simulator observations from reusable operational budgets, constrained optimization, graph state encoding, and counterfactual action screening. We evaluate the method in CAGE Challenge 4, where agents operate under budgets for Mean Time to Recover (MTTR), false-positive response, and firewall change-management disruption. Across the benchmark, every unconstrained method violates the SOC downtime budget in 100% of evaluated episodes, with mean downtime proxy costs of 311–430 against a budget of 50. This complements prior CAGE Challenge 4 findings by showing that reward-only learning lacks operational discipline. Constrained MAPPO-GAT (C-MAPPO-GAT) isolates Lagrangian operational-cost control and budgetaware screening, while ACD3 -GAT adds budget context, CVaR tail-risk estimation, opponent-belief state, and Graph Counterfactual Risk Propagation (G-CRP). The replicated comparison includes three 200-episode seeds for IPPO, MAPPO-GAT, C-MAPPO-GAT, and ACD3 -GAT. C-MAPPO-GAT reduces downtime violation from 100% to 0.3% and mean downtime cost from 355.4 to 15.5 relative to MAPPO-GAT. ACD3 -GAT reduces mean downtime cost to 48.2 with a 13.8% violation rate, placing it on the safety-contract frontier rather than at the most conservative compliance point. Topology-seed and coupled adaptive Red-process stress tests preserve this contrast and show lower worst adaptive degradation for safety-constrained policies than reward-only MAPPO-GAT.

1. Introduction The volume and velocity of network-security incidents facing modern enterprises have outpaced human response capacity (Vyas et al., 2023). Security Operations Centers (SOCs) must triage thousands of alerts daily, manually correlating events, isolating hosts, and reimaging compromised systems, and adjusting firewall policies while maintaining the availability of mission-critical services. Intelligent network security agents that can recommend, screen, or execute these responses are therefore an active and growing research priority for Civilian SOC automation and enterprise network management. This setting is intrinsically structured and sequential: in CAGE Challenge 4, five CAGE “Blue” agents act over a 500-step episode from partial binary observations of a changing enterprise network, choosing action types and topology-dependent targets while the simulator’s CAGE “Red” process continues to discover, escalate, and disrupt services. A response action can have delayed, competing effects, whether it is a Restore, which may remove a compromised session while taking a host offline, or BlockTrafficZone, which may protect one mission path while disrupting another. Furthermore, a missed response may only become visible many steps later. Therefore, an autonomous network ∗ Corresponding author

[email protected] (J.L.L.d.J. Silva) https://oxaala.com.br (J.L.L.d.J. Silva)

ORCID (s):

security response requires long-horizon multi-agent coordination that accounts for partial observability, invalid-action constraints, and non-stationarity due to opposing processes. The learning objective must therefore capture both containment of malicious activity and the operational acceptability of the resulting intervention pattern. Existing autonomous cyber-response research using reinforcement learning (RL) typically optimizes a scalar security reward without imposing explicit operational constraints. In practice, every response action carries an operational cost. For example, a host R ESTORE action takes the system offline for the duration of reimaging. Repeated B LOCKT RAFFICZ ONE and A LLOW T RAFFIC Z ONE commands create firewall-policy violations under change-management governance, while R ESTORE actions issued without clear malicious evidence constitute false-positive responses that burden SOC analysts. If an RL agent learns to issue excessive Restore actions to clear compromised sessions, which is a sensible strategy for maximizing security reward in the simulator, it may simultaneously exhaust the organization’s Mean Time to Recover (MTTR) budget, flood the change management system, and generate analyst fatigue. In our evaluation, every unconstrained learning method reaches a downtime-budget violation rate of 100% (𝑃 (violDT ) = 1.00), with mean per-episode downtime proxy costs of 311–430 Restore-action events against an episode budget of 50. This pattern complements the central finding of the CAGE Challenge 4 evaluation (Kiely et al.,

Preprint submitted for review. Final version may differ from this manuscript.

Page 1 of 30

Safety-Contract Graph MARL

2025b), where carefully engineered heuristics outperformed the submitted multi-agent reinforcement learning (MARL) agents because they encoded valid-action handling, missionphase traffic discipline, and selective incident-response logic that reward-only MARL did not reliably discover. Our contribution is to make that discipline explicit as a measurable safety contract rather than leaving it implicit in manually engineered policies. As a direct consequence, the learned agent becomes an auditable decision-support component whose actions can be checked against local MTTR, firewall-change, and false-positive response budgets. We also introduce ACD3 -GAT (Adaptive Constrained Counterfactual Decisioning with a Graph Attention Network), a safety-contract graph MARL framework for autonomous network security response. The framework builds on established components used in the CAGE cyber-range evaluation (TTCP CAGE Working Group, 2023; Kiely et al., 2025b), Graph Attention Networks (Veličković et al., 2018; Sandoval et al., 2025), Proximal Policy Optimization (PPO) and MAPPO-style multi-agent policy optimisation (Schulman et al., 2017; Yu et al., 2022), graph-based generalisation (King et al., 2025), and constrained reinforcement learning (Altman, 1999; Achiam et al., 2017). Our contribution is to combine them around a specific deployability problem, where the SOC operational budgets must be represented explicitly during training, action screening, and evaluation. In this formulation, the downtime/MTTR, falsepositive response, and firewall change disruptions are not treated as incidental side effects of the reward function, but as operational quantities that the policy must learn to respect and that the evaluator can audit. We introduce C-MAPPO-GAT as a controlled constrained instantiation that combines a MAPPO centralized critic, a GAT observation encoder, Lagrangian operational-cost advantages, and a budget-exhaustion fallback under the same SOC contract. This is a baseline that isolates the effect of adding explicit operational-cost control before the broader ACD3 -GAT architecture adds budget context, tail-risk estimation, opponent-state information, and counterfactual action screening. The framework separates reusable safety-contract components from simulator-specific interfaces. Operational budgets, Lagrangian cost learning, graph-structured encoding, counterfactual screening, and tail-risk accounting form the reusable safety-contract machinery. By contrast, the observation parser and valid-action mapping are treated as environment interfaces. The empirical claims in this paper are therefore kept deliberately grounded in the CAGE-4 evaluation and the reported robustness stress tests. Autonomous network-security response is treated here as a constrained Decentralized Partially Observable Markov Decision Process (Dec-POMDP). The formulation optimizes response policies not only for CAGE-4 reward, but also for operational acceptability under SOC budget constraints. We therefore attach three budget counters to the response process: (i) R ESTORE action is counted against the downtime and MTTR budget (𝐵down =50), (ii) a response

taken without visible malicious evidence is counted against the false-positive budget, which represents analyst burden (𝐵f p =10), and (iii) traffic-control actions are counted against the firewall-disruption budget, which represents change-management pressure (𝐵f w =20). The same counters are used during training, screening, and evaluation, and an episode is acceptable only if the agent contains the simulated intrusion without exceeding any SOC budget. The framework turns this contract into an expert-system layer for graph MARL. Operational budgets, Lagrangian cost learning, action-screening rules, and safety-labelled trajectory logging are treated as reusable decision-support components, while the CAGE-4 observation parser and valid-action mapping remain simulator-specific interfaces. This separation keeps the empirical claims grounded in CAGE-4 while allowing the contract layer to be reused with other observation formats and action schemas. Our resulting architecture, the ACD3 -GAT, combines the host-subnet graph encoders, a factorized target-action policy, operational cost critics, CVaR-based tail-risk estimation, override signals, Lagrangian cost learning, and Graph Counterfactual Risk Propagation (G-CRP). We also introduce the C-MAPPO-GAT as a controlled, constrained graph-MARL baseline, which isolates the effect of explicit operational-cost control before the broader ACD3 architecture adds budget context, opponent-state information, tail-risk accounting, and counterfactual action screening. In this work, we evaluate reward-only, graph-attentive, Lagrangian-constrained, and ACD3 policies under the same safety-contract metrics, with explicit seed and episode reporting. We also introduce Temporal Contract Graph Shielding (TCGS) is evaluated as a diagnostic extension that estimates whether a candidate action is likely to push the episode beyond a SOC budget given the recent history of alerts, actions, rewards, and accumulated costs. TCGS learns this future budget-violation risk from safety-labelled trajectory histories and uses the prediction to screen frozen ACD3 GAT action proposals before execution. It is therefore reported as evidence that temporal contract-risk prediction can support action screening, rather than as a claim that we have trained a new end-to-end TCGS policy. Experiments on CAGE Challenge 4 show that the replicated core benchmark has three 200-episode seeds for Independent Proximal Policy Optimisation (IPPO), Multi-Agent PPO with a GAT encoder (MAPPO-GAT), constrained MAPPO-GAT, and ACD3 -GAT. The results show that the Constrained MAPPO-GAT reduces downtime violation from 100% to 0.3% and mean downtime cost from 355.4 to 15.5 relative to MAPPOGAT. ACD3 -GAT reduces mean downtime cost to 48.2 with a 13.8% violation rate, placing the integrated method on a broader safety-contract frontier rather than at the most conservative compliance point. This result separates two claims: ACD3 -GAT defines the general safety-contract architecture, while the experiments

Preprint submitted for review. Final version may differ from this manuscript.

Page 2 of 30

Safety-Contract Graph MARL

identify which configurations already deliver reliable operational compliance. We have also conducted robustness stress tests, which strengthen this interpretation, since the constrained MAPPO-GAT preserves the downtime contract under topology-seed shifts, and the coupled adaptive Redprocess stress test shows lower worst-case degradation for constrained and ACD3 policies than for reward-only MAPPO-GAT.

2. Related Work 2.1. Autonomous Network Security Environments The growth of autonomous cyber-response research has driven a proliferation of simulation environments (Vyas et al., 2023). CybORG (Standen et al., 2021) underpins the CAGE challenge series and provides the enterprise network model used in this work. The CAGE Challenge 4 is the first multi-agent CAGE variant, requiring five cooperating CAGE “Blue” response agents to operate against a persistent CAGE “Red” process across 500-step episodes (TTCP CAGE Working Group, 2023; Kiely et al., 2025b). The principal finding of the CAGE 4 evaluation, that carefully engineered heuristics outperformed the submitted multi-agent reinforcement learning (MARL) agents, motivating our focus on operationally constrained training. Those heuristics encoded valid-action handling, event filtering, and missionphase traffic discipline, and selective Restore/Remove behavior that reward-only MARL agents did not learn reliably. Beyond CAGE studies have also shown that Proximal Policy Optimisation (PPO)-family response policies can degrade under unseen networks and opposing strategies (Wolk et al., 2022). Together, these results motivate a deployability question that is distinct from reward generalisation, and focus on whether a learned responder remains within the operational budgets that a SOC would impose on its actions.

2.2. MARL for Network Security Response Multi-agent reinforcement learning has been applied to network intrusion response (Vyas et al., 2023), with most prior work using independent actor-critic variants (de Witt et al., 2020) or centralised critics (Yu et al., 2022). The published CAGE 4 analysis reports that the best default CAGE Challenge 4 (CC4) top-team entry was heuristic (−113 ± 35 mean return) while the top-team MARL entry scored −193± 84 over 100 episodes of 500 steps (Kiely et al., 2025a). The −101 ± 36 value is the constant-network-size reference setting, which is useful as a reference point but not the default CC4 score (Kiely et al., 2025a). Our work addresses the same gap from a deployability perspective by imposing explicit operational constraints. Recent Large Language Model (LLM)-based CAGE 4 agents achieve approximately −2,888 with role prompting (GPT-o1-mini) (Castro et al., 2025). Hierarchical MARL for CAGE 4 decomposes the response into sub-policies for investigation and recovery, improving convergence and reporting interpretable metrics such as clean-machine ratio, precision, and false positives (Singh et al., 2024). A recent LLM-based CAGE 4 work instead

emphasizes explainability, natural-language observation formatting, and communication among LLM/RL teams (Castro et al., 2025). These lines complement our contribution because they improve task decomposition, reasoning, or communication, whereas ACD3 -GAT turns operational harms into explicit episode-level SOC budgets and optimizes or screens policies against those budgets.

2.3. Constrained and Safe Reinforcement Learning The Constrained MDPs (Altman, 1999) formalize budget constraints via Lagrangian relaxation, with theoretical convergence results under standard regularity conditions. Furthermore, the Constrained policy optimization extends this idea to deep policy gradient settings with trust-region style constraint handling (Achiam et al., 2017). Our neural MARL setting is non-convex, so we use this machinery as a practical budget-enforcement mechanism rather than as a proof of formal constraint satisfaction. Additionally, Safe reinforcement learning (RL) benchmarks evaluate singleagent constraint satisfaction (Ray et al., 2019). Therefore, we extend this to multi-agent network-security response with three simultaneous SOC constraints. Our Lagrangian update (dual step size 𝜂𝜆 =0.01) follows the application of projected subgradient methods on the dual variables.

2.4. Graph Neural Networks in Security Graph-structured representations have been applied to network intrusion detection and malware classification, often through message-passing neural networks (Gilmer et al., 2017) and inductive neighborhood aggregation such as GraphSAGE (Hamilton et al., 2017). Temporal graph networks provide a general memory-based framework for dynamic graphs (Rossi et al., 2020), and cyber detection systems have used spatio-temporal graph neural networks (GNNs) for smart-grid intrusion localization (Haghshenas et al., 2022) and network-intrusion detection (Van Langendonck et al., 2024). Those works primarily solve detection, localization, or dynamic-graph prediction problems. Within autonomous network response, attentive graph agents have already shown that Graph Attention Network (GAT) policies can exploit network topology and adapt across changed network structures (Sandoval et al., 2025), while graph-based RL agents represent ACD observations and actions as attributed graphs to improve zero-shot topology generalisation (King et al., 2025). Accordingly, graph attention is the perception layer in our system, while the new problem formulation is a safetycontract graph MARL, where the key evaluation question is whether response actions remain within explicit SOC budgets. This positioning also clarifies the role of C-MAPPOGAT in our benchmark. The MAPPO (Yu et al., 2022), GAT encoders (Veličković et al., 2018), and constrained MDP/Lagrangian safety methods (Altman, 1999; Achiam et al., 2017) are established components, but C-MAPPOGAT is introduced here as their controlled SOC-budget

Preprint submitted for review. Final version may differ from this manuscript.

Page 3 of 30

Safety-Contract Graph MARL Table 1 Positioning relative to closely related autonomous cyber-defence research. The comparison emphasises the evaluation axis rather than ranking prior systems by return or topology generalisation. Citations in the first column identify representative work for each research line. Research line

Primary objective

Deployability axis addressed here

CAGE/CybORG benchmarks and Beyond CAGE (Standen et al., 2021; TTCP CAGE Working Group, 2023; Kiely et al., 2025b; Wolk et al., 2022)

Cyber-range evaluation, reward generalisation, and comparison with engineered heuristics

We add explicit episode-level SOC budgets and violation rates, showing that reward-improving MARL can remain operationally non-deployable.

Hierarchical and LLM-assisted CAGE-4 agents (Singh et al., 2024; Castro et al., 2025)

Task decomposition, convergence, clean-machine ratio, precision/false positives, reasoning, and communication

We treat downtime, false-positive response, and firewall-change disruption as governance constraints optimized or screened during response.

Topology-adaptive GAT and graph-RL defenders (Veličković et al., 2018; Sandoval et al., 2025; King et al., 2025)

Graph representation, topology adaptation, Graph attention is used as the perception layer inside a and zero-shot generalisation of constrained safety-contract policy rather than as the sole cyber-defence policies novelty or the headline claim.

Dynamic graph memory, intrusion Temporal graph and detection, and attack localisation cyber-detection GNNs (Rossi et al., 2020; Haghshenas et al., 2022; Van Langendonck et al., 2024)

The response problem adds multi-agent action selection and operational costs; TCGS is a diagnostic step toward temporal contract-risk screening.

Asynchronous cyber-range MARL (Jankowski, 2026)

The proposed safety-contract layer can be ported to richer environments once their action traces support SOC cost accounting.

Simulator realism, continuous-time telemetry, and Sim2Real evaluation

instantiation for CAGE-4 response. It is therefore not treated as a named prior algorithm; it is the constrained baseline that isolates the effect of explicit operational costs before the broader ACD3 -GAT architecture adds budget context, tailrisk accounting, opponent state, and counterfactual screening.

2026). Our contribution is orthogonal to that simulatorrealism direction, as the safety-contract layer introduced here can be instantiated in CAGE-4 today and can also serve as the governance layer for future temporal or asynchronous cyber-range environments.

2.5. Risk-Sensitive Reinforcement Learning

3. Evaluation Scope and Operational Safety Problem

Conditional Value-at-Risk (CVaR) optimisation targets the tail of the return distribution (Rockafellar and Uryasev, 2000). This is particularly important in cybersecurity, where a single catastrophic episode (full critical-zone compromise) may outweigh many successful ones. We integrate CVaR episode reweighting into the MARL training loop via batchlevel importance weights.

2.6. Adaptive Opposing-Policy Evaluation as Robustness Context Adaptive opposing-policy evaluation is a standard way to probe whether an autonomous response policy is brittle to changes in the simulated source of malicious activity, but it is rarely formalised in CAGE-style benchmarks. We retain this as a robustness extension in the evaluation protocol, closest in spirit to self-play in multi-agent game-playing AI (Bansal et al., 2018) and population-based training (Jaderberg et al., 2019), adapted to the asymmetric CAGE Blue–Red setting of enterprise network-security response. Recent asynchronous cyber-range work argues that simulator-to-SOC transfer also requires continuous-time events, noisy telemetry, and richer hypervisor-backed evaluation (Jankowski,

3.1. Network and Asset Model We adopt the CAGE Challenge 4 enterprise network model (TTCP CAGE Working Group, 2023; Kiely et al., 2025b). The network comprises nine subnets spanning internet-facing, contractor, office, administrative, restricted, and operational zones. Mission-critical assets reside in two “operational zones” whose compromise or unavailability directly degrades the simulated mission score. All response actions are abstract simulation primitives, where no exploit code, credentials, malware, vulnerability details, or real network infrastructure are involved. The Red process’s actions (discover, exploit, escalate, degrade, impact, withdraw) are likewise simulator-internal operations. This paper is therefore a civilian enterprise SOC decisionsupport study conducted inside a closed simulator, and it provides no real-world intrusion capability.

3.2. Opposing Process Model The CAGE Red process begins with a session on the contractor subnet (simulating a supply-chain foothold) and attempts lateral movement toward restricted and operational

Preprint submitted for review. Final version may differ from this manuscript.

Page 4 of 30

Safety-Contract Graph MARL

zones. We evaluate against three Red-process strategy classes: 1. Finite-state machine (FSM) Red process: the default CAGE-4 scripted process that selects targets probabilistically using a fixed host-state transition matrix (Kiely et al., 2025b). 2. Discovery Red: a variant that emphasises reconnaissance before impact, creating longer dwell time. 3. Learned Red (adaptive): a PPO-trained Red process adapted online against a frozen Blue policy, used exclusively for exploitability evaluation.

3.3. Operational Safety: SOC Budget Constraints A response policy is not considered operationally successful merely by achieving high security reward. Real SOC governance imposes three categories of operational constraint that map directly to our cost signals (Section 5.2): The budgets are deliberately explicit stress-test thresholds for the CAGE-4 simulator rather than universal SOC constants; in deployment they would be set by local service-level objectives, change-control policy, and analyst capacity.

(1) Mean Time to Recover (MTTR) — 𝑐𝑡down . Host

reimaging (Restore) renders a machine unavailable for several minutes in practice. Repeated reimaging of hosts violates MTTR service-level agreements (SLAs) and degrades green-user access to services. We proxy this with a count of Restore actions per episode, budgeted at 𝐵down =50. fp

(2) False-Positive Response Rate — 𝑐𝑡 . Restore or Remove actions issued in the absence of clear malicious evidence constitute false-positive responses. They consume analyst time for post-action review, create unnecessary disruption, and erode operator trust in the autonomous system. Budget: 𝐵f p =10.

(3) Firewall Change-Management Policy — 𝑐𝑡fw .

BlockTrafficZone and AllowTrafficZone actions modify network topology. In enterprise environments, such changes require change-management approval and create audit trails. Excessive firewall churn violates change-management governance. Budget: 𝐵fw =20.

credentials, or exploit code are involved. Red-process actions are abstract primitives with no operational meaning outside the simulator. The anonymised replication package contains no offensive capability.

4. Theoretical Foundations 4.1. Markov Decision Process Definition 4.1 (Markov Decision Process (MDP)). A Markov Decision Process (MDP) is a tuple (, , 𝑃 , 𝑅, 𝛾, 𝜌0 ) where  is the state space,  the action space, 𝑃 ∶  ×  → Δ() the transition kernel, 𝑅 ∶  ×  → ℝ the reward function, 𝛾 ∈ [0, 1) the discount factor, and 𝜌0 the initial state distribution. The discounted return from step 𝑡 is 𝐺𝑡 = ∑𝑇 −1−𝑡 𝑘 𝑘=0 𝛾 𝑟𝑡+𝑘 . The state-value and action-value functions under policy 𝜋 are: (1)

𝜋

(2)

𝑄 (𝑠, 𝑎) = 𝔼𝜋 [𝐺𝑡 ∣ 𝑠𝑡 =𝑠, 𝑎𝑡 =𝑎].

The advantage function 𝐴𝜋 (𝑠, 𝑎) = 𝑄𝜋 (𝑠, 𝑎)−𝑉 𝜋 (𝑠) has zero expectation under 𝜋.

4.2. Policy Gradient and Actor-Critic Theorem 4.1 (Policy Gradient (Sutton and Barto, 2018)). For any differentiable policy 𝜋𝜃 , ] [𝑇 −1 ∑ ∇𝜃 log 𝜋𝜃 (𝑎𝑡 ∣ 𝑠𝑡 ) 𝐴𝜋𝜃 (𝑠𝑡 , 𝑎𝑡 ) . (3) ∇𝜃 𝐽 (𝜃) = 𝔼𝜋𝜃 𝑡=0

The temporal-difference residual at step 𝑡 is: (4)

𝛿𝑡 = 𝑟𝑡 + 𝛾(1 − 𝑑𝑡 )𝑉𝜙 (𝑠𝑡+1 ) − 𝑉𝜙 (𝑠𝑡 ), where 𝑑𝑡 ∈ {0, 1} indicates episode termination.

4.3. Independent Advantage Actor–Critic Independent Advantage Actor–Critic (IA2C) is included as a reward-only actor–critic baseline; it is the independentagent version of Advantage Actor–Critic (A2C) (Mnih et al., 2016). Each Blue agent maintains its own actor–critic network and optimiser, with no weight sharing and no centralised critic. For agent 𝑖, the policy and value function are conditioned only on the local observation: 𝑎𝑖𝑡 ∼ 𝜋𝜃𝑖 (⋅ ∣ 𝑜𝑖𝑡 ),

Safety contract. A CAGE∑Blue policy satisfies the safety

contract in an episode iff 𝑡 𝑐𝑡𝑘 ≤ 𝐵𝑘 for all three constraints. The violation rate 𝑃 (viol𝑘 ) is the primary safety metric: a deployed autonomous response system must meet a pre-specified violation tolerance with confidence intervals, audit logs, and an escalation path for budget exhaustion. We therefore treat lower violation rate and tighter confidence bounds as deployability evidence rather than claiming formal certification or a mathematical safety guarantee for nonconvex neural policies.

𝑉 𝜋 (𝑠) = 𝔼𝜋 [𝐺𝑡 ∣ 𝑠𝑡 =𝑠],

𝑉𝜙𝑖 (𝑜𝑖𝑡 ) ∈ ℝ.

(5)

Here 𝑡 indexes the environment step and 𝑖 ∈ {1, … , 𝑁} indexes a Blue agent. The local observation is 𝑜𝑖𝑡 , the sampled discrete action is 𝑎𝑖𝑡 , 𝜋𝜃𝑖 is agent 𝑖’s actor with parameters 𝜃𝑖 , and 𝑉𝜙𝑖 is its scalar value critic with parameters 𝜙𝑖 . The dot in 𝜋𝜃𝑖 (⋅ ∣ 𝑜𝑖𝑡 ) denotes the full categorical distribution over the agent’s valid actions conditioned on 𝑜𝑖𝑡 . The discounted return target used by the implementation is

Ethical statement. All experiments are conducted entirely within the CybORG/CAGE-4 simulator. No real networks,

𝑅̂ 𝑖𝑡 =

𝑇∑ −1

𝛾 𝜏−𝑡 𝑟𝑖𝜏 ,

𝜏=𝑡

(6)

𝛾 = 0.99, Preprint submitted for review. Final version may differ from this manuscript.

Page 5 of 30

Safety-Contract Graph MARL

where 𝜏 is a summation index over future steps and 𝑇 is the episode horizon; episode termination resets the recursion. The baseline advantage is

Clipped value loss. To prevent catastrophic valuefunction updates:

𝑉𝜙,clip = 𝑉𝜙old (𝑠𝑡 ) + Δ𝑉𝑡 , ( ) Δ𝑉𝑡 = clip 𝑉𝜙 (𝑠𝑡 ) − 𝑉𝜙old (𝑠𝑡 ), −𝜖, 𝜖 ,

= 𝑅̂ 𝑖𝑡 − 𝑉𝜙𝑖 (𝑜𝑖𝑡 ), 𝐴̂ 𝑖,A2C 𝑡 = 𝐴̃ 𝑖,A2C 𝑡

𝑖 − 𝜇𝐴 𝐴̂ 𝑖,A2C 𝑡

𝜎𝐴𝑖 + 𝜀

(7)

.

𝑖 and 𝜎 𝑖 are the empirical mean and standard where 𝜇𝐴 𝐴 deviation of agent 𝑖’s batch advantages, and 𝜀 is a small positive constant used only for numerical stability. The peragent objective is the vanilla actor–critic loss ] [ 𝑖IA2C = − 𝔼𝑡 log 𝜋𝜃𝑖 (𝑎𝑖𝑡 ∣ 𝑜𝑖𝑡 ) 𝐴̃ 𝑖,A2C 𝑡 )2 ] 𝑐vf [( 𝑖 𝔼𝑡 𝑉𝜙𝑖 (𝑜𝑡 ) − 𝑅̂ 𝑖𝑡 + (8) 2 [ ( )]

− 𝑐ent 𝔼𝑡 𝐻 𝜋𝜃𝑖 (⋅ ∣ 𝑜𝑖𝑡 )

,

Here 𝔼𝑡 denotes the empirical average over rollout timesteps and 𝐻(𝜋𝜃𝑖 ) is the action-distribution entropy used to encourage exploration. We use 𝑐vf = 0.5, 𝑐ent = 0.01, gradient clipping at 0.5, and RMSprop learning rate 7 × 10−4 . Unlike PPO and MAPPO, IA2C does not use a clipped policy ratio; it is included to show how a simpler independent actor–critic behaves under the same CAGE-4 observation, action-mask, reward, and cost-logging protocol.

4.4. Generalised Advantage Estimation The Generalised Advantage Estimation (GAE) advantage is computed with 𝛾=0.99 and 𝜆GAE =0.95 via the backward recursion (Schulman et al., 2016): 𝐴̂ 𝑇 = 0, 𝐴̂ 𝑡 = 𝛿𝑡 + 𝛾𝜆GAE (1 − 𝑑𝑡 ) 𝐴̂ 𝑡+1 ,

(9) 𝑡 = 𝑇 −1, … , 0, (10)

with bootstrapped return target 𝑅̂ 𝑡 = 𝐴̂ 𝑡 + 𝑉𝜙 (𝑠𝑡 ).

[ ( )] VF (𝜙) = 21 𝔼𝑡 max 𝑒2𝑡 , 𝑒̄2𝑡 ,

Total PPO objective. (𝜃, 𝜙) = CLIP (𝜃) + 𝑐vf VF (𝜙) − 𝑐ent 𝐻[𝜋𝜃 ], (15) where 𝑐vf weights the critic loss and 𝑐ent weights the entropy bonus. We use 𝑐vf =0.5, 𝑐ent =0.005 for ACD3 -GAT and 𝑐ent =0.01 for the MAPPO variants. Parameters are updated with Adam (𝜂=3 × 10−4 ), gradient norm clipped at 0.5, over 4 optimisation epochs. MAPPO-family baselines use minibatch size 64; the reported ACD3 -GAT optimisation applies the same PPO objective over its concatenated episode batch as specified in Section 6.5.

4.6. Multi-Agent Proximal Policy Optimisation (MAPPO) with Centralised Critic Multi-Agent Proximal Policy Optimisation (MAPPO) adopts Centralised Training with Decentralised Execution (CTDE) (Yu et al., 2022), following the centralised-critic tradition in multi-agent actor-critic methods (Lowe et al., 2017). A centralised value function 𝑉𝜙 (𝐨𝑡 ) has access to all 𝑁=5 agents’ observations 𝐨𝑡 = (𝑜1𝑡 , … , 𝑜𝑁 𝑡 ) during training, while each actor 𝜋𝜃𝑖 operates on local 𝑜𝑖𝑡 at execution. The centralised value function is: 𝑉𝜙 (𝐨𝑡 ) = 𝑓𝜙 (𝐳𝑡 ),

Proximal Policy Optimisation (PPO) updates the policy via a clipped surrogate that prevents destructively large steps (Schulman et al., 2017). Define the importancesampling ratio: 𝜋𝜃 (𝑎𝑡 ∣ 𝑠𝑡 ) . 𝜋𝜃old (𝑎𝑡 ∣ 𝑠𝑡 )

(14)

𝑒𝑡 = 𝑉𝜙 (𝑠𝑡 ) − 𝑅̂ 𝑡 , 𝑒̄𝑡 = 𝑉𝜙,clip − 𝑅̂ 𝑡 .

4.5. Proximal Policy Optimisation

𝜌𝑡 (𝜃) =

(13)

(11)

The numerator is the current policy probability of the sampled action and the denominator is the probability under the frozen behaviour policy 𝜃old that generated the rollout.

𝐳𝑡 =

5 ⨁

(16)

Enc𝑖𝜙 (𝑜𝑖𝑡 ) ∈ ℝ320 .

𝑖=1

⨁

Here denotes concatenation of five 64-dimensional peragent embeddings into the fused critic input 𝐳𝑡 , and 𝑓𝜙 ∶ ℝ320 → ℝ is a two-layer multi-layer perceptron (MLP). This separates training and execution information: the critic sees the five-agent fused observation during optimisation, whereas each Blue actor samples from its own local observation or context and the valid CybORG action mask at execution time.

Policy loss.

[ ( )] CLIP (𝜃) = −𝔼𝑡 min 𝑞𝑡 , 𝑞̄𝑡 , 𝑞𝑡 = 𝜌𝑡 (𝜃)𝐴̂ 𝑡 , ( ) 𝑞̄𝑡 = clip 𝜌𝑡 (𝜃), 1−𝜖, 1+𝜖 𝐴̂ 𝑡 ,

4.7. Constrained Markov Decision Process (12) 𝜖 = 0.2.

Here 𝑞𝑡 is the unclipped policy-gradient term and 𝑞̄𝑡 is the same term after clipping the likelihood ratio to the interval [1 − 𝜖, 1 + 𝜖].

Definition 4.2 (Constrained Markov Decision Process (CMDP) (Altman, 1999)). A Constrained Markov Decision Process (CMDP) augments an MDP with 𝐾 cost functions 𝑐 𝑘 ∶  ×  → ℝ≥0 and budget constraints 𝐵𝑘 > 0. The feasible policy set is Π = {𝜋 ∶ 𝐽𝑐𝑘 (𝜋) ≤ 𝐵𝑘 , ∀𝑘}, where ∑ 𝐽𝑐𝑘 (𝜋) = 𝔼𝜋 [ 𝑡 𝛾 𝑡 𝑐𝑡𝑘 ].

Preprint submitted for review. Final version may differ from this manuscript.

Page 6 of 30

Safety-Contract Graph MARL

Proposition 4.1 (Strong duality (Altman, 1999)). Under the linear programming relaxation of the discounted CMDP, max min (𝜋, 𝝀) = min max (𝜋, 𝝀), 𝜋

𝝀≥𝟎

𝝀≥𝟎

where (𝜋, 𝝀) = 𝐽𝑟 (𝜋) −

∑

𝜋

(17)

𝑘 𝜆𝑘 (𝐽𝑐𝑘 (𝜋) − 𝐵𝑘 ).

This proposition motivates the Lagrangian update used in the algorithm, but the implemented neural MARL problem is non-convex and partially observed. Accordingly, the paper reports empirical episode-level violation rates rather than claiming a formal strong-duality guarantee for ACD3 GAT.

4.8. Graph Neural Networks GraphSAGE. The MAPPO-GNN baseline uses the same

where 𝑚 indexes the attention head, 𝑊𝑚(𝓁) is the headspecific projection, and ELU denotes the exponential linear unit. In the implementation, the PyTorch Geometric GAT layers use their default self-loop augmentation, so  (𝑢) includes the node itself during message passing. Our encoder uses two layers: Layer 1 with 𝐾1 =4 heads and per-head dimension 16 (output ∈ ℝ64 ); Layer 2 with 𝐾2 =1 head and dimension 64 (output ∈ ℝ64 ). The global graph embedding is obtained by mean-pooling: ∑ 64 𝐠 = |𝑉1 | 𝑣∈𝑉 𝐡(2) 𝑣 ∈ℝ .

Gated Recurrent Unit. The Gated Recurrent Unit (GRU) cell maps input 𝐱𝑡 ∈ ℝ𝑑𝑥 and state 𝐡𝑡−1 ∈ ℝ𝑑ℎ as follows (Cho et al., 2014): 𝐳𝑡 = 𝜎(𝑊𝑧 𝐱𝑡 + 𝑈𝑧 𝐡𝑡−1 ),

observation-to-graph parser as the GAT models, but replaces attention with a two-layer GraphSAGE encoder. For layer 𝓁, GraphSAGE aggregates neighbouring embeddings and applies a learned linear map: ( ( [ ])) (𝓁) (𝓁) (𝓁) . 𝐡(𝓁+1) = LN ReLU 𝑊 𝐡 ‖ mean 𝐡 𝑣∈ (𝑢) 𝑣 𝑢 sage 𝑢 (18) Here 𝑢 is the node being updated, 𝑣 indexes neighbouring (𝓁) nodes in  (𝑢), ‖ denotes vector concatenation, and 𝑊sage is the learned linear map at layer 𝓁. LN denotes layer normalisation and ReLU denotes the rectified linear unit. The implementation uses hidden dimension 64 in the first layer, output dimension 64 in the second layer, and global mean pooling over nodes to obtain the per-agent embedding.

Graph Attention Networks. Graph neural networks can be

understood as neural message-passing architectures (Gilmer et al., 2017); GraphSAGE provides an inductive neighbourhood-aggregation baseline (Hamilton et al., 2017). A Graph Attention Network (GAT) layer computes attention-weighted neighbourhood aggregation (Veličković et al., 2018). Given node features {𝐡(𝓁) 𝑢 }, the un-normalised attention score from 𝑣 to 𝑢 is: ( (𝓁)⊤ [ (𝓁) (𝓁) ]) , (19) 𝑒(𝓁) 𝑊 𝐡𝑢 ‖ 𝑊 (𝓁) 𝐡(𝓁) 𝑢𝑣 = LeakyReLU0.2 𝐚 𝑣 Here 𝑢 is the destination node, 𝑣 is a neighbour contributing a message, 𝑊 (𝓁) is the layer-𝓁 feature projection, and 𝐚(𝓁) is the learned attention vector. The scores are normalised via softmax over  (𝑢): (𝓁) 𝛼𝑢𝑣 =∑

exp(𝑒(𝓁) 𝑢𝑣 ) (𝓁) 𝑤∈ (𝑢) exp(𝑒𝑢𝑤 )

.

(20)

(𝓁) The coefficient 𝛼𝑢𝑣 is therefore the normalised attention weight assigned to neighbour 𝑣 when updating node 𝑢. The updated representation with 𝐾𝓁 attention heads (‖ = concatenation): ( ( )) 𝐾𝓁 ∑ (𝓁) 𝐡(𝓁+1) = LN ELU ‖ 𝛼𝑢𝑣,𝑚 𝑊𝑚(𝓁) 𝐡(𝓁) , (21) 𝑢 𝑣 𝑚=1 𝑣∈ (𝑢)

(22)

𝐫𝑡 = 𝜎(𝑊𝑟 𝐱𝑡 + 𝑈𝑟 𝐡𝑡−1 ), ̃𝐡𝑡 = tanh(𝑊ℎ 𝐱𝑡 + 𝑈ℎ (𝐫𝑡 ⊙ 𝐡𝑡−1 )), 𝐡𝑡 = (1 − 𝐳𝑡 ) ⊙ 𝐡𝑡−1 + 𝐳𝑡 ⊙ ̃𝐡𝑡 .

(23) (24)

Here 𝐳𝑡 is the update gate, 𝐫𝑡 the reset gate, ̃𝐡𝑡 the candidate state, 𝜎 the logistic sigmoid, and ⊙ element-wise multiplication. We use 𝑑𝑥 =64 (graph embedding dimension) and 𝑑ℎ =32 (opponent latent dimension).

5. Problem Formulation 5.1. CAGE-4 as a Constrained Decentralized Partially Observable Markov Decision Process (Dec-POMDP) We instantiate the Dec-POMDP formalism (Oliehoek and Amato, 2016) (Section 4) with 𝑁=5 CAGE “Blue” agents, episode length 𝑇 =500, and 𝛾=0.99. The global simulator state is not observed directly by the response policies. At each step, each Blue agent receives a local binary observation, selects a discrete response action, and the joint Blue action is applied together with the CAGE Red transition dynamics in CybORG. The learning problem is therefore to optimise decentralised Blue policies from partial observations while satisfying episode-level operational budgets. The MDP notation in Section 4 is used as background: the hidden simulator state 𝑠𝑡 defines the transition process, but the implemented policies condition on local observations 𝑜𝑖𝑡 or on the derived ACD3 context vector. Thus equations written in state notation should be read as their partially observed implementation counterparts after replacing 𝑠𝑡 by 𝑜𝑖𝑡 , 𝐨𝑡 , or ctx𝑖𝑡 as specified below.

Notation used throughout. Subscript 𝑡 indexes an envi-

ronment step, superscript 𝑖 indexes a Blue agent, superscript 𝑘 indexes an operational constraint, and 𝑒 indexes an episode. The policy 𝜋𝜃 is executed independently by each Blue agent. The formal objective is written in terms of mean team reward and episode-level operational budgets, while the implemented ACD3 PPO update stores per-agent reward and cost streams and then reports the same costs after aggregating over agents and time. Rewards are discounted when

Preprint submitted for review. Final version may differ from this manuscript.

Page 7 of 30

Safety-Contract Graph MARL

𝑐𝑡𝑖,f w = 𝟏[𝜏(𝑎𝑖𝑡 ) ∈ {BlockZone, AllowZone}], (29) [ ] ∑ 𝑖,f p 𝑖,down ⋅𝟏 (𝑝𝑗 [ℎ] ∨ 𝑛𝑗 [ℎ]) = 0 , (30) 𝑐𝑡 = 𝑐𝑡

constructing PPO advantages; operational costs are recorded as undiscounted totals because the budgets are governance limits rather than reward-shaping terms. Thus 𝑟𝑖𝑡 denotes the ∑ simulator reward observed by agent 𝑖, 𝑟̄𝑡 = 15 𝑖 𝑟𝑖𝑡 is the team reward used in the objective, 𝑐𝑡𝑖,𝑘 is the immediate cost ∑ 𝑖,𝑘 proxy for agent 𝑖, and 𝑐𝑡𝑘 = 𝑖 𝑐𝑡∑ denotes the audited team cost at step 𝑡. The episode total 𝑡 𝑐𝑡𝑘 is compared with budget 𝐵𝑘 . This distinction is important for reproducibility: the optimiser uses discounted reward advantages, while the reported safety metrics and Lagrange multiplier updates use undiscounted episode cost totals.

Observation space. All agents share a uniform observa-

tion dimension 𝑑=210. The observation 𝑜𝑖𝑡 ∈ {0, 1}210 has the structure: [ ] 𝑜𝑖𝑡 = 𝜙𝑡 ‖ 𝐵0 ‖ 𝐵1 ‖ 𝐵2 ‖ 𝑀 , (25) 𝜙𝑡 ∈ {0, 1, 2}, 𝐵𝑗 ∈ {0, 1}59 , 𝑀 ∈ {0, 1}32 . Here 𝜙𝑡 is the mission phase (Preplanning, MissionA, MissionB), 𝐵𝑗 is subnet block 𝑗, and 𝑀 encodes inter-agent messages (4 agents × 8 bits). Agents 0–3 each monitor one subnet (blocks 𝐵1 , 𝐵2 are zero-padded); agent 4 monitors three subnets (all three blocks are real). Each 59-bit block encodes: [ ] 𝐵𝑗 = 𝑠𝑗 ‖ 𝑏𝑗 ‖ 𝑚𝑗 ‖ 𝑝𝑗 ‖ 𝑛𝑗 , (26) 𝑠𝑗 , 𝑏𝑗 , 𝑚𝑗 ∈ {0, 1}9 , 𝑝𝑗 , 𝑛𝑗 ∈ {0, 1}16 . Here 𝑠𝑗 is the subnet one-hot vector, 𝑏𝑗 marks blocked subnets, 𝑚𝑗 stores communications-policy mismatch, and 𝑝𝑗 [ℎ] (resp. 𝑛𝑗 [ℎ]) equals 1 if a malicious process (resp. network connection) event has been detected on host ℎ in subnet 𝑗. Hosts are ordered: server hosts ℎ ∈ {0, … , 5} followed by user hosts ℎ ∈ {6, … , 15}. This is the observation-to-graph interface used throughout the paper: the flat 210-bit observation is environment-specific, while the resulting host–subnet graph representation is the reusable input expected by the graph encoders and Graph Counterfactual Risk Propagation (G-CRP) screen.

Action space. Each agent selects from |𝑖 | ≤ 242 discrete actions. Actions decompose into 𝐾=9 types:

𝑗,ℎ

where 𝜏(𝑎) denotes the action type of 𝑎. The indicator in (30) is zero when at least one malicious flag is active, so a Restore action is counted as false-positive only when no alert evidence is visible in the decoded observation. The team ∑ cost audited in the benchmark is 𝑐𝑡𝑘 = 𝑖 𝑐𝑡𝑖,𝑘 . These are proxy labels derived from simulator action names and visible alerts: R ESTORE contributes downtime, R ESTORE without visible alert evidence contributes false-positive response cost, and B LOCKT RAFFICZ ONE and A LLOW T RAFFICZ ONE contribute firewall-change cost. A violation of any one budget is a violation of the episode safety contract. Definition 5.2 (Safety contract). Policy 𝜋 satisfies the safety contract in episode 𝑒 iff 𝑇∑ −1

with  = {down, f w, f p}, 𝐵down =50, 𝐵f w =20, 𝐵f p =10. The ∑ per-constraint violation rate is 𝑃 (viol𝑘 ) = ℙ( 𝑡 𝑐𝑡𝑘 > 𝐵𝑘 ).

5.3. Constrained Joint Objective

max 𝐽𝑟 (𝜋) = 𝔼𝜋

[𝑇 −1 ∑

𝜋

s.t.

] 𝑡

𝛾 𝑟̄𝑡 ,

𝑡=0

𝐽𝑐𝑘 (𝜋) = 𝔼𝜋

[𝑇 −1 ∑

(32)

] 𝑐𝑡𝑘

≤ 𝐵𝑘 ,

∀𝑘.

𝑡=0

∑ where 𝑟̄𝑡 = 15 5𝑖=1 𝑟𝑖𝑡 is the mean team reward. Operational costs are audited as undiscounted episode totals, matching the budget accounting used by the shield and evaluation metrics.

Lagrangian relaxation following the constrained objective. (𝜋, 𝝀) = 𝐽𝑟 (𝜋) −

(27)

∑

( ) 𝜆𝑘 𝐽𝑐𝑘 (𝜋) − 𝐵𝑘 ,

(33)

𝑘∈

A LLOW Z ONE, D EPLOYD ECOY, O THER}.

𝝀 ≥ 𝟎.

For compact notation, B LOCKZ ONE and A LLOW Z ONE denote the simulator actions B LOCKT RAFFICZ ONE and A L LOW T RAFFIC Z ONE . The time durations are 𝛿 ∶  → {1, 2, 3, 5}: Sleep, Monitor, BlockZone, and AllowZone take one simulator step; Analyse and DeployDecoy take two; Remove takes three; Restore takes five.

Dual gradient ascent. Multipliers are updated after each episode batch: [ ] 𝜆𝑘 ← 𝜆𝑘 + 𝜂𝜆 (𝐽̂𝑐𝑘 − 𝐵𝑘 ) + , 𝑀 𝑇𝑒 −1 1 ∑∑ 𝑘 𝐽̂𝑐𝑘 = 𝑐 . 𝑀 𝑒=1 𝑡=0 𝑒,𝑡

5.2. Operational Cost Signals Definition 5.1 (Three operational cost signals). At step 𝑡 for agent 𝑖, with action 𝑎𝑖𝑡 and observation 𝑜𝑖𝑡 : 𝑐𝑡𝑖,down = 𝟏[𝜏(𝑎𝑖𝑡 ) = Restore],

(31)

∀𝑘 ∈ ,

𝑡=0

 = {SLEEP, M ONITOR, A NALYSE, R EMOVE, R ESTORE, B LOCKZ ONE,

𝑐𝑡𝑘 ≤ 𝐵𝑘 ,

𝜂𝜆 = 0.01, (34)

where 𝑀 is the number of complete episodes in the PPO update batch.

(28)

Preprint submitted for review. Final version may differ from this manuscript.

Page 8 of 30

Safety-Contract Graph MARL

in Equation (32) by combining four mechanisms. First, each agent converts its binary CAGE-4 observation into a host– subnet graph and encodes it with a Graph Attention Network a graph 𝐺𝑡𝑖 = (𝑉 , 𝐸, 𝑋) from 𝑜𝑖𝑡 : (GAT). Second, the graph embedding is concatenated with a • Nodes: One node per host slot per monitored subnet recurrent opponent embedding, the remaining budget state, (16 slots: 6 server, 10 user), plus one subnet-level and an uncertainty signal to form the policy context. Third, node. Agents 0–3: |𝑉 |=17. Agent 4: |𝑉 |=51. a factorised actor proposes an action type and target, while a set of reward, cost, tail-risk, and exploitability critics sup• Edges: Bidirectional star between each host node and plies PPO training signals. Fourth, before execution, a graphits subnet node; inter-subnet edges for agent 4 where risk shield evaluates admissible counterfactual actions with the communication policy permits. 𝑄safe and replaces unsafe proposals when the safety contract • Node features: 𝐱𝑣 ∈ ℝ9 defined below. requires it. Figures 1 and 2 give the architectural and closed-loop Node feature vector. For node 𝑣 corresponding to host views used during training and evaluation. Figure 1 should index ℎ in subnet 𝑗: be read from left to right as the implemented data path. [ usr sub The CAGE-4 network state first becomes the binary Blue 𝐱𝑣 = 𝑥srv , 𝑥 , 𝑥 , 𝑝 [ℎ], 𝑗 𝑣 𝑣 𝑣 observation 𝑜𝑖𝑡 ; the parser converts that simulator-specific (35) ] ⊤ 9 𝑛𝑗 [ℎ], 𝑚̄ 𝑗 , 𝐞⊤ ∈ ℝ , vector into a host–subnet graph 𝐺𝑡 = (𝑉 , 𝐸, 𝑋). The GAT 𝜙𝑡 encoder produces the graph embedding 𝐠𝑡 , and the policy srv usr where 𝑥𝑣 =1 iff ℎ ∈ {0, … , 5}, 𝑥𝑣 =1 iff ℎ ∈ {6, … , 15}, context concatenates 𝐠𝑡 with recurrent opponent state 𝐳𝑡 , ̄ 𝑗 is the mean 𝑥sub 𝑣 =1 for the subnet summary node, 𝑚 budget state 𝐛𝑡 , and uncertainty 𝐮𝑡 before the factorised actor communications-policy mismatch in subnet 𝑗, and samples an action type and target. The lower path in the same 𝐞𝜙𝑡 ∈ {0, 1}3 is the mission-phase one-hot vector. figure is the safety path: the shield scores candidate actions with 𝑄safe , using graph risk, operational cost multipliers, and GAT encoder. Applying Equations (19)–(21) with 𝑑in =9, remaining budgets, and only then emits the action sent to 64 𝐾1 =4 heads, per-head dimension 16 (Layer 1 output: ℝ ), CybORG. The feedback arrows therefore correspond exactly 64 then 𝐾2 =1 head, dimension 64 (Layer 2 output: ℝ ): to the two quantities that make ACD3 -GAT different from ∑ reward-only graph MARL: executed-action costs update the 1 (36) 𝐠𝑖𝑡 = 𝐡(2) ∈ ℝ64 . remaining episode budgets at the next decision step, while |𝑉 | 𝑣∈𝑉 𝑣 batch-level costs update the Lagrange multipliers used by the 5.5. Factorised Action Policy next PPO optimisation. Every action 𝑎 ∈ {0, … , |𝑖 | − 1} decomposes into a Figure 2 translates the same architecture into the temtype 𝜏(𝑎) ∈  and a target index 𝜈(𝑎) ∈ {0, … , 𝑀𝜏(𝑎) − 1}, poral order of one interaction step: observe 𝑜𝑖𝑡 , parse 𝐺𝑡𝑖 , where 𝑀𝜏 is the number of valid targets for type 𝜏 and encode 𝐠𝑖𝑡 , build ctx𝑖𝑡 , propose 𝑎prop , screen it into 𝑎exec , 𝑀max = max𝜏 𝑀𝜏 . execute in CybORG, store reward and costs, and update The factorised policy is: budgets and multipliers on their respective time scales. The environment produces observations, as the policy proposes type tgt 𝜋𝜃 (𝑎 ∣ ctx𝑡 ) = 𝜋𝜃 (𝜏(𝑎) ∣ ctx𝑡 ) ⋅ 𝜋𝜃 (𝜈(𝑎) ∣ ctx𝑡 , 𝜏(𝑎)), a response action, the shield selects the executable action, the CybORG returns reward and operational costs, and the (37) remaining budgets and Lagrange multipliers determine how ⊤ ctx + 𝛼 ) ⋅ 𝟏 exp(𝑤 the next decision is evaluated. The Algorithm 1 gives the 𝜏 𝜏 type 𝜏 𝜋𝜃 (𝜏 ∣ ctx) = ∑ , (38) ⊤ same procedure in executable order. The equations below 𝜏 ′ exp(𝑤𝜏 ′ ctx + 𝛼𝜏 ′ ) ⋅ 𝟏𝜏 ′ define each quantity used in that loop. Together with the exp(𝑓𝜈 (ctx, 𝐞𝜏 )𝜈 ) ⋅ 𝟏𝜈 tgt problem notation in Section 5, they specify the observation 𝜋𝜃 (𝜈 ∣ ctx, 𝜏) = ∑ , (39) 𝜈 ′ exp(𝑓𝜈 (ctx, 𝐞𝜏 )𝜈 ′ ) ⋅ 𝟏𝜈 ′ parser, graph encoder, recurrent context, actor factorisation, critic targets, composite advantage, action shield, and dual where 𝟏𝜏 , 𝟏𝜈 mask invalid types/targets, 𝐞𝜏 ∈ ℝ16 is a update used by the reported implementation. The figure learnable type embedding, and 𝑓𝜈 ∶ ℝ120+16 → ℝ𝑀max is is deliberately architectural rather than decorative: learned a two-layer MLP. modules are the GAT, recurrent context, actor, and critic The joint log-probability is log 𝜋𝜃 (𝑎 ∣ ctx) = heads; deterministic or governance modules are the obsertype tgt log 𝜋𝜃 (𝜏(𝑎)) + log 𝜋𝜃 (𝜈(𝑎)), and the entropy decomposes vation parser, budget accounting, and Graph Counterfactual as 𝐻[𝜋𝜃 ] = 𝐻[𝜋 type ] + 𝔼𝜏 [𝐻[𝜋 tgt (⋅ ∣ 𝜏)]]. Risk Propagation (G-CRP) shield. The graph parser and valid-action layout are environment interfaces, while the 3 constrained policy, graph encoder, cost critics, and shielding 6. ACD -GAT equations are the reusable ACD3 method components.

5.4. Graph Observation Encoding Graph construction. For each agent 𝑖 and step 𝑡, we build

6.1. Method Overview

ACD3 -GAT solves the constrained Decentralized Partially Observable Markov Decision Process (Dec-POMDP) Preprint submitted for review. Final version may differ from this manuscript.

Page 9 of 30

Safety-Contract Graph MARL ACD3 -GAT architecture: perception, constrained policy, and action screening perception

Blue observation oti ∈ {0, 1}210

CAGE-4 network state

Graph parser Gt = (V, E, X)

GAT encoder gt = GAT(Gt )

policy and screening

ACD3 context [gt ‖zt ‖bt ‖ut ]

safety-contract feedback

Factorised actor π(τ)π(ν|τ)

Operational contract remaining budget bk (t) dual prices λk

Graph-risk shield a * = arg maxQsafe

Blue action

̂ t , a) − ∑λk Ĉ k (a) − βU(G ̂ t , a) Qsafe (Gt , a) = − S(G k

graph state

budget context

risk score

observed costs

Figure 1: ACD3 -GAT system architecture. Partial CAGE-4 observations are parsed into host–subnet graphs, encoded with graph attention, combined with opponent, budget, and uncertainty context, and passed to a factorised actor. The graph-risk shield screens the proposed action with current safety-contract multipliers; observed costs update the remaining budgets and the next dual step. The parser and action layout are simulator-specific interfaces, while the graph encoder, constrained policy, and safety-contract feedback are reusable components. Closed-loop safety-contract update in ACD3 -GAT

1. Observe oti ∈ {0, 1}210 alerts, hosts, mission phase

2. Encode context Gt = (V, E, X), gt = GAT(Gt ) ctxt = [gt ‖zt ‖bt ‖ut ]

3. Propose action aprop ∼ πθ (a ∣ ctxt ) = π(τ)π(ν ∣ τ)

constrained advantage à t =  rt − ∑λk  ct k − 0.1 tCVaR − 0.05 exploit − 0.01 override t t k

6. Update optimiser λk ← [λk + ηλ (J ̂ck − Bk )] + 3 PPO on à ACD t

5. Execute in CybORG (ot + 1 , rt , ctk ) ∼ P(⋅ ∣ ot , aexec ) bk (t+1) = bk (t) − ctk

4. Screen action aexec = arg max Qsafe a∈safe

Qsafe = − Ŝ − ∑λk Ĉ k − βÛ k

Figure 2: Closed-loop safety-contract update used by ACD3 -GAT. Each decision observes the cyber range, encodes graph and budget context, proposes and screens an action, executes the selected response in CybORG, and stores the resulting reward, costs, and context. Remaining budgets update at the next step, while Lagrange multipliers update after the PPO episode batch, separating execution-time safety from slower policy learning.

Preprint submitted for review. Final version may differ from this manuscript.

Page 10 of 30

Safety-Contract Graph MARL

6.2. From Safety Contract to Policy Update

6.3. Context Vector

The implemented update follows directly from the constrained objective in Equation (32). For a fixed multiplier vector 𝝀, maximising the Lagrangian in Equation (33) is equivalent to maximising a reward signal in which operational costs reduce the policy advantage. Using the policygradient identity in Equation (3), the unconstrained reward advantage 𝐴̂ 𝑟𝑡 is therefore replaced by a Lagrangian advantage ∑ 𝑐 Lag 𝜆𝑘 𝐴̂ 𝑡 𝑘 , (40) 𝐴̂ 𝑡 = 𝐴̂ 𝑟𝑡 −

The ACD3 -GAT policy conditions on a 120-dimensional context vector assembling four representations: [ ] ctx𝑖𝑡 = 𝐠𝑖𝑡 ‖ 𝐳𝑡𝑖 ‖ 𝐛𝑡 ‖ 𝐮𝑡 ∈ ℝ64+32+16+8 , (44)

𝑘∈{down,fw,fp} 𝑐

where 𝐴̂ 𝑟𝑡 and 𝐴̂ 𝑡 𝑘 are both computed with the same GAE recursion but from different scalar streams: simulator reward 𝑐 for 𝐴̂ 𝑟𝑡 and operational cost proxy 𝑐𝑡𝑖,𝑘 for 𝐴̂ 𝑡 𝑘 . This is the first point where the safety contract enters the policy update: actions that improve security reward can still be discouraged when they increase expected downtime, firewall disruption, or false-positive Restore cost. ACD3 -GAT then augments this Lagrangian advantage with three additional signals that are present in the implementation. CVaR reweighting emphasises the worst-return episodes in the PPO batch, adaptive opposing-policy evaluation can contribute an exploitability signal, and the override term penalises excessive shield or human-governance burden. The resulting scalar used in the PPO surrogate is 3 Lag 𝐴̂ ACD = 𝐴̂ 𝑡 − 𝛽 𝐴̂ CVaR 𝑡 𝑡

exploit , − 𝜂 𝐴̂ 𝑡 − 𝜇 𝐴̂ override 𝑡 (𝛽, 𝜂, 𝜇) = (0.1, 0.05, 0.01).

(41)

Before entering the clipped PPO objective, this advantage is normalised within the batch:

where each component is derived from 𝑜𝑖𝑡 , the recurrent episode history, or the current safety-contract state.

Graph embedding 𝐠𝑖𝑡 ∈ ℝ64 . Obtained by the two-layer GAT encoder (Equations 19–36) applied to 𝐺𝑡𝑖 .

Opponent embedding 𝐳𝑡𝑖 ∈ ℝ32 . A GRU (Equations 22– 24) encodes the recent observation trajectory: 𝑖 𝐳𝑡𝑖 = GRU64→32 (𝐠𝑖𝑡 , 𝐳𝑡−1 ),

This provides an implicit, online estimate of the CAGE Red process without requiring explicit strategy identification. When Red-process strategy labels 𝑦 ∈ {0, … , 3} and a non-zero auxiliary weight are available (e.g., from league training), an auxiliary classification loss opp = −

3 ∑

𝐴̃ 𝑡 =

𝐴̂ 𝑡

− 𝜇𝐴

𝜎𝐴 + 𝜀

,

(42)

𝜀 = 10−8 . Equation (12) is then applied with 𝐴̃ 𝑡 in place of the standard reward-only advantage. The multiplier update closes the loop. After a PPO batch, the observed episode cost estimate 𝐽̂𝑐𝑘 is compared with its budget 𝐵𝑘 : [ ] 𝜆𝑘 ← 𝜆𝑘 + 𝜂𝜆 (𝐽̂𝑐𝑘 − 𝐵𝑘 ) + , (43) 𝜂𝜆 = 0.01. If a cost remains below budget, the corresponding multiplier is unchanged or decreases toward zero through projection; if it exceeds budget, the multiplier increases and future PPO updates penalise that cost more strongly through Equation (40). In parallel, the action shield in Section 6.6 applies the current remaining budgets 𝑏𝑘 (𝑡) at decision time. Thus the method has two safety mechanisms with different time scales: Lagrangian learning shapes the policy across batches, while the shield prevents budget-exhausting actions during an episode.

(46)

𝑦𝑗 log 𝑦̂𝑗 (𝐳𝑡 )

𝑗=0

encourages 𝐳𝑡 to encode discriminative opponent information. Here 𝑗 indexes the Red-process class, 𝑦𝑗 is the one-hot target label, and 𝑦̂𝑗 (𝐳𝑡 ) is the classifier’s predicted probability for class 𝑗 from the recurrent state.

Budget embedding 𝐛𝑡 ∈ ℝ16 . The remaining operational budgets are embedded via a linear layer: ) ( 𝐛𝑡 = tanh 𝑊𝑏 𝐪𝑡 + 𝐝𝑏 , [ ]⊤ 𝑏 (𝑡) 𝑏 (𝑡) 𝑏 (𝑡) , 𝐪𝑡 = 𝐵down , 𝐵fw , 𝐵fp down

ACD3

(45)

𝐳0𝑖 = 𝟎.

fw

(47)

fp

𝑊𝑏 ∈ ℝ16×3 ,

𝐝𝑏 ∈ ℝ16 . ∑ 𝑘 where 𝑏𝑘 (𝑡) = max(0, 𝐵𝑘 − 𝑡−1 𝜏=0 𝑐𝜏 ). The network therefore receives normalised remaining budgets in [0, 1] rather than raw episode totals.

Uncertainty signal 𝑢𝑡 ∈ [0, 1] and embedding 𝐮𝑡 ∈ ℝ8 .

The entropy of the action-type distribution conditioned on 𝐠𝑡 alone (computed before the full context, avoiding circularity) is: ( ) 𝐻 sof tmax(𝑊𝑢 𝐠𝑖𝑡 + 𝐝𝑢 ) 𝑢𝑡 = , (48) log 𝐾 𝑊𝑢 ∈ ℝ9×64 ,

𝐝𝑢 ∈ ℝ9 ,

𝐾 = 9.

normalised to [0, 1]. A learned affine map 𝐮𝑡 = 𝑊ue 𝑢𝑡 + 𝐝ue ∈ ℝ8 embeds it into the context. High 𝑢𝑡 indicates that the graph state alone is insufficient to determine the action type; it is used as contextual evidence for the policy and confidence monitor. In the control loop, 𝑢𝑡 is therefore not a separate objective: it is a compact uncertainty feature that enters the context vector and can trigger the governance fallback when the configured confidence or G-CRP uncertainty thresholds are exceeded.

Preprint submitted for review. Final version may differ from this manuscript.

Page 11 of 30

Safety-Contract Graph MARL

6.4. Multi-Objective Critic Set ACD3 -GAT maintains four distinct critic heads, all con-

ditioned on ctx𝑡 :

𝑘 ∈ {1, 2, 3},

(51)

𝑉exploit (ctx𝑡 ) ∈ ℝ.

(52)

Each critic head is a two-layer MLP with hidden dimension 64. The reward head predicts the security-return baseline, the three cost critics correspond to downtime, firewall-change cost, and false-positive response cost, the CVaR head supports tail-risk accounting, and the exploitability head stores the adaptive Red-process signal when enabled. The cost critics are stacked as 𝐕𝑐 = (𝑉𝑐1 , 𝑉𝑐2 , 𝑉𝑐3 )⊤ ∈ ℝ3 .

Composite ACD3 Advantage

6.5. Per-objective GAE advantages. Using the recursion in Equations (9)–(10):

𝐴̂ 𝑟𝑡 = GAE({𝑟𝑖𝜏 }, 𝑉𝑟 ; 𝛾=0.99, 𝜆GAE =0.95), 𝑐 𝐴̂ 𝑘 = GAE({𝑐 𝑖,𝑘 }, 𝑉 𝑘 ; 𝛾=0.99, 𝜆GAE =0.95). 𝜏

𝑐

(53) (54)

During ACD3 optimisation these streams are computed per Blue agent from the stored rollout of that agent; the reported episode metrics aggregate the same cost proxies across agents and time.

CVaR advantage. In each batch of 𝑀=8 episodes ranked by return, the 𝑘∗ = max(1, ⌊0.1𝑀⌋) worst episodes receive weight 𝑤𝑒 =1∕𝑘∗ ; all others receive 𝑤𝑒 =0: (55)

= 𝑤𝑒 𝐴̂ 𝑟𝑡 . 𝐴̂ CVaR 𝑡

This is an episode-level reweighting of the policy-gradient signal rather than a separate distributional value estimator.

Exploitability advantage. When adaptive opposing-

policy evaluation is enabled, a short Red PPO mini-loop is run every 𝐼exploit episodes (default 50; 3 updates × 2 episodes each): exploit ̄ 𝐴̂ 𝑡 = 0.1 𝛿,

𝛿̄ = 𝑈Blue (0) − 𝑈Blue (3),

3 𝐴̂ ACD = 𝐴̂ 𝑟𝑡 − 𝑡

(50)

𝑉CVaR (ctx𝑡 ) ∈ ℝ,

𝑡

above into Equation (41) gives the implemented scalar advantage:

(49)

𝑉𝑟 (ctx𝑡 ) ∈ ℝ, 𝑉𝑐𝑘 (ctx𝑡 ) ∈ ℝ,

ACD3 composite advantage. Substituting the terms

(56)

where 𝑈Blue (𝑘) is Blue’s mean return after 𝑘 Red-process adaptation steps and 𝛿̄ is the running mean over the last 10 stored measurements. If no adaptive opposing-policy measurement has been stored, this term is zero. In the reported experiments, coupled adaptive Red-process evaluation is used primarily as a post-training stress test rather than as the headline optimisation target.

𝐴̂ override = 𝟏[shield triggered at 𝑡] − 𝜔, ̄ 𝑡 where 𝜔̄ is the mean override rate in the current batch.

(57)

𝑐 𝜆𝑘 𝐴̂ 𝑡 𝑘 − 0.1 𝐴̂ CVaR 𝑡

(58)

𝑘=1 exploit − 0.01 𝐴̂ override . − 0.05 𝐴̂ 𝑡 𝑡

Normalised before use: 𝐴̃ 𝑡 =

3 𝐴̂ ACD − 𝜇𝐴 𝑡

𝜎𝐴 + 𝜀

,

(59)

−8

𝜀 = 10 .

PPO update. Equation (12) is applied to 𝐴̃ 𝑡 with 𝜖=0.2,

𝑐vf =0.5, 𝑐ent =0.005, and 4 epochs over the concatenated episode batch. The same update also fits the reward, cost, CVaR, and exploitability critic heads using clipped value losses.

6.6. Budget-Aware Counterfactual Shield Remaining budget. The remaining budget is defined as: 𝑏𝑘 (𝑡) = max(0, 𝐵𝑘 −

𝑡−1 ∑

(60)

𝑐𝜏𝑘 )

𝜏=0

Admissible action set. The reported shield uses the cost

proxies as a hard exhaustion guard: once a budget has been depleted, actions with positive immediate proxy cost for that budget are blocked. Costly actions before exhaustion remain admissible, but are penalised by the Lagrangian and G-CRP score. { safe (𝑡) = 𝑎 ∈ 𝑖 ∶ 𝑏𝑘 (𝑡) > 0 (61) } ∨ 𝑐̂𝑘 (𝑎) = 0, ∀𝑘 , where 𝑐̂𝑘 (𝑎) is the immediate cost proxy (Equations 28–30). In this expression, 𝑖 is agent 𝑖’s discrete action set, 𝑏𝑘 (𝑡) is the remaining budget for constraint 𝑘 before action execution, and the condition is applied for all three operational costs.

Shielded proposal. The policy first proposes an action

from the factorised actor. For efficiency, the reported G-CRP shield ranks one representative candidate per action type. It accepts the proposal when its action type is the top ranked admissible type; otherwise it replaces the proposal with the highest-ranked admissible type, or falls back to SLEEP when no safe action is available: 𝜋prop (𝑎 ∣ 𝑜𝑖𝑡 ) = 𝜋𝜃 (𝑎 ∣ ctx𝑡 ), ( ) 𝑎exec = Shield 𝑎prop , 𝐺𝑡 , safe (𝑡) .

Override advantage. The centred override indicator penalises excessive analyst burden:

3 ∑

(62)

Here 𝜋prop is the stochastic policy distribution before screening, 𝑎prop is the sampled proposal, and 𝑎exec is the action actually submitted to CybORG after applying the admissible set and G-CRP ranking. When the shield replaces the

Preprint submitted for review. Final version may differ from this manuscript.

Page 12 of 30

Safety-Contract Graph MARL

proposed action, the implementation stores and updates PPO on the executed action by recomputing its log-probability under the current policy and action mask. Thus the trajectory logger, cost accounting, and likelihood ratio are all tied to the action actually submitted to CybORG. The shield is nevertheless a non-differentiable intervention and can bias the policy-gradient estimate, this is why override frequency is recorded and penalised in Equation (57). The rule blocks after the relevant remaining budget is exhausted, so episode-level compliance is evaluated empirically through the violation-rate metrics rather than asserted analytically from the shield alone.

6.7. G-CRP: Graph Counterfactual Risk Propagation The rule shield (61) reacts to cost proxies but cannot estimate how the network’s compromise state will evolve. GCRP provides predictive risk scoring via graph propagation.

Node risk beliefs. Given the proc and net flags in 𝑜𝑖𝑡 : 𝑝̂𝑣 (𝑡) = 12 𝑝𝑗 [ℎ(𝑣)] + 12 𝑛𝑗 [ℎ(𝑣)] ∈ [0, 1].

(63)

Here ℎ(𝑣) maps host node 𝑣 to its decoded host slot, while 𝑝𝑗 [ℎ] and 𝑛𝑗 [ℎ] are the process-alert and network-alert bits for host ℎ in subnet 𝑗.

Action effects on beliefs. Defensive actions modify beliefs before propagation:

Restore(𝑣) ∶ 𝑝̂𝑣 ← 0.05 𝑝̂𝑣 ,

(64)

Remove(𝑣) ∶ 𝑝̂𝑣 ← 0.30 𝑝̂𝑣 ,

(65)

Block ∶ 𝛼 ← 0.10 𝛼,

(66)

Allow ∶ 𝛼 ← 1.05 𝛼,

(67)

Decoy(𝑣) ∶ 𝑝̂𝑣 ← 0.70 𝑝̂𝑣 , 𝑝̂𝑤 ← 0.85 𝑝̂𝑤 ∀𝑤 ∈  (𝑣),

(68)

where 𝛼=0.3 is the default edge influence.

Independent cascade propagation. For 𝐿=2 steps: ⎛ ⎞ ∏ (𝑙) ⎟ ⎜𝑝̂(𝑙) , 1 − 𝑝̂(𝑙+1) = max (1 − 𝛼 𝑝 ̂ ) 𝑣 𝑢 ⎟. ⎜ 𝑣 𝑢∈ in (𝑣) ⎝ ⎠

(69)

The max enforces monotonicity: compromise probability does not decrease through propagation alone. The superscript (𝑙) indexes the propagation depth, and  in (𝑣) is the set of nodes with directed influence into 𝑣 under the constructed host–subnet graph. The reported shield uses 𝐿 = 2 as a local neighbourhood screen: one hop captures immediate subnet-to-host effects and the second hop captures the next propagation opportunity without turning the shield into a rollout search.

Predicted security risk. With asset weights 𝜔𝑣 {2, 1, 0.5} for server/user/subnet nodes: ∑ (𝐿) 𝑣∈𝑉 𝜔𝑣 𝑝̂𝑣 (𝑡 ∣ 𝑎) ̂ 𝑆(𝐺𝑡 , 𝑎) = . ∑ 𝑣 𝜔𝑣

∈

(70)

̂ 𝑡 , 𝑎) is a normalized post-action graphThe resulting 𝑆(𝐺 risk score: larger values indicate higher predicted residual compromise risk after applying candidate action 𝑎 and propagating for 𝐿 steps.

Q-safe score and action selection. ̂ 𝑡 , 𝑎) − 𝑄safe (𝐺𝑡 , 𝑎) = −𝑆(𝐺

∑

𝜆𝑘 𝐶̂𝑘 (𝑎) − 𝛽 𝑈̂ (𝐺𝑡 , 𝑎),

𝑘

(71) (72)

∗

𝑎 = arg max 𝑄safe (𝐺𝑡 , 𝑎), 𝑎∈safe (𝑡)

where 𝐶̂𝑘 (𝑎) uses the same cost proxies as (28)–(30), and 𝛽=0.1 weights the budget-risk penalty 𝑈̂ . For deterministic G-CRP, 𝑈̂ (𝐺𝑡 , 𝑎) = 1 if the immediate proxy cost would exceed any remaining budget; otherwise it is the mean normalised proxy cost across the three budgets. For learned G-CRP, the same symbol denotes the predicted violation probability. In the reported configuration 𝛽 = 0.1, matching the G-CRP shield code and the ACD3 configuration. The cascade coefficients and asset weights are hand-specified operational priors rather than calibrated causal estimates; they define the deterministic screen used in this study and should be recalibrated before transfer to a different cyber range or enterprise topology. The article reports deterministic G-CRP as the deployed action-screening rule; the learned G-CRP fit is treated as a supervised model-fitting result, not as a separate policy-performance claim.

Learned G-CRP fitting objective. When the learned G-

CRP component is trained from collected trajectories, the model predicts next-step process-alert probabilities, nextstep network-alert probabilities, immediate cost proxies, and violation probability. Its supervised loss is ) ) ( ( G-CRP = BCE 𝐩̂ proc , 𝐩proc + BCE 𝐩̂ net , 𝐩net (73) + 0.1 ‖𝐜̂ − 𝐜‖22 + BCE(𝑣, ̂ 𝑣), where BCE denotes binary cross-entropy, hatted quantities are model predictions, unhatted quantities are supervised targets from the next logged transition, 𝐜 is the immediate cost-proxy vector, and 𝑣 is the binary violation label. This loss matches the replication-package trainer and is reported only as a component fit; the policy benchmark uses the deterministic G-CRP screen unless explicitly stated otherwise.

6.8. ACD3 -TCGS: Temporal Contract Graph Shielding The deterministic G-CRP screen estimates immediate graph risk from the current observation. Temporal Contract Graph Shielding (TCGS) adds a learned recurrent contractrisk model that asks a different question: given the recent trajectory history and a candidate action, how likely is the episode to cross an operational budget within the next decision horizon? TCGS is implemented as an opt-in diagnostic extension. It reads existing safety-labelled trajectories, trains a small recurrent risk model, and writes separate evaluation

Preprint submitted for review. Final version may differ from this manuscript.

Page 13 of 30

Safety-Contract Graph MARL

traces without modifying completed checkpoints or training outputs. For each step, the feature vector combines the decoded binary observation, immediate operational costs, cumulative cost normalized by the budget, remaining budget fraction, mission phase, action-type frequencies, and scaled reward: [ 𝐱𝑡 = 𝑜𝑡 ‖ 𝐜𝑡 ‖ 𝐂𝑡 ∕𝐁 ‖ [𝐁 − 𝐂𝑡 ]+ ∕𝐁 (74) ] ‖ onehot(𝜙𝑡 ) ‖ 𝐦(𝑎𝑡 ) ‖ 𝑟̄𝑡 ∕100 ∈ ℝ233 .

with 𝜖 = 0.05 in the reported diagnostic. If the proposal is rejected, the shield searches the valid action set and chooses the candidate with the smallest lexicographic risk tuple Ψ𝑡 (𝑎). The implemented ranking ( Ψ𝑡 (𝑎) = 𝜓𝑡𝐵 (𝑎), 𝜓𝑡𝑃 (𝑎), 𝑝down (𝑎), 𝑡 ) 𝑝̄𝑡 (𝑎), 𝑐̄𝑡 (𝑎) , 𝜓𝑡𝐵 (𝑎) = 𝟏[∃𝑘 ∶ 𝐶𝑡𝑘 + 𝑐̂𝑘 (𝑎) > 𝐵𝑘 ], 𝜓𝑡𝑃 (𝑎) = 𝟏[𝑝down (𝐻𝑡 , 𝑎) > 𝜖], 𝜙

Here 𝐂𝑡 is the cumulative episode cost before the candidate action, 𝐁 is the three-budget vector, and 𝐦(𝑎𝑡 ) is the normalized action-type count vector used by the implementation. For candidate action 𝑎, TCGS forms the length-𝐿 history ( ) 𝐻𝑡 (𝑎) = 𝐱𝑡−𝐿+1 , … , 𝐱𝑡−1 , 𝐱𝑡 (𝑎) , 𝐿 = 8. (75) The recurrent model is a Gated Recurrent Unit encoder with hidden dimension 96 and two heads: 𝐡𝑡 = GRU𝜙 (𝐻𝑡 (𝑎)), 𝓵 𝑡 = 𝑊𝑣 𝐡𝑡 , 𝐝̂𝑡 = ReLU(𝑊𝑑 𝐡𝑡 ),

(76)

Here 𝓵 𝑡 is the three-dimensional vector of violation logits and 𝐝̂𝑡 is the non-negative predicted normalized future cost increment. Applying a logistic sigmoid to logit 𝓁𝑡𝑘 estimates the probability that budget 𝑘 will be violated within horizon 𝐻 = 100: ] [ | ∑ | 𝑘 𝑘 max 𝑐𝑠 > 𝐵𝑘 | 𝐻𝑡 (𝑎) . (77) 𝑝𝜙 (𝐻𝑡 , 𝑎) = Pr | 𝜙 𝜏∈[𝑡,𝑡+𝐻] 𝑠≤𝜏 | In Equation (77), 𝑠 and 𝜏 are step indices inside the future prediction window and 𝐵𝑘 is the corresponding operational budget. The second head 𝐝̂𝑡 estimates the normalized future cost increment. The supervised training objective is ∑ ( ) TCGS = BCE 𝓁𝑡𝑘 , 𝑦𝑘𝑡 𝑘 (78) + 0.1 ‖𝐝̂𝑡 − 𝐝𝑡 ‖22 , where 𝑦𝑘𝑡 indicates whether budget 𝑘 is crossed within the prediction horizon and 𝐝𝑡 is the future normalized cost increment. At evaluation time, a frozen ACD3 -GAT policy proposes 𝑎prop . TCGS accepts the proposal only if the predicted downtime risk is below the deployability threshold and the immediate proxy cost does not exhaust a budget: [ ] 𝜒𝑡 (𝑎) = 𝟏 𝑝down (𝐻 , 𝑎) ≤ 𝜖 𝑡 𝜙 (79) [ 𝑘 ] ⋅ 𝟏 𝐶𝑡 + 𝑐̂𝑘 (𝑎) ≤ 𝐵𝑘 , ∀𝑘 . Here 𝜒𝑡 (𝑎) is an accept/reject indicator for candidate action 𝑎, 𝐶𝑡𝑘 is the cumulative cost already incurred in the current episode, and 𝑐̂𝑘 (𝑎) is the immediate cost proxy for that candidate. { 𝑎prop , 𝜒𝑡 (𝑎prop ) = 1, 𝑎exec = (80) arg min𝑎∈valid Ψ𝑡 (𝑎), 𝜒𝑡 (𝑎prop ) = 0.

(81)

𝑝down (𝑎) = 𝑝down (𝐻𝑡 , 𝑎), 𝑡 𝜙 ∑ 𝑝𝑘𝜙 (𝐻𝑡 , 𝑎), 𝑝̄𝑡 (𝑎) = 13 𝑘

𝑐̄𝑡 (𝑎) =

∑

𝑐̂𝑘 (𝑎)

𝑘

orders candidates first by hard budget feasibility 𝜓𝑡𝐵 , then by downtime-risk threshold feasibility 𝜓𝑡𝑃 , followed by predicted downtime risk, mean predicted violation risk, and immediate proxy cost. It evaluates one representative valid candidate per action type for speed. Thus TCGS is not an end-to-end retrained policy in this paper; it is a frozenpolicy temporal shield that tests whether learned contractrisk prediction can improve action screening online.

6.9. Override Readiness Three conditions can trigger the SLEEP fallback: max 𝜋(𝑎 ∣ ctx𝑡 ) < 𝜏conf ,

(82)

∃𝑘 ∶ 𝑏𝑘 (𝑡) = 0 ∧ 𝑐̂𝑘 (𝑎) > 0, [ ] ̂ 𝑡 , 𝑎) > 𝜏ood , Std𝑎∈ 𝑆(𝐺

(83)

𝑎

probe

(84)

with 𝜏conf ∈ {0.0, 0.15} depending on the ablation and 𝜏ood =0.7. Here 𝜏conf is the minimum policy-confidence threshold, probe is the finite set of candidate actions probed by the G-CRP uncertainty check, and 𝜏ood is the threshold on the standard deviation of predicted graph risk across those ∑ probes. The override burden 𝐽override = 𝔼[ 𝑡 𝟏[override]] is penalised via 𝜇 𝐴̂ override in (58). 𝑡

6.10. Training Algorithm Algorithm 1 writes the training horizon as 𝐸train because the experiments use different horizons (30, 100, and 200 episodes in the main benchmark, with 300-episode replications for the longer-horizon robustness check).

7. Experiments 7.1. Study Design The empirical study is organised around three questions that follow directly from the safety-contract formulation:

constraints. Do explicit operational constraints reduce budget violations without eliminating security utility? Operational

Preprint submitted for review. Final version may differ from this manuscript.

Page 14 of 30

Safety-Contract Graph MARL

Algorithm 1 ACD3 -GAT Training 1: Initialise 𝜃, 𝜙𝑟 , {𝜙𝑐𝑘 }, 𝜙CVaR , 𝜙expl , 𝝀 = 𝟎 2: for episode 𝑒 = 1, … , 𝐸train do 3: Reset; 𝐳0𝑖 ← 𝟎32 ; 𝑏𝑘 (0) ← 𝐵𝑘 ; batch ← ∅ 4: for 𝑡 = 0, … , 499 do 𝑖 ) 5: 𝐺𝑡𝑖 ←PARSEO BS(𝑜𝑖𝑡 ); 𝐠𝑖𝑡 ←GAT(𝐺𝑡𝑖 ); 𝐳𝑡𝑖 ←GRU(𝐠𝑖𝑡 , 𝐳𝑡−1 6: 𝑢𝑡 ← 𝐻(sof tmax(𝑊𝑢 𝐠𝑖𝑡 + 𝐝𝑢 ))∕ log 9; ctx𝑖𝑡 ← [𝐠𝑖𝑡 ‖𝐳𝑡𝑖 ‖𝐛𝑡 ‖𝐮𝑡 ] 7: Sample 𝑎𝑖𝑡 ∼ 𝜋𝜃 (⋅ ∣ ctx𝑖𝑡 ) 8: if confidence trigger fires (82) then 9: 𝑎𝑖𝑡 ← SLEEP 10: else 11: 𝑎𝑖𝑡 ← Shield(𝑎𝑖𝑡 , 𝐺𝑡 , safe (𝑡)) 12: end if ∑ 13: Execute joint action; observe 𝑟𝑖𝑡 , 𝑐𝑡𝑖,𝑘 ; 𝑐𝑡𝑘 ← 𝑖 𝑐𝑡𝑖,𝑘 ; 𝑏𝑘 (𝑡+1) ← max(0, 𝑏𝑘 (𝑡) − 𝑐𝑡𝑘 ) 14: end for 15: batch←batch∪{𝑒} 16: if |batch| = 8 then 3 17: Compute 𝐴̂ ACD via Eq. (58); normalise 𝑡 18: Run PPO update (4 epochs over the concatenated episode batch) 19: 𝜆𝑘 ← [𝜆𝑘 + 0.01(𝐽̂𝑐𝑘 − 𝐵𝑘 )]+ 20: batch← ∅ 21: end if 22: if adaptive Red-process evaluation enabled and 𝑒 mod 𝐼exploit = 0 then 23: Red mini-loop; update 𝛿̄ 24: end if 25: end for

Graph structure. Does graph structure, through graph

neural network (GNN) and Graph Attention Network (GAT) encoders, improve over flat encoders in tail behaviour and mean return?

Operational discipline. What does the gap between en-

gineered CAGE 4 heuristics and naive reactive rules reveal about operational discipline?

7.2. Environment and Setup We use CAGE Challenge 4 (TTCP CAGE Working Group, 2023) with 5 CAGE “Blue” agents, episode length 𝑇 =500, and a finite-state machine CAGE “Red” process as the baseline source of simulated malicious activity. All agents receive uniform 210-dimensional binary observations (padded to equal length across all agents). Operational budgets: 𝐵down =50, 𝐵fp =10, 𝐵fw =20 per episode. These values are fixed simulator stress-test thresholds for this study, not universal SOC constants. A production transfer would require selecting 𝐵𝑘 from local governance constraints and rerunning the same safety-labelled evaluation, ideally with a budget-sensitivity sweep. Reference points from prior work. In the default CAGE Challenge 4 (CC4) evaluation, the published benchmark table reports a top heuristic reference of −113 ± 35 and a top MARL reference of −193 ± 84 (Kiely et al., 2025a). The −101 ± 36 score corresponds to the constant-network-size reference rather than the default CC4 column (Kiely et al., 2025a). The best Large Language Model (LLM) agent (GPT-o1-mini, role prompting) scored ≈ −2888 (Castro et al., 2025). We include the default

CC4 values as reference lines in our figures and treat them as prior-work references, not direct baselines, because our experiments use a safety-labelled training/evaluation wrapper and explicit cost accounting. The LLM score is reported only as a literature reference because the published result does not provide action-level traces compatible with our downtime, firewall, and false-positive safety accounting. Hyperparameters: PPO clip 𝜖=0.2, 𝛾=0.99, 𝜆GAE =0.95, 𝜂𝜆 =0.01, 𝛼CVaR =0.1. All learning methods use hidden dimension 64. ACD3 -GAT uses 𝑀=8 episodes per update, (𝛽, 𝜂, 𝜇) = (0.1, 0.05, 0.01) for CVaR, exploitability, and override terms, and an exploitability mini-loop every 50 episodes when enabled. ACD3 -GAT and C-MAPPO-GAT use 𝑐ent = 0.005; the other MAPPO-family baselines (MAPPO-MLP, MAPPO-GNN, MAPPO-GAT, and CVaRMAPPO) use 𝑐ent = 0.01. All MAPPO-family learners use mini-batches of 64. The comparison controls the main training horizon, PPO hyperparameters, and hidden dimension, but it is not parameter-count matched: graph encoders and factorised heads have different capacity from flat MLP policies. We therefore interpret encoder and architecture rows as matched-protocol comparisons, not as capacity-normalised dominance claims. All baseline rows are presented as implemented policy families under the same safety logger; prior published CAGE-4 heuristic and LLM scores are used only to calibrate the reward scale because their public outputs do not expose the action-level traces required for our operational-cost audit. Metric semantics and reporting conventions. Down∑ time cost is the undiscounted episode total 𝑡 𝑐𝑡down ; an episode violates the downtime contract iff this total exceeds

Preprint submitted for review. Final version may differ from this manuscript.

Page 15 of 30

Safety-Contract Graph MARL

𝐵down = 50. Figures may plot either this raw cost or the ratio ∑ down ∕𝐵down ; captions state which is used. CVaR-10% is 𝑡 𝑐𝑡 the empirical mean return of the worst 10% of evaluated episodes, not merely the 10th percentile. The Pareto figures show the non-dominated points among observed methods, not a continuous optimum of the non-convex MARL objective. Confidence bands and box widths appear only when multiple seeds exist; methods with one seed or short component-check horizons are plotted without inferential claims and are labelled by their seed/episode counts in the table. For rare violation rates, the episode count should be read together with the seed count: for example, 𝑃 (violDT ) = 0.003 over 600 episodes corresponds to only a small number of observed violations and is therefore evidence of strong empirical compliance in this benchmark, not a formal probability guarantee for deployment. All operational costs are audited as undiscounted totals because budgets are governance limits, while reward learning still uses discounted returns. All learned policies sample only from the valid CybORG action mask exposed by the environment wrapper. If a budget shield changes an action, the stored trajectory records the executed action label and recomputed costs; the replication records also store the run configuration, random seed, topology seed when available, git commit, package freeze, and safety-labelled per-step trajectory needed to recompute the benchmark tables. Evaluation protocol. Each comparison reports its seed count and episode horizon explicitly. The study combines five evidence streams. First, reward-only baselines quantify the operational failure mode: Independent Proximal Policy Optimisation (IPPO), Multi-Agent PPO with a multi-layer perceptron encoder (MAPPO-MLP), and MAPPO-GAT are each evaluated at a 200-episode horizon, with IPPO and MAPPO-GAT replicated across three seeds. Second, CMAPPO-GAT, the constrained safety baseline introduced in this paper, is evaluated over three 200-episode seeds to test whether Lagrangian costs and budget-aware action screening reduce SOC safety-contract violations. Third, ACD3 GAT is evaluated as the integrated architecture, combining graph attention, budget context, CVaR weighting, and override with three 200-episode seeds. Fourth, two additional 300-episode replications are run for MAPPO-GAT, C-MAPPO-GAT, and ACD3 -GAT to test whether the main safety conclusion survives a longer horizon. Fifth, short 30-episode ACD3 +AskHuman and ACD3 +deterministic GCRP component runs verify that these components operate inside the safety-contract regime and separate component behaviour from the core replicated comparison. This structure separates replicated comparisons from component checks and keeps each quantitative claim tied to its corresponding evaluation horizon.

7.3. Baselines We evaluate the following policy families:

• Rule-based: Monitor by default; Restore on proc alert; Block on persistent net alert • IA2C: Independent Advantage Actor–Critic with MLP encoder. Prior method: (Mnih et al., 2016) (Equations 5–8) • IPPO: Independent Proximal Policy Optimisation with MLP encoder. Prior method: (de Witt et al., 2020) • MAPPO-MLP: Multi-Agent PPO with a centralised critic and MLP encoder. Prior method: (Yu et al., 2022) • MAPPO-GNN: Multi-Agent PPO with a centralised critic and GraphSAGE encoder. Encoder prior: (Hamilton et al., 2017) • MAPPO-GAT: Multi-Agent PPO with a centralised critic and GAT encoder. Encoder prior: (Veličković et al., 2018) (ours) • C-MAPPO-GAT: Constrained MAPPO-GAT, introduced here as a controlled safety-contract baseline with Lagrangian cost learning (Altman, 1999; Achiam et al., 2017) and the configured hard budget-exhaustion fallback (ours) • CVaR-MAPPO: Conditional Value-at-Risk (CVaR)MAPPO with episode-level CVaR reweighting, following the CVaR tail-risk objective (Rockafellar and Uryasev, 2000) and treated as a safety component rather than a separate safety-contract policy claim • ACD3 -GAT: graph-attentive constrained response policy with budget context, CVaR weighting, and override mechanisms (ours; three 200-episode seeds) For MAPPO-GAT and C-MAPPO-GAT, the GAT encoder in Equations 19–36 instantiates Enc𝑖𝜙 in the centralised critic of Equation 16; the actor still executes from each agent’s local observation. For ACD3 -GAT, the same graph embedding 𝐠𝑖𝑡 is concatenated with opponent, budget, and uncertainty context in Equation 44 before the factorised action policy and safety shield are applied.

7.4. Benchmark Method Instantiations All learned methods use the CAGE-4 observation and action spaces defined in Section 5, and they differ only in the policy update, encoder, critic information, and safety machinery. This subsection fixes the mapping between Table 2 and the implemented algorithms.

Non-learning reference policies. Sleep is the determin-

istic policy 𝜋(𝑎𝑖𝑡 = SLEEP ∣ 𝑜𝑖𝑡 ) = 1. Random samples uniformly from the valid action mask 𝑖valid (𝑜𝑖𝑡 ):

• Sleep: always-sleep lower bound

𝜋rand (𝑎 ∣ 𝑜𝑖𝑡 ) = |𝑖valid (𝑜𝑖𝑡 )|−1 𝟏[𝑎 ∈ 𝑖valid (𝑜𝑖𝑡 )].

(85)

• Random: uniform random valid actions Preprint submitted for review. Final version may differ from this manuscript.

Page 16 of 30

Safety-Contract Graph MARL

Here 𝑖valid (𝑜𝑖𝑡 ) is the CybORG action mask for agent 𝑖 in observation 𝑜𝑖𝑡 , so invalid simulator actions receive zero probability. The rule-based policy is a deterministic alert policy: monitor by default, restore when process alerts are present, and block traffic when persistent network alerts are visible. These policies use no learned parameters; their returns and safety metrics are computed with the same logger and cost proxies as the learned methods.

advantage by an episode-level tail weight. For a batch of 𝑀 episodes and tail fraction 𝛼 = 0.1,

IA2C and IPPO. IA2C follows Equations 5–8. IPPO uses

Here 𝑒 indexes episodes in the PPO batch, 𝑤𝑒 is the binary tail weight assigned to the worst-return episodes, and is the reward-only MAPPO advantage. This row 𝐴̂ MAPPO 𝑡 therefore tests tail-risk reweighting alone, without Lagrange constraints or action shielding.

the same independent per-agent information pattern, but replaces the vanilla actor–critic objective with clipped PPO: 𝜋𝜃𝑖 (𝑎𝑖𝑡 ∣ 𝑜𝑖𝑡 ),

𝑉𝜙𝑖 (𝑜𝑖𝑡 ),

𝑖IPPO = CLIP +𝑐vf VF 𝑖 𝑖 −𝑐ent 𝐻[𝜋𝜃𝑖 ], (86)

with GAE advantages from Equations 9–10. The actor parameters 𝜃𝑖 , critic parameters 𝜙𝑖 , local observation 𝑜𝑖𝑡 , and entropy/value-loss weights are the same quantities defined for IA2C and PPO in Section 4. The IPPO row uses an MLP encoder; the Factorized-IPPO row keeps the same independent PPO update but replaces the flat actor with the type–target factorisation in Equation 37.

MAPPO encoder variants. MAPPO-MLP, MAPPO-

GNN, and MAPPO-GAT all use the centralised critic in Equation 16 and the clipped PPO objective in Equation 15. Their only architectural difference is the per-agent encoder: ⎧MLP(𝑜𝑖 ), 𝑡 ⎪ 𝑖 𝑖 Enc𝜙 (𝑜𝑡 ) = ⎨GraphSAGE(𝐺𝑡𝑖 ), ⎪GAT(𝐺𝑖 ), ⎩ 𝑡

MAPPO-MLP, MAPPO-GNN, (87) MAPPO-GAT.

Here Enc𝑖𝜙 denotes the per-agent encoder whose output is sent to the centralised critic during training; 𝐺𝑡𝑖 is the graph parsed from the same local observation 𝑜𝑖𝑡 . GraphSAGE and GAT are defined in Equations 18 and 19–36, respectively. Actors execute decentralised local policies, while the critic uses the concatenated five-agent embedding during training.

Opponent-conditioned IPPO. Opp-IPPO keeps the inde-

pendent PPO loss but augments each agent’s policy state with a recurrent opponent embedding: 𝑖 𝐳𝑡𝑖 = GRU(Enc(𝑜𝑖𝑡 ), 𝐳𝑡−1 ),

𝑎𝑖𝑡 ∼ 𝜋𝜃𝑖 (⋅ ∣ 𝑜𝑖𝑡 , 𝐳𝑡𝑖 ).

(88)

Here 𝐳𝑡𝑖 is the recurrent state for agent 𝑖, initialised to zero at the start of each episode and updated from that agent’s encoded local observation. This is the same recurrent mechanism later reused in the ACD3 context vector, but without safety budgets, CVaR weighting, or shielding.

CVaR-MAPPO. CVaR-MAPPO keeps the MAPPO centralised critic and PPO update, but multiplies the policy

𝑘∗ = max(1, ⌊𝛼𝑀⌋), { 1∕𝑘∗ , 𝑒 ∈ worst 𝑘∗ episodes, 𝑤𝑒 = (89) 0, otherwise, -MAPPO = 𝑤 𝐴̂ MAPPO . 𝐴̂ CVaR 𝑒 𝑡 𝑡

Constrained MAPPO-GAT. C-MAPPO-GAT is our con-

strained MAPPO-GAT instantiation: it keeps the MAPPOGAT encoder and centralised critic, adds cost-advantage penalties for the three SOC budgets, and applies the configured hard budget-exhaustion fallback: ∑ 𝑐 -MAPPO = 𝐴̂ 𝑟 − 𝜆𝑘 𝐴̂ 𝑡 𝑘 , 𝐴̂ C 𝑡 𝑡 𝑘 (90) 𝜆𝑘 ← [𝜆𝑘 + 𝜂𝜆 (𝐽̂𝑐𝑘 − 𝐵𝑘 )]+ . 𝑐 Here 𝐴̂ 𝑟𝑡 is the reward advantage, 𝐴̂ 𝑡 𝑘 is the advantage computed from cost stream 𝑘, 𝜆𝑘 is the corresponding Lagrange multiplier, and 𝐽̂𝑐𝑘 is the observed batch cost estimate compared with budget 𝐵𝑘 . This is the safety-contract baseline against which ACD3 -GAT is interpreted.

ACD3 variants. ACD3 -GAT uses the full context vector

Equation 44, factorised policy Equation 37, composite advantage Equation 58, and shielded execution Equation 62. The AskHuman and deterministic G-CRP rows are short configuration checks of the override and action-screening components inside the same ACD3 scaffold; they are not presented as separately optimised policy families.

7.5. G-CRP Shield Diagnostics To isolate the contribution of Graph Counterfactual Risk Propagation (Section 6.7), we define three shield variants while holding the policy class (ACD3 -GAT) fixed:

Rule-based shield. The reactive shield blocks actions whose cost proxy is positive for an exhausted budget; it uses no prediction and no graph reasoning.

Deterministic G-CRP. The physics-based shield uses the cascade propagation model in Equation 69, selects arg max 𝑄safe (Equation 71), and requires no training.

Learned G-CRP. The supervised extension trains a

GraphSAGE model on collected trajectories to predict next-step alert state, cost, and violation probability before action screening. The deterministic G-CRP path is included as part of the ACD3 action-screening design and is evaluated in a short configuration check. The learned G-CRP model is trained

Preprint submitted for review. Final version may differ from this manuscript.

Page 17 of 30

Safety-Contract Graph MARL

in the replication package as the supervised risk-estimation extension of the deterministic screen. The quantitative policy comparison reports the deterministic G-CRP path so that each policy-performance claim is tied to the actionscreening rule used in the evaluated runs. When that component is evaluated, the intended measures are violation rate, mean return, override rate, violation-probability calibration, and cost-prediction error.

7.6. Robustness Extensions The robustness evaluation tests whether the safety contract survives outside the matched training and evaluation seed setting: • In-distribution (ID): 5 topology seeds used during training • Topology out-of-distribution (OOD): 5 unseen seeds (different host counts) • Red-process OOD protocol: a separate evaluation with Red policies not used in Blue training or tuning, reserved for future quantitative claims beyond the present split-labelled seed-variation artifacts • Mission OOD: altered mission phase schedule • Adaptive Red process: stress curve – Red PPO trained 20 rounds against frozen Blue policies The present benchmark reports the ID setting and two robustness checks. First, IPPO, MAPPO-GAT, and constrained MAPPO-GAT are evaluated on unseen topology seeds for 20 episodes per seed, using two seeds per split and the same finite-state Red process. Second, a Red PPO league is trained for 20 updates against frozen Blue policies, with sampled Red-process actions coupled into the CybORG transition loop. The split-labelled Red-process artifacts collected in this replication package use the same CAGE finitestate Red-process family with seed variation and are therefore not interpreted as held-out Red-policy evidence. Heldout Red-process and mission-schedule variants remain part of the evaluation protocol but are not used as quantitative claims here.

8. Results 8.1. Evaluation Context All experiments use CAGE Challenge 4 (TTCP CAGE Working Group, 2023) with 𝑁=5 CAGE “Blue” agents, 𝑇 =500 steps, and the finite-state machine CAGE “Red” process. All agents receive a uniform 210-dimensional binary observation vector. Operational budgets: 𝐵down =50, 𝐵fp =10, 𝐵f w =20 per episode. The main replicated comparison uses three 200-episode seeds for IPPO, MAPPO-GAT, constrained MAPPO-GAT, and ACD3 -GAT. MAPPO-MLP provides a one-seed flat-encoder reference at the same horizon. For the three principal graph-based methods, two additional 300-episode replications assess whether the central safety ordering persists when training is extended. Shorter

architectural component runs (100 or 30 episodes) are used only to interpret component behaviour and are labelled separately from the replicated comparison; they are not used to claim a new best policy.

8.2. Safety and Return Table 2 summarises the safety-contract benchmark. The table reports the complete direct comparison under our safety-labelled evaluation protocol: it includes the non-learning baselines, IA2C, IPPO, MAPPO with MLP/GraphSAGE/GAT encoders, factorised and opponentconditioned IPPO variants, CVaR-MAPPO, C-MAPPOGAT, and ACD3 -GAT. The figures provide the same evidence at different levels of resolution. Figure 3 summarizes the benchmark through four linked views: budget violation, budget overrun, return–downtime tradeoff, and tail-risk gap. Figure 4 then unpacks that dashboard into four benchmark views: panel (a) asks whether the episode exceeds 𝐵down ; panel (b) identifies which actionderived costs are responsible for the separation; panel (c) places the same methods on a return–downtime plane; and panel (d) checks whether mean return is hiding poor worst-tail outcomes. Figure 5 connects the final summaries to training dynamics rather than treating the table as a single endpoint, and Figure 6 makes the replication status visible by separating three-seed comparisons from one-seed or short component checks. Finally, Figure 9 reports the coupled adaptive Red-process stress test, while Figure 7 visualises the deterministic graph counterfactual model used by the shield. Published CAGE-4 and LLM numbers appear only as prior-work reference lines or table notes; they are not treated as direct baselines because they lack the same safety-labelled action traces and budget accounting. Component-check rows. The final two ACD3 rows in Table 2 are implementation checks of the governance layer rather than separate policy claims. Both reuse the same trained ACD3 -GAT checkpoint for a 30-episode, singleseed check while emphasizing a different screening path: the AskHuman escalation gate or the deterministic G-CRP shield. Under the default confidence setting (𝜏conf = 0.0), the AskHuman gate does not separate the aggregate metrics in this short window, so the two rows coincide. They are included to show that these screening hooks preserve the learned reward and compliance profile under the short component check; attributing performance differences between the hooks would require a longer targeted evaluation. Read together, Table 2 and Figure 3 establish the central empirical pattern. In the dashboard’s compliance panel, the unconstrained learners cluster at full downtime-budget violation, meaning that the policies cross 𝐵down in every evaluated episode even when their reward is competitive. The adjacent over-budget panel explains the scale of the failure: the same methods consume several multiples of the allowed MTTR budget, whereas C-MAPPO-GAT moves below the budget and ACD3 -GAT moves close to it on average. The Pareto panel then shows the price of that movement. Reward-only MAPPO-GAT and random exploration sit high

Preprint submitted for review. Final version may differ from this manuscript.

Page 18 of 30

Safety-Contract Graph MARL Table 2 Benchmark summary for the safety-contract evaluation. Episode and seed columns make the replication depth explicit. “Ep.” is the total number of evaluated episodes included for each row. 𝑅̄ is mean episode return; CVaR10% is the worst-tail mean return averaged over per-run metrics; 𝑃 (violDT ) is downtime-budget violation rate; 𝑐̄DT is mean downtime cost; Catast. is the catastrophic episode rate under the alert-threshold proxy. † marks the most conservative safety-contract compliance row in this benchmark; ⋆ marks the integrated ACD3 -GAT method architecture on the broader safety-contract frontier. Citations in the method column identify prior algorithmic families; C-MAPPO-GAT and ACD3 -GAT are introduced here as safety-contract instantiations built from those components. Group

Method

Ep.

S

𝑅̄

Nonlearn.

Sleep Random Rule-based (Kiely et al., 2025b)

140 140 140

3 3 3

−6,792 −5,149 −8,124

−8,984 0.000 −6,737 1.000 −10,456 1.000

0.0 0.000 426.1 0.107 115.9 0.000

Actor-critic

IA2C (Mnih et al., 2016) IPPO (Schulman et al., 2017)

100 600

1 3

−4,948 −4,254

−6,691 1.000 −6,299 1.000

420.6 0.070 314.4 0.012

MAPPO enc.

MAPPO-MLP (Yu et al., 2022) MAPPO-GNN (Yu et al., 2022; Hamilton et al., 2017) MAPPO-GAT (Yu et al., 2022; Veličković et al., 2018)

200 100 600

1 1 3

−3,937 −4,387 −3,979

−6,226 1.000 −6,108 1.000 −5,864 1.000

316.2 0.015 406.2 0.010 355.4 0.035

Arch.

Fact.-IPPO (Schulman et al., 2017) Opp-IPPO (Schulman et al., 2017; Cho et al., 2014)

100 100

1 1

−5,378 −4,648

−7,428 1.000 −7,015 1.000

311.1 0.120 321.2 0.000

Safety

CVaR-MAPPO (Yu et al., 2022; Rockafellar and Uryasev, 2000) 100 C-MAPPO-GAT (Yu et al., 2022; Veličković et al., 2018; 600 Altman, 1999) † ACD3 -GAT (Yu et al., 2022; Veličković et al., 2018; Altman, 600 1999; Rockafellar and Uryasev, 2000) ⋆

1 3

−5,131 −6,992

−6,509 1.000 −9,694 0.003

429.6 0.090 15.5 0.000

3

−8,144

−11,074 0.138

48.2 0.003

ACD3 +AskHuman ACD3 +det. G-CRP

1 1

−7,901 −7,901

−10,460 0.100 −10,460 0.100

36.8 0.000 36.8 0.000

ACD3 comp.

in return but far to the right of the downtime boundary; C-MAPPO-GAT moves into the feasible operational region at a return cost; and ACD3 -GAT occupies an intermediate frontier point that reflects the integrated architecture rather than the most conservative compliance setting. The final dashboard panel makes the same argument for tail behaviour, which is that the gap between mean return and CVaR-10% remains large enough that average reward cannot serve as the sole deployment criterion.

compliance. C-MAPPO-GAT reaches 𝑃 (violDT ) = 0.003 across 600 episodes from three seeds. This corresponds to two downtime-budget violations in the replicated core sample, providing empirical compliance evidence under the reported benchmark but not a certified chance constraint. ACD3 -GAT reaches 𝑃 (violDT ) = 0.138 with mean downtime cost 48.2 across three 200-episode seeds: below the episode budget on average, but with less reliable episode-level compliance than C-MAPPO-GAT. Every unconstrained learning method in Table 2 achieves 𝑃 (violDT ) = 1.000. The 200-episode IPPO/MAPPO rows still consume 314–355 downtime-cost units per episode, while constrained MAPPO-GAT consumes 15.5, a 95–96% reduction against a budget of 50. ACD3 -GAT consumes 48.2, an 84–86% reduction in mean downtime cost relative to the same unconstrained policies, with a 13.8% violation rate. This places the integrated policy on the safety-contract frontier rather than at the most conservative compliance point: it demonstrates the full ACD3 architecture, while Constraint

30 30

CVaR

𝑃DT

𝑐̄DT

Cat.

C-MAPPO-GAT—introduced here as the constrained MAPPO/GAT/Lagrangian instantiation of the same safety-contract idea—represents the strongest compliance configuration in the present benchmark. The cost savings do not come for free: constrained MAPPO-GAT and ACD3 -GAT both have lower return than the unconstrained MAPPO variants. The resulting Pareto tradeoff is explicit: operational safety costs security reward, but a policy with 𝑃 (violDT ) = 1.000 cannot be considered deployable under the stated SOC contract. Figure 4 decomposes this distinction across violation probability, operational cost, return–downtime tradeoff, and tail-risk gap. Panel (a) turns downtime into a governance event–crossing the episode budget–rather than a continuous training metric. Panel (c) then shows why the safety result is not simply a lower-score variant of the same policy: the constrained methods move the system into a different operating regime, where lower reward is exchanged for remaining inside or near the MTTR budget. Panel (b) explains that downtime Restore cost is the dominant operational harm separating the methods, while firewallchange and false-positive costs remain important secondary contract dimensions. Panel (d) keeps the interpretation honest: even when a method improves mean return, its worst episodes can remain much worse than its average behaviour.

Graph structure and tail behaviour. In the benchmark,

MAPPO-MLP at 200 episodes (−3,937) remains ahead

Preprint submitted for review. Final version may differ from this manuscript.

Page 19 of 30

Safety-Contract Graph MARL

Figure 3: Main empirical story in four focused panels. The colors and markers identify the representative methods shown in the dashboard; the complete benchmark, including IA2C, MAPPO-MLP/GNN, Factorized-IPPO, Opp-IPPO, and CVaR-MAPPO, is reported in Table 2. (a) Reward-only policies violate the downtime budget in nearly every episode; C-MAPPO-GAT is the most reliable safety-contract configuration, while ACD3 -GAT is evaluated as the integrated safety-contract architecture. (b) ∑ The same methods consume multiple MTTR budgets per episode; the dashed boundary corresponds to 𝑡 𝑐𝑡down ∕𝐵down = 1. (c) The return–downtime Pareto surface exposes the operational price of constraint compliance and the current ACD3 -GAT frontier position. (d) The gap between mean return and CVaR-10% shows that tail episodes remain a first-class risk even when average reward improves.

of MAPPO-GAT across 600 episodes (−3,979), while MAPPO-GAT has the strongest CVaR-10% score among the unconstrained MAPPO encoder variants. The evidence therefore supports a narrower conclusion: graph structure improves tail behaviour, but the raw-return advantage of graph attention has not emerged in the three-seed MAPPOGAT comparison. Figure 5 is useful here because it shows that the reward-only policies learn quickly toward higher return while remaining operationally unsafe, whereas the constrained curves occupy a lower-return region because the policy update is also paying for downtime, firewall, and false-positive costs. Figure 6 complements this by showing the spread of seed-level returns and marking which rows are replicated enough to support the core comparison. The visual message is therefore not that graph attention dominates every metric, but that graphaware and constrained variants expose different axes of performance: representation affects tail return, while operational constraints affect deployability.

Longer-horizon replication. Table 3 reports the two ad-

ditional 300-episode replications for MAPPO-GAT, constrained MAPPO-GAT, and ACD3 -GAT. The longer horizon preserves the main ordering. Reward-only MAPPO-GAT remains operationally non-deployable: 𝑃 (violDT ) = 1.000 and mean downtime cost 357.1, despite strong mean return (𝑅̄ = −4,033). Constrained MAPPO-GAT remains the most reliable safety-contract policy, with 𝑃 (violDT ) = 0.007, mean downtime cost 10.1, and zero catastrophic episodes across the two 300-episode replications. ACD3 -GAT remains close to the downtime budget on average (𝑐̄DT = 48.8), with a 14.3% episode-violation rate. The extension strengthens the central safety-contract result and preserves the same frontier interpretation: the integrated method reduces operational harm, while the constrained baseline remains the most conservative compliance configuration in the reported evaluations.

Preprint submitted for review. Final version may differ from this manuscript.

Page 20 of 30

Safety-Contract Graph MARL

Downtime-budget violation rate 100.0%

Violation rate (%)

100

100.0%

75 50 25

13.8%

5% safety target

0

p

0.3%

dom

Slee

d

O

IPP

-base

Ran

Rule

T

-GA

PPO

MA

APP

C-M

AT

O-G

AC

D -G 3

AT

(a) Downtime-budget violation.

(b) Operational cost decomposition.

Sleep Random Rule-based IPPO MAPPO-GAT C-MAPPO-GAT

−5000

ACD3 -GAT IA2C MAPPO-MLP MAPPO-GNN Factorized-IPPO Opp-IPPO CVaR-MAPPO

−6000

−7000

−2000

Episode return

−4000

Mean episode return R ̄

Mean vs CVaR-10\% Episode Returns

0

Security–Operations Pareto Frontier

−4000 −6000 −8000

ACD3 +AskHuman ACD3 +det. G-CRP ACD3 -TCGS Bdown = 50

−8000

Mean return CVaR-10\%

−10000

Bdown = 50

0

100

200

300

Mean downtime cost per episode

O

400

IPP

M

(c) Return–downtime frontier.

T

-GA

PO AP

C-M

T

-GA

PO AP

3 -G A

D AC

T

(d) Mean return and CVaR-10%.

Figure 4: Safety-contract benchmark decomposed into violation, operational cost, return–cost frontier, and tail-risk views. (a) A ∑ violation is the episode-level event 𝑡 𝑐𝑡down > 𝐵down ; the dashed line marks a 5% safety-contract target. (b) Downtime (Restore), firewall churn (Block/Allow), and false-positive Restore costs are undiscounted episode totals. (c) The vertical boundary marks the downtime budget 𝐵down = 50; points left of the boundary satisfy the mean MTTR budget. (d) CVaR-10% is the mean return of the worst 10% of episodes, showing that reward improvements do not automatically remove worst-tail episodes. Table 3 Longer-horizon replication check for the three principal graph-based learning methods. The table reports two additional 300-episode replications per method, evaluated separately from the balanced three-seed benchmark in Table 2. Method

Ep.

S

𝑅̄

CVaR

𝑃DT

𝑐̄DT

Cat.

MAPPO-GAT C-MAPPO-GAT ACD3 -GAT

600 600 600

2 2 2

−4,033 −6,962 −8,132

−5,848 −9,594 −11,082

1.000 0.007 0.143

357.1 10.1 48.8

0.063 0.000 0.002

Preprint submitted for review. Final version may differ from this manuscript.

Page 21 of 30

Safety-Contract Graph MARL A. Learning dynamics

B. Mean and worst-tail return MAPPO-GAT

Episode return (moving average)

−3000 −4000 Random

−5000

IPPO

−6000 Sleep

−7000

C-MAPPO-GAT

Rule-based

−8000 −9000

ACD3 -GAT 95% seed band

IPPO MAPPO-GAT C-MAPPO-GAT

−10000 0

25

ACD3 -GAT

50

75

100

125

150

Training episode

175

200

−12000 −10000 −8000 −6000 −4000

Episode return

Figure 5: Learning dynamics and final return-risk context for the principal learning methods. (a) Moving-average episode returns over the common 200-episode benchmark window, with shaded seed bands for replicated methods. Non-learning baselines are shown only as muted scale references. (b) Final mean return and CVaR-10% for the same methods. This pairing should be read with the violation figures because reward alone is not the deployment criterion. Seed Consistency: Robustness of Method Ranking Across Replicates Published CC4 refs

Published LLM ref

1--2 seeds

−8000

1--2 seeds

−6000

1--2 seeds

1--2 seeds

1--2 seeds

1--2 seeds

−4000

1--2 seeds

1--2 seeds

−2000 1--2 seeds

Per-seed mean return

0

Boxes denote replicated methods; diagnostics with 1--2 seeds are labelled.

2C

IA

O

IPP

P MA

PO

LP

-M

P MA

PO

NN

-G

P MA

PO

AT

-G

F

PO

-IP

d ize

or act

Op

O

PP p-I

A -G PO

P MA

C-

T C

PO AP

-M

R Va

3 -G

D AC

AT 3 +A

D AC

skH

an um

et.

3 +d

D AC

G-

P CR

3 -T

D AC

S CG

Figure 6: Seed-consistency audit for learning methods. Each point is one seed’s mean episode return and boxes summarise the across-seed distribution. Single-seed and short component-check methods are explicitly marked, separating replicated comparisons from exploratory component checks.

8.3. Operational Discipline and Heuristic Baselines A central finding from the CAGE4 competition (Kiely et al., 2025b) is that engineered heuristic agents outperformed the submitted MARL agents. Our local rule-based baseline is not a reproduction of the winning heuristic. It is a deliberately simple reactive policy that isolates the role of operational discipline. Our rule-based agent achieves 𝑅̄ ≈ −8,124 – worse than doing nothing (Sleep: ≈ −6,792) and far below random (≈ −5,149). The mechanism is clear from the cost breakdown: 𝑐̄DT ≈ 115 (rule-based) vs 0 (sleep) and 425 (random). The rule agent triggers R ESTORE on every malicious-process alert, eventually exhausting 𝐵down = 50 because the cost proxy counts one unit per Restore action. Each subsequent

Restore action is a budget violation, and the accumulated cost overwhelms the security benefit. This motivates constrained learning: naive rules fail because they lack the observation engineering, valid-action filtering, mission-phase traffic discipline, and selective response logic that made the top CAGE4 heuristics effective. Unconstrained RL fails for a complementary reason: it discovers high-impact actions such as R ESTORE but does not internalise the operational budget. The target for any deployable autonomous response policy is therefore not just high security reward, but safety-contract compliance.

8.4. Architectural and Component Analysis Encoder architectures. The MAPPO family covers three encoder architectures. At the reported horizons:

Preprint submitted for review. Final version may differ from this manuscript.

Page 22 of 30

Safety-Contract Graph MARL

• MLP: 𝑅̄ = −3,937, CVaR10% = −6,226 at 200 episodes. Best raw return; no structural inductive bias. • GNN (GraphSAGE): 𝑅̄ = −4,387, CVaR10% = −6,108 at 100 episodes. Graph aggregation reduces tail episodes.

Table 4 Learned G-CRP fitting result. The table reports the supervised risk-estimation component trained from safety-labelled trajectories; policy performance in the benchmark is tied to the deterministic G-CRP screen used by the evaluated runs.

• GAT: 𝑅̄ = −3,979, CVaR10% = −5,864 across 600 episodes from three seeds. Best reported MAPPO encoder CVaR-10%; The attention mechanism adds interpretability at modest extra cost. The graph encoders improve the tail metric relative to the MLP run (−6,108 for GNN and −5,864 for GAT vs −6,226 for MLP), while the MLP still leads in mean return. In this benchmark, graph structure contributes primarily to tailrisk behaviour and interpretability rather than to raw-return dominance.

Factorized action head. Factorized-IPPO (−5,378 at 100

episodes) underperforms replicated IPPO (−4,254 across 600 episodes). The factorized head (type × target) has more parameters and slower apparent convergence in this short 100-episode component run, especially with the GAT encoder. However, its catastrophic episode rate (0.120 vs 0.012 for IPPO) is higher, so the factorized variant remains an instability to revisit in longer ablations.

Opponent embedding. Opp-IPPO (−4,648 at 100

episodes) trails replicated IPPO (−4,254 across 600 episodes) in return but achieves catastrophic rate 0.000 vs 0.012 for IPPO. This is consistent with the GRU opponent embedding acting as a useful regulariser for worst episodes, although the short horizon prevents a replicated architectural conclusion.

CVaR reweighting. CVaR-MAPPO (−5,131, CVaR =

−6,509) vs MAPPO-MLP (−3,937, CVaR = −6,226 at 200 episodes): at this training horizon, episode-level return-tail reweighting by itself is less effective than explicit operational-cost constraints. This supports the paper’s central design choice: tail-risk accounting is a useful component of ACD3 , but SOC deployability is driven primarily by the safety contract.

ACD3 component checks. The 30-episode AskHuman

and deterministic G-CRP component runs both reach 𝑃 (violDT ) = 0.100, 𝑐̄DT = 36.8, and 𝑅̄ = −7,901. This is safer than the reported ACD3 -GAT result on violation rate (0.100 vs 0.138), but the horizon is shorter and the two variant metrics are identical at this summary level. These short-horizon results show that the ACD3 switches can operate within the safety-contract regime early in training. They are component checks, not attribution evidence: because the AskHuman and deterministic G-CRP rows collapse to identical aggregate metrics, they are not used to claim separate causal effects for the two switches. Their role is to verify that the override and deterministic G-CRP paths execute within the same accounting framework as

Quantity

Value

Source trajectory files Valid transition pool Sampled transitions Training epochs Loss at epoch 5 Loss at epoch 10

6 45,000 6,000 12 0.2638 0.2606

the replicated policies; causal attribution would require per-step override reasons, proposed and executed actions, 𝑄safe values, uncertainty estimates, and remaining-budget state at the decision point.

Learned G-CRP fitting. Table 4 reports the supervised

fitting result for the learned G-CRP model. The model was trained from six longer-horizon trajectory files, using 6,000 sampled transitions from a 45,000-transition valid pool. The loss decreased from 0.2638 at epoch 5 to 0.2606 at epoch 10, indicating that the learned risk-propagation component can be fitted from the collected safety-labelled trajectories. This establishes the supervised risk-estimation path from the collected trajectories. The policy benchmark itself uses the deterministic G-CRP screen, keeping policy performance claims tied to the action-screening rule used during the reported evaluations. Figure 7 should therefore be read as a mechanism figure, not as another benchmark row. Each panel applies a different candidate response action to the same graph belief state and propagates risk with the deterministic cascade in Equation (69). The comparison illustrates how G-CRP changes the action-screening problem: actions are not ranked only by immediate reward or immediate cost, but by the post-action graph-risk field that would be handed back to the policy loop. This is the counterfactual calculation behind 𝑄safe in Equation (71).

Temporal Contract Graph Shielding diagnostic.

ACD3 -TCGS extends the action-screening question from one-step graph risk to future contract risk. The temporal model was trained on 40,000 length-8 sequences sampled from 180 safety-labelled episodes across 9 trajectory files. On the held-out split, it achieves near-perfect discrimination for future downtime-budget violation (AUCdown = 0.99998, Brier score = 0.00162, calibration error = 0.00177); firewall-disruption and false-positive Restore risks show similarly high discrimination (AUCf w = 0.99827 and AUCf p = 0.99995). At the diagnostic threshold 𝜖 = 0.05, the fitted model accepts approximately 32.9% of candidate histories, with an observed future downtime-violation rate of 0.11% among accepted samples. We also ran a frozen-policy ACD3 -GAT + TCGS screening diagnostic. The completed frozen-policy

Preprint submitted for review. Final version may differ from this manuscript.

Page 23 of 30

Safety-Contract Graph MARL

1.00

Baseline (Sleep)

Remove v0

Δp̄ = − 0.00

Δp̄ = − 0.08

0.75

1.0

0.50

0.8

0.25 0.00

0.6

Restore v0

BlockTraffic

Δp̄ = − 0.10

0.4

Δp̄ = − 0.14

0.75

Risk level

Compromise probability p̂v

1.00

Compromise probability p̂v

Graph counterfactual risk propagation: node risk p̂v (t+2 | a)

0.2

0.50

0.0

0.25 0.00

Node

Node

Figure 7: Graph Counterfactual Risk Propagation (G-CRP) under alternative response actions. Each panel applies one counterfactual action to the same graph observation and propagates risk for two steps using the independent-cascade model in Equation (69). This figure illustrates the deterministic rollout-free shield model used by the reported policy runs; learned G-CRP is the supervised risk-estimation extension trained from logged trajectories. ACD3 -TCGS: predictive temporal contract shielding

A. Downtime violation

B. Utility retained

C. Temporal shield behavior

P(violDT )

0.10 0.05

0.05 0.00

0.00 AP C-M

T

GA PO-

Episode return

0.14

0.15

−8000 −9000 −10000 Mean return CVaR-10%

−11000 AC

3 -G A

D

T A

G 3 -T C CD

S

Rate / normalized cost

−7000

AC

3 -G A

D

T A

G 3 -T C CD

T

S

-GA PPO -MA

1.0 0.8

ε = 0.05 0.79

0.69

0.6 0.4 0.2 0.0

Override rate

DT cost / budget

C

3

Figure 8: Predictive temporal contract shielding diagnostic for ACD -TCGS. (a) The policy-level diagnostic compares downtimebudget violation for C-MAPPO-GAT, ACD3 -GAT, and the frozen ACD3 -GAT policy screened by the temporal contract-risk model. (b) Mean return and CVaR-10% are reported on the same policies so the shield is interpreted as a safety–utility intervention rather than a violation-only post-processing step. (c) The final panel reports the deployability threshold 𝜖 = 0.05, override rate, and normalized downtime cost for the temporal shield.

diagnostic contains 100 episodes. It is included as singleseed policy-level diagnostic evidence. Mean downtime cost is 39.3, with downtime-violation rate 5.0%; falsepositive violation is 8.0% and catastrophic rate is 0.0%. The mean return is −7,331 with CVaR10% = −10,590. The firewall-change budget remains the active weakness (𝑐̄fw = 105.2, violation rate 31.0%), so these results are used as temporal-shield evidence rather than as a headline claim that ACD3 -TCGS dominates the replicated policies. Figure 8 visualizes this diagnostic frontier.

8.5. Tail-Risk and Catastrophic Episodes We define a catastrophic episode as one in which the mean alert level exceeds threshold 𝜃alert =8, corresponding to coordinated multi-host compromise that threatens mission continuity. Table 2 reports the catastrophic episode rate for all methods. CVaR reweighting alone is insufficient at this horizon. CVaR-MAPPO achieves a catastrophic rate of 0.090 – higher than MAPPO-MLP (0.015 at 200 episodes) and MAPPO-GNN (0.010 at 100 episodes). Its CVaR10% (−6,509) is also worse than the reported MAPPOMLP and MAPPO-GAT rows. This confirms the finding

Preprint submitted for review. Final version may differ from this manuscript.

Page 24 of 30

Safety-Contract Graph MARL

from Section 8.4: at the reported training horizon, the episodic diversity that CVaR needs to reweight has not yet accumulated. Graph structure improves worst-tail return, not every tail proxy. MAPPO-GAT achieves the best CVaR-10% among the reported unconstrained MAPPO encoder rows (−5,864, catastrophic rate 0.035), outperforming MAPPOMLP (−6,226, 0.015) and MAPPO-GNN (−6,108, 0.010) on worst-tail return. The catastrophic-rate proxy does not improve in the same ordering: MAPPO-GAT has a higher catastrophic rate than MAPPO-MLP and MAPPO-GNN. Therefore, the graph attention improves the worst-tail return metric in this benchmark, but it does not by itself remove catastrophic alert-threshold episodes. Constraints are the most reliable observed mechanism for reducing operational harm. C-MAPPO-GAT achieves catastrophic rate 0.000 and CVaR-10% = −9,694. The lower catastrophic rate reflects an important tradeoff: the Lagrangian policy accepts lower mean return in exchange for bounded operational cost, which removes alert-threshold catastrophic episodes in the C-MAPPO-GAT evaluations. The reported ACD3 -GAT result achieves catastrophic rate 0.003, but its CVaR-10% is −11,074, worse than C-MAPPOGAT. This boundary is informative, because composing graph encoding, CVaR reweighting, Lagrangian constraints, and shielding does not automatically dominate the cleaner constrained baseline. The 30-episode AskHuman and deterministic G-CRP component runs have lower catastrophic rate and lower downtime violation than the reported ACD3 result, but their horizon is too short to identify a stable component effect. Additional ACD3 replications and component ablations are therefore needed to separate whether the penalty comes from override behavior, G-CRP screening, CVaR weighting, or insufficient training horizon.

8.6. Robustness Stress Tests The robustness studies explores a narrower question than the main benchmark. Once a policy has been trained, does the safety contract survive controlled changes in topology seed or adaptive Red-process behaviour? They are interpreted as stress tests rather than as a replacement for the replicated ID benchmark in Table 2.

Topology-seed stress. On unseen topology seeds, the

safety contrast remains intact. Across 120 evaluation episodes, constrained MAPPO-GAT preserves 𝑃 (violDT ) = 0.000, mean downtime cost 14.9, and catastrophic rate 0.000. IPPO and MAPPO-GAT remain non-deployable under the same stress: both have 𝑃 (violDT ) = 1.000, with mean downtime costs 223.6 and 345.7, respectively. The topology shift does not harm MAPPO-GAT’s reward (𝑅̄ = −3,517 vs −3,548 on the matched ID evaluation). Thefore, the reward robustness alone is insufficient when the operational contract is exhausted in every episode. For constrained MAPPO-GAT, topology stress produces a stable safety profile (𝑅̄ = −7,286, CVaR10% = −9,433) while keeping the MTTR budget satisfied. ACD3 -GAT is analysed on the matched-layout benchmark and adaptive

Red-process stress test. The topology-OOD comparison is restricted to policies whose action heads are already layout-stable under the evaluated unseen topology seeds.

Coupled adaptive Red process. Figure 9 reports the

adaptive Red-process stress test. For each frozen Blue policy, a Red PPO learner is trained for 20 updates and its sampled actions are injected into the CybORG transition loop. The raw-return panel is useful for scale, while the degradation panel compares how much each Blue policy worsens relative to its first Red-process update. MAPPO-GAT has the largest worst degradation, since its Blue return drops by 1,147 points at the worst update and finishes 835 points below its starting value. The Constrained MAPPO-GAT is much more stable, with a worst degradation of 476 points and an end-of-run change of only −8 points. ACD3 -GAT starts from a lower raw-return level, but its worst degradation is 541 points and the curve recovers to finish 680 points above its first update. This supports a bounded robustness interpretation: explicit safety machinery is associated with lower worst policy-return degradation than reward-only MAPPO-GAT, while constrained MAPPO-GAT remains the most reliable safety-contract policy among the evaluated methods.

8.7. Synthesis Across the evaluated methods, reward-only learning is consistently non-deployable under the stated SOC contract: 𝑃 (violDT ) = 1.000 for every unconstrained learner, and the replicated IPPO/MAPPO rows use 314–355 downtime-cost units per episode against a budget of 50. Lagrangian safety machinery changes that operating regime. C-MAPPO-GAT reduces downtime cost to 15.5 and downtime-budget violation to 0.3%, while ACD3 -GAT reduces mean downtime cost to 48.2 with a 13.8% episode-violation rate. The difference is central to the interpretation, because the C-MAPPO-GAT configuration introduced here is the most reliable safetycontract policy observed in the benchmark, whereas ACD3 GAT is the broader architecture for integrating graph perception, safety contracts, tail-risk accounting, and counterfactual action screening. The experiments also explain why simple alternatives are insufficient. Naive reactive rules can perform worse than doing nothing (𝑅̄ ≈ −8,124 vs. −6,792 for Sleep) because they apply restoration without the valid-action, observation-processing, and mission-policy discipline that made engineered CAGE 4 heuristics effective. Rewardonly MARL fails from the opposite direction, because it learns high-impact security interventions without learning when those interventions exhaust operational budgets. Together, the results answer the central question of the paper: autonomous network-security policies cannot be evaluated on reward alone, and operational safety contracts must be represented and optimized explicitly. Future ACD3 GAT evaluations should extend the component ablations and make the factorised target head layout-stable for topology generalisation.

Preprint submitted for review. Final version may differ from this manuscript.

Page 25 of 30

Safety-Contract Graph MARL

(a) Blue return under coupled Red

Blue episode return (×103 )

0

(b) Degradation summary MAPPO-GAT

-1

Worst drop Final change

-1147 -835

-2 -3

-541

ACD3 -GAT

+680

-4 -5

-476

C-MAPPO-GAT

-8

-6 1

5

10

Red PPO update

15

20

1000

500

0

500

Return change from first Red update

Figure 9: Coupled adaptive Red-process stress test. Each curve evaluates a frozen Blue policy while a Red PPO learner adapts for 20 updates and sampled Red actions are injected into the CybORG transition loop. (a) Raw Blue-policy return under adaptation. (b) Worst and final change in Blue-policy return relative to the first Red-process update, which compares degradation despite different baseline return scales. The figure is a robustness stress test over one Blue-policy seed per method, not a multi-seed exploitability guarantee.

The empirical evidence extends beyond a single algorithm comparison. Together, the experiments characterise a difficult safety-contract MARL problem: non-learning policies expose the cost of naive response rules; IPPO and MAPPO variants show that reward-only learning discovers operationally harmful interventions; graph encoders test whether network structure improves representation and tail behaviour; constrained MAPPOGAT isolates the effect of explicit Lagrangian safety machinery; ACD3 -GAT integrates graph perception, budget context, CVaR tail-risk accounting, override signals, and counterfactual action screening; short AskHuman and deterministic G-CRP evaluations examine component behaviour; topology-seed and coupled adaptive Redprocess stress tests ask whether the safety contrast survives controlled robustness checks; and the two 300-episode replications test whether the principal ordering persists beyond the balanced 200-episode benchmark. This breadth matters because the contrasts are scientifically informative: graph attention improves the worst-tail return but not raw return dominance; CVaR reweighting alone does not improve the tail at the reported horizon; the integrated ACD3 policy reduces harm while the constrained safety machinery is the component that most consistently changes the operational outcome.

allowable MTTR budget per episode. In a real SOC, this would manifest as hundreds of unnecessary host reimages per shift, violation of SLA commitments, and systematic availability degradation of the very assets being protected. Constrained MAPPO-GAT reduces the mean downtime cost to 15.5 (< 𝐵down ) and the violation rate to 0.3% across three 200-episode seeds. ACD3 -GAT also brings mean downtime cost below budget (48.2), with a 13.8% episodeviolation rate, so the integrated architecture occupies a different point in the design space: it demonstrates the full safety-contract stack, whereas C-MAPPO-GAT is the new controlled constrained configuration that most reliably satisfies the SOC contract in the present benchmark. The two additional 300-episode replications strengthen rather than soften this interpretation, since constrained MAPPOGAT remains near-zero on downtime violations (0.7%), reward-only MAPPO-GAT remains at 100% violation, and ACD3 -GAT remains close to budget on average while still violating in 14.3% of episodes. C-MAPPO-GAT incurs a return penalty of approximately 3,050 points relative to the highest-return unconstrained learner, MAPPO-MLP (−6,992 vs. −3,937). ACD3 -GAT incurs approximately 4,200 points (−8,144 vs. −3,937), consistent with the broader set of safety signals active in the integrated safety-contract stack. We argue that this tradeoff is not only acceptable but necessary: a policy that achieves high security reward while violating operational budgets is not deployable, regardless of its average performance.

9.2. The Operational Safety Finding

9.3. Role of ACD3 -GAT

9. Discussion 9.1. Evidence Across the Experiments

The most important result is not which method achieves the highest return, but rather that every unconstrained method fails the safety contract on every episode. Mean downtime costs of 311–430 against a budget of 50 mean that unconstrained agents consume roughly 6–9× the

ACD3 -GAT functions as the integrated method architecture, while C-MAPPO-GAT is the strongest compliance row in the current benchmark. Its role is methodological, since it specifies a general safety-contract

Preprint submitted for review. Final version may differ from this manuscript.

Page 26 of 30

Safety-Contract Graph MARL

architecture with graph-structured perception over network entities, Lagrangian cost learning, explicit budget context, counterfactual action screening, tail-risk accounting, and override signals. C-MAPPO-GAT is the strongest safety-contract configuration because it isolates the new combination of MAPPO, a GAT encoder, and Lagrangian operational-cost control, satisfying the downtime contract most reliably in the reported evaluations. ACD3 -GAT is the extensible architecture: it unifies the reusable components needed when a response policy must reason over changing topology, opposing-process adaptation, uncertain action consequences, and human-governed operational budgets. The 13.8% downtime-violation rate therefore identifies the key stabilization target for the integrated policy while preserving the broader method contribution.

the contrast between reward-only MARL and explicit operational-cost control, not return-based ranking, broad topology generalisation, or factorial attribution of every ACD3 component. Three replicated seeds support stability of the safety contrast for the core methods, and rare violation rates are reported with raw episode counts rather than as formal chance-constraint certificates. Robustness results are controlled stress tests under the same cost accounting. Larger seed counts, hierarchical task decomposition, layout-stable topology evaluation for ACD3 -GAT, and head-to-head comparison with recent graph-RL and hierarchical MARL systems under a shared safety logger are complementary follow-on work rather than prerequisites for the central deployability finding.

9.4. Failure Mechanism of Unconstrained MARL

grounded in CybORG/CAGE-4, where actions, observations, CAGE Red-process behaviour, and mission scoring are controlled and reproducible. The cost proxies fp (𝑐𝑡down , 𝑐𝑡f w , 𝑐𝑡 ) map the simulator primitives onto SOCrelevant governance quantities: service recovery burden, firewall-change burden, and false-positive response burden. This makes the safety contract auditable inside the benchmark and gives practitioners a clear template for replacing the proxies with organisation- specific measurements in a live range, SOAR platform, or enterprise digital twin.

The mechanism is clear from the cost breakdown. Unconstrained MAPPO-MLP issues 316 downtime-cost units per episode (vs. budget 50); IPPO issues 314 and MAPPOGAT issues 355. The rule-based heuristic, despite being the lowest-returning baseline, incurs 117 downtime-cost units because it applies Restore selectively. Random exploration averages 424 downtime-cost units—worse than the constrained agent because it occasionally selects Restore by chance. This refines the intuition from the published CAGE 4 analyses: the competition heuristics succeeded because they encoded observation handling, invalid-action avoidance, mission-phase firewall discipline, and selective response policies. Our naive rule baseline lacks that structure and becomes operationally harmful. Unconstrained RL fails from the other side, because it finds Restore and BlockTraffic attractive because they genuinely reduce the CAGE Red process’s presence in the short term, but the Lagrangian penalties in our framework counterbalance that exploitable pattern with explicit operational budgets.

9.5. Graph Encoder Evidence In the benchmark, MAPPO-MLP at 200 episodes (−3,937) slightly outperforms MAPPO-GAT across 600 episodes (−3,979), while MAPPO-GNN remains available at 100 episodes (−4,387). This is consistent with slower convergence of graph encoders due to their larger parameter count and the need to learn structural attention weights. These data support a specific graph-encoder conclusion rather than a broad raw-return dominance conclusion. Critically, MAPPO-GAT achieves the best CVaR-10% among the reported unconstrained MAPPO encoder rows (−5,864), and MAPPO-GNN also improves the tail relative to MAPPO-MLP, suggesting that graph structure is presently a tail-risk and interpretability result rather than a mean-return result.

9.6. Scope of Evidence and Transfer Claim boundary. This study is intentionally scoped to safety-contract evaluation of autonomous network-security response in CAGE-4. The primary inferential target is

Simulator-grounded evidence. The study is intentionally

Robustness and attribution scope. The replicated core

comparison covers IPPO, MAPPO-GAT, and constrained MAPPO-GAT, and ACD3 -GAT at three 200-episode seeds. Two additional 300-episode replications cover MAPPOGAT, constrained MAPPO-GAT, and ACD3 -GAT, and are reported as a longer-horizon replication check rather than as a replacement for the balanced core table. The topology-seed stress test covers IPPO, MAPPO-GAT, and constrained MAPPO-GAT; the coupled adaptive Red-process stress test covers MAPPO-GAT, constrained MAPPO-GAT, and ACD3 -GAT. These choices define the evidence that a replicated safety-contract comparison, a longer-horizon check of the principal ordering, and controlled robustness probes. The 30-episode AskHuman and deterministic G-CRP component rows are therefore treated only as execution-path checks; their identical aggregate summaries do not identify which switch caused the observed behaviour. The next attribution layer is naturally factorial: C-MAPPOMLP, budget-context only, hard-shield only, Lagrangianonly, CVaR-only, opponent-context, and G-CRP-screening variants can be used to decompose the integrated ACD3 policy into its causal components. That decomposition is an extension of the present benchmark rather than a prerequisite for the core result that explicit safety machinery changes the operational regime.

Temporal and operational transfer. The 500-step,

𝛾=0.99 formulation provides a controlled horizon for comparing policies under identical mission dynamics.

Preprint submitted for review. Final version may differ from this manuscript.

Page 27 of 30

Safety-Contract Graph MARL

Longer incident campaigns, analyst workflows, ticketing delays, and organisation-specific change-control rules can be incorporated by changing the cost functions and episode horizon while preserving the same constrained Dec-POMDP and safety-contract machinery. ACD3 -TCGS is the first step in that direction, where rather than screening only the immediate proxy cost or a two-hop deterministic graph cascade, it learns from trajectory histories to predict future budget exhaustion before the action is executed. The current evidence is frozen-policy temporal-shield evidence, not an end-to-end retrained TCGS policy result. That boundary is important, but the result is still informative. It shows that the safety-labelled trajectories produced by the benchmark contain enough temporal structure to support predictive contract-risk screening.

Proxy cost signals. The false-positive Restore proxy (Re-

store with no active alerts) is defined from visible alert evidence because the Blue response agents act under partial observability. This choice matches the operational question faced by SOC automation: whether an action was justified by available evidence at decision time. In a deployment audit, the same definition can be paired with ground-truth incident labels when post-incident forensics are available.

and chance-constraint sensitivity. The Lagrangian update controls expected episode cost, not violation probability directly. The empirical violation rate is therefore a measured outcome rather than a formal chanceconstraint guarantee. Changing 𝐵down , 𝐵fw , or 𝐵f p would change both the feasible policy set and the shield activation pattern. The CAGE-4 budgets used here are therefore best read as reproducible stress-test thresholds; operational adoption would instantiate the same method with locally chosen service, change-management, and analyst-capacity budgets. Budget

Interpretation boundary. The safety contract is an eval-

uation and training discipline inside CAGE-4: it exposes when a policy exhausts MTTR, firewall-change, or falsepositive budgets and provides the decision record needed for audit. Graph attention is reported as topology-aware representation learning, and G-CRP is reported as the deterministic counterfactual risk screen used by the shield. The learned G-CRP model is included in the replication package as the next risk-estimation component in the same framework; the reported policy performance is tied to the deterministic screen unless explicitly stated otherwise.

Prior-work references. The published CC4 heuristic and

LLM scores are important context, but they are not direct safety-contract baselines. They use different reported outputs and do not expose the per-action Restore, Block/Allow, and false-positive traces required by our cost metrics. We therefore use them to calibrate the reward scale, while the main claims are restricted to methods evaluated with the same safety-labelled protocol.

9.7. Implications for Autonomous SOC Deployment Read as an expert-system result, the framework provides a deployable decision-support pattern rather than a claim of autonomous production readiness, because we encode SOC governance as budgets, train and evaluate policies against those budgets, log every executed action and cost proxy, and route budget exhaustion or low-confidence states to human governance. Our results suggest three practical principles for deploying such agents: 1. Define operational budgets before training. Unconstrained training consistently produces operationally harmful policies. Budget values (𝐵𝑘 ) should be set by SOC governance and treated as explicit violation constraints with audit metrics, not as hidden soft reward preferences. The CAGE4 values used here are stress-test thresholds; a SOC deployment should tune them from servicelevel objectives, change-management policy, and analyst capacity, then rerun the same safety-labelled evaluation. 2. Validate robustness after satisfying the safety contract. Fixed scripted Red processes can overestimate policy robustness, so adaptive opposing-policy and topology-shift studies should follow the ID safety benchmark before any deployment claim. In the reported stress tests, constrained MAPPO-GAT preserves zero downtime violations under topology-seed shift and degrades less than reward-only MAPPOGAT under the coupled adaptive Red-process stress test. 3. Support human override. The ACD3 -GAT override mechanism (triggered by low confidence, budget exhaustion, or OOD states) provides a governance layer that maintains human authority over high-stakes decisions. AskHuman variants measure the cost of that governance layer and connect the learned policy to a SOC approval workflow.

10. Conclusion Autonomous network security agents trained without explicit operational constraints systematically violate SOC budgets in the evaluated CAGE 4 setting, creating downtime costs roughly 6–9× the allowable MTTR budget. This is not merely a reward-engineering failure; it reflects a structural mismatch between scalar security reward and operational deployability. ACD3 -GAT addresses this through a unified safetycontract design: Lagrangian-constrained multi-agent PPO that optimizes under MTTR, false-positive, and firewall-policy budgets; graph encoders for structured host–subnet observations; a factorised type–target action policy; a multi-objective training objective for reward, operational cost, and tail-risk accounting; and a budgetaware counterfactual shield. On CAGE Challenge 4, the C-MAPPO-GAT constrained configuration introduced in

Preprint submitted for review. Final version may differ from this manuscript.

Page 28 of 30

Safety-Contract Graph MARL

this work achieves a 95–96% reduction in operational harm while remaining within budget 99.7% of episodes—a qualitative shift from policy that is operationally harmful to one that is operationally auditable. Two additional 300-episode replications preserve the same ordering: constrained MAPPO-GAT remains near-zero on downtime violations, while reward-only MAPPO-GAT continues to violate the downtime budget in every episode. ACD3 -GAT reduces mean downtime harm but still violates the budget in 13.8% of episodes. This identifies a clear division between contribution types: C-MAPPO-GAT is the strongest compliance configuration in the reported benchmark and demonstrates that the MAPPO/GAT/Lagrangian composition is a powerful safety-contract baseline, while ACD3 -GAT is the general architecture for combining graph perception, safety contracts, tail-risk accounting, opponent context, and counterfactual action screening. Topology-seed and coupled adaptive Red-process stress tests reinforce the same conclusion, which is that explicit safety machinery, not reward alone, determines whether an autonomous response policy remains operationally auditable under stress. The core finding has direct implications for SOC governance: autonomous network-security agents must carry explicit operational constraints before they can be considered for deployment. More broadly, ACD3 -GAT treats network security response as one instance of a wider class of safetycontract MARL problems. For instance, multiple agents act over structured entities, local actions have operational side effects, and deployment requires respecting budgets that are not reducible to reward. The anonymised replication package provides safety-labelled trajectories, trained checkpoints, evaluation metadata, and robustness protocols so that the reported claims can be independently regenerated and extended. The ACD3 -TCGS diagnostic shows how those trajectories can also train a temporal contract-risk shield that predicts future budget exhaustion before executing a proposed action. The natural next research agenda is to instantiate the same safety-contract framework at larger scale, use layout-stable target heads for broader topology transfer, evaluate learned Red-process variants in the main robustness loop, integrate the learned G-CRP risk model into policy evaluation, and pair the safety contract with temporal graph memory or asynchronous cyber-range simulators when moving from CAGE-style episodes to richer SOC telemetry (Rossi et al., 2020). Continuous-time cyber-range evaluation provides the complementary simulator direction (Jankowski, 2026).

CRediT Author Contribution Statement Jose Luis Silva: The author was responsible for the conceptualization and methodology of the study, developed the software, conducted the formal analysis and investigation, curated the data, prepared the original draft, reviewed and edited the manuscript, and produced the visualizations.

Declaration of Competing Interest The authors declare no competing financial or personal interests.

Data Availability Software, safety-labelled trajectories, trained model checkpoints, and robustness-evaluation metadata are available from the corresponding author upon request.

References Achiam, J., Held, D., Tamar, A., Abbeel, P., 2017. Constrained policy optimization, in: Proceedings of the 34th International Conference on Machine Learning, pp. 22–31. Altman, E., 1999. Constrained Markov decision processes. CRC Press. Bansal, T., Pachocki, J., Sidor, S., Sutskever, I., Mordatch, I., 2018. Emergent complexity via multi-agent competition, in: International Conference on Learning Representations. Castro, S.R., Campbell, R., Lau, N., Villalobos, O., Duan, J., Cardenas, A.A., 2025. Large language models are autonomous cyber defenders. arXiv preprint arXiv:2505.04843 . Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation, in: Conference on Empirical Methods in Natural Language Processing, pp. 1724–1734. Gilmer, J., Schütt, K.T., Thomas, S., Brendan Faber, F., Lilienfeld, O.A., Schindler, C.R., 2017. Neural message passing for quantum chemistry, in: Proceedings of the 34th International Conference on Machine Learning, pp. 1263–1272. Haghshenas, S.H., Hasnat, M.A., Naeini, M., 2022. A temporal graph neural network for cyber attack detection and localization in smart grids. arXiv preprint arXiv:2212.03390 . Hamilton, W.L., Ying, R., Leskovec, J., 2017. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems 30. Jaderberg, M., Czarnecki, W.M., Dunning, I., Marris, L., Lever, G., Castañeda, A.G., Beattie, C., Rabinowitz, N.C., Morcos, A.S., Ruderman, A., Sonnerat, N., Green, T., Deason, L., Leibo, J.Z., Silver, D., Hassabis, D., Kavukcuoglu, K., Graepel, T., 2019. Human-level performance in 3d multiplayer games with population-based reinforcement learning. Science 364, 859–865. Jankowski, I., 2026. Event-driven temporal graph networks for asynchronous multi-agent cyber defense in NetForge_RL. arXiv preprint arXiv:2604.09523 . Kiely, M., Ahiskali, M., Borde, E., et al., 2025a. Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A CAGE Challenge 4 perspective, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 28907–28913. URL: https://ojs. aaai.org/index.php/AAAI/article/view/35158. Kiely, M., Bowman, D., Ahiskali, M., et al., 2025b. CAGE Challenge 4: A scalable multi-agent reinforcement learning gym for autonomous cyber defence. AI Magazine 46, e70021. URL: https://doi.org/10.1002/aaai. 70021, doi:10.1002/aaai.70021. King, I.J., Bowman, B., Huang, H.H., 2025. Automated cyber defense with generalizable graph-based reinforcement learning agents. arXiv preprint arXiv:2509.16151 . Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., Mordatch, I., 2017. Multiagent actor-critic for mixed cooperative-competitive environments, in: Advances in Neural Information Processing Systems. Mnih, V., Badia, A.P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., Kavukcuoglu, K., 2016. Asynchronous methods for deep reinforcement learning, in: Proceedings of the 33rd International Conference on Machine Learning, pp. 1928–1937. Oliehoek, F.A., Amato, C., 2016. A concise introduction to decentralized POMDPs. SpringerBriefs in Intelligent Systems, Springer.

Preprint submitted for review. Final version may differ from this manuscript.

Page 29 of 30

Safety-Contract Graph MARL Ray, A., Achiam, J., Amodei, D., 2019. Benchmarking safe exploration in deep reinforcement learning, in: arXiv preprint arXiv:1910.01708. Rockafellar, R.T., Uryasev, S., 2000. Optimization of conditional value-atrisk. Journal of Risk 2, 21–41. Rossi, E., Chamberlain, B., Frasca, F., Eynard, D., Monti, F., Bronstein, M., 2020. Temporal graph networks for deep learning on dynamic graphs. arXiv preprint arXiv:2006.10637 . Sandoval, I.O., Symes Thompson, I., Mavroudis, V., Hicks, C., 2025. An attentive graph agent for topology-adaptive cyber defence. arXiv preprint arXiv:2501.14700 . Schulman, J., Moritz, P., Levine, S., Jordan, M., Abbeel, P., 2016. Highdimensional continuous control using generalized advantage estimation, in: 4th International Conference on Learning Representations. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O., 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 . Singh, A.V., Rathbun, E., Graham, E., Oakley, L., Boboila, S., Chin, P., Oprea, A., 2024. Hierarchical multi-agent reinforcement learning for cyber network defense. arXiv preprint arXiv:2410.17351 . Standen, M., Lucas, M., Bowman, D., Richer, T.J., Kim, J., Marriott, D., 2021. CybORG: A gym for the development of autonomous cyber agents. arXiv preprint arXiv:2108.09118 . Sutton, R.S., Barto, A.G., 2018. Reinforcement learning: An introduction. 2 ed., MIT Press. TTCP CAGE Working Group, 2023. CAGE Challenge 4. https://github. com/cage-challenge/cage-challenge-4. Van Langendonck, L., Castell-Uroz, I., Barlet-Ros, P., 2024. PPT-GNN: A practical pre-trained spatio-temporal graph neural network for network security. arXiv preprint arXiv:2406.13365 . Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y., 2018. Graph attention networks, in: 6th International Conference on Learning Representations. Vyas, S., Hannay, J.E., Bolton, A., Burnap, P., 2023. Autonomous cyber defence: A survey, in: arXiv preprint arXiv:2303.04926. de Witt, C.S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P.H.S., Sun, M., Whiteson, S., 2020. Is independent learning all you need in the StarCraft multi-agent challenge?, in: arXiv preprint arXiv:2011.09533. Wolk, M., Applebaum, A., Dennler, C., Dwyer, P., Moskowitz, M., Nguyen, H., Nichols, N., Park, N., Rachwalski, P., Rau, F., Webster, A., 2022. Beyond CAGE: Investigating generalization of learned autonomous network defense policies. arXiv preprint arXiv:2211.15557 . Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., Wu, Y., 2022. The surprising effectiveness of PPO in cooperative multi-agent games. Advances in Neural Information Processing Systems 35.

Preprint submitted for review. Final version may differ from this manuscript.

Page 30 of 30