arXiv CSopen access

COHORT: Collaborative Orchestration for Hardening via Offensive Replay on Emulated Topologies

COHORT: Collaborative Orchestration for Hardening via Offensive Replay on Emulated Topologies Chen Frydmana , Aviram Zilbermanb,a,∗ , Rubin Kriefa , Abed Showgana , Andres Murilloc , Sekiya Motoyoshic , Asaf Shabtaia , Yuval Elovicia and Rami Puzisa a Ben-Gurion University of The Negev, Beer Sheva, Israel b Jerusalem College of Technology, Jerusalem, Israel

arXiv:2606.30479v1 [cs.NI] 29 Jun 2026

c Fujitsu

ARTICLE INFO

ABSTRACT

Keywords: Network Emulation Automatic Network Mitigation Adversary Emulation Cumulative Mitigations Proactive Defense Multi Agent System

Mitigating an observed adversary in an enterprise network typically takes weeks of expert work: an analyst derives a mitigation tailored to that adversary, validates it without breaking production, and verifies it disrupts the specific attack. The procedure relies on expert judgment and cannot safely be exercised against the production network. COHORT is the first end-to-end framework to automate this procedure for deployable mitigations. A role-decomposed multi-agent LLM workflow proposes candidates, implements them as real device commands, and refines them through a critique loop, all on a high-fidelity GNS3 emulator running real vendor firmware (firewall, switch, router). Each candidate is evaluated by offensive replay: re-executing the original adversary on the mitigated network for a paired comparison against the unmitigated baseline, rather than the reward-signal or expert-judgment proxies used in prior simulation, hybrid, and configuration-generation work. Two further checks complement replay: a connectivity-regression check (LAN ping and internet HTTP probe) rejects mitigations that disrupt legitimate LAN or internet connectivity, and a cumulative evaluation stacks approved mitigations onto a persistent state to surface compound effects. Across three topologies and four attack scenarios (ransomware, lateral movement, DNS exfiltration, data theft), 46.7% of generated mitigations both disrupt the attack and preserve connectivity under replay, 4.4× the rate of a singleagent baseline using the same model and tool access. A demo video walking through the framework is available with our released artifacts (Appendix 8.3).

1. Introduction Given an adversary profile reconstructed from forensic analysis, threat-intelligence feeds, or red-team exercises, the defender must derive a deployable mitigation tailored to that adversary, validate it without breaking production while verifying that it disrupts the specific attack. Two properties of this workflow drive its cost: it relies on expert knowledge, and it cannot safely be exercised on the production network, since experimentally adjusting firewall rules, routing policy, or host configuration risks breaking legitimate operation. Existing reactive tooling (IDS/IPS, SIEM, SOAR) executes curated or playbook-driven responses for known attack patterns; even AI-assisted SOAR operates within a maintained playbook set rather than synthesizing new controls Lyu, Gharakheili and Sivaraman (2024); Islam, Babar and Nepal (2019). Threat intelligence informs analysts on new attacks but does not generate per-incident defenses. Breach-andattack simulation (BAS) replays curated adversary techniques against the unmodified network to validate detection coverage rather than to test newly proposed mitigations. The IBM Cost of a Data Breach report measures mean breach containment times of multiple weeks IBM Security (2025); ∗ Corresponding author

ORCID (s): 0009-0006-1554-992X (C. Frydman); 0000-0002-8137-7314 (A. Zilberman); 0009-0000-9507-4368 (R. Krief); 0009-0006-6997-5043 (A. Showgan); 0000-0001-6965-2283 (A. Murillo); 0009-0008-6542-2681 (S. Motoyoshi); 0000-0003-0630-4059 (A. Shabtai); 0000-0002-9641-128X (Y. Elovici); 0000-0002-7229-3899 (R. Puzis)

Frydman et al.: Preprint submitted to Elsevier

per-incident mitigation derivation is a cost driver inside that window. We present COHORT, the first end-to-end framework that autonomously generates and validates deployable mitigations. To remove the expert-knowledge bottleneck, a multi-agent LLM workflow proposes, implements, and reviews each candidate mitigation as real device configuration commands. To avoid production exposure during evaluation, the workflow operates on a high-fidelity emulated enterprise network running real vendor firmware (firewall, switch, router) in GNS3, with Caldera Applebaum, Miller, Strom, Korber and Wolf (2016); Chang, Cao, Teoh, Ear and Xu (2025) executing ATT&CK-aligned adversary actions. Validation rests on offensive replay: from an established foothold, the same Caldera adversary is reexecuted against the network with the candidate mitigation in place, and a judge compares the per-step attack outcome to the unmitigated baseline. Three further mechanisms shape generation and evaluation: an iterative implementation– critique loop catches configuration-text errors before replay; a connectivity-regression check (a LAN ping and an internet HTTP probe) rejects mitigations that disrupt legitimate LAN or internet connectivity; a cumulative evaluation stacks approved mitigations on a persistent network state to surface compound effects. We target post-compromise enterprise networks against scripted, intelligence-equipped adversaries; adaptive adversaries are out of scope by construction (Section 3).

Page 1 of 17

Prior work covers parts of this workflow but not all of a high-fidelity emulator with real device commands rather it: simulation and hybrid platforms (CybORG/CAGE Kiely, than the production network. Ahiskali, Borde, Bowman, Bowman, Van Bruggen, Cowan, • Multi-agent decomposition. Lift from 10.7% to 46.7% Dasgupta, Devendorf, Edwards et al. (2025), CyberBatMSR over a matched single-agent baseline (same model, tleSim Team. (2021), CSLE Hammar and Stadler (2022)) inputs, tool access). An ablation decomposes the lift: cannot test real configurations, configuration-generation sysrole specialization accounts for roughly three-quarters, tems (Stackelberg planning Speicher, Steinmetz, Hoffmann, the iterative critique loop for the remaining quarter. Backes and Künnemann (2019), NetConfEval Wang et al. • Cumulative evaluation. A non-regression acceptance (2024a)) never confront an active adversary, emulation rule accumulates approved mitigations on a persistent netplatforms (CyGIL Li, Fayad and Taylor (2021)) target work state, exposing compound effects that per-mitigation attacker training rather than defender mitigation, concurrent evaluation cannot surface. LLM-based defense agents Saroui, Marchioro and Olivereau (2025) skip replay-based re-validation, and commercial BAS 2. Background and Related Work replays curated rather than generated mitigations. None We organize prior work along the six dimensions COcovers the full workflow on a high-fidelity emulator using HORT integrates: enterprise defense, automated configureal device configuration commands (Section 2). ration, multi-agent LLM reasoning, autonomous defense The mitigation workflow is decomposed into role-specialized agents, the environments used to evaluate them, and comLLM agents. A suggester proposes deployable defenses mercial tooling. Each subsection ends by identifying the (firewall ACLs, routing-policy changes, host-hardening comcomponent of COHORT’s workflow that the corresponding mands). An implementer translates a selected mitigation into line of work omits. device commands within a bounded per-cycle command budget. A critic reviews each implementation before replay; 2.1. Enterprise Network Defense the implementation–critique loop runs to a fixed iteration Modern enterprise defenses span firewalls, IDS/IPS, limit before the mitigation is accepted or discarded. Two SIEM, SOAR, and EDR Lyu et al. (2024) under a fundafurther agents support evaluation: a judge compares premental security–availability tradeoff: a control that disrupts and post-mitigation outcomes via Caldera replay, and a an attack but breaks legitimate services is not deployable. summarizer produces a human-readable artifact. The rolePre-deployment validation reasons over abstract infrastrucspecialized design is contrasted against two ablations using ture models, while security orchestration and AI-driven the same model and tool access (Section 4.6, Section 5.1): incident response automate the reactive side with curated a single-agent baseline that consolidates these roles into one playbooks Islam et al. (2019); Hamadanian, Arzani, Fouladi, conversation, and a multi-agent variant with the iterative Kakarla, Fonseca, Billor, Cheema, Nkposong and Chandra critique loop disabled. (2023); neither synthesizes a new attack-specific mitigation, We evaluate COHORT across small, medium, and large nor validates one by re-executing the observed adversary enterprise topologies emulated in GNS3, four attack sceagainst the modified network. narios (Section 3), and 1782 mitigation attempts, all driven by GPT-5.4 mini. The multi-agent framework reaches an overall mitigation success rate (MSR) of 46.7%, where MSR is the rate of mitigations that both reduce attack progress (ME > 0) and pass both connectivity probes (Section 5.2). This is 4.4× the single-agent baseline of 10.7%, with the gap attributable to role specialization and the iterative critique loop (Section 5.3). Stacking approved mitigations on a persistent network state plateaus at 52% cumulative attackstep reduction by the fourth defense (Section 5.3). We do not claim transfer to adaptive adversaries, Windows endpoints, model families beyond GPT-5.4 mini, or vendor families beyond FortiGate, Cisco IOS, and Open vSwitch (Section 3.4); calibration against expert-derived mitigations is open (Section 6.5). All evaluation runs in an isolated emulator; production deployment of validated mitigations is a separate human-driven step that falls outside the framework. Contributions. • End-to-end framework that addresses both bottlenecks. The first framework that autonomously generates, deploys, and validates deployable mitigations against the specific observed adversary, removing the expertknowledge dependency and confining experimentation to

2.2. Automated Configuration and Network Management Automating network configuration is challenging due to ambiguity in intent, configuration conflicts, and unintended side effects Bringhenti, Marchetto, Sisto and Valenza (2023). Recent LLM-based tools translate natural-language intents into device configurations: NetConfEval Wang et al. (2024a) benchmarks intent-to-config translation; GeNet Ifland, Krief, Zilberman, Duani, Ohana, Murillo, Manor, Lavi, Hikichi, Shabtai, Elovici and Puzis (2025) assists engineers with topology and configuration updates; LLM-NetCFG Lira, Caicedo and da Fonseca (2024) targets zero-touch service management; and recent multi-vendor intent-driven generators Wang, He, Zhao, Xuan, Sun, Qi, Liang, Zhuang and Liao (2026) extend the paradigm. Recent surveys and validation tests across these systems assess intent fulfillment, syntactic correctness, and policy compliance; however, they do not evaluate how the generated configuration affects a re-executed adversary Hong, Tu and Hong (2025).

2

training Rigaki, Lukáš, Catania and Garcia (2023). Concurrent work begins to standardize evaluation for intrusionresponse systems on public NIDS datasets Marchioro, Saroui and Olivereau (2025), but defensive agents otherwise operate in abstract simulators or execute reactive playbooks, and adversary emulation platforms execute attacks without proposing or validating mitigations (Table 2).

2.3. Large Language Models and Multi-Agent Reasoning in Cybersecurity large language models (LLMs) enable contextual reasoning over heterogeneous security data Zhang, Bu, Wen, Liu, Fei, Xi, Li, Yang, Zhu and Meng (2025) and have been applied to alert triage, threat explanation, and candidate remediation Wang, Xie, Zhang, Wang, Zhang and Cui (2024b); Kaheh, Kholgh and Kostakos (2023), but mostly in humanin-the-loop workflows and remain susceptible to hallucinations that fabricate threats or miss real ones, motivating explicit validation Sood, Zeadally and Hong (2025). Multiagent architectures decompose complex tasks into specialized roles Li, Hammoud, Itani, Khizbullin and Ghanem (2023); Hong, Zhuge, Chen, Zheng, Cheng, Wang, Zhang, Wang, Yau, Lin et al. (2023); An, Yang, Lu, Li, Ren, Huang, Wang, Zhao, Kang, Ding et al. (2024); Luo, Sun, Liu, Niyato, Yu, Atiquzzaman and Dustdar (2026), with iterative refinement and coordinated reasoning across agents Du, Li, Torralba, Tenenbaum and Mordatch (2024). Prior work focused on analysis and recommendation, scoring defenses through RL reward signals, expert judgment, or averages over independent attack runs. Isolating a particular mitigation’s effect on the targeted adversary requires re-executing the same adversary against the just-mitigated network, a step that these methods omit.

2.5. Simulated and Emulated Network Environments for Adversarial Emulation Safe evaluation of autonomous cyber defense strategies requires isolated environments that can faithfully reproduce adversarial behavior without risk to production systems. The literature spans three approaches with a fidelity–speed tradeoff. Network simulation uses abstract graph-based or Markov-decision-process representations Yamin, Katt and Gkioulos (2020); attack and defense execute as programmatic state transitions, so simulators cannot surface the side effects of real configuration changes. Network emulation instantiates real operating systems, protocols, and devices in virtualized hardware, with backends ranging from containers and VMs to full hardware virtualization booting vendor firmware images (e.g., GNS3 Neumann (2015)); only the latter can evaluate deployable mitigations as they would behave on production hardware, building on the digital-twin paradigm of safe security testing on virtual replicas Eckhart and Ekelhart (2018). Hybrid platforms couple a fast simulation surrogate with an emulated backend Molina-Markham, Miniter, Powell and Ridley (2021); Janisch, Pevnỳ and Lisỳ (2023); Oesch, Chaulagain, Weber, Dixson, Sadovnik, Roberson, Watson and Austria (2024). Beyond fidelity, we evaluate environments along four validation properties. The first two of these properties are operationalized in Section 4: mitigation validation (replaybased attack disruption) and connectivity regression testing (verifying that legitimate LAN and internet reachability is preserved after a mitigation is deployed). Deployable mitigations require that defenses be expressed as real device configuration commands rather than abstract state transitions, so that what is validated is what would deploy. Cumulative mitigations compose multiple independently proposed defenses on a persistent network state, capturing synergistic or interfering effects that per-mitigation evaluation misses. Across these environments, none provides all four validation properties: simulators cannot evaluate deployable mitigations, and even emulators that support offensive replay Kouremetis et al. (2024); Hammar and Stadler (2022) do not pair it with connectivity regression testing or cumulative mitigation evaluation. Our framework addresses all four gaps by integrating Caldera adversary emulation with a GNS3-based emulated enterprise network that boots real vendor device images, enabling attack replay after mitigation deployment, automatic connectivity validation, and cumulative evaluation of stacked defenses (Table 1).

2.4. Autonomous Cyber Defense Autonomous defensive agents vary in their degree of autonomy and adaptability Stakhanova, Basu and Wong (2007): scripted agents execute fixed, predetermined action sequences with no runtime reasoning; planning-based agents dynamically construct action sequences by reasoning over a maintained model of current network state and goals; reinforcement learning (RL) agents learn policies through environmental interaction but remain opaque and difficult to transfer across environments Nguyen and Reddi (2021); and large language model (LLM)-based agents leverage pre-trained world knowledge and natural language reasoning to generate, critique, and explain actions, addressing the explainability and transferability limitations of RLonly approaches Castro, Campbell, Lau, Villalobos, Duan and Cardenas (2025). RL-based agents learn mitigation policies in simulated enterprise environments Wang and Dechene (2024), while LLM-based defenders (e.g., AgentNIRS Saroui et al. (2025), which generates firewall rules from NIDS alerts) and hybrid RL+LLM systems Loevenich, Adler, Huerten and Lopes (2025) drive containment in simulators, SOAR pipelines, and cyber ranges (Table 2). On the offensive side, Caldera has been used to evaluate cyber deception against AI-based attackers Kouremetis, Lawrence, Alford, Cheuvront, Davila, Geyer, Haigh, Michalak, Murphy and Russo (2024), and pre-trained LLMs themselves have been deployed as autonomous attackers in cybersecurity RL environments such as NetSecGame, matching or exceeding purpose-trained RL baselines without per-environment

3

Table 1 Autonomous cyber defense environments compared on fidelity, attack/defense features, and the validation properties required to evaluate deployable mitigations. ✓ = supported; × = not supported; ~ = partial or unclear. Attack / Defence Features

Validation Properties

Mitigation Connectivity Cumulative Open Custom. Vendor Custom. topo. dev. imgs attacks Deployable validation regression mit. source

Platform

Network type

Emulation-based MIRAGE (2024) CyGIL† (2021)

Emu., Sim.1 Emulation

~ ~

× ×

✓ ✓

× ×

✓ ×

× ×

× ×

× ×

Hybrid (simulation + emulation) Cyberwheel (2024) FARLAND (2021) CSLE (2022) NASimEmu† (2023)

Hybrid Sim., Hybrid Hybrid Hybrid

✓ ✓ ✓ ✓

~ × × ~

✓ ~ ~ ~

× ~ × ×

~ ✓ ✓ ×

× × × ×

× × × ×

✓ × ✓ ✓

Simulation-based CybORG / CAGE (2021; 2025; 2024) CybORG++ (2024) PrimAITE (2024) CyberBattleSim (2021)

Simulation3 Simulation Simulation Simulation

~ ~ ✓ ✓

× × × ×

~ ~ ✓ ~

× × × ×

~ ~ ~ ~

× × × ×

× × × ×

✓ ✓ ✓ ✓

COHORT (this work)

Emulation

✓

✓

✓

✓

✓

✓

✓

~2

Deployable = defenses expressed as real device configuration commands rather than abstract state transitions. † Attack-only platform; no defender side. 1 https://www.mitre.org/our-impact/intellectual-property/cyberlayer 2 See Section 8.3 for artifact and code release status.

3 CybORG also supports an AWS-based emulation backend; CAGE Challenges, the standard benchmarks built on it, run only the simulation

mode.

We use Caldera as the replay platform; equivalent BAS or adversary-emulation platforms emitting a comparable operational report can substitute. Adaptive adversaries that re-plan in response to observed defenses are excluded (Section 3.4); fixing the adversary across baseline and postmitigation runs is what makes the comparison attributable to the deployed mitigation rather than to attacker variation.

2.6. Commercial Adversary Emulation and Continuous Validation Industry tooling spans three overlapping categories: Breach-and-Attack Simulation (AttackIQ AttackIQ, Cymulate Cymulate, Picus Security Picus, SafeBreach SafeBreach, and SCYTHE SCYTHE); continuous attack-path and exposure management (XM Cyber XMCyber and Pentera Pentera); and high-fidelity cyber ranges (SimSpace SimSpace and Cyberbit Cyberbit). These platforms replay curated adversary playbooks against live or emulated infrastructure to surface control gaps; some now integrate AI assistants for triage and remediation orchestration SafeBreach; Pentera, but the autonomous generation of novel configuration mitigations and their replay-based re-validation remain outside their scope.

3.1. Adversary Model Knowledge The adversary has a foothold on an endpoint

and is seeded with targeted intelligence: valid credentials for one LAN host, the IP of an exfiltration target, and sensitive file paths and extensions. All other facts (additional hosts, services, credentials) must be acquired through on-host reconnaissance and lateral movement during the attack. The adversary has no visibility into the mitigation framework.

Capabilities The adversary operates with normal user

3. Threat Model

privileges and executes MITRE ATT&CK techniques using tools staged on the foothold host. The operator control channel is delivered out of band from the production data path and lies outside the defender’s mitigation surface, so agent connectivity is never a defensive target; by contrast, exfiltration traffic traverses the production path and is in scope. Mitigations are credited for blocking exfiltration on the production path, not for severing the control channel, thereby preventing trivial wins and forcing engagement with the actual data flow. Implementation specifics (Caldera Sandcat agent, GNS3 interface naming) are deferred to Caldera Linux Agent.

We consider the security of enterprise networks in the post-compromise phase. The defender’s goal is to harden the network so that an observed adversary, when re-executed against the modified topology, makes less progress without breaking legitimate operation. We evaluate four attack scenarios (Table 6): Thief (data theft), Ransomware, DNS Exfiltration, and Lateral Movement.

Threat profile COHORT targets scripted, intelligence-

equipped adversaries that have established a foothold and acquired targeted reconnaissance, replayable from forensic analysis, threat-intelligence feeds, or red-team exercises. 4

Table 2 Comparison of autonomous cyber defense (ACD) systems and LLM-based network configuration systems. ✓ = supported; × = not supported; ~ = partial or limited. System

Agent

Environment

Suggest

Deploy

Config-level

Open source

LLM-based defensive systems AgentNIRS (2025) LLM CyberAlly (2025) LLM + KG Castro et al. (2025) LLM ShieldGPT (2024b) LLM

Dataset Live cyber range Simulation Advisory only

✓ ✓ ✓ ✓

✓ × ~ ×

✓ × × ×

✓ × ✓ ✓

Hybrid RL + LLM systems SecurityBot† (2024) RL + LLM Loevenich et al. (2025) DRL + LLM

Simulation Simulation

~ ~

~ ~

× ×

× ×

LLM-based network configuration systems (no adversary in the loop) NetConfEval (2024a) LLM Benchmark GeNet (2025) LLM (multimodal) Dataset LLM-NetCFG (2024) LLM (local) Use-case demo ConfGen (2026) LLM + RAG Dataset

✓ ✓ ✓ ✓

~ × ✓ ×

✓ ✓ ✓ ✓

✓ ✓ × ×

Emulation-based defensive systems MIRAGE (2024) Rule-based

Emulation

~

✓

×

×

COHORT (this work)

Emulation

✓

✓

✓

~1

LLM (multi-agent)

Environment refers to where mitigations are deployed/validated, not training. Config-level = mitigation expressed as device configuration commands rather than abstract state transitions. † SecurityBot operates as both attacker and defender; all other rows are defenders. 1 See Section 8.3 for artifact and code release status.

2. legitimate connectivity is preserved, verified by two independent probes from the compromised host (a LAN ping to an internal target and an HTTP request to an internet endpoint), both of which must succeed. Equivalently, letting Φ ∶  → {0, 1} denote the connectivity gate, Φ( ) = 𝟏[LAN ping passes ∧ HTTP probe passes], the success indicator is 𝟏[ME(𝑚; 𝑡 , ) > 0 ∧ Φ(𝑡+1 ) = 1], with 𝑡 , 𝑡+1 , and  as introduced in Section 4.2. We report this binary outcome as the Mitigation Success Rate (MSR), the continuous reduction as the Mitigation Effectiveness (ME), and the compounding effect across stacked mitigations as the Cumulative ME (Section 5.2). The criterion is step-level by design. A step-level signal exposes which parts of the kill chain a mitigation disrupts, the granularity needed to drive the iterative critic loop; objective denial follows when the disrupted step lies on the only viable path to the objective (for example, blocking the final T1041 exfiltration step prevents data from leaving the network). Step-level disruption that contributes little to the aggregate effect remains visible through ME and Cumulative ME.

3.2. Defender Model The defender corresponds to a SOC operator with administrative authority over their own infrastructure. The framework consumes a structured operational report from executing a known adversary profile (threat-intelligence feeds, the Caldera Stockpile, or red-team exercises) against an emulation of the production network; forensic-grade reconstruction of a real-world breach is not assumed.

Emulation prerequisite A high-fidelity emulation of the

production network is presupposed as input; automated production-to-emulation generation is an open problem (Section 6.5). The prerequisite is met by organizations that already maintain a staging environment or pre-production lab.

Failure modes Perfect execution is not assumed. The agent

pipeline can produce invalid configurations, disrupt legitimate connectivity, or generate operationally ineffective controls. The connectivity-preservation gate (Success Criterion, below) excludes mitigations that break connectivity. Connectivity-preserving but ineffective mitigations (rules at the wrong hop, evaluated in the wrong order, or targeting primitives the adversary does not use) surface as ME ≤ 0 in the replay; Section 5.3 characterizes the recurring patterns.

3.4. Scope and Generalization The reported numbers generalize to scripted adversaries with observable TTPs whose execution profile can be replayed deterministically. They do not generalize to: • Adaptive adversaries that observe deployed defenses and re-plan. Replay-based validation is non-adaptive by construction (Section 6.1); evaluation against adaptive

3.3. Success Criterion A mitigation succeeds on a run if both of the following hold: 1. it strictly reduces the Attack Step Success Rate (ASSR) compared to the unmitigated baseline (ME > 0); and

5

attackers requires a co-evolving measurement contract, left open (Section 6.5). • TTPs absent from MITRE ATT&CK or whose execution Caldera cannot emulate. • Heterogeneous endpoint OSes. The current evaluation is Linux-only (Section 5.1); whether MSR transfers to Windows or ICS environments is open. • Network-device heterogeneity. The evaluation uses FortiGate, Cisco IOS, and Open vSwitch images (Firewall, Router, Switch); transfer to other vendor families (e.g., Palo Alto, Juniper, Aruba) is open. • Operational regression beyond connectivity probes, including latency, application-layer disruption, and uservisible behavior beyond LAN/internet reachability.

expands into commands; (3) refine: the implementer/critic alternation produces a sequence 𝑚(0) , 𝑚(1) , … , 𝑚(𝑗) with 𝑗 ≤ 𝐾 and |𝑚(𝑖) | ≤ 𝐵 for every 𝑖, halting on critic approval or at 𝑗 = 𝐾, with 𝑚 ∶= 𝑚(𝑗) . The post-mitigation state is 𝑡+1 = 𝜏(𝑡 , 𝑚). The cycle’s binary outcome is the Success Criterion of Section 3 evaluated on (𝑡 , 𝑡+1 , ).

Constraints (i) Per refinement iteration, |𝑚(𝑖) | ≤ 𝐵; the

cycle therefore issues at most 𝐾 ⋅ 𝐵 commands across all iterations (Section 5.1). (ii)  is held fixed across the baseline replay 𝜌(𝑡 , ) and the post-mitigation replay 𝜌(𝑡+1 , ) used to compute ME (Section 3). (iii) The suggester is prompted to exclude indicator-based blocking, monitoringonly responses, and allowlists permitting the observed exfiltration protocol; this is a design constraint on the policy class, enforced by prompting rather than by syntactic projection. (iv) The single-mitigation environment is reset to a baseline snapshot between cycles; the cumulative environment is not (Section 4.7).

3.5. Out of Scope • Attacks on the framework or its supporting infrastructure. • Initial-access and pre-compromise phases of the attack lifecycle. • Insider threats and physical attacks. • Virtualization infrastructure failures.

Cumulative Variant The cumulative environment main-

tains a persistent state  (0) ,  (1) , … . After the 𝑘-th candidate 𝑚𝑘 is implemented and replayed on 𝜏( (𝑘−1) , 𝑚𝑘 ), it is accepted,  (𝑘) = 𝜏( (𝑘−1) , 𝑚𝑘 ), only if (a) Cumulative ME does not decrease relative to  (𝑘−1) , and (b) Φ(𝜏( (𝑘−1) , 𝑚𝑘 )) = 1; otherwise  (𝑘) =  (𝑘−1) .

4. COHORT 4.1. Framework Scope and Purpose COHORT focuses on post-compromise enterprise defense: it proposes network-level mitigations and validates them by replaying the original adversary in an emulated environment, with no production exposure during evaluation. Fidelity derives from three design choices: (i) control and data planes run real vendor firmware on the firewall, switch, and router; (ii) endpoints run as Ubuntu containers; (iii) Caldera executes real commands, producing genuine traffic and host-level artifacts. We do not model enterprisescale traffic load, configuration drift, Windows endpoints (Section 6.3), or live synchronization with production; the framework operates as a staging environment on static baseline snapshots. The fidelity claim is scoped to deployable mitigations (Section 3.4).

4.3. Agent Roles and Responsibilities The mitigation workflow is implemented using a team of role-specialized LLM-based agents (Figure 1) Liu, Li, Zhang, Wang, He, Hong, Liu, Zhang, Song, Zhu, Cheng, Wang, Wang, Luo, Jin, Zhang, Liu, Chen, Zhang, Yu, Shi, Li, Wu, Teng, Jia, Xu, Xiang, Lin, Liu, Liu, Su, Sun, Berseth, Nie, Foster, Ward, Wu, Gu, Zhuge, Liang, Tang, Wang, You, Wang, Pei, Yang, Qi and Wu (2025).

Mitigation Suggester The mitigation suggester analyzes

the adversary’s operational report, along with the network topology and configuration state, to propose configuration changes that disrupt the observed attack capabilities. Suggested mitigations must be reusable, generalizable, and expressed at the level of enterprise controls (e.g., segmentation policies, routing constraints, firewall rules).

4.2. Problem Formalization Setting Let  denote the space of network configuration

states (device configurations, ACLs, routing policies, hostlevel configuration). Let  denote a fixed adversary scenario executed by Caldera, and 𝜌 the procedure that runs  against a state and returns an operational report  (ASSR is a function of , defined in Section 5.2). Let  be the device configuration command space; a mitigation is a finite sequence 𝑚 ∈  ∗ , applied via a state transition 𝜏 ∶  ×  ∗ →  . Let 𝐵 ∈ ℕ be the per-refinement command budget and 𝐾 ∈ ℕ the implementer/critic refinement cap (Section 5.1).

Mitigation Implementer The mitigation implementer translates a selected mitigation into concrete configuration changes within the emulated environment, modifying device configurations or logical topology elements.

Critic The critic reviews implemented mitigations prior to

offensive replay, identifying configuration errors, policy violations, or deviations from the intended mitigation objective. When issues are detected, it provides feedback to the mitigation implementer, who revises the configuration accordingly. This review–revision loop halts on critic approval or at the iteration cap 𝐾; the empirical lift over the no-critic ablation (Section 5.3) justifies running it.

Mitigation Cycle A mitigation cycle starts from a baseline

state 𝑡 and produces a candidate 𝑚 in three stages: (1) replay: 𝑡 = 𝜌(𝑡 , ); (2) suggest: the suggester reads 𝑡 and (per Section 4.3, Context Management) its prior suggestions, and proposes a candidate strategy that the implementer 6

Caldera Operation Report

Emulated Network Topology

Evaluation

Next mitigation

Task

Mitigation Suggester Agent

Mitigation Implementer Agent

Critic Agent

Implements mitigation

Selects mitigation

Approves / Reached mitigation iteration limit

Judge Agent

Reached mitigations limit / No mitigations left

Disapproves

Network Device Configuration

Send command to device

Emulated Network Environment (GNS3)

Summarizer Agent

Mitigations Summary

Execution Logs

Calculation of mitigation effectiveness

Caldera Adversary Emulation

Cumulative Hardened Emulated Network Topology

Figure 1: Multi-agent automatic mitigation framework showing the overall architecture and agent interactions.

Mitigation Suggesting. The suggester proposes a candidate mitigation. Mitigation Implementation. The implementer translates the candidate into device commands. Critic Review. The critic reviews the implementation; refinement runs up to 𝐾 iterations. Evaluation. The judge replays the attack on the mitigated environment to determine whether the mitigation succeeded. Connectivity Validation. Two Caldera probes (a LAN ping and an internet HTTP check) verify reachability; both must pass for the mitigation to count toward MSR (Section 5.2). Cumulative Replay. Successful mitigations are replayed onto the persistent cumulative project (Section 4.7).

Judge Whereas the critic reviews the implemented config-

uration prior to attack execution, the judge evaluates the resulting attack outcome. It replays the attack scenario in the post-mitigation environment using Caldera and compares the resulting attack step success rate (ASSR) to the pre-mitigation baseline; a lower rate indicates a successful mitigation. The judge also categorizes each mitigation into strategic classes (e.g., Host Hardening, Network-Based Traffic Filtering).

Summarizer After all mitigation cycles are complete, the

summarizer reads the judge’s per-mitigation evaluations (rather than the full multi-round conversation) and produces a compact retrospective covering the attack scenario, attempted mitigations and outcomes, cross-mitigation observations, connectivity validation results, and the device commands executed by the implementer.

4.6. Single-Agent Baseline To isolate the contribution of the multi-agent architecture, we evaluate a single-agent baseline that consolidates the Suggester, Implementer, and Critic roles into a single LLM; the Judge and Summarizer roles are unchanged. The single agent shares the same model (GPT-5.4 mini), inputs (operational report, network topology, configuration state), and device-configuration access as the multi-agent system. Figure 2 shows the simplified workflow.

4.4. Context Management Each agent’s context scope is tailored to its role. The mitigation suggester receives the history of its own prior suggestions, allowing it to avoid proposing strategies it has already tried; outcomes from earlier mitigations are deliberately excluded both to prevent the suggester from exploiting variations of already-successful strategies and to keep its context (and therefore token cost) bounded across long sessions. In contrast, the implementer and critic agents receive only the current mitigation context, with no knowledge of prior mitigations, since each mitigation is evaluated independently.

4.7. Cumulative Mitigation Evaluation Per-mitigation evaluation reveals individual effectiveness but not how defenses interact when stacked. The cumulative evaluation (Figure 3) captures these compound effects: a mitigation that passed the per-mitigation Evaluation with connectivity preserved is replayed onto a persistent cumulative project, an emulated network instance that accumulates approved defenses. After each cumulative replay, the mitigation is accepted into the project only if it (i) does not regress the Cumulative Mitigation Effectiveness (Cumulative ME, Section 5.2) and (ii) preserves basic connectivity; otherwise

4.5. Mitigation Cycle Each mitigation is evaluated through the following phases: Initialization. The single-mitigation environment is reverted to the baseline snapshot. Attack Execution. The attack scenario is executed, producing the operational report. 7

Caldera Operation Report

Emulated Network Topology

segmentation. The large enterprise topology adopts a hierarchical three-tier architecture: five core routers interconnect four floor segments, each terminated by a dedicated floor router, a floor-level firewall, and a floor L3 switch, with twelve departmental access switches distributed across the floors. These three topologies approximate smalloffice, mid-market-branch, and multi-department enterprisecampus deployments, respectively.

Evaluation

Next mitigation

Task

Single Agent

Judge Agent

Summarizer Agent

Finish Reached mitigations limit / No mitigations left

Network Device Configuration

Mitigations Summary

Send command

Execution Logs

Calculation of mitigation effectiveness

Emulated Network Environment (GNS3)

Caldera Adversary Emulation

Cumulative Hardened Emulated Network Topology

Attack Scenarios We focus on adversary objectives that

pose concrete, recurring risks to enterprise environments. All four attack scenarios are implemented as Caldera adversaries with MITRE ATT&CK-aligned abilities. Together, these adversaries span distinct tactical objectives across the attack lifecycle: • Thief, a multi-stage attack involving local file discovery, staging, archiving, and exfiltration over the C2 channel, stressing access controls and egress constraints (adapted from the Thief adversary pack in the MITRE Caldera Stockpile plugin Stockpile; Harrell, Chan, Han, Voss, Danke, Ji, Brobin and Esprit (2023)). • Ransomware, which performs system reconnaissance and file discovery on the compromised host, then pivots via SSH to a dedicated victim container (SSH Victim Container) to encrypt sensitive files with AES, deploy a ransom note, and establish cron-based persistence, before exfiltrating the encryption key to C2. • DNS Exfiltration, which base64-encodes sensitive file contents and exfiltrates them as DNS subdomain queries to bypass HTTP egress controls, then removes artifacts. • Lateral Movement, which scans the LAN for SSH services, brute-forces credentials against a dedicated victim container (SSH Victim Container), pivots in via SSH, enumerates the remote host and collects files, then exfiltrates them to C2. All attacks include a dedicated C2 communication link (see Section 3.1) and end with the ability to validate LAN and internet connectivity. The C2 server orchestrates attacks against a compromised Caldera Linux Agent running the Caldera Sandcat agent; the Lateral_Movement and Ransomware adversaries additionally target a dedicated SSH Victim Container that serves as the SSH lateral movement destination. Per-step breakdowns of each adversary, including tactic and MITRE ATT&CK technique identifiers, are provided in the Adversary Attack Steps appendix (Table 6).

Figure 2: Single-agent baseline workflow for mitigation suggestion, implementation, and self-validation.

the project is rolled back to the previously accepted state. Because acceptance requires non-regression rather than strict improvement, non-interfering mitigations are retained even when they add no measurable ASSR reduction, yielding a defense-in-depth posture.

Failure modes Each role has a characteristic failure shape.

The suggester proposes generalizable controls that do not bind the adversary to the actual primitive, for example, disabling a specific binary while equivalent alternatives remain available. The implementer produces syntactically valid configurations whose effective scope is wrong: rules placed at the wrong hop, evaluated in the wrong order, or scoped to a path that does not intersect the attacker’s read or write targets. The critic catches errors visible in the configuration text (syntax errors, policy violations, deviations from stated intent) but cannot distinguish a syntactically correct rule that fails in placement from one that succeeds, since placement effects only surface during replay. Section 5.3 reports the empirical breakdown by mitigation category.

5. Evaluation 5.1. Experimental Setup All agent roles (mitigation suggester, mitigation implementer, and critic) use GPT-5.4 mini (via Azure OpenAI) with the default sampling temperature of 1.0. The SingleAgent baseline (Section 4.6) uses a single GPT-5.4 mini instance to play all roles in a single conversation. Caldera replay is deterministic given a fixed scenario and network state, so the five repetitions per condition (below) capture only the variance attributable to LLM sampling at this temperature.

Experimental Conditions To disentangle the contribu-

tion of role specialization from that of iterative critique, we evaluate three runtime conditions that share the same model (GPT-5.4 mini), the same Caldera adversaries, and the same empirical validation pipeline: • Single-Agent. A single GPT-5.4 mini instance plays the mitigation suggester, implementer, and critic roles within one conversation (Section 4.6). There is no iterative critique loop: the agent proposes, implements, and selfchecks in a single linear trajectory.

Network Topologies Three representative enterprise net-

work topologies of increasing complexity are used (Figures 10–12). The small enterprise topology consists of one FortiGate firewall and a single layer-2 switch connecting workstations and servers, representing a low-complexity baseline environment. The medium enterprise topology introduces a two-segment network behind a centralized firewall, with two distribution routers and two layer-2 switches enabling inter-segment routing and realistic VLAN-based

8

Original EN

Single Mitigations EN

Cumulative Mitigations EN

Original EN

Initial vulnerability Initial testing state

M1 Egress Mitigation Block success

Initial state

Initial state

M2 Restrict write Mitigation access failure

Initial state

Initial state

Mitigation failure M1

Mitigation M1 success mitigation Apply M1

Replay attack

M3 Restrict upload Mitigation success tools

Single EN

✗

Cumulative EN

Apply M3

Create snapshot

Apply mitigation

Revert snapshot

Figure 3: Evaluation workflow showing the parallel per-mitigation evaluations (independent, rolled-back) and the cumulative evaluation (persistent cumulative project with sequential mitigation replay). A demo video walking through this diagram can be found in Appendix 8.3.

(ASSR) achieved by a deployed mitigation, with the symbols of Section 4.2:

• Multi-Agent without Critic. The multi-agent graph is executed with role-specialized suggester and implementer agents, and the implementation–critique iteration is constrained to a single pass, so the critic’s feedback is never fed back into a revision. The implementer’s initial output is taken as-is, without acting on the critic’s advice. • Multi-Agent. The full framework, including the critic and up to five implementation–critique iterations per mitigation. Each of the 12 scenarios (4 attacks × 3 topologies) is repeated five times per condition, with a limit of 10 mitigations per run, up to 5 implementation–critique iterations per mitigation, and a budget of 20 device commands per implementation cycle, instantiating 𝐾 = 5 and 𝐵 = 20 in the formalization of Section 4.2. Across the three conditions we collected 1 782 mitigation attempts across 179 runs: 600 mitigations (60 runs) Single-Agent, 599 (60 runs) MultiAgent without Critic, and 583 (59 runs) Multi-Agent. Subcap mitigation counts reflect the multi-agent graph’s earlyexit edge when the suggester proposes no further distinct candidates; one Multi-Agent replicate was excluded after its summary file failed to write. In all three conditions, the suggester operates under an additional evaluation-time restriction: candidate mitigations may not alter the network topology (no adding, removing, or substituting devices, links, or services). The restriction is motivated by deployability and by frequent agent failures observed when attempting topology-altering changes during early system testing.

ME(𝑚; 𝑡 , ) = ASSR(𝑡 , )−ASSR(𝜏(𝑡 , 𝑚), ) (1) where ASSR( , ) is the ratio of successfully executed Caldera abilities to attempted abilities when adversary  is replayed against state  , with each ability scored as success, failure, or timeout. ME = ASSR(𝑡 , ) signifies complete mitigation (post-mitigation ASSR reduced to zero) and ME = 0 indicates no effect. ME can also be negative, where the attempt actively worsens defense, and the attack succeeds at a higher rate than at baseline.

Mitigation Success Rate (MSR) MSR is the probability

that a proposed mitigation works: the fraction of attempts that both disrupt the attack (ME > 0) and preserve legitimate connectivity (Φ = 1, Section 3). MSR =

𝑁ME>0 ∧ Φ=1 × 100% 𝑁total

(2)

where 𝑁total is the total number of mitigation attempts and the numerator counts attempts whose 𝑖-th cycle satisfies ME(𝑚𝑖 ; 𝑡𝑖 , ) > 0 and Φ(𝜏(𝑡𝑖 , 𝑚𝑖 )) = 1.

Cumulative Mitigation Effectiveness (MEcumulative ) While ME and MSR measure each mitigation independently (with rollback between attempts), the cumulative evaluation measures how defenses compound when deployed together. Cumulative ME after 𝑘 approved mitigations is

5.2. Evaluation Criteria

MEcumulative,𝑘 = ASSR( (0) , ) − ASSR( (𝑘) , ) (3)

Our evaluation is grounded in operational outcomes: every mitigation is validated by replaying the attack on the mitigated network and comparing the resulting ASSR to the pre-mitigation baseline.

where  (0) , … ,  (𝑘) is the sequence of cumulative-project states defined by the acceptance rule of Section 4.2 (Cumulative Variant). Cumulative ME accumulates as a step function: each accepted mitigation either reduces ASSR by a discrete amount

Mitigation Effectiveness (ME) Mitigation effectiveness

quantifies the reduction in the Attack Step Success Rate 9

100

MSR (regression-checked) MSR without regression check

80 60

39% 41%

40 20 0

47%

Mean cumulative ME (%)

Mitigation Success Rate (%)

100

55%

11% 15% Single-Agent

Multi-Agent without Critic

80 60 40 20 0

Multi-Agent

Figure 4: MSR by runtime condition, pooled across attacks and topologies. The hatched bar shows the rate without the connectivity-regression check.

2

4

Multi-Agent (n=60) Multi-Agent without Critic (n=60) Single-Agent (n=60) 6 8 10

Number of Sequential Defenses

Figure 5: Cumulative ME by condition, pooled across attacks.

Mean cumulative ME (%)

100

or has no incremental effect on top of the defenses already deployed on the cumulative project. Aggregate cumulativeME behavior across all runs is reported in Figures 5 (by condition) and 6 (by topology).

5.3. Results MSR Results Figure 4 reports the headline MSR (per-

attack split in Appendix 8.6). Across all attacks, MSR rises from 10.7% (Single-Agent) to 38.6% (Multi-Agent without Critic) to 46.7% (Multi-Agent). Every pairwise difference is statistically significant under Holm-Bonferroni correction (full pairwise statistics in Appendix 8.5). Decomposed against the Single-Agent baseline, role specialization alone (Multi-Agent without Critic) accounts for roughly threequarters of the total gain over Single-Agent, and the iterative critique loop contributes the remaining one-quarter. The same ranking holds at the run level: across the run’s up-toten sequential attempts, Multi-Agent produces at least one working mitigation in 98% of runs and Multi-Agent without Critic in 95%, against 78% for Single-Agent. Connectivity preservation moves non-monotonically: Multi-Agent without Critic improves it slightly over Single-Agent (94.3% vs. 92.0%), while the critic trades it down to 88.0% in exchange for higher MSR. The critic’s more aggressive controls occasionally break a probe while still blocking the attack.

80 60 40 20 0

2

4

6

Small (n=60) Medium (n=60) Large (n=60) 8 10

Number of Sequential Defenses

Figure 6: Cumulative ME by topology, all conditions pooled.

Cumulative Mitigation Effectiveness Figure 5 character-

izes how defenses compound when stacked. Multi-Agent plateaus at ≈ 52% cumulative ME pooled across attacks, Multi-Agent without Critic at ≈ 47%, and Single-Agent at ≈ 28%. The topology view (Figure 6) shows that, once enough mitigations are stacked, the final cumulative ME is comparable across small, medium, and large enterprise networks. The topology slows the rate of ME accumulation more than it bounds the achievable ceiling. The cumulative track follows a greedy policy: each accepted mitigation is permanently retained on the cumulative project, and earlier choices are not revisited as later mitigations are added.

Multi-agent decomposition lifts MSR 4.4× over a Single-Agent baseline. Single-Agent 10.7%, MultiAgent without Critic 38.6%, Multi-Agent 46.7%. Role specialization accounts for roughly three-quarters of the lift; the iterative critique loop accounts for the remaining quarter.

MSR by Topology Figure 7 reports MSR by topology for

the Multi-Agent condition. We do not detect a topology effect: small enterprise 48.9% (𝑛 = 188), medium 48.5% (𝑛 = 200), large 42.6% (𝑛 = 195); Welch’s ANOVA is non-significant (𝑝 = 0.37), with all pairwise comparisons non-significant under Holm correction (smallest 𝑝 = 0.21). The cumulative-ME view (Figure 6) is consistent: all three topologies plateau by the fourth sequential defense, at ≈ 43% (small), ≈ 45% (medium), and ≈ 39% (large).

Cost The lift carries a cost (Table 5, Per-Mitigation Cost).

Per mitigation, Multi-Agent averages $0.47 and 124 s of wall-clock time, compared to $0.09 and 60 s for SingleAgent (≈5× cost and 2× wall-clock time). Multi-Agent without Critic falls in between at $0.22 and 96 s. Total experiment cost across all 1 782 mitigations was $464. 10

Mitigation Success Rate (%)

100

to account for hidden intermediaries on the routed path. The failed host-centric mitigations exhibit an analogous targeting error: the control is correctly applied but does not bind to the attacker’s actual primitive. The dominant patterns are disabling a specific binary (e.g., find, tar, python3) while the attacker’s tooling has equivalent alternates; tightening filesystem permissions on a path that does not intersect the attack’s actual read or write targets; and hardening SSH authentication when the validated attack path is already authenticated.

80 60

48.9% (n=188)

48.5% (n=200)

Small Enterprise

Medium Enterprise

42.6% (n=195)

40 20 0

Large Enterprise

Figure 7: MSR by enterprise topology (small/medium/large) for the Multi-Agent condition. Error bars indicate the standard error of the mean.

Host-centric controls dominate; network filtering looks right on paper but rarely fires. Host Hardening, the largest bucket at 𝑛 = 341, reaches 63.3% MSR, while Network-Based Traffic Filtering, the secondlargest by volume at 𝑛 = 139, reduces the attack only 5.8% of the time. Network filtering fails not from wrong intent but from rules placed at the wrong hop or evaluated in the wrong order.

82.1% (n=39) 63.3% (n=341) 53.8% (n=13) 25.0% (n=20) 15.8% (n=19) 9.1% (n=11) Host-based 5.8% (n=139) Network-based

Application Control Host Hardening Host-Based Intrusion Prevention Host-Based Traffic Filtering Network Isolation Network-Based Traffic Inspection Network-Based Traffic Filtering 0

20

40

60

80

Mitigation Success Rate (%)

100

Figure 8: MSR by mitigation category for the Multi-Agent condition, pooled across topologies. Category labels are assigned for each mitigation and normalized to a shared reporting vocabulary (Table 3). Error bars indicate the standard error of the proportion.

6. Discussion 6.1. Attribution and the Optimization-Evaluation Gap Caldera replays each adversary scenario unchanged across pre- and post-mitigation runs (Section 3). Holding the attack fixed makes per-mitigation ASSR change attributable to the deployed mitigation rather than to concurrent attacker adaptation. The same property creates an optimizationevaluation gap: the suggester reads the operational report from the same replay that the judge later scores, so candidates can, in principle, be tuned to the evaluation steps. Several design properties bound this gap. The success criterion pairs ME > 0 with a connectivity check (Section 5.2), so trivial wins like severing all egress fail outright. Mitigations are credited against the attack’s productionpath actions, not the operator’s C2 channel (Section 3.1), which forces engagement with the data flow rather than the attacker’s infrastructure. The suggester is restricted to reusable enterprise-level controls, with indicator-based blocking and monitoring-only responses excluded by prompt (Section 4.2). The framework’s outputs are reviewed by engineers before deployment, not auto-applied, so final acceptance rests on human review rather than on the metric.

No detectable topology effect. Differences between small (48.9%), medium (48.5%), and large (42.6%) topologies are not statistically significant.

Mitigation Categories To characterize which classes of

defense the system produces, we group mitigations into high-level categories based on their defensive approach. These categories can be broadly grouped into networkcentric controls (enforced at the network perimeter or within the infrastructure), host-centric controls (applied directly to endpoints), and reactive/operational measures. Figure 8 presents the per-category MSR for the Multi-Agent condition, pooled across topologies; Table 3 gives definitions, examples, and counts. Two high-level patterns are visible. First, host-centric controls dominate in both volume and effectiveness: Host Hardening (the largest single bucket at 𝑛 = 341) achieves 63.3% MSR, Application Control 82.1% (𝑛 = 39), and Host-Based Intrusion Prevention 53.8% (𝑛 = 13, too few attempts to rank reliably against the larger buckets). Second, network-based filtering dominates in volume but not in effectiveness: Network-Based Traffic Filtering is the secondlargest bucket (𝑛 = 139) but only 5.8% of its attempts reduce the attack. Inspection of the failed network-filtering mitigations shows a recurring failure mode: the control is syntactically correct and aligned with best-practice designs (VLAN segmentation, inter-VLAN blocks, router ACLs, egress firewall rules) but is not enforced at the correct point in the network hierarchy, is evaluated in the wrong rule order (e.g., a deny rule after an allow rule on FortiGate), or fails

6.2. Failure Analysis: Where Mitigations Miss Host Hardening reaches 63.3% MSR (𝑛 = 341) and Application Control 82.1% (𝑛 = 39), while Network-Based Traffic Filtering reaches only 5.8% (𝑛 = 139) (Section 5.3). Host-local controls succeed because the implementer has the resident context (file, process, user, binary) it acts on, so misses concentrate in cases where the chosen primitive does not bind the attacker’s: disabling find when tar suffices, hardening SSH when the validated path is already authenticated. Network controls fail because rules are reasoned about as syntactic objects rather than bound to a routed path: rules 11

Table 3 Mitigation categories, MITRE D3FEND mappings, and per-category MSR for the Multi-Agent condition, pooled across topologies. Category

Definition

Examples

D3FEND

N

MSR (%)

Application Control

Per-account execute restrictions on specific applications

Per-user execute-denylist of attacker tooling (sshpass, nmap, dig, base64 + Python exfil staging); allowlists permitting only business-required network utilities chmod -x on attacker-used utilities (sshpass, curl, openssl, dig); SSH key-only authentication with password auth disabled; per-user write/permission tightening on staging paths fail2ban SSH event-thresholding and account lockout on authentication-failure bursts; cron-based watchdog kills of reconnaissance binaries Host iptables/nftables egress rules denying outbound HTTPS to C2 destinations while preserving LAN ICMP and HTTP validation FortiGate / Open vSwitch inter-VLAN ACLs; east-west SSH restrictions and floor-router segmentation between department LANs FortiGate DNS-security inspection blocking long / high-entropy subdomains and query-burst anomalies FortiGate outbound HTTPS denies to C2 endpoints; OpenWrt UCI rules blocking DNS-tunnel ports (53, 8853); destination/IP allowlists

D3-EAL D3-EDL

39

82.1

D3-LFP D3-SPP D3-UAP

341

63.3

D3-ANET D3-AL D3-PT

13

53.8

D3-OTFh

20

25.0

D3-BDI D3-NI

19

15.8

D3-DNSTA 11 D3-DNSDL D3-PMAD D3-OTF 139 D3-NTF D3-FRDDL

9.1

Host Hardening

Host-level reduction of attacker capability via permission and credential restriction Host-Based Active host-runtime defenses Intrusion Prevention that detect and respond to in-progress attack actions

Host-Based Traffic Filtering

Endpoint-local egress firewall enforcement

Network Isolation

Network-segmentation controls limiting lateral reach

Network-Based Traffic Inspection

Application-layer inspection at network devices

Network-Based Traffic Filtering

Perimeter firewall egress and proxy controls

5.8

h D3-OTF enforced on the endpoint rather than at a network device.

at the wrong hop, allow before deny, or scoped past hidden intermediaries. The cumulative-ME plateau (52% pooled, 43%, 45%, and 39% across topology sizes; ANOVA 𝑝 = 0.37) is consistent with the same bound: stacked greedy defense saturates against the attack steps that no single host- or network-side control covers under our connectivity contract.

6.5. Open Questions First, the cumulative project produces a persistent hardened network state at the end of each attack’s track. Reusing it as the starting topology for a different adversary would let the framework measure cross-attack transfer directly: whether mitigations stacked against one attack continue to disrupt a second, distinct adversary. Iterating this handoff across multiple adversaries extends the cumulative track into a sequential multi-attack evaluation, in which each attack inherits the hardened state left by the previous one. Second, this cumulative track currently uses a greedy stacking policy in which each accepted mitigation is retained on the cumulative project and never revisited. Nongreedy variants (lookahead over candidate mitigations, beam search across cumulative-project trajectories, or a revisitand-replace operator that swaps earlier choices when later mitigations subsume them) would allow the framework to probe whether the observed cumulative-ME plateau is local to the greedy policy or near a global ceiling for stacked defense. Third, grounding the suggester in MITRE D3FEND or organization-specific incident-response playbooks would constrain proposals to a vetted defensive vocabulary, align autonomous mitigations with approved response actions, and produce mitigations whose primitives are immediately recognizable to security operations staff. Fourth, automated generation of emulated topologies from production network configurations would reduce the manual configuration burden associated with high-fidelity replicas. While LLMs can assist with such setup Ifland et al.

6.3. Linux-Only Endpoints GNS3 supports Windows endpoint images, but Windows guests are substantially more compute-intensive than Linux containers in emulation, so we restrict the evaluation to Linux endpoints. Whether the reported MSR transfers to Windows-hosted environments is open, and the exposure is uneven across mitigation categories. Host Hardening and Application Control depend on platform-specific binaries, policy primitives, and default services that differ substantially between Linux and Windows; Network-Based Traffic Filtering operates on the same TCP/IP path and inherits the same placement-bound constraints (Section 6.2).

6.4. Ability-Level Granularity MSR and ME are reported at the granularity of Caldera abilities. When a single ability bundles multiple discrete actions, intra-ability disruption is not resolved: a partially broken ability still counts as a successful step. This biases reported disruption downward rather than upward, since a mitigation that thwarts most but not all of a bundled ability fails the ME > 0 check. 12

(2025); Wang et al. (2024a), fully automated productionto-emulation generation remains an open engineering challenge, and is a precondition for scaling the framework to networks beyond hand-curated topologies. Fifth, all agents are backed by GPT-5.4 mini (Section 8.1.3); whether the reported MSR transfers to other model families is open. Sixth, the single-agent baseline ablates the multi-agent architecture but does not calibrate the framework against expert manual work. A controlled study with security engineers producing mitigations under comparable constraints would situate the reported MSR relative to a human reference point.

Castro, S.R., Campbell, R., Lau, N., Villalobos, O., Duan, J., Cardenas, A.A., 2025. Large language models are autonomous cyber defenders, in: 2025 IEEE Conference on Artificial Intelligence (CAI), IEEE. pp. 1125–1132. Chang, C., Cao, M., Teoh, K., Ear, E., Xu, S., 2025. Characterizing caldera’s cyber attack emulation capabilities. Military Cyber Affairs 8, 5. Cyberbit, . Cyberbit Cyber Range. https://www.cyberbit.com. Accessed: 2026-04. Cymulate, . Cymulate Exposure Validation Platform. https://cymulate.com. Accessed: 2026-04. Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., Mordatch, I., 2024. Improving factuality and reasoning in language models through multiagent debate, in: Forty-first international conference on machine learning. Eckhart, M., Ekelhart, A., 2018. Towards security-aware virtual environments for digital twins, in: Proceedings of the 4th ACM Workshop on Cyber-Physical System Security, pp. 61–72. Emerson, H., Bates, L., Hicks, C., Mavroudis, V., 2024. Cyborg++: An enhanced gym for the development of autonomous cyber agents. arXiv preprint arXiv:2410.16324 . Hamadanian, P., Arzani, B., Fouladi, S., Kakarla, S.K.R., Fonseca, R., Billor, D., Cheema, A., Nkposong, E., Chandra, R., 2023. A holistic view of ai-driven network incident management, in: Proceedings of the 22nd ACM Workshop on Hot Topics in Networks, pp. 180–188. Hammar, K., Dhir, N., Stadler, R., 2024. Optimal defender strategies for cage-2 using causal modeling and tree search. arXiv preprint arXiv:2407.11070 . Hammar, K., Stadler, R., 2022. Intrusion prevention through optimal stopping. IEEE Transactions on Network and Service Management 19, 2333–2348. Harrell, B., Chan, M., Han, H., Voss, K., Danke, G., Ji, L., Brobin, O., Esprit, K., 2023. Testing Data Exfiltration with MITRE Caldera. https:

7. Conclusion We presented COHORT, the first end-to-end framework to autonomously generate and validate deployable mitigations against an observed adversary in emulated network. The two bottlenecks of expert dependency and productionexposure risk are addressed by a multi-agent LLM generation pipeline and a high-fidelity emulator running real vendor firmware; each candidate is validated by offensive replay of the original Caldera adversary on the mitigated network. Across three topologies and four attack scenarios, COHORT outperforms a single-agent baseline (Section 5.3). The same autonomy opens new attack surfaces: a routine hardening run surfaced a long-acknowledged containerescape weakness, making staging isolation and human review of accepted mitigations preconditions for safe operation. Results are scoped to scripted Caldera adversaries on a Linux-only lab with FortiGate, Cisco IOS, and Open vSwitch under GPT-5.4 mini, and connectivity preservation is verified by two probes; cross-attack transfer, adaptive adversaries, other model families, calibration against expertderived mitigations, and finer operational regression remain open (Section 6.5).

//medium.com/@mitrecaldera/testing-data-exfiltration-with-mitre-c aldera-aa7dfe3fc9bd.

Hong, J., Tu, N.V., Hong, J.W.K., 2025. A comprehensive survey on llmbased network management and operations. International Journal of Network Management 35, e70029. Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S.K.S., Lin, Z., et al., 2023. Metagpt: Meta programming for a multi-agent collaborative framework, in: The twelfth international conference on learning representations. IBM Security, 2025. Cost of a data breach report. URL: https://www.ibm. com/reports/data-breach. accessed: 2025-01-14. Ifland, B., Krief, R., Zilberman, A., Duani, E., Ohana, M., Murillo, A., Manor, O., Lavi, O., Hikichi, K., Shabtai, A., Elovici, Y., Puzis, R., 2025. GeNet: A Multimodal LLM-Based Co-Pilot for Network Topology and Configuration , in: 2025 IEEE 45th International Conference on Distributed Computing Systems Workshops (ICDCSW), IEEE Computer Society, Los Alamitos, CA, USA. pp. 117–122. URL: https://do i.ieeecomputersociety.org/10.1109/ICDCSW63273.2025.00026, doi:10.1109/ICDCSW63273.2025.00026. Islam, C., Babar, M.A., Nepal, S., 2019. A multi-vocal review of security orchestration. ACM Computing Surveys 52, 1–45. Janisch, J., Pevnỳ, T., Lisỳ, V., 2023. Nasimemu: Network attack simulator & emulator for training agents generalizing to novel scenarios, in: European Symposium on Research in Computer Security, Springer. pp. 589–608. Kaheh, M., Kholgh, D.K., Kostakos, P., 2023. Cyber sentinel: Exploring conversational agents in streamlining security tasks with gpt-4. arXiv preprint arXiv:2309.16422 . Kiely, M., Ahiskali, M., Borde, E., Bowman, B., Bowman, D., Van Bruggen, D., Cowan, K., Dasgupta, P., Devendorf, E., Edwards, B., et al., 2025. Exploring the efficacy of multi-agent reinforcement learning for autonomous cyber defence: A cage challenge 4 perspective, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 28907–28913. Kim, M., Wang, J., Moore, K., Goel, D., Wang, D., Mohsin, A., Ibrahim, A., Doss, R., Camtepe, S., Janicke, H., 2025. Cyberally: Leveraging llms and knowledge graphs to empower cyber defenders, in: Companion

LLM usage Grammar was edited using Grammarly and Claude.

References An, K., Yang, F., Lu, J., Li, L., Ren, Z., Huang, H., Wang, L., Zhao, P., Kang, Y., Ding, H., et al., 2024. Nissist: An incident mitigation copilot based on troubleshooting guides. arXiv preprint arXiv:2402.17531 . Applebaum, A., Miller, D., Strom, B., Korber, C., Wolf, R., 2016. Intelligent, automated red team emulation, in: Proceedings of the 32nd Annual Conference on Computer Security Applications, pp. 363–373. AttackIQ, . AttackIQ Security Optimization Platform. https://www.attack iq.com. Accessed: 2026-04. AutoGen, . AutoGen. https://github.com/microsoft/autogen. Autonomous Resilient Cyber Defence Programme, 2024. PrimAITE: Primary-AI training environment. https://github.com/Autonomous- R esilient-Cyber-Defence/PrimAITE. Bringhenti, D., Marchetto, G., Sisto, R., Valenza, F., 2023. Automation for network security configuration: State of the art and research trends. ACM Computing Surveys 56, 1–37.

13

Proceedings of the ACM on Web Conference 2025, pp. 2851–2854. Kouremetis, M., Lawrence, D., Alford, R., Cheuvront, Z., Davila, D., Geyer, B., Haigh, T., Michalak, E., Murphy, R., Russo, G., 2024. Mirage: cyber deception against autonomous cyber attacks in emulation and simulation. Annals of Telecommunications 79, 803–817. Li, G., Hammoud, H., Itani, H., Khizbullin, D., Ghanem, B., 2023. Camel: Communicative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems 36, 51991– 52008. Li, L., Fayad, R., Taylor, A., 2021. Cygil: A cyber gym for training autonomous agents over emulated network systems. arXiv preprint arXiv:2109.03331 . Lira, O.G., Caicedo, O.M., da Fonseca, N.L., 2024. Large language models for zero touch network configuration management. IEEE Communications Magazine 63, 146–153. Liu, B., Li, X., Zhang, J., Wang, J., He, T., Hong, S., Liu, H., Zhang, S., Song, K., Zhu, K., Cheng, Y., Wang, S., Wang, X., Luo, Y., Jin, H., Zhang, P., Liu, O., Chen, J., Zhang, H., Yu, Z., Shi, H., Li, B., Wu, D., Teng, F., Jia, X., Xu, J., Xiang, J., Lin, Y., Liu, T., Liu, T., Su, Y., Sun, H., Berseth, G., Nie, J., Foster, I., Ward, L., Wu, Q., Gu, Y., Zhuge, M., Liang, X., Tang, X., Wang, H., You, J., Wang, C., Pei, J., Yang, Q., Qi, X., Wu, C., 2025. Advances and challenges in foundation agents: From brain-inspired intelligence to evolutionary, collaborative, and safe systems. arXiv:2504.01990. see Section 15.2.2. Loevenich, J., Adler, E., Huerten, T., Lopes, R.R.F., 2025. Design and evaluation of an autonomous cyber defence agent using drl and an augmented llm. Computer Networks 262, 111162. Luo, H., Sun, G., Liu, Y., Niyato, D., Yu, H., Atiquzzaman, M., Dustdar, S., 2026. A trustworthy agentic multi-llm network: Challenges, solutions, and a use case. IEEE Wireless Communications . Lyu, M., Gharakheili, H.H., Sivaraman, V., 2024. A survey on enterprise network security: Asset behavioral monitoring and distributed attack detection. IEEE Access . Marchioro, T., Saroui, R., Olivereau, A., 2025. Network intrusion response systems: towards standardized evaluation of intrusion response, in: ESORICS 2025: 30th European Symposium on Research in computer security. Molina-Markham, A., Miniter, C., Powell, B., Ridley, A., 2021. Network environment design for autonomous cyberdefense. arXiv preprint arXiv:2103.07583 . Neumann, J.C., 2015. The book of GNS3: build virtual network labs using Cisco, Juniper, and more. No Starch Press. Nguyen, T.T., Reddi, V.J., 2021. Deep reinforcement learning for cyber security. IEEE Transactions on Neural Networks and Learning Systems 34, 3779–3795. Oesch, S., Chaulagain, A., Weber, B., Dixson, M., Sadovnik, A., Roberson, B., Watson, C., Austria, P., 2024. Towards a high fidelity training environment for autonomous cyber defense agents, in: Proceedings of the 17th Cyber Security Experimentation and Test Workshop, pp. 91– 99. Pentera, . Pentera Automated Security Validation. https://pentera.io. Accessed: 2026-04. Picus, . Picus Security Validation Platform. https://www.picussecurity.com. Accessed: 2026-04. Rigaki, M., Lukáš, O., Catania, C.A., Garcia, S., 2023. Out of the cage: How stochastic parrots win in cyber security environments. arXiv preprint arXiv:2308.12086 . SafeBreach, . SafeBreach Breach and Attack Simulation. https://www.safe breach.com. Accessed: 2026-04. Saroui, R., Marchioro, T., Olivereau, A., 2025. Agentnirs: An llm-driven agent for network intrusion response, in: ACSAC 2025-Annual Computer Security Applications Conference Workshops. SCYTHE, . SCYTHE Adversary Emulation Platform. https://scythe.io. Accessed: 2026-04. SimSpace, . SimSpace Cyber Force Platform. https://www.simspace.com. Accessed: 2026-04. Sood, A.K., Zeadally, S., Hong, E., 2025. The paradigm of hallucinations in ai-driven cybersecurity systems: Understanding taxonomy, classification

outcomes, and mitigations. Computers and Electrical Engineering 124, 110307. Speicher, P., Steinmetz, M., Hoffmann, J., Backes, M., Künnemann, R., 2019. Towards automated network mitigation analysis, in: Proceedings of the 34th ACM/SIGAPP symposium on applied computing, pp. 1971– 1978. Stakhanova, N., Basu, S., Wong, J., 2007. A taxonomy of intrusion response systems. International Journal of Information and Computer Security 1, 169–184. Standen, M., Lucas, M., Bowman, D., Richer, T.J., Kim, J., Marriott, D., 2021. Cyborg: A gym for the development of autonomous cyber agents. arXiv preprint arXiv:2108.09118 . Stockpile, . MITRE Caldera plugin: Stockpile. https://github.com/mitre /stockpile. Team., M.D.R., 2021. Cyberbattlesim. https://github.com/microso ft/cyberbattlesim. Created by Christian Seifert, Michael Betser, William Blum, James Bono, Kate Farris, Emily Goren, Justin Grana, Kristian Holsheimer, Brandon Marken, Joshua Neil, Nicole Nichols, Jugal Parikh, Haoran Wei. Wang, C., et al., 2024a. Netconfeval: Can llms facilitate network configuration?, in: Proceedings of the 20th International Conference on emerging Networking EXperiments and Technologies (CoNEXT), ACM. pp. 467– 481. Wang, J., He, B., Zhao, J., Xuan, Y., Sun, H., Qi, Q., Liang, J., Zhuang, Z., Liao, J., 2026. Llm-powered intent-driven configuration generation for multi-vendor networks. IEEE Transactions on Network and Service Management . Wang, M., Dechene, R., 2024. Multi-agent actor-critics in autonomous cyber defense. arXiv preprint arXiv:2410.09134 . Wang, T., Xie, X., Zhang, L., Wang, C., Zhang, L., Cui, Y., 2024b. Shieldgpt: An llm-based framework for ddos mitigation, in: Proceedings of the 8th Asia-Pacific Workshop on Networking, pp. 108–114. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., et al., 2024. Autogen: Enabling next-gen llm applications via multi-agent conversations, in: First conference on language modeling. XMCyber, . XM Cyber Continuous Exposure Management. h t t p s : //www.xmcyber.com. Accessed: 2026-04. Yamin, M.M., Katt, B., Gkioulos, V., 2020. Cyber ranges and security testbeds: Scenarios, functions, tools and architecture. Computers & Security 88, 101636. Yan, Y., Zhang, Y., Huang, K., 2024. Depending on yourself when you should: Mentoring llm with rl agents to become the master in cybersecurity games. arXiv preprint arXiv:2403.17674 . Zhang, J., Bu, H., Wen, H., Liu, Y., Fei, H., Xi, R., Li, L., Yang, Y., Zhu, H., Meng, D., 2025. When llms meet cybersecurity: A systematic literature review. Cybersecurity 8, 55.

8. Appendix 8.1. Experimental Setup 8.1.1. Emulation Platform GNS3 version 2.2.57 for our network emulation environment.

8.1.2. Multi-Agent System AutoGen version 0.7.5 Wu, Bansal, Zhang, Wu, Li, Zhu, Jiang, Zhang, Zhang, Liu et al. (2024); AutoGen. 8.1.3. Language Model All agents were backed by OpenAI GPT-5.4 mini, snapshot version 2026-03-17, served via an Azure OpenAI Global Standard deployment. 14

Table 5 Per-mitigation cost by runtime condition (mean ± std). Tokens are summed across all agent-role LLM calls within a mitigation (which may include multiple implementation–critique iterations); duration is the wall-clock window from the first to the last context file in the mitigation directory; cost applies GPT-5.4-mini Azure pricing of $0.75 per 1M input tokens and $4.50 per 1M output tokens.

Table 4 Per-attack and overall pairwise condition comparisons on the strict joint MSR (attack disrupted and connectivity preserved). Each test is a Welch’s t-test on run-level mean MSR (one observation per run; 𝑛 = 59 / 𝑛 = 60 / 𝑛 = 60 for Multi-Agent / Multi-Agent without Critic / Single-Agent). Each cell shows the MSR difference (Δpp, improvement) and 𝑝-value, with Holm-Bonferroni correction across the three pairs (𝛼 = 0.05). Comparison Single-Agent → Multi-Agent without Critic Multi-Agent without Critic → Multi-Agent Single-Agent → Multi-Agent

DNS Exfiltration Lateral Movement Ransomware +12.0 (𝑝=0.038) +1.3 (𝑝=0.83) +13.3 (𝑝=0.0025)

+44.0 (𝑝<10−7 ) +4.7 (𝑝=0.42) +48.7 (𝑝<10−9 )

+24.0 (𝑝<10−4 ) +14.4 (𝑝=0.022) +38.4 (𝑝<10−6 )

Thief

Condition

Overall

+31.6 +27.9 (𝑝<10−4 ) (𝑝<10−14 ) +11.4 +7.9 (𝑝=0.13) (𝑝=0.036) +43.0 +35.8 (𝑝<10−6 ) (𝑝<10−19 )

Multi-Agent Multi-Agent without Critic Single-Agent

100

Mitigation Success Rate (%)

8.1.4. Azure Server Specifications Experiments ran on a Microsoft Azure Standard D8s v3 instance (8 vCPUs, 32 GiB RAM) running Ubuntu 24.04.2 LTS. 8.1.5. Caldera MITRE Caldera version 5.3.0. 8.1.6. Caldera Linux Agent A custom Docker container on gns3/ubuntu:noble running the Caldera Sandcat client. The agent reaches the Caldera C2 over HTTPS (port 8443) via a dedicated outof-band management interface, so C2 connectivity survives mitigations applied to the topology interface.

80

N

Input (k)

Output (k)

Duration (s)

Cost/mit ($)

591

609 ± 546

3.8 ± 1.4

124 ± 50

0.47 ± 0.41

279.72

600

273 ± 195

2.6 ± 0.4

96 ± 45

0.22 ± 0.15

130.04

604

109 ± 77

1.8 ± 0.3

60 ± 18

0.09 ± 0.06

54.43

Multi-Agent Multi-Agent without Critic Single-Agent Failed regression check

66%

60

60%

64%

59%

56% 55%

49% 42%

51% 41%

40

32%

20

27% 25% 15%

15%

18%

13%

11%

13%

Lateral Movement

Ransomware

0

37%

37%

27%

DNS Exfil

Total ($)

11% Thief

5%

Figure 9: MSR by runtime condition and attack scenario. Hatched bar: rate without the connectivity-regression check.

8.3. Data and Code Availability In the spirit of open science, we release the following artifacts at https://github.com/user32133/cohort and https://cohort-experiments-app.streamlit.app/: • Agent prompts for all roles (Suggester, Implementer, Critic, Judge, Summarizer, and the single-agent baseline). • GNS3 topology files and device configurations for the small, medium, and large environments. • Docker images for the containers used in the experiments, including images prepared but not used in the final evaluation. • Caldera adversaries (Thief, Ransomware, DNS Exfiltration, Lateral Movement) and the abilities that compose them, with ATT&CK mappings. • Conversation logs and post-mitigation operational reports from every run analyzed in this work. A demo video walking through the framework is available at https://youtu.be/8Lj6rXAOwM8. The multi-agent runtime is AutoGen (Section 8.1.2). The agent code and the GNS3 automation code cannot be placed in the public domain; however, organizations involved in this research are open to reviewing disclosure requests submitted through the corresponding author.

8.1.7. SSH Victim Container A second container used as the lateral-movement target by the Lateral_Movement and Ransomware adversaries. It runs OpenSSH on port 22 with password authentication for a non-root user with a deliberately weak password, and prepopulates the user’s home directory with sensitive files for collection and encryption. The entry point emits periodic ICMP/HTTP background traffic to simulate an active host. No Caldera agent is installed. 8.1.8. Firewall FortiGate version 7.0.14. 8.1.9. Switch Open vSwitch 3.3.7. 8.1.10. Router Cisco 3640 running IOS 12.4(25d).

8.2. Adversary Attack Steps Table 6 enumerates, in execution order, the Caldera abilities that make up each adversary, together with their MITRE ATT&CK tactic and technique identifiers. The final two steps in every adversary (LAN and internet connectivity checks) are shared validation abilities used to confirm that baseline network reachability is preserved after mitigation. The full ability definitions and downloadable Caldera adversary YAMLs are released alongside this paper as described in Section 8.3.

8.4. Network Topologies The medium enterprise network topology was inspired by Simple Network Layout by Gilbert Nims (GNS3 Marketplace).

15

Table 6 Caldera adversary steps for the four post-compromise scenarios. The final two rows in every adversary (LAN/Internet connectivity checks) are shared validation abilities. The Thief adversary is adapted from the Thief pack in the MITRE Caldera Stockpile plugin Stockpile; Harrell et al. (2023); the two final connectivity-validation steps are our addition. (a) Lateral_Movement adversary steps.

(b) Thief adversary steps.

#

Step

Tactic

Tech. ID

Description

#

Step

Tactic

Tech. ID

Description

1

Scan target for open SSH port

Discovery

T1046

nmap confirms port 22 is open on the lateral target

1

Create staging directory

Collection

T1074.001

Create a local directory to hold files for exfiltration

2

SSH password brute force

Credential Access

T1110.003

Spray a short password list against SSH until a credential succeeds

2

Find files

Collection

T1005

Locate files deemed sensitive on the compromised host

3

SSH lateral movement

Lateral Movement

T1021.004

Log into the remote host with the discovered credential

3

Stage sensitive files

Collection

T1074.001

Copy the discovered files into the staging directory

4

Exfiltration

T1560.001

Enumerate users, processes, and network config on the remote host

Compress staged directory

Archive the staging directory with a standard utility

5

Exfil staged directory

Exfiltration

T1041

Locate sensitive files on the remote host and copy them back via SCP

Upload the archive to the C2 server over the agent channel

6

LAN Connectivity Check (Ping)

Discovery

T1018

Validate that LAN reachability is preserved

7

Internet Connectivity Check (HTTP)

Discovery

T1016

Validate that outbound internet reachability is preserved

4

Remote host enumeration

Discovery

Collect files from remote host

Collection

6

Exfil remotely collected file to C2

Exfiltration

T1041

Upload the collected file to the Caldera C2 server

7

LAN Connectivity Check (Ping)

Discovery

T1018

Validate that LAN reachability is preserved

8

Internet Connectivity Check (HTTP)

Discovery

T1016

Validate that outbound internet reachability is preserved

5

T1082

T1005

(c) DNS_Exfil adversary steps.

(d) Ransomware adversary steps.

#

Step

Tactic

Tech. ID

Description

#

Step

Tactic

Tech. ID

Description

1

Find files

Collection

T1005

Locate files deemed sensitive on the compromised host

1

System Recon

Discovery

T1082

2

Encode sensitive data for DNS exfiltration

Defense Evasion

T1027

Base64-encode the contents of a discovered file into DNS-safe chunks

Gather basic system information before ransomware deployment

2

Find files

Collection

T1005

Locate files deemed sensitive for encryption

3

Exfiltrate data via DNS queries

Exfiltration

T1048.003

Send encoded chunks as subdomain lookups to bypass HTTP egress controls

3

Encrypt sensitive files

Impact

T1486

Generate a random AES key and encrypt the discovered files

4

Remove DNS exfil artifacts

Defense Evasion

T1070.004

Delete local staging files and clear shell history to cover tracks

4

Write ransom note

Impact

T1491.001

Drop a ransom note on the victim host

5

Plant cron persistence

Persistence

T1053.003

5

LAN Connectivity Check (Ping)

Discovery

T1018

Validate that LAN reachability is preserved

Install a cron job that beacons back to the C2 server

6

Exfil encryption key to C2

Exfiltration

T1041

6

Internet Connectivity Check (HTTP)

Discovery

T1016

Validate that outbound internet reachability is preserved

Upload the AES encryption key to the C2 server over HTTP

7

LAN Connectivity Check (Ping)

Discovery

T1018

Validate that LAN reachability is preserved

8

Internet Connectivity Check (HTTP)

Discovery

T1016

Validate that outbound internet reachability is preserved

16

Internet Attacker IP: 203.0.113.1

fortigate-firewall WAN: 7.0.14.1 LAN: 192.168.1.1

OpenvSwitch-1

eth4 PC1

e0 PC2

eth5

IP=192.168.1.3 DEFAULT_GATEWAY=192.168.1.1 UbuntuComputer

eth6

UbuntuSSHVictim

e0

e0 PC3

PC4

CALDERA-MGMT

Figure 10: Small enterprise network topology.

The large enterprise network topology was inspired by Enterprise Network Lab: Bank Project by Kiki Oyewole (GNS3 Marketplace).

8.5. Pairwise Statistical Comparisons Table 4 reports per-attack and overall pairwise condition comparisons referenced in Section 5.3. Each test is a Welch’s t-test on run-level mean MSR (one observation per run; n=60/ 60/ 59for Single-Agent / Multi-Agent without Critic / Multi-Agent), with Holm-Bonferroni correction across the three pairs.

8.6. MSR by Attack Scenario Figure 9 disaggregates the headline MSR (Figure 4) by attack scenario. The hatched bar shows the rate without the connectivity-regression check. Sample sizes range from 138 to 150 mitigations per cell. Multi-Agent MSR ranges from 27% on DNS Exfiltration to 60% on Lateral Movement; DNS Exfiltration’s lower rate is structural, reflecting that its egress channel (DNS) bypasses the network-based egress controls the suggester most often proposes.

8.7. Per-Mitigation Cost Table 5 reports per-mitigation token usage, duration, and dollar cost by runtime condition. Each mitigation may comprise multiple implementation–critique iterations; the row aggregates across all such iterations within the mitigation. Cost is computed at GPT-5.4-mini Azure list pricing ($0.75 per 1M input tokens, $4.50 per 1M output tokens); the Total column sums across all mitigations in the condition.

17

UbuntuSSHVictim-1

CALDERA_MGMT cisco-router-R1

PC4

cisco-router-R2 UbuntuComputer

Port0: 192.168.3.1 Port1: 192.168.1.3 Port2: 10.0.0.1 e0PC2

PC5

eth2 eth4

e0

Port0: 192.168.2.1 Port1: 192.168.4.2 Port2: 10.0.0.2 f0/0

OpenvSwitch-1 eth0

f0/0

f2/0

eth6 eth5

f2/0

f1/0

eth2 eth5 eth3 eth6 eth4

eth0

f1/0

IP:192.168.2.3 Gateway: 192.168.2.1 PC9

eth0

OpenvSwitch-2 eth1

e0 PC7 e0

e0

PC6

PC10

e0

PC11

e0 Port2

e0 PC3

Port3

e0 PC8

fortigate-firewall

Port1: 7.0.14.1 Port2: 192.168.1.1 Port3: 192.168.4.1

Port1

caldera-br Cloud1 Attack IP: 203.0.113.1

Figure 11: Medium enterprise network topology (inspired by Simple Network Layout by Gilbert Nims, GNS3 Marketplace).

Marketing - 192.168.10.0/24 Accountant - 192.168.20.0/24 Finance - 192.168.30.0/24 Management - 192.168.40.0/24 Research - 192.168.50.0/24 HR - 192.168.60.0/24 Customer - 192.168.70.0/24 Logistic - 192.168.80.0/24 ICT - 192.168.90.0/24 SwitchX - 192.168.100.0/24 ServerRoom - 192.168.110.0/24 Admin - 192.168.120.0/24

Marketing-PC-5

Marketing-PC-4

Marketing-PC-1

Marketing-PC-6

e0

e0

Accountant-PC-5

Accountant-PC-6

Accountant-PC-1 e0

Accountant-PC-2

Accountant-PC-3

All VPCs have access to the core router. Floor routers perform NAT. Internal communication between floors is routed through the floor router interfaces eth2 and eth3. No access from the core router to the internet.

e0

e5

Accountant-PC-4

*Static IPs* Marketing - 192.168.10.0/24 Accountant - 192.168.20.0/24 Finance - 192.168.30.0/24 Management - 192.168.40.0/24 Research - 192.168.50.0/24 HR - 192.168.60.0/24 Customer - 192.168.70.0/24 Logistic - 192.168.80.0/24 ICT - 192.168.90.0/24 SwitchX - 192.168.100.0/24 ServerRoom - 192.168.110.0/24 Admin - 192.168.120.0/24

Marketing-PC-3

Marketing-PC-2

e6 e1

vLan 110

Marketing-SW

FileServer

DVR

vLan 100

cameraB1 e0

e0

Finance-PC-1

Finance-PC-5

Finance-PC-2 e0

Server-Room-SW

Cloud1

e4 e1 Finance-PC-4

Accountant-SW

.2-.7 IP: 192.168.120.3 Gateway: 192.168.120.1 Admin-PC-2

e0 Attacker IP: 203.0.113.1

192.168.10.0/24

192.168.20.0/24

192.168.110.0/24

e0

SwitchX(PoE) e0

docker0

eth1 eth2

192.168.100.0/24

Finance-SW

e0

Finance-PC-6

UbuntuComputer CALDERA-MGMT

vLan 120

CameraA

e4 e5 Finance-PC-3 e6 e0

Floor1-L3SW

e0

192.168.30.0/24

10.2.1.0/30

NAT

Floor1-FW NAT

10.2.4.0/30

Floor4-Router e1

192.168.120.0/24

e4

e0

e0 Admin-PC-3

Admin-PC-4

e0

e6

eth3

trunk port

Admin-PC-5 e0

eth0

Floor4-FW 10.1.1.0/30

Admin-PC-6

Floor4-L3SW

10.1.4.0/30

CoreRouterB

e2

eth1

e0

Floor1-Router

Admin-SW

eth2

CoreRouterA

eth3

NAT

CoreRouterE

Routers perform inter-LAN communication using Router-on-a-stick technics.

e2 Manager-PC-4

Manager-PC-1

e2

10.0.4.0/30 10.3.12.0/30 10.0.1.0/30

CoreRouterC

e0 Manager-PC-5

Manager-PC-2e4 e5 e0 e6

Manager-PC-6

Management-SW

10.0.2.0/30

e2

10.1.2.0/30 10.1.3.0/30

10.2.2.0/30

192.168.40.0/24

ICT-PC-3

ICT-PC-4

ICT-PC-5

ICT-PC-6

ICT-SW

10.2.3.0/30

NAT

NAT

Floor3-Router

192.168.90.0/24

e0

10.3.23.0/30

e3

e3 ip: 10.layer.floor.host/30 Floor2-L3SW

Research-PC-1

Research-PC-4

ICT-PC-2

Floor3-FW

10.0.3.0/30

Floor2-FW

Floor2-Router e2

Manager-PC-3 e0

ICT-PC-1 10.3.34.0/30

CoreRouterD

Floor3-L3SW eth3

Upstream interfaces: WAN downstream interfaces: LAN

192.168.80.0/24

eth2 e0 Research-PC-5 Research-PC-2 e0 Research-PC-6

Research-SW e4 e5 e6

e0

Customer-SW

Research-PC-3

Logistics-PC-1

Logistics-PC-2

e0 e2 e4 Logistics-PC-3 e6 e0

Logistics-PC-4

Logistics-PC-5 e0

Logistics-PC-6

Logistics-SW

192.168.70.0/24

HR-SW 192.168.60.0/24

e0 HR-PC-1 HR-PC-2 e4 UbuntuSSHVictim-1 192.168.50.0/24

e0 HR-PC-4 HR-PC-5

Costumer-PC-1

e6 Costumer-PC-2

Costumer-PC-4

Costumer-PC-5

HR-PC-6

Costumer-PC-3

e0

Costumer-PC-6

Figure 12: Large enterprise network topology (inspired by Enterprise Network Lab: Bank Project by Kiki Oyewole, GNS3 Marketplace).

18