arXiv CSopen access

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response

2026-07-30

SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response Lehan Wang1,3 , Boli Chen1 , Ruixue Ding1 , Pengjun Xie1 , Jinwei Huang2 , Zhendong Liu2 , Shuo Wang2 , Tao Lei2 , Xin Ouyang2 , Xiaomeng Li3 1

Tongyi Lab, Alibaba Group 3

2

Alibaba Cloud Computing, Alibaba Group

The Hong Kong University of Science and Technology

arXiv:2607.26791v1 [cs.CR] 29 Jul 2026

Abstract Large Language Model (LLM) agents are increasingly adopted in real-world security operations with access to host artifacts and command-line interfaces (CLIs), making it critical to thoroughly assess their security capabilities. However, existing cybersecurity benchmarks focus on pre-compromise settings where agents are placed in a clean and idealized environment before an attack occurs. This leaves the post-compromise setting underexplored. To address this gap, we introduce S EC R ESPOND, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. Given a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, agents are required to produce forensic reports on intrusions, baseline risks, and vulnerability risks, together with a remediation plan. We instantiate this task across 10 cyber ranges, each constructed from a distinct compromised cloud host, spanning 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. We evaluate 23 frontier LLMs on the OpenCode agent harness. Experimental results show that although current agents can reliably uncover the problems exposed by alerts, they struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This reveals a fundamental bottleneck in building agents for real-world incident response. The benchmark is publicly available at https://github.com/Alibaba-NLP/qqr/tree/ma in/data/secrespond.

1

Introduction

Large Language Model (LLM) agents, equipped with iterative reasoning and autonomous tool-use abilities (OpenAI, 2026; Qwen, 2026; Z.ai, 2026), demonstrate remarkable potential in tackling complex, professional real-world tasks (Merrill et al., 2026; Ye et al., 2026; Li et al., 2026b), among which cybersecurity is a significant deployment target. With access to host artifacts and command-line interfaces (CLIs), LLM agents are increasingly expected to assist in real-world security operations from vulnerability discovery to incident response (Wang et al., 2025; Deason et al., 2025). Therefore, it is critical to assess how reliably current agents can fulfill such pipelines and how they can be involved in human workflows. Recently, several cybersecurity benchmarks have been developed to cover various offensive-security subtasks. Zhang et al. (2025) and Shao et al. (2024) assess the offensive capabilities of LLM agents with CTF challenges, while Zhu et al. (2025) and Wang et al. (2025) evaluate agents’ vulnerability discovery and exploitation abilities using real-world web applications or software projects. Recent efforts such as CyberModelArena (Wiz, 2026) have begun to assess harness-model combinations across the full attack lifecycle. In parallel, defensive benchmarks evaluate LLMs on security knowledge, vulnerability discovery, and patching. Jing et al. (2024) collects multiple-choice question-answer pairs to assess LLMs’ defensive knowledge, while Meta (2025) evaluates LLMs on vulnerability patching. However, these existing security benchmarks focus on pre-compromise settings, where the agent operates in a clean and idealized environment to find, exploit, or patch weaknesses before an attack occurs (Zhang et al., 2025; Wang et al., 2025). Even on the defensive side, existing works simplify the problem to reasoning over isolated alerts (Deason et al., 2025), system logs (Wu et al., 2025), or incident reports (Lin et al., 2025) rather than confronting the compromised system with real artifacts left by the attack, such as persistence mechanisms, deliberately erased traces, and noisy concurrent host activity. This leaves the post-compromise workflow underexplored, which includes investigating a successful intrusion on the compromised host, reconstructing the attack chain and remediating the system. As post-compromise response covers the full incident-response cycle, evaluating LLM agents on this pipeline measures 1

whether they can actively discover problems within the compromised disk and respond effectively to live alerts and real intrusion artifacts, thereby serving as the prerequisite to facilitating agent assistance for security operations in production. To address this gap, we introduce S EC R ESPOND, the first benchmark for evaluating LLM agents on the post-compromise incident-response workflow. The agent is given a forensic disk snapshot of a compromised host with the alerts, vulnerability scans, and baseline checks reported by a host security product. Given these inputs, the agent is required to investigate the host, reconstruct what happened, and produce one progress file and four reports covering the detected intrusion, vulnerabilities and baseline risks, along with a remediation plan to resolve them. S EC R ESPOND instantiates the above task in 10 cyber range scenarios, each built from a frozen snapshot of a distinct, fully instantiated cloud-host environment compromised through an end-to-end attack over real network protocols. Together, these ranges cover the diversity and complexity of real-world intrusion scenarios across 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. Moreover, we propose a hierarchical rubric that decomposes each range into detailed checkpoints that validate whether each problem is comprehensively discovered and effectively handled, and grade them with the LLM-as-a-Judge method. We further design a fivedimensional Capability (CAP) taxonomy including intrusion entity, persistence mechanism, baseline risk, vulnerability risk, and investigation & response quality, and map each checkpoint to its corresponding capability items. In this way, checkpoint results from heterogeneous ranges become comparable and can be aggregated through the capability taxonomy to benchmark model capabilities. In total, S EC R ESPOND defines 52 capability items and 280 checkpoints, reflecting fine-grained model performance in the full incident response workflow from investigation to remediation. We conduct extensive experiments on a representative agent harness, OpenCode (OpenCode, 2026), paired with 23 different large language models (LLMs), spanning multiple model families and successive releases within each family. Our experiments reveal that current agents can surface the problems exposed by alerts but struggle to investigate the forensic disk proactively and to complete the remediation required for incident response. Even the strongest model, Claude Opus 4.7, achieves only 72.4% averaged over detection and planning, leaving malicious artifacts untouched and remediation incomplete, especially in the ranges where the attack chain grows longer and broader. All the models score higher on detection than on planning, with the gap widening to 34.7% for GPT-5.5, since agents tend to apply the obvious first fix but rarely complete the remaining remediation. Furthermore, we observe a significant imbalance across capability dimensions. Average detection reaches 75.4% on intrusion entity but only 58.8% on persistence mechanism, whereas the average planning score peaks at 55.3% on baseline risk and reaches only 31.8% on investigation & response quality. This exposes a fundamental bottleneck in versatile incident-response models. Our contributions are as follows: • A benchmark for post-compromise incident response. We release S EC R ESPOND, the first benchmark targeting the full post-compromise response workflow, which comprises 10 cyber ranges built on fully instantiated cloud-host environments, covering 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. • A hierarchical capability evaluation framework. We propose a hierarchical LLM-as-a-Judge framework where each task is decomposed into fine-grained checkpoints manually designed by security experts. The resulting 280 checkpoints are graded along both detection and planning axes and mapped to 52 capability items in our five-dimensional Capability (CAP) taxonomy, supporting per-capability diagnosis and cross-range analysis. • An empirical study of frontier agents on real-world post-compromise tasks. We evaluate 23 LLMs on a representative agent harness, OpenCode, spanning multiple model families and successive releases within each family, with each task independently judged by three strong proprietary LLMs to mitigate individual bias. • Key findings on incident response capability of agent models. Our study shows that current agents reliably discover alerted problems but struggle with proactive investigation and complete remediation. We also observe that agent models’ incident response capabilities are significantly imbalanced across different dimensions.

2

Related Work

2.1

Agent Benchmarks

To comprehensively evaluate the capabilities of LLM agents from multiple aspects, many benchmarks have been developed. To validate agent models’ coding abilities, SWE-Bench (Jimenez et al., 2024) tests the

2

Table 1: Comparison with existing cybersecurity benchmarks. We compare S EC R ESPOND with representative cybersecurity benchmarks along the following aspects: Scope denotes the security task; Post-Compromise indicates whether the benchmark targets the real-world scenario after a host has already been attacked; Real Filesystem denotes whether the task is grounded in a host disk rather than synthesized logs or text; Multi-Step Analysis indicates whether solving the task requires multiple investigative steps; Cross-File Analysis summarizes whether the evidence should be correlated across multiple files; Rubric-Based Evaluation indicates whether the evaluation metrics use fine-grained rubrics; CLI-Compatible denotes whether the benchmark task can be executed through a command-line interface. Benchmark SecBench (Jing et al., 2024) NYU CTF Bench (Shao et al., 2024) CyberSecEval (Wan et al., 2024) CyBench (Zhang et al., 2025) CVE-Bench (Zhu et al., 2025) AutoPatchBench (Meta, 2025) CyberGym (Wang et al., 2025) CyberSOCEval (Deason et al., 2025) ExCyTIn-Bench (Wu et al., 2025) CyberModelArena (Wiz, 2026) S EC R ESPOND (Ours)

Scope

Post-Compromise

Real Filesystem

Multi-Step Analysis

Cross-File Analysis

Rubric-Based Evaluation

CLICompatible

Knowledge CTF Offensive CTF CVE Patch Vuln SOC Threat Detection Offensive

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗

✗ ✗ ✗ ✗ ✗ ✓ ✓ ✗ ✗ ✓

✗ ✓ ✓ ✓ ✓ ✓ ✓ ✗ ✓ ✓

✗ ✓ ✗ ✓ ✗ ✓ ✓ ✗ ✓ ✓

✗ ✗ ✓ ✗ ✗ ✗ ✗ ✗ ✗ ✗

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓

Forensic

✓

✓

✓

✓

✓

✓

model on resolving real GitHub issues, SWE-Bench-Pro (Deng et al., 2025) extends it to more realistic longhorizon problems that require examining multiple files and modifying substantial code, and TerminalBench (Merrill et al., 2026) targets interactive shell execution in a terminal environment. To evaluate agents in GUI environments, WebArena (Zhou et al., 2024) places agents in realistic web applications to conduct web navigation, while OSWorld (Xie et al., 2024) extends to real desktop environments where agents are supposed to perceive screenshots and operate applications. To benchmark agents in multi-turn conversational interaction, τ-bench (Yao et al., 2024) prompts agents to serve a simulated user in customerservice dialogues, τ 2 -bench (Barres et al.) establishes a dual-control environment in which the user can also modify the shared state, and Vitabench (He et al., 2025) scales the setting to versatile real-world applications including food delivery, in-store consumption, and online travel services. Recently, agentic harnesses such as OpenClaw and OpenCode have encouraged evaluation of agents operating within these scaffolds. CocoaBench (Team et al., 2026) evaluates agents’ compositional capability in vision, search, and coding on long-horizon tasks, ClawBench (Zhang et al., 2026) constructs daily tasks on live websites, and Claw-Eval (Ye et al., 2026) and Claw-Eval-Live (Li et al., 2026a) cover end-to-end workflows across service orchestration, multimodal interaction, and professional dialogue. Despite these broad evaluations, existing agent benchmarks target software engineering, web, and general productivity tasks, largely overlooking security operations. 2.2

Benchmarking LLMs in Cybersecurity

Evaluating cybersecurity capabilities of LLMs and LLM agents has drawn growing attention, typically from both offensive and defensive aspects. On the offensive side, early work (Li et al., 2024) assesses intrinsic security knowledge of LLMs with curated multiple-choice questions. These benchmarks focus on knowledge probing rather than automatic operational abilities. Subsequently, Cybench (Zhang et al., 2025) and NYU CTF Bench (Shao et al., 2024) place LLM agents inside Docker containers to evaluate their offensive abilities through interactive CTF challenge solving, and CyberGym (Wang et al., 2025) extends the scenario to vulnerability discovery and exploitation in real-world software projects, where LLM agents uncover zero-day bugs that human maintainers have missed. Recent work, CyberModelArena (Wiz, 2026), unifies the full attack lifecycle into a single benchmark of five categories and evaluates different combinations of agent designs and underlying models to analyze their independent and joint offensive capabilities. On the defensive side, SecBench (Jing et al., 2024) measures LLMs’ defensive knowledge in network, endpoint, application, and cloud security, while CyberSOCEval (Deason et al., 2025) evaluates LLMs from the perspective of Security Operations Center (SOC) analysts on understanding malware analysis and threat intelligence reports. AutoPatchBench (Meta, 2025) evaluates AI-assisted patch generation for repairing vulnerabilities. ExCyTIn-Bench (Wu et al., 2025) evaluates LLM agents on cyber threat investigation by reasoning over alerts and security logs. However, as shown in Table 1, none of the existing work reproduces actual end-to-end host intrusion that leaves forensic artifacts, including persistence mechanisms, partially cleaned traces, and noisy concurrent legitimate activity. As a result, there remains a gap in evaluating whether LLM agents can investigate a post-compromise disk snapshot and produce detection and remediation analysis.

3

Figure 1: Overview of S EC R ESPOND. In our benchmark task, the agent is expected to investigate a cyber range and produce incident-response reports. Each cyber range includes a forensic disk snapshot of a real, fully instantiated cloud host compromised through an end-to-end attack over real network protocols, together with security analytics comprising alerts and scan findings from a security product. To evaluate the agent’s incident-response capability, we organize the required skills into a capability taxonomy and map each capability to a list of checkpoints for each range. These checkpoints serve as evaluation rubrics for the LLM-as-a-Judge framework. 2.3

Evaluation Methodology

Cybersecurity benchmarks predominantly rely on deterministic verification with a fixed ground truth, such as multiple-choice accuracy (Li et al., 2024), CTF flag matching (Zhang et al., 2025), or verifying the agent’s execution output (Wang et al., 2025; Meta, 2025). Such protocols cannot be directly applied to our benchmark, where the task output is open-ended investigation reports. Therefore, we adopt the LLM-as-a-Judge method (Gu et al., 2026), which decouples the target into verifiable units, such as the rubric items for trajectory evidence in Claw-Eval (Ye et al., 2026), thus making it well-suited for open-ended tasks. To keep the judgment reliable and interpretable, we design a hierarchical LLM-as-aJudge framework that decomposes each task into fine-grained checkpoints, which are further mapped to capability items in our five-dimensional capability taxonomy, enabling fine-grained analysis within each range and capability-level comparison across ranges.

3

S EC R ESPOND Benchmark

S EC R ESPOND evaluates LLM agents’ ability to investigate a post-compromise host snapshot and produce comprehensive forensic reports on intrusion analysis, baseline assessment, and vulnerability verification, together with a structured remediation plan. In this section, we first describe the benchmark task and explain how we construct cyber ranges on real, fully instantiated cloud hosts across diverse attack types and operating systems (§3.1). Then, we define a capability taxonomy that specifies what the agent should be able to accomplish in the incident-response process (§3.2). Finally, we introduce the evaluation framework, which decomposes each range into fine-grained checkpoints that link concrete forensic evidence to capability items, enabling hierarchical, cross-range evaluation (§3.3). 3.1 3.1.1

Task Construction Overview of Tasks

To comprehensively evaluate the agents’ incident response abilities, we construct 10 cyber ranges covering multiple attack scenarios. A cyber range includes a frozen, reproducible, read-only forensic disk snapshot

4

Table 2: Overview of the cyber ranges. We construct 10 cyber ranges in total, spanning 4 entry points and 5 operating systems. Each range reproduces a complete multi-stage attack chain from initial access through privilege escalation, persistence, and impact, covering 21 ATT&CK techniques. Range

Entry type

OS

ATT&CK techniques

Attack chain

SSH-Miner

Baseline Weak

CentOS 7

T1110, T1021.004, T1059.004, T1552, T1053.003, T1546.004, T1543.002, T1548.003, T1496

SSH brute force → mining and persistence (crontab/bashrc/systemd)

ShiroFastjson

Known CVE

CentOS 8

T1190, T1059.004, T1505.003, T1548.003, T1098.004, T1053.003, T1543.002, T1496

Shiro default key + Fastjson → webshell → privilege escalation → mining

Log4j-RCE

Known CVE

CentOS 8

T1190, T1059.006, T1505.003, T1548.003, T1098.004, T1053.003, T1546.004, T1543.002, T1556, T1070, T1496

Log4j RCE → webshell + persistence

DockerEscape

Baseline Weak

Ubuntu 20.04

T1190, T1611, T1059.004, T1098.004, T1053.003, T1546.004, T1543.002, T1071.001, T1496

Exposed Docker API → container escape → host takeover

Redis-RCE

Baseline Weak

Ubuntu 22.04

T1190, T1021.004, T1059.004, T1552, T1098.004, T1053.003, T1546.004, T1496

Unauthenticated Redis → SSH-key write → mining

JenkinsRCE

Business code

Ubuntu 22.04

T1190, T1059.004, T1548.001, T1548.003, T1574.006, T1053.003, T1071.001, T1496

Jenkins Script Console → command execution → mining

Next.jsRCE

Known CVE

Ubuntu 22.04

T1190, T1059.007, T1505.003, T1548.001, T1574.006, T1098.004, T1053.003, T1546.004, T1543.002, T1496

Next.js CVE → SUID privilege escalation → LD_PRELOAD rootkit

NPMWorm

Supply chain

Ubuntu 22.04

T1195.002, T1059.007, T1059.006, T1552, T1098.004, T1053.003, T1543.002, T1071.001, T1496

Malicious npm package → worm propagation + credential theft

ASP.NETViewState

Known CVE

Windows Server

T1190, T1505.003, T1552, T1071.001

ASP.NET ViewState → RCE → MSSQL backdoor + WMI persistence

RDPServiceAbuse

Baseline Weak

Windows Server

T1110, T1552, T1071.001

Exposed RDP → password spraying → weak service DACL abuse → service restart for SYSTEM payload + disguised service & scheduled task + credential dump and C2

of a host that has been compromised by a complete, end-to-end attack delivered over real network protocols, along with a set of security analytics including real-time alerts and scan findings from a security product. The agent is expected to investigate the exposed alerts and the snapshot, to reconstruct the intrusion from the traces the attack leaves behind. These ranges cover different operating systems, entry points, ATT&CK techniques, and attack stages, which are selected to mimic the distribution in actual incidents based on a correlation analysis of alerts, vulnerabilities, and baselines from 372 real compromised cloud hosts. In detail, the ranges are summarized in Table 2. The forensic disk snapshot typically covers all the directories of the compromised host, and contains the naturally generated intrusion artifacts left by the attack which the agent can read as the primary forensic evidence, including logs, configuration changes, and residual files. Beyond the disk snapshot, the agent also receives three types of security analytics from the host security product on the victim machine: (i) real-time alerts from the anomaly detection engine monitoring suspicious processes, network connections, and login events, (ii) vulnerability findings from scans of installed packages and applications, and (iii) baseline check results from configuration checks. Given these inputs, the agent is required to produce one progress file and four reports: an intrusion report reconstructing the attack chain covering entry point, lateral movement, persistence mechanisms, and final impact, a vulnerability report verifying which exploited weaknesses are present on the host, a baseline report assessing host configuration against security baselines, and a remediation plan with recommended steps and reference commands for executing the fixes. The overview of S EC R ESPOND is demonstrated in Figure 1. 3.1.2

Design Principles

As described above, each task is centered on a cyber range that the agent should investigate. To reproduce the operational properties of real incidents, we build each range as three decoupled layers. The blueprint layer specifies the evaluation purpose of the scenario, including the targeted stack, the attack chain

5

Figure 2: Construction Pipeline of Cyber Range. The construction pipeline comprises 12 stages, with human experts involved in key steps to audit and validate the results. annotated with MITRE ATT&CK technique IDs, and the capability items the scenario aims to cover. The instance layer realizes the blueprint on a victim host under three constraints: (i) real vulnerabilities in the form of disclosed CVEs or configuration flaws rather than synthetic bugs, (ii) real attacks delivered through network protocols rather than directly placing malicious files, and (iii) real intrusion traces produced naturally by the attack. The checklist layer converts the resulting artifacts into checkpoints, each with a pass/fail criterion, an evidence source, and a capability mapping. 3.1.3

Construction Pipeline.

The above three layers are realized by a 12-stage construction pipeline shown in Figure 2: blueprint design, attack script development, victim instance curation, victim environment deployment, victim readiness verification, pre-attack clean snapshot, attack environment deployment, attack execution, attack verification, security data collection, remediation validation, and post-attack snapshot. Among these stages, human experts are involved in auditing the plausibility of the attack chain and capability coverage of the blueprint, reviewing the code quality and attack authenticity of the attack scripts, verifying that alert coverage meets the standard during data collection, and checking the completeness of the final range. To distinguish alert-driven shallow analysis from deep forensic capability in each range, we include 30–40% of attack actions that can trigger security-product alerts, while the remaining 60–70% leave silent file modification and configuration artifacts. The alerts provide anchor points to begin reasoning, while the hidden evidence on disk forces investigation and discovery. These analytics are collected through the security platform API after the attack and provided as static files. 3.2

Capability Taxonomy

To benchmark agent models’ incident response capability, we organize the required capabilities to investigate and remediate a compromised host into a five-dimensional Capability (CAP) taxonomy. • Intrusion Entity (ENT) denotes the ability to locate and handle the artifacts left by an intrusion, spanning malicious processes, malicious or tampered files, and malicious network entities. • Persistence Mechanism (PER) indicates the ability to recover the mechanisms an attacker installs to survive reboot and cleanup, including scheduled tasks, services, shell and environment initialization, account and privilege backdoors, web-component tampering, database backdoors, and kernel- or loader-level techniques. • Baseline Risk (BAS) is the ability to examine host configuration against security baselines, covering SSH hardening, database and middleware access control, credential safety, privilege and service-permission auditing, container and orchestration configuration, web-service configuration, and cloud-resource configuration. • Vulnerability Risk (VUL) represents the ability to confirm the weaknesses on the host, spanning Java component vulnerabilities, web-application code vulnerabilities, container-escape vulnerabilities, and front-end framework vulnerabilities. • Investigation & Response Quality (Q) reflects the ability to produce a thorough, well-evidenced, and trustworthy investigation, covering entry-point localization, attack-chain reconstruction, attacker-information extraction, honesty, investigation comprehensiveness, cross-language and cross-service tracing, remediation completeness, and business-impact assessment. In total, the taxonomy comprises 52 items across five dimensions as shown in Table 3, including 12 in ENT, 21 in PER, 7 in BAS, 4 in VUL, and 8 in Q. These capability items represent the benchmark target and form the foundation to build compromised ranges.

6

Table 3: The S EC R ESPOND evaluation taxonomy with 52 capability items across five CAP dimensions. Dimension

Intrusion Entity

Persistence Mechanism

Baseline Risk Vulnerability Risk Investigation & Response Quality

3.3

Capability Items ENT-F01 Webshell file; ENT-F02 Linux malware file; ENT-F03 Malicious SO / kernel module; ENT-F04 Residual data file; ENT-F05 Tampered-file restoration; ENT-F06 Windows malware file; ENT-N01 Attacker-IP inbound block; ENT-N02 Malicious-IP outbound block; ENT-N03 Malicious-domain block; ENT-P01 Mining process; ENT-P02 C2 beacon process; ENT-P03 Windows malicious process PER-A01 authorized_keys tampering; PER-A02 sudoers implant; PER-A03 Rogue local account; PER-D01 MySQL backdoor; PER-D02 MSSQL backdoor; PER-E01 profile.d / udev rule; PER-E02 WMI event subscription; PER-H01 ld.so.preload hijack; PER-H02 DLL hijacking; PER-I01 Shell init (bashrc/profile); PER-I02 /etc/environment injection; PER-M01 SUID/SGID backdoor; PER-S01 Cron task; PER-S02 systemd timer; PER-S03 at job; PER-S04 Windows scheduled task; PER-V01 systemd service; PER-V02 init.d / rc.local; PER-V03 Windows service; PER-W01 Nginx config tampering; PER-W02 App-container config tampering BAS-01 SSH hardening; BAS-02 DB / middleware access control; BAS-03 Credential safety; BAS-04 Privilege / service audit; BAS-05 Container / orchestration config; BAS-06 Web-service config; BAS-07 Cloud-resource config VUL-01 Java component vuln; VUL-02 Web-app code vuln; VUL-03 Container-escape vuln; VUL-04 Node.js / front-end framework vuln Q-01 Entry-point localization; Q-02 Attack-chain reconstruction; Q-03 Attacker-info extraction; Q-04 Honesty & confidence calibration; Q-05 Investigation thoroughness; Q-06 Cross-language/service tracing; Q-07 Remediation-verification completeness; Q-08 Business-impact assessment

Checkpoint Decomposition

To evaluate the capabilities above, we decompose them into fine-grained checkpoints within each range to measure whether the corresponding capability is fulfilled in the specific scenario. A checkpoint (CHK) is the smallest scoring unit, specifying the evidence required to discover a finding in a specific range. Each checkpoint matches one or more items in the CAP taxonomy, and multiple checkpoints may be assigned to the same CAP. The capability taxonomy is operationalized through 280 checkpoints in total, as detailed in Table 6. We first define CHK-score, which grades each CHK along two axes, detection and planning. Detection measures whether the agent correctly uncovers problems on the compromised host, including intrusion entities, baseline risks, and vulnerability risks. Planning validates whether the agent proposes a correct and complete remediation plan that resolves the existing problems. The detection axis has a total score of 3, comprising discovery of the problem, sufficiency of the supporting evidence, and correctness of attribution to the root cause. The planning axis has a maximum score of 2, comprising correctness of the proposed remediation step and completeness of the plan to fully resolve the attack artifacts. The normalized score of a single checkpoint on axis a is defined as the CHK-score, which represents the agent model’s performance at the checkpoint level: CHK-scoreca =

sca ∈ [0, 1], Ma

a ∈ {det, plan}

(1)

where Mdet = 3 and Mplan = 2 are the maximum scores of the two axes, and sca ∈ {0, 1, . . . , M a } is the score assigned to checkpoint c on axis a. Since each checkpoint is anchored to a capability item, checkpoint results from different ranges become comparable. We aggregate the checkpoints mapped to the same capability item into a capability-level score, which we define as the CAP-score. The CAP-score on a given axis is the ratio of the achieved scores to the maximum scores over all checkpoints mapped to the corresponding capability, computed as: CAP-scorea =

1

∑ CHK-scoreca × 100%

a CCAP a c∈CCAP

a where CCAP is the set of checkpoints mapped to the capability item.

7

(2)

Table 4: The range-level CHK-scorera of Detection and Planning for each range. “Det” denotes CHK-scorerdet , the percentage of achieved detection scores over all its checkpoints, averaged across plan three LLM judges; “Plan” denotes the corresponding CHK-scorer on the planning dimension. Model Claude Opus 4.7† Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

SSH-Miner Redis-RCE Docker-Escape Jenkins-RCE Shiro-Fastjson Log4j-RCE Next.js-RCE NPM-Worm

ASP.NET- RDP-SerViewState vice-Abuse

Overall

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det Plan Det

Plan

Det Plan

77.8 75.3 72.5 75.8 77.8 67.6 65.2 57.5 61.3 66.7 58.5 75.4 75.8 76.3 60.9 75.8 70.5 70.5 65.7 60.4 71.5 69.1 63.3

83.3 48.0 61.7 44.1 76.5 27.4 27.4 25.5 21.5 38.3 25.5 58.8 56.9 61.8 53.0 50.0 36.2 54.9 51.0 34.3 61.8 36.3 38.3

82.4 79.6 70.4 68.5 86.1 78.2 75.9 57.4 59.7 55.1 62.5 80.1 60.2 67.1 65.7 87.5 86.6 66.2 66.2 63.0 72.7 62.0 56.5

69.6 60.8 55.9 61.8 75.6 49.0 47.1 34.3 43.1 38.2 34.3 62.8 49.0 56.9 48.1 76.5 71.6 52.9 58.8 27.4 64.8 51.9 30.4

90.8 92.1 79.2 83.3 84.2 68.0 66.7 69.4 54.2 60.7 57.4 91.2 75.5 75.9 65.3 79.2 75.9 77.3 61.6 88.4 75.9 74.5 61.6

75.5 62.3 57.0 63.2 78.1 35.1 41.2 20.2 34.2 43.0 39.5 86.0 51.8 50.0 37.8 64.0 57.0 64.1 46.5 79.0 59.7 56.2 48.2

78.9 66.7 71.9 67.2 62.6 73.7 69.0 55.0 73.1 60.2 56.1 71.4 50.3 63.2 50.9 72.5 71.9 64.3 67.8 76.6 55.6 63.7 47.4

70.2 60.6 62.3 59.7 55.3 33.4 19.4 29.0 39.4 39.5 32.5 53.5 34.3 59.7 47.4 51.7 69.3 32.5 49.2 53.5 43.0 49.1 32.5

71.6 66.7 69.6 76.8 73.8 63.9 80.5 56.3 65.9 43.0 50.1 82.5 49.4 73.1 68.9 70.4 82.0 60.0 43.5 53.1 51.6 55.1 38.5

52.5 43.2 55.2 60.9 71.9 27.0 63.0 38.5 36.4 28.7 27.6 63.5 42.7 55.2 47.9 46.4 70.8 44.3 32.3 40.6 46.4 41.6 22.9

88.4 88.4 85.9 87.9 78.3 96.5 91.9 85.8 88.4 72.2 80.8 90.4 88.4 87.9 80.3 90.4 88.9 84.8 82.8 84.9 88.9 76.8 76.3

78.7 68.5 71.3 81.5 39.8 33.3 42.6 41.7 44.5 32.4 51.9 64.8 64.8 49.1 68.6 60.1 75.0 54.6 57.4 55.6 69.4 38.0 45.4

77.8 76.5 66.7 71.6 65.7 58.5 65.7 51.0 71.9 46.4 57.5 93.1 76.5 75.2 58.5 82.4 70.9 73.2 64.7 63.4 63.1 66.4 57.8

60.1 64.9 60.8 70.2 69.1 40.4 44.7 36.9 44.0 32.1 31.5 61.9 62.4 57.1 40.5 63.1 61.9 67.9 55.4 31.5 53.6 45.2 42.3

— 81.5 69.8 68.7 41.0 55.8 65.5 57.8 51.3 43.0 41.9 75.8 50.4 80.1 36.2 84.9 73.5 65.0 61.8 52.4 43.3 47.6 31.9

— 60.9 65.4 53.8 49.4 41.0 37.2 32.7 34.6 11.5 26.3 60.9 31.4 64.1 21.1 48.7 65.5 38.4 37.2 36.5 35.9 32.0 23.1

83.3 77.8 56.1 58.1 73.8 81.3 85.9 67.2 56.6 47.5 49.0 61.6 67.2 71.2 41.9 84.9 81.8 68.2 70.2 64.7 46.0 42.9 46.5

63.9 61.1 47.2 66.7 54.7 53.7 52.8 47.2 50.9 28.7 42.6 52.8 38.9 47.2 54.6 41.7 45.3 41.7 31.5 38.0 49.1 35.2 33.3

79.0 78.2 69.5 74.7 69.7 70.7 72.5 60.7 66.2 54.8 57.4 76.3 64.9 73.2 59.2 78.0 75.6 68.4 63.5 65.8 62.1 60.1 52.4

37.9 49.3 27.3 29.5 41.7 19.7 31.9 26.5 22.0 22.7 17.4 27.3 26.5 32.6 16.7 40.9 35.6 31.8 21.2 33.3 23.5 14.4 12.9

59.9 77.8 53.1 89.5 53.7 63.6 58.6 49.4 79.6 53.1 60.5 42.0 54.9 62.3 63.6 51.8 53.7 54.9 51.2 51.3 52.5 42.6 43.8

65.7 58.0 56.4 59.1 61.2 36.0 40.7 33.2 37.1 31.5 32.9 59.2 45.9 53.4 43.6 54.3 58.8 48.3 44.0 43.0 50.7 40.0 32.9

† Claude Opus 4.7 returned a safety refusal on every NPM-Worm attempt and produced no report; its overall score is therefore computed as the average over the remaining nine ranges.

4

Experiments

4.1

Experimental Settings

We evaluate 23 large language models spanning 8 model series and their successive versions, with the full list reported in Table 4. We include multiple versions of the same series to track how incident response capability evolves across releases. All models are driven by the same agent harness, OpenCode (OpenCode, 2026), and receive identical task inputs (§3.1.1), instruction prompt (see Appendix C.1), and tool set to ensure a fair comparison. Each model is accessed using default parameters with reasoning enabled. 4.2

Evaluation Metrics

Following the LLM-as-a-Judge paradigm, we score each generated report against the range checklist defined in §3.3. To reduce single-judge bias, we adopt three strong proprietary models, Claude Opus 4.7 (Anthropic, 2026), Gemini 3.1 Pro (The Gemini Team, 2026), and GPT-5.4 Pro (OpenAI, 2026), as independent judges. Each judge runs in the same OpenCode harness, receives the instruction prompt (see Appendix C.3) and the checkpoint list with criteria, and uses a bash tool set to read the reports produced by the assessed model. Each judge then assigns a score for every checkpoint, and the CHK-scoreca (Eq. 1) for each checkpoint is averaged over the three judges. We then aggregate the checkpoint-level CHK-scores to the range level. For each range r and axis a ∈ {det, plan}, the range-level CHK-scorera is the mean value of the achieved CHK-scoreca over the range’s checkpoints: CHK-scorera =

∑a CHK-scoreca

c∈Cr

|Cra |

× 100%,

(3)

where Cra is the set of checkpoints in range r for axis a. We report the range-level CHK-score (Eq. 3) for range-specific analysis, and the CAP-score (Eq. 2) for range-aggregated analysis. 4.3

Range-Level Analysis across Attack Scenarios

Through an in-depth analysis of the agent models’ performance across different attack scenarios as demonstrated in Table 4, we derive five primary research findings. Finding 1: Detection consistently outperforms planning, which is the bottleneck across all ranges. All the models achieve a higher range-level CHK-score on Detection than on Planning, as demonstrated in Figure 3 (a). For example, GPT-5.5 scores 70.7% on detection but only 36% on planning, while Claude Sonnet 4.5 presents the narrowest gap of 8.5%. This implies that for a large portion of checkpoints the agent models can successfully detect the problems from the disk snapshot, but fail to provide a comprehensive approach to fixing them. This gap is largely attributed to incomplete fixes instead of 8

Figure 3: (a) Detection consistently outperforms Planning across all models. The x-axis denotes the Detection score and the y-axis presents the Planning score. The dashed line marks “Detection=Planning”. All models fall below the diagonal, demonstrating that detection always exceeds planning. (b) Version evolution within each model series. Most model families improve across versions, but the progress is not consistent. wrong actions. The agent models often advise correct commands initially, such as terminating a malicious process or removing a dropped file. However, in the follow-up steps, they fail to carry out the remaining remediation, such as rotating the leaked credentials, blocking the outbound channel, or confirming that the business service stays healthy after the cleanup. This suggests that proficiency in detection cannot ensure success in handling an incident thoroughly. In conclusion, current agents are better at finding attack traces or vulnerabilities than producing correct, complete, and verified remediation suggestions. Finding 2: Model performance varies widely, with only a few models leading consistently across attack scenarios and demonstrating superior forensic and incident-response capabilities. Claude Opus 4.7 achieves the highest average range-level CHK-score across ranges (79.0% for detection and 65.7% for planning), followed by Claude Opus 4.6 (78.2%/58.0%), GLM-5.1 (76.3%/59.2%), and Qwen3.7 Plus (75.6%/58.8%), whereas Gemini 3.1 Pro, MiniMax M2.5, and Gemini 3 Flash present the lowest results. Open models such as GLM and DeepSeek outperform the GPT and Gemini series, indicating that proprietary models do not guarantee stronger incident-response capabilities. Overall, performance on this task is highly uneven across models, making model selection an important factor in security incident-response applications. Finding 3: Performance generally improves as models evolve across versions, but the improvement is not universal. As shown in Figure 3 (b), several model families improve steadily across successive releases. For instance, the CHK-scores of the Claude Opus series increase across three releases, from 69.5%/56.4% for Opus 4.5 to 78.2%/58.0% for Opus 4.6 and 79.0%/65.7% for Opus 4.7. Similarly, GLM-5.1 surpasses GLM-5 by 11.4%/13.3% (76.3%/59.2% vs. 64.9%/45.9%), DeepSeek V4 Pro improves over DeepSeek V3.2 by 14.0%/9.8% (73.2%/53.4% vs. 59.2%/43.6%), and Qwen3.7 Plus outperforms Qwen3.6 Plus by 7.2%/10.5% (75.6%/58.8% vs. 68.4%/48.3%). However, this trend is not universal. For example, Kimi K2.6 falls below K2.5 on the planning dimension (43.0% vs. 50.7%), and Gemini 3.1 Pro also underperforms Gemini 3 Flash (54.8%/31.5% vs. 57.4%/32.9%). Finding 4: Range difficulty varies, and model performance declines as the attack chain grows longer and broader. No model achieves complete detection and remediation on any single range. The overall CHK-score across models on each range reveals a clear difficulty stratification that aligns with the structural complexity of each scenario. Agent models generally show strong performance on Log4j-RCE, Docker-Escape, and Redis-RCE ranges with a single entry point, a linear attack chain, and common persistence exposed in alerts such as cron jobs or systemd services. Log4j-RCE is the easiest range because the included CVE is well-known and already familiar to the models. Ranges like SSH-Miner, Next.js-RCE, and Jenkins-RCE additionally include a wider host baseline, multi-step privilege escalation, cross-service correlation and more diverse persistence, leading to a decrease in both detection and planning scores. The hardest ranges are Shiro-Fastjson, RDP-Service-Abuse, NPM-Worm, and ASP.NET-ViewState, which combine a broad attack surface, cross-service or cross-runtime correlation, and deliberately disguised persistence. The complex attack traces in these ranges make it difficult for agent models to uncover potential issues and especially challenging to propose comprehensive remediation methods. Finding 5: Agent models inherently struggle with proactive detection of silent intrusions and generating complete, verified remediation plans. Across all ranges, all the agent models are typically confronted with two primary weaknesses: failure to uncover silent intrusions beyond the exposed attack

9

Table 5: The CAP-scorea of agent models and the traditional agentless scanning method across the five capability dimensions, namely Intrusion Entity (ENT), Persistence Mechanism (PER), Baseline Risk (BAS), Vulnerability Risk (VUL), and Investigation & Response Quality (Q). For each dimension, “Detect” denotes CAP-scoredet , the percentage of achieved detection scores over all checkpoints mapped to that capability dimension (following the mapping in Table 6), averaged across three LLM judges; “Plan” denotes CAP-score plan , the corresponding percentage on the planning axis. ENT

Model

PER

BAS

VUL

Q

Detect Plan Detect Plan Detect Plan Detect Plan Detect Plan Agentless Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

43.6 85.4 86.0 76.5 81.6 72.8 78.4 78.3 69.9 73.8 65.2 66.5 84.1 67.8 78.1 69.1 84.1 88.4 77.1 80.2 72.6 72.0 66.0 61.5

— 61.9 58.1 60.8 62.7 58.4 31.1 38.3 29.2 36.0 33.8 35.3 61.8 42.1 51.7 42.9 52.4 62.3 47.1 44.6 43.9 52.4 34.1 33.3

2.1 67.8 72.1 56.9 58.7 56.1 71.3 76.4 47.6 67.6 48.7 47.8 68.4 54.2 69.0 47.2 79.0 66.7 57.5 49.5 57.9 49.5 43.7 38.4

— 48.7 45.8 44.0 44.9 44.1 28.8 35.2 24.4 39.0 24.1 23.5 46.7 36.6 45.8 30.1 46.7 49.4 32.2 31.5 33.9 36.0 30.3 22.0

20.7 80.8 78.6 75.8 84.2 66.2 63.6 73.5 61.8 62.4 52.1 58.5 81.1 67.8 76.1 57.7 80.9 82.4 74.7 61.9 66.7 57.7 68.6 55.2

— 74.8 67.6 63.5 69.8 70.8 46.8 51.9 46.4 44.2 34.5 36.2 65.7 54.1 59.4 50.0 63.7 68.9 62.6 49.0 48.6 52.9 51.2 38.7

50.0 76.2 79.2 72.6 79.2 68.1 61.5 61.5 53.3 42.2 37.0 62.2 78.5 68.9 86.7 69.6 62.2 68.9 54.8 47.4 55.5 65.9 63.7 40.0

— 72.6 63.3 54.4 67.8 66.7 40.0 45.6 37.8 17.8 31.1 43.3 71.1 58.8 62.2 54.5 47.8 60.0 52.3 44.4 41.1 60.0 38.9 35.5

— 73.6 68.8 62.4 66.7 71.6 61.4 64.0 55.2 62.9 43.8 45.4 75.5 56.5 66.2 54.8 69.1 64.6 58.0 52.0 55.7 54.9 52.8 42.8

— 53.6 34.4 33.3 34.4 66.7 24.4 35.5 15.5 27.8 12.2 12.2 41.1 25.5 42.2 36.7 32.2 34.4 24.5 30.0 26.7 42.3 32.3 14.4

traces and to suggest thorough and verified remediation. This represents a common capability ceiling, with detailed evidence for each specific scenario provided in Appendix D.1. On the detection side, the agent models can find the issues directly reflected by the exposed alerts, but fail to discover the hidden problems beyond the obvious trace. Silent intrusions that leave no running process or network footprint are frequently overlooked, and the agent models rarely scan the whole host proactively. This shows that they tend to respond to the given clues instead of actively investigating the system. On the planning side, the agent models usually propose the most obvious fix rather than completing the comprehensive remediation. The remediation plans often stop at removing the visible artifact and leave the secondary risks or undetected persistence mechanisms untouched. More importantly, the models rarely recommend verifying whether the proposed fixes truly work or whether the normal services still run after the cleanup, leaving the remediation partial and unverified. 4.4

Capability-Level Analysis

Following our capability taxonomy described in § 3.2, we aggregate CHK-score into CAP-score using the mapping between range checkpoints and capability items (see Table 6). The aggregated results are shown in Table 5, from which we derive the following findings. Finding 1: Models reliably identify intrusion entities but struggle to uncover persistence mechanisms. As observed in Table 5, models are strongest at detecting Intrusion Entity (ENT), where most models exceed a 70% CAP-score. Leading models such as Qwen3.7 Plus and Claude Opus 4.6 achieve 88.4% and 86.0%, respectively. The strength in ENT can be attributed to the nature of malicious entities, which are concrete objects that trigger visible alerts. Models can discover them through routine inspection and by tracing the alerted trail. By contrast, detection collapses on Persistence Mechanism (PER), which has the lowest average detection CAP-score across models. This is because persistence mechanisms can be left in less obvious locations with no active signal, including cron jobs, systemd units, shell-init files, and platform-specific hooks. Uncovering them requires the model to enumerate the whole host rather than simply follow a visible trace. Thus, the ability to conduct a more proactive and systematic investigation represents one of the primary directions for improvement. Finding 2: Models remediate standardized risks effectively but perform weakly on intrusion entity and persistence mechanism cleanup. In terms of planning, models demonstrate superior CAP-scores

10

Figure 4: The CAP-score for each capability (ENT, PER, BAS, VUL, Q) of the representative models. In detection, Qwen3.7 Plus is strongest at entity identification, Claude Sonnet 4.6 at baseline assessment, Qwen3.7 Max at persistence detection, DeepSeek V4 Pro at vulnerability verification, and GLM-5.1 at investigation quality. In planning, Claude Opus 4.7 shows the strongest capabilities on average. when remediating Baseline Risk (BAS) or Vulnerability Risk (VUL), with Claude Opus 4.7 reaching 74.8% on BAS and 72.6% on VUL. These two dimensions are the easiest for remediation planning because their remediation is standardized, such as patching the affected component, disabling an exposed service, or rotating a default credential. Interestingly, models are strongest at detecting ENT but remediate it less successfully. This is because uncovering a malicious entity may require only a single observation, whereas cleaning it up involves a more complex, multi-step process. In particular, models’ capability to address PER remains relatively weak in remediation planning because the low detection score implies that many persistence mechanisms are never discovered and therefore cannot be properly remediated. Finding 3: Investigation & Response Quality is consistently among the weakest capabilities across models in both detection and planning, reflecting inadequate and unverified incident handling. The Investigation & Response Quality dimension validates process-level completeness and soundness of the whole incident response procedure, covering entry point localization, attack-chain reconstruction, honesty and confidence calibration, investigation thoroughness, remediation verification completeness, and business impact assessment. The CAP-scores on the quality dimension are low on both the detection and planning axes. Even the highest detection result reaches only 75.5%, falling behind other dimensions, and almost all models fall below 50% on planning except Claude Opus 4.7 and Claude Sonnet 4.5. From the detection perspective, even when models detect a concrete artifact, they still fail to reconstruct the full attack chain because of the challenge of correlating evidence across services and time. In terms of planning, models are inclined to address the immediate problem rather than handle the incident thoroughly, rarely verifying the effectiveness and completeness of the proposed remediation. Finding 4: Different models excel at different capabilities, but none of them dominates across all five dimensions. Figure 4 shows that the best detection model differs for every single dimension. Qwen3.7 Plus is strongest at entity identification with 88.4% on ENT, Qwen3.7 Max at persistence detection with 79.0% on PER, Claude Sonnet 4.6 at baseline assessment with 84.2% on BAS, DeepSeek V4 Pro at vulnerability verification with 86.7% on VUL, and GLM-5.1 at investigation quality with 75.5% on Q. In terms of planning, Claude models achieve the best CAP-score in four of the five dimensions, namely Opus 4.7 on BAS (74.8%) and VUL (72.6%), Sonnet 4.6 on ENT (62.7%), and Sonnet 4.5 on Q (66.7%), while Qwen3.7 Plus reaches the highest planning score of 49.4% on PER. This highlights that there is no universally best model and that model selection for incident response should be driven by the capabilities required for deployment rather than by a single overall ranking. Finding 5: Agents substantially outperform the traditional agentless detection baseline on average, and the gap is largest on detecting persistence mechanism. We also compare the agent models against an agentless baseline1 used in the traditional incident-response workflow, which is a production scanner that inspects a host’s disk snapshot without installing any runtime agent. Given the same host disk image, the scanner performs detection purely by static pattern matching driven by a predefined knowledge base, and reports four categories of findings after scanning. Concretely, it flags malicious samples by hash and signature matching against a malware database, identifies software vulnerabilities by 1 https://help.aliyun.com/zh/security-center/user-guide/use-the-agentless-detection-feature

11

Figure 5: Changes in the range-level Detection and Planning CHK-scores of representative models on each range after providing our designed skill. Each value denotes the difference between runs with and without the skill.

Figure 6: The Detection and Planning CAP-scores across the five capability dimensions for four representative models with and without our designed skill. enumerating installed packages and matching their versions against the CVE database, detects baseline risks by checking configuration entries against a fixed set of hardening rules, and finds sensitive files by pattern-matching for exposed credentials and keys. Unlike the agent models, agentless detection is detection-only, yielding a static findings list without providing remediation suggestions or reconstructing the attack. We map the agentless-detection findings to our capability dimensions and observe that the scanner reaches 43.6% on ENT and 50.0% on VUL, but only 20.7% on BAS and 2.1% on PER. Agents outperform this traditional security scanner on average across the four comparable dimensions. This is because the scanner reports only isolated static artifacts and fails to analyze persistence mechanisms, reconstruct attack chains, or attribute entry points. In contrast, agents can actively probe the host and connect scattered artifacts across files into an intrusion chain, thereby uncovering persistence mechanisms hidden in the system. Furthermore, unlike the scanner, agent models provide root-cause analyses and remediation plans.

12

Figure 7: (a) Inter-Judge Agreement. For each pair of judges, the value denotes the percentage of checkpoints on which both judges assign the same score, and ρ is the Spearman rank correlation between their scores. (b) Each Judge’s Difference from the Average. For each judge (row) and the representative model of each family (column), we compute the average difference between that judge’s score and the mean score of the three judges across all checkpoints. 4.5

Augmenting Agents with Procedural Priors

A typical failure mode is procedural: agents fail to follow a systematic investigation routine. For example, they follow the alerted trail instead of scanning the whole host, and the remediation plans often stop at the most obvious fix, which is reflected in their low scores on planning and on the Investigation & Response Quality dimension. We therefore distill the operational experience of the security incident-response team into procedural priors encoded as skills and provide them to the agents, allowing us to examine the performance upper bound that such priors alone can achieve. Notably, the designed skills contain only procedural priors rather than range-specific knowledge. Concretely, we encode the general routines by which experts investigate threats and organize fixes, together with report and remediation templates. The skills are distilled from a deployed incident-response product used for real customer cases and do not contain the ground truth of any range. The checklist items, expected findings, malicious file paths, and CVE lists are all excluded, ensuring that integrating such skills does not introduce knowledge leakage. As shown in Figure 5, our designed skills improve the agent models’ CHK-scores, particularly on the planning axis. The largest gains occur for models whose planning scores are initially low. For example, GPT-5.4 rises from 26% to 83% on the SSH-Miner CHK-score and increases by 43% on Docker-Escape, while GPT-5.4 Pro gains 45% on Jenkins-RCE and 52% on SSH-Miner. From the capability perspective in Figure 6, our skill can address the blind spots of strong models. For example, although Claude Opus 4.7 performs relatively weakly on persistence, the designed skill improves the PER CAP-score from 68% to 88% in Detection and from 49% to 75% in Planning. Similarly, the skill benefits GLM-5.1 most on the Investigation & Response Quality dimension. However, the skill does not ensure improvement. GLM-5.1 degrades by 12% in detection on Shiro-Fastjson and by 11% on Docker-Escape. This is probably because such broad-scope ranges include attacker traces beyond the skill’s enumerated categories, so the model stops at its boundary rather than continuing to explore and recover these long-tail traces. Overall, these results indicate that a procedural prior mitigates substantial failure, particularly on the planning axis, but cannot eliminate it. Closing this gap requires agents to learn from the outcome of their own investigation rather than executing a predefined routine, which is potentially the next step for this task. 4.6

LLM Judgment Analysis

Agreement among LLM Judges. LLM-as-a-Judge can be biased toward assigning higher scores to longer or more structured answers (Zheng et al., 2023). To reduce this effect, we grade each checkpoint on applicable axes with three judges from three different developers, namely Gemini 3.1 Pro, Claude Opus 4.7, and GPT-5.4 Pro, using the same rubric, and compute their average as the final score. However, this average is reliable only if the three judges are consistent with one another and do not share the same bias. We assess this consistency in Figure 7 using all checkpoint scores from the three judges. As shown in Figure 7 (a), the three judges agree on most checkpoints. Each pair assigns the same score on 72% to 75% of all checkpoints, and their rankings are highly correlated (ρ = 0.86–0.87). These results indicate that the rubric is interpreted consistently across the three judges and support the reliability of the final scores as measures of model capability. Moreover, Figure 7(b) shows that the main difference among the judges is strictness rather than a preference for specific models. The GPT judge scores the lowest, while the Claude and Gemini judges score slightly above the average. Since this tendency in strictness affects all models in the same way, averaging the scores from three judges can remove this influence, and

13

the model ranking is not affected. More importantly, no judge gives noticeably higher scores to its own model family, indicating that self-preference bias is small. Agreement with Human Experts. To validate the agreement between the LLM judges and human experts, we randomly select 60 checkpoints covering all 10 ranges and ask a security expert to grade 10 trajectories for each checkpoint under the same rubric without seeing the judge results. Detection is scored from 0 to 3, and planning is scored from 0 to 2. The human expert and LLM judges agree closely, with a Pearson correlation of 0.96, a quadratic-weighted Cohen’s kappa (κ) of 0.94, and a mean absolute error (MAE) of only 0.15. For 98% of the checkpoints, the scores from the human expert and the LLM judges differ by at most one point. Moreover, the LLM judges are slightly stricter than the human expert and are therefore less likely to overstate the evaluation results.

5

Conclusion

In this paper, we introduce S EC R ESPOND, the first benchmark for evaluating LLM agents on the postcompromise incident-response workflow. Each task requires the agent to analyze a forensic disk snapshot of a compromised host together with the alerts, vulnerability scans, and baseline checks reported by a host security product, and deliver forensic reports on intrusions, baseline risks, and vulnerability risks, along with a remediation plan. We instantiate this task across 10 cyber ranges, each built from a distinct compromised cloud host, together covering 4 entry-point types, 21 ATT&CK techniques, and 5 operating systems. By evaluating 23 frontier LLMs on the OpenCode agent harness, we observe that current agents can reliably uncover the problems exposed by alerts but struggle to proactively investigate the disk for silent intrusions and to produce comprehensive, verified remediation plans, with no model achieving complete detection and remediation on any single range. This gap reveals a fundamental bottleneck in building agents for real-world incident response. We hope S EC R ESPOND provides a foundation for developing security agents that investigate and remediate incidents thoroughly rather than merely reacting to shallow alerts.

Ethics Statement Third-Party Model Evaluation. The third-party models included in our evaluation were assessed solely for academic research. All reported results were obtained in a controlled experimental environment and are presented only for capability comparison and technical discussion. The inclusion of model and provider names and scores does not imply any official position or endorsement by the corresponding providers or brands. Neither outputs obtained from these services nor the resulting evaluation records are used to train, fine-tune, or distill any competing model. Security Research and Use Restrictions. All benchmark data were generated in isolated experimental environments under our authorization and control, rather than collected from production systems. Each cyber range is a host that we provisioned, compromised, and snapshotted for this purpose. The released forensic and security-product artifacts are synthetic and contain no data from customer, production, or real-world business systems. We sanitized every disk snapshot by replacing real credentials, keys, and personal data with placeholders, retaining only the forensic artifacts required for analysis. The attack techniques, CVEs, and exploitation methods represented in the scenarios are derived solely from public knowledge bases and vulnerability disclosures, such as MITRE ATT&CK and public vulnerability databases. The benchmark contains forensic evidence rather than runnable exploit chains; it includes no proprietary or internally developed advanced attack techniques and no incremental attack information beyond public sources, and it cannot be directly used to conduct attacks against real-world systems. The benchmark is intended exclusively for defensive security research. Any use for unauthorized intrusion, destructive testing, offensive activity, or any other purpose that violates applicable laws or regulations is strictly prohibited.

References Anthropic. Introducing claude opus 4.7, 2026. URL https://www.anthropic.com/news/claude-opus-4-7. V Barres, H Dong, S Ray, X Si, and K Narasimhan. τ 2 -bench: Evaluating conversational agents in a dual-control environment, 2025. URL https://arxiv. org/abs/2506.07982. Lauren Deason, Adam Bali, Ciprian Bejean, Diana Bolocan, James Crnkovich, Ioana Croitoru, Krishna Durai, Chase Midler, Calin Miron, David Molnar, et al. Cybersoceval: Benchmarking llms capabilities for malware analysis and threat intelligence reasoning. arXiv preprint arXiv:2509.20166, 2025. 14

Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, et al. Swe-bench pro: Can ai agents solve long-horizon software engineering tasks? arXiv preprint arXiv:2509.16941, 2025. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. The Innovation, 7(6), 2026. Wei He, Yueqing Sun, Hongyan Hao, Xueyuan Hao, Zhikang Xia, Qi Gu, Chengcheng Han, Dengchang Zhao, Hui Su, Kefeng Zhang, et al. Vitabench: Benchmarking llm agents with versatile interactive tasks in real-world applications. arXiv preprint arXiv:2509.26490, 2025. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, pp. 54107–54157, 2024. Pengfei Jing, Mengyun Tang, Xiaorong Shi, Xing Zheng, Sen Nie, Shi Wu, Yong Yang, and Xiapu Luo. Secbench: A comprehensive multi-dimensional benchmarking dataset for llms in cybersecurity. arXiv preprint arXiv:2412.20787, 2024. Chenxin Li, Zhengyang Tang, Mingxin Huang, Yunlong Lin, Shijue Huang, Shengyuan Liu, Bowen Ye, Rang Li, Lei Li, Benyou Wang, et al. Claw-eval-live: A live agent benchmark for evolving real-world workflows. arXiv preprint arXiv:2604.28139, 2026a. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, AnnKathrin Dombrowski, Shashwat Goel, Long Phan, et al. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218, 2024. Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, et al. Skillsbench: Benchmarking how well agent skills work across diverse tasks. arXiv preprint arXiv:2602.12670, 2026b. Xihuan Lin, Jie Zhang, Gelei Deng, Tianzhe Liu, Tianwei Zhang, Qing Guo, and Riqing Chen. Ircopilot: Automated incident response with large language models. arXiv preprint arXiv:2505.20945, 2025. Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868, 2026. Meta. Introducing autopatchbench: A benchmark for ai-powered security fixes, 2025. URL https: //engineering.fb.com/2025/04/29/ai-research/autopatchbench-benchmark-ai-powered-securit y-fixes/. OpenAI. Introducing gpt-5.4, 2026. URL https://openai.com/index/introducing-gpt-5-4. OpenAI. Introducing gpt-5.4, 2026. URL https://openai.com/index/introducing-gpt-5-4/. OpenCode. Opencode: The open source ai coding agent, 2026. URL https://opencode.ai/. Qwen. Qwen3.7: The agent frontier, 2026. URL https://qwen.ai/blog?id=qwen3.7. Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, et al. Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security. Advances in Neural Information Processing Systems, 37:57472–57498, 2024. CocoaBench Team, Shibo Hao, Zhining Zhang, Zhiqi Liang, Tianyang Liu, Yuheng Zha, Qiyue Gao, Jixuan Chen, Zilong Wang, Zhoujun Cheng, et al. Cocoabench: Evaluating unified digital agents in the wild. arXiv preprint arXiv:2604.11201, 2026. The Gemini Team. Gemini 3.1 pro: A smarter model for your most complex tasks, 2026. URL https: //blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/. Shengye Wan, Cyrus Nikolaidis, Daniel Song, David Molnar, James Crnkovich, Jayson Grace, Manish Bhatt, Sahana Chennabasappa, Spencer Whitman, Stephanie Ding, et al. Cyberseceval 3: Advancing the evaluation of cybersecurity risks and capabilities in large language models. arXiv preprint arXiv:2408.01605, 2024. Zhun Wang, Tianneng Shi, Jingxuan He, Matthew Cai, Jialin Zhang, and Dawn Song. Cybergym: Evaluating ai agents’ cybersecurity capabilities with real-world vulnerabilities at scale. arXiv e-prints, pp. arXiv–2506, 2025. 15

Wiz. Cyber model arena, 2026. URL https://www.wiz.io/cyber-model-arena. Yiran Wu, Mauricio Velazco, Andrew Zhao, Manuel Raúl Meléndez Luján, Srisuma Movva, Yogesh K Roy, Quang Nguyen, Roberto Rodriguez, Qingyun Wu, Michael Albada, et al. Excytin-bench: Evaluating llm agents on cyber threat investigation. arXiv preprint arXiv:2507.14201, 2025. Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh J Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. Advances in Neural Information Processing Systems, 37: 52040–52094, 2024. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. tau-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong, et al. Claw-eval: Towards trustworthy evaluation of autonomous agents. arXiv preprint arXiv:2604.06132, 2026. Z.ai. Glm-5.1: Towards long-horizon tasks, 2026. URL https://z.ai/blog/glm-5.1. Andy K Zhang, Neil Perry, Riya Dulepet, Joey Ji, Celeste Menders, Justin Lin, Eliot Jones, Gashon Hussein, Samantha Liu, Donovan Jasper, et al. Cybench: A framework for evaluating cybersecurity capabilities and risks of language models. In International Conference on Learning Representations, volume 2025, pp. 25094–25243, 2025. Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan Lu, Wendong Xu, Yunzhuo Hao, Songcheng Cai, Xiaochen Wang, et al. Clawbench: Can ai agents complete everyday online tasks? arXiv preprint arXiv:2604.08523, 2026. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pp. 15585–15606, 2024. Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, et al. Cve-bench: a benchmark for ai agents’ ability to exploit real-world web application vulnerabilities. arXiv preprint arXiv:2503.17332, 2025.

16

A

Details on Range Construction

SSH-Miner. The ssh-miner range is built on a CentOS 7 host with weak SSH baseline settings and a weak root password. An external attacker brute-forces the root credential with hydra, drops a mining binary, installs three persistence mechanisms spanning scheduled tasks, shell initialization, and system services, and clears the authentication log. The agent receives this post-attack disk snapshot with (i) five alerts covering the brute-force success, miner execution, suspicious crontab, autorun change, and outbound miner connection, (ii) an empty vulnerability list since there is no CVE-driven entry, and (iii) baseline checks flagging the SSH, MySQL, and credential-strength baselines. With these inputs, the required reports are expected to cover (i) an intrusion report covering the full attack chain and intrusion entities such as the mining binary, hydra residue, log backup, and attacker IP, (ii) a vulnerability report confirming the absence of a CVE-driven entry, (iii) a baseline report flagging the SSH and credential baseline weaknesses, and (iv) a remediation plan prioritizing miner termination, persistence cleanup, SSH hardening, and credential rotation. ssh-miner/disk/ |-- usr/ |-- opt/ |-- ... |-- etc/ | |-- ssh/sshd_config | |-- sudoers.d/devops | |-- my.cnf | \-- systemd/system/ | |-- system-update-helper.service | \-- multi-user.target.wants/ | \-- system-update-helper.service |-- home/ | \-- devops/.bash_history |-- root/ | \-- .bashrc |-- tmp/ | \-- .X11-unix/ | |-- .xorg | \-- .xorg.log \-- var/ |-- spool/cron/root \-- www/html/index.html

Redis-RCE. The redis-rce range is built on an Ubuntu 22.04 host that exposes an unauthenticated Redis server running as root to the public internet. The attacker abuses the Redis server to write an SSH key into the root account, logs in, opens a reverse shell, and deploys a cryptominer disguised as a Redis background-save process, with persistence mechanisms including the injected key, a malicious crontab, an SSH login hook, and a boot script, with the miner’s timestamp forged to resist timeline analysis. The agent receives this snapshot with (i) two alerts covering only the reverse shell and its outbound connection, (ii) an empty vulnerability list since the entry is a misconfiguration rather than a CVE, and (iii) no baseline findings, leaving the persistence silent on disk. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the Redis-to-root chain and intrusion entities such as the miner, injected key, and malicious crontab, (ii) a vulnerability report concluding the absence of a CVE-driven entry, (iii) a baseline report flagging the Redis unauthorized-access and SSH root-login weaknesses, and (iv) a remediation plan prioritizing miner termination, persistence removal, and Redis and SSH hardening. redis-rce/disk/ |-- opt/ |-- tmp/ |-- usr/ |-- ... |-- etc/ | |-- init.d/redis-watchdog | |-- redis/redis.conf | |-- ssh/sshd_config | \-- sudoers.d/deploy |-- home/ | \-- deploy/.bash_history |-- root/ | \-- .ssh/

17

| |-- authorized_keys | \-- rc `-- var/ |-- spool/cron/crontabs/root \-- tmp/ |-- .redis-bgsave \-- .redis-bgsave.log

Docker-Escape. The docker-escape range is built on an Ubuntu 20.04 host where Docker Remote API is exposed without authentication. The attacker reaches the API, launches a privileged container that mounts the host filesystem to escape to the host, injects an SSH key, and runs a cryptominer hidden as a Docker health tool, with persistence including a crontab, a systemd service, a shell-initialization hook, the injected key, and a long-running backdoor container. The agent receives this snapshot with (i) three alerts covering the mining process, the suspicious crontab, and the anomalous systemd service, (ii) an empty vulnerability list since the exposed API is a misconfiguration, and (iii) no baseline findings, so the API exposure, container escape, and backdoor container must be recovered from Docker events and host forensics. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the API-to-host-takeover chain and entities such as the miner, the escape container, and the backdoor container, (ii) a vulnerability report concluding no CVE-driven entry, (iii) a baseline report flagging the exposed Docker API and excessive permissions, and (iv) a remediation plan prioritizing miner and backdoor-container removal, persistence cleanup, and firewalling Docker TCP access. docker-escape/disk/ |-- usr/ |-- home/ |-- tmp/ |-- ... |-- etc/systemd/system/ | |-- docker-health-agent.service | \-- docker.service.d/ | \-- override.conf |-- opt/ | |-- .docker/ | | |-- .health-monitor | | \-- .health-monitor.log | \-- app/ | \-- docker-compose.yml |-- var/spool/cron/crontabs/ | \-- root \-- root/ |-- .bashrc \-- .ssh/ \-- authorized_keys

Jenkins-RCE. The jenkins-rce range is built on an Ubuntu 22.04 host running Jenkins whose authorization misconfiguration leaves the Groovy Script Console reachable without authentication. The attacker obtains unauthenticated remote code execution through the console, deploys a cryptominer masquerading as a Jenkins agent, exfiltrates Jenkins credentials, and installs redundant persistence spanning a cron job, a rogue root-equivalent account, a SUID-root binary, a sudoers backdoor, a library-preload injection, and a Jenkins startup hook. The agent receives this snapshot with (i) five alerts covering the Script Console access, the masqueraded miner, the new privileged account, the cron backdoor, and the outbound mining connection, and (ii) no explicit vulnerability or baseline checks. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the unauthenticated RCE-to-mining chain and entities such as the miner, the exfiltrated secrets, and the multiple persistence implants, (ii) a vulnerability report attributing the entry to a Jenkins authorization misconfiguration, (iii) a baseline report flagging the anonymous-administrator setting and weak administration password, and (iv) a remediation plan prioritizing miner termination, removal of all persistence, credential rotation, and revoking anonymous permissions. jenkins-rce/disk/ |-- home/ |-- opt/ |-- tmp/ |-- usr/ |-- ... |-- etc/

18

| |-- hosts | |-- environment | |-- passwd | |-- sudoers | \-- cron.d/jenkins-update |-- usr/local/bin/.update |-- var/ | |-- cache/jenkins/ | | |-- .update | | |-- .x.so | | \-- .j.tgz | \-- jenkins_home/ | |-- config.xml | |-- init.groovy.d/zz-update.groovy | \-- secrets/master.key \-- root/.ssh/authorized_keys

Shiro-Fastjson. The shiro-fastjson range is built on a CentOS 8 host running a Java web application on Tomcat that uses a default key and a vulnerable Fastjson library. The attacker gains code execution through Shiro deserialization, drops two JSP webshells hidden among static assets, steals database credentials and dumps a user table, escalates to root by abusing a sudo rule without passwords, attempts to uninstall the security agent, and deploys a miner behind persistence including a systemd service, crontab, shell-initialization hook, injected key, sudoers backdoor, a MySQL UDF backdoor, an at job, and rc.local. The agent receives this snapshot with (i) fifteen alerts spanning initial access, encodedcommand execution, security-agent tampering, privilege escalation, and persistence, (ii) an empty vulnerability list since the Shiro and Fastjson flaws are library-level components beyond the scanner, and (iii) baseline findings flagging the operating system, Tomcat, and MySQL baselines and several weak-password checks. With these inputs, the required outputs are expected to include (i) an intrusion report reconstructing the Shiro-to-root chain and entities such as the two webshells, the miner, the database backdoor, and the data dump, (ii) a vulnerability report identifying the vulnerable Shiro and Fastjson components, (iii) a baseline report flagging the Tomcat, MySQL, and credential weaknesses, and (iv) a remediation plan prioritizing removal of the webshells, miner, and all persistence mechanisms, together with version upgrades and database lockdown. shiro-fastjson/disk/ |-- home/ |-- ... |-- etc/ | |-- my.cnf | |-- profile.d/java-env.sh | |-- rc.d/rc.local | |-- sudoers.d/ | | |-- 99-java-ops | | \-- tomcat | \-- systemd/system/java-app-monitor.service |-- opt/ | |-- .cache/ | | |-- .java-updater | | \-- .java-updater.log | |-- ruoyi/application.yml | \-- tomcat/webapps/ROOT/ | |-- WEB-INF/lib/ | | |-- fastjson-1.2.68.jar | | \-- shiro-core-1.7.0.jar | \-- static/ | |-- css/error.jsp | \-- js/analytics.jsp |-- root/.ssh/authorized_keys |-- tmp/.sql_dump |-- usr/lib64/mysql/plugin/lib_mysqludf_json.so \-- var/spool/cron/root

Log4j-RCE. The log4j-rce range is built on a CentOS 8 host whose web application bundles a vulnerable Log4j library exploitable through a JNDI lookup (CVE-2021-44228). The attacker triggers the JNDI injection to launch a webshell, overwrites a deploy script invoked through sudo to escalate to root, deploys a miner, exfiltrates a database dump, and truncates the authentication log to cover its tracks, 19

with persistence including a systemd service, a crontab, an injected key, and a shell-initialization hook. The agent receives this snapshot with (i) six alerts covering the Log4j RCE, the webshell, the privilege escalation, the miner, the crontab backdoor, and the log tampering, (ii) a vulnerability check reporting a CVE for the Log4j library, and (iii) baseline findings flagging SSH root-login, weak passwords, and the sudo rule without password protection. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the Log4j-to-root chain and entities such as the webshell, miner, data dump, and tampered deploy script, (ii) a vulnerability report confirming the Log4j RCE and prescribing the upgrade, (iii) a baseline report flagging the SSH and sudo weaknesses, and (iv) a remediation plan prioritizing the Log4j upgrade, webshell and miner removal, persistence cleanup, and credential rotation. log4j-rce/disk/ |-- bin/ |-- usr/ |-- lib/ |-- home/ |-- boot/ |-- sbin/ |-- ... |-- etc/ | |-- profile.d/java-env.sh | |-- ssh/sshd_config | |-- sudoers.d/app-deploy | \-- systemd/system/java-gc-helper.service |-- opt/webapp/ | |-- lib/ | | |-- log4j-api-2.14.1.jar | | \-- log4j-core-2.14.1.jar | |-- logs/app.log | |-- VulnWebApp.java | |-- deploy.sh | \-- webshell.py |-- var/ | |-- cache/.java-gc | |-- log/auth.log | \-- spool/cron/crontabs/root |-- tmp/ | |-- .db_dump | |-- .escalation_log | \-- .privesc_proof \-- root/.ssh/authorized_keys

Next.js-RCE. The nextjs-rce range is built on an Ubuntu 22.04 host running a Next.js application vulnerable to a server-component deserialization RCE. The attacker obtains code execution as the application user, drops a Node.js webshell on a spoofed debugger port, escalates to root through a command injection in a SUID-root helper, installs an LD_PRELOAD rootkit, runs a miner, and exfiltrates a database dump, with persistence including the loader-preload rootkit, a shell-initialization hook, a systemd service, a crontab, and an injected key. The agent receives this snapshot with (i) six alerts covering the miner, the suspicious Node child process, the webshell, the crontab, the systemd service, and the outbound mining connection, (ii) an empty vulnerability list, requiring the agent to infer vulnerabilities from the disk snapshot, and (iii) baseline findings flagging the SSH configuration, database exposure, a non-standard SUID binary, and a loose sudo rule. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the RCE-to-rootkit chain and entities such as the malicious shared library, the miner, the webshell, and the data dump, (ii) a vulnerability report identifying the framework vulnerability from the dependency manifest, (iii) a baseline report flagging the SSH, database, and privilege-escalation weaknesses, and (iv) a remediation plan prioritizing rootkit and miner removal, persistence cleanup, framework upgrade, and SUID and sudo hardening. nextjs-rce/disk/ |-- bin/ |-- boot/ |-- lib/ |-- sbin/ |-- snap/ |-- srv/ |-- usr/lib/ |-- ... |-- etc/

20

| |-- ld.so.preload | |-- profile.d/node-env.sh | |-- ssh/sshd_config | |-- sudoers.d/node-ops | |-- systemd/system/node-gc-helper.service | \-- postgresql/14/main/ | |-- pg_hba.conf | \-- postgresql.conf |-- opt/webapp/ | |-- .env | |-- package.json | |-- node_modules/next/package.json | \-- .next/static/chunks/debug.js |-- usr/ | |-- lib/x86_64-linux-gnu/.libnode_helper.so | \-- local/bin/backup-tool |-- var/ | |-- cache/.node-gc | |-- cache/.node-gc.log | \-- spool/cron/crontabs/root |-- tmp/ | |-- .pg_dump | |-- .escalation_log | \-- .privesc_proof \-- root/.ssh/authorized_keys

NPM-Worm. The npm-worm range is built on an Ubuntu 22.04 host running a Node.js application whose unauthenticated deployment webhook passes its parameter straight into npm install, letting the attacker install a malicious package from the attacker host. The package’s installed hook drops a worm that scans the internal network, harvests several classes of credentials, and injects itself into other projects to propagate, after which a leaked cloud access key is used to deploy a miner, with persistence including a crontab, shell and login initialization hooks, a systemd service, injected keys, and the infected project. The agent receives this snapshot with (i) two alerts covering the worm-download command and the web application spawning an abnormal child process, (ii) an empty vulnerability list, and (iii) baseline findings flagging the Ubuntu host baseline, a weak root password, the Nginx-exposed unauthenticated webhook, and several sensitive-configuration leaks such as a plaintext npm token, an unencrypted SSH key, and loose file permissions. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the supply-chain-to-worm chain and entities such as the malicious package, the worm script, the stolen-credential cache, and the miner, (ii) a vulnerability report attributing the entry to the unauthenticated webhook and unsigned package install, (iii) a baseline report flagging the weak credential, exposed webhook, and sensitive configuration findings, and (iv) a remediation plan prioritizing worm and miner termination, package and propagation removal, credential rotation, webhook authentication, and disabling install scripts. npm-worm/disk/ |-- tmp/ |-- usr/ |-- ... |-- etc/ | |-- profile.d/node-env.sh | |-- sudoers.d/developer | \-- systemd/system/npm-cache-gc.service |-- home/ | \-- developer/ | |-- .bashrc | |-- .git-credentials | |-- .npmrc | \-- .ssh/ | |-- authorized_keys | \-- id_rsa |-- opt/ | |-- .node-helpers/ | | |-- .npm-gc | | \-- .npm-gc.log | |-- internal-tools/package.json | \-- webapp/ | |-- .env | |-- package-lock.json

21

| \-- node_modules/@corp-utils/logger/ | |-- package.json | \-- scripts/setup.js |-- root/ | \-- .ssh/authorized_keys \-- var/ |-- spool/cron/crontabs/developer \-- tmp/.npm-cache/ |-- scan_results.txt |-- sysinfo.json |-- npmrc/.npmrc |-- ssh_keys/id_rsa |-- env_files/.env |-- git_credentials/.git-credentials \-- bash_history/.bash_history

ASP.NET-ViewState. The aspnet-viewstate range is built on a Windows Server 2019 host running an ASP.NET application whose backup configuration file is downloadable and leaks the hardcoded machineKey. The attacker forges a signed ViewState payload to execute code as SYSTEM, drops an ASPX webshell, dumps and parses LSASS to recover credentials, runs a C2 beacon masqueraded as an audit service, and clears event logs and shadow copies, with Windows-specific persistence spanning a scheduled task, a WMI event subscription, a sticky-keys/IFEO backdoor, a DLL search-order hijack, and a malicious MSSQL logon trigger. The agent receives this snapshot with (i) three alerts covering the webshell, the beacon process, and the WMI subscription, (ii) a vulnerability check reporting the ViewState deserialization with the backup exposure and application pool misconfigurations, and (iii) baseline findings flagging the weak database password, the enabled command execution stored procedure, and the hardcoded machineKey. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the ViewState-to-SYSTEM chain and entities such as the webshell, the credential-dump residue, the beacon, and the five persistence objects, (ii) a vulnerability report confirming the ViewState RCE and the related misconfigurations, (iii) a baseline report flagging the MSSQL and machineKey weaknesses, and (iv) a remediation plan prioritizing webshell and beacon removal, teardown of the Windows persistence, machineKey rotation, application-pool de-privileging, and credential rotation. aspnet-viewstate/disk/ |-- Windows/ | |-- System32/ | | |-- Tasks/Microsoft/Windows/Maintenance/AuditTask | | |-- wbem/Repository/OBJECTS.DATA | | \-- ... | \-- Temp/ | |-- lsass.dmp | |-- creds.txt | |-- plant-trigger.sql | \-- crm-app.tar.gz |-- Program Files/ | \-- Notepad++/ | |-- notepad++.exe | \-- version.dll |-- ProgramData/ | |-- WindowsAudit/ | | |-- WindowsAuditSvc.exe | | |-- last-beacon.log | | |-- schtasks-fired.log | | |-- wmi-fired.log | | \-- mssql-trigger-fired.log | \-- ... |-- ... \-- inetpub/wwwroot/CRM/ |-- web.config |-- web.config.bak \-- help.aspx

RDP-Service-Abuse. The rdp-service-abuse range is built on a Windows Server 2019 host that exposes RDP to the public internet with a weak helpdesk password. The attacker sprays the exposed RDP service, lands a logon as the low-privilege helpdesk user, and abuses a weak service DACL to rewrite the service bin path to a PowerShell payload and restart it, gaining execution as SYSTEM. Then, the attacker installs 22

persistence with a disguised service and a scheduled task, leaves credential dumps, and beacons to a C2 endpoint. The agent receives this snapshot with (i) seven alerts clustered on the post-exploitation stage covering the encoded-PowerShell execution, the LSASS-dump tooling, and the anomalous service registry write, (ii) an empty vulnerability list since the entry is an RDP weak-credential and servicemisconfiguration chain rather than a CVE, and (iii) no baseline findings, leaving the RDP authentication timeline and the DACL only recoverable from the event logs in the disk snapshot. With these inputs, the required reports are expected to cover (i) an intrusion report reconstructing the RDP-spraying-to-SYSTEM chain and discovering the disguised service, scheduled task, credential dump, and C2 indicator, (ii) a vulnerability report concluding the absence of a CVE-driven entry, (iii) a baseline report flagging the helpdesk weak password and RDP exposure with the weak service DACL, and (iv) a remediation plan prioritizing helpdesk credential rotation, a more restrictive service DACL, removal of the persistence, the credential residue, and C2 blocking. rdp-service-abuse/disk/ |-- Windows/System32/ |-- ... |-- ProgramData/ | |-- CorpBackup/ | | |-- corp-backup-svc.ps1 | | \-- system-stage.ps1 | \-- WindowsHealth/ | |-- WindowsHealthSvc.ps1 | \-- last-beacon.log |-- Users/ | \-- helpdesk/NTUSER.DAT |-- Windows/Temp/ | |-- procdump.exe | |-- lsass.dmp | \-- creds.txt \-- Windows/System32/ |-- Tasks/Microsoft/Windows/Maintenance/HealthSync |-- config/ | |-- SYSTEM | \-- SAM \-- winevt/Logs/ |-- Security.evtx |-- System.evtx |-- Microsoft-Windows-TerminalServices-RemoteConnectionManager%4Operational.evtx \-- Microsoft-Windows-TerminalServices-LocalSessionManager%4Operational.evtx

23

B

Details on Capability Taxonomy

Table 6: The mapping between capability items and range checklist. Each cell lists the checkpoint indices (CHK-n) in that range mapped to the item. Item

Capability Name

SSH-Miner Redis-RCE Docker-Escape

Intrusion Entity (ENT) ENT-F01 Webshell file ENT-F02 Linux malware file ENT-F03 Malicious SO / kernel module ENT-F04 Residual data file ENT-F05 Tampered-file restoration ENT-F06 Windows malware file ENT-N01 Attacker-IP inbound block ENT-N02 Malicious-IP outbound block ENT-N03 Malicious-domain block ENT-P01 Mining process ENT-P02 C2 beacon process ENT-P03 Windows malicious process Persistence Mechanism (PER) PER-A01 authorized_keys tampering PER-A02 sudoers implant PER-A03 Rogue local account PER-D01 MySQL backdoor PER-D02 MSSQL backdoor PER-E01 profile.d / udev rule PER-E02 WMI event subscription PER-H01 ld.so.preload hijack PER-H02 DLL hijacking PER-I01 Shell init (bashrc/profile) PER-I02 /etc/environment injection PER-M01 SUID/SGID backdoor PER-S01 Cron task PER-S02 systemd timer PER-S03 at job PER-S04 Windows scheduled task PER-V01 systemd service PER-V02 init.d / rc.local PER-V03 Windows service PER-W01 Nginx config tampering PER-W02 App-container config tampering

2,3

Jenkins-RCE Shiro-Fastjson Log4j-RCE Next.js-RCE NPM-Worm ASP.NET-ViewState

2

2,3

2

3,4

4

9 6

20 23

23 24

1

1

1

12 11 10 1

1,4 3

5,8

5 21

38 39 3

2

3,36 2 4 5

3,4 2,6

1

7,8,9

5

16,17

30 31

38 37

2,4,8 6 7 7

13,15 3

1

1,5

14

3 3 8 10

14

9

23

6

24

4,22 6 16,17 12,13,14 4 8,15 12

13

11

16

13 12

7 5,23 3,20

20

15

19

13

20 8 7

19,25

21 9,10,11 13,14

14,15

14,15

17,18 22

12

10,11,12

21,22 6,8,9 10

8,21

Baseline Risk (BAS) BAS-01 SSH hardening BAS-02 DB / middleware access control BAS-03 Credential safety BAS-04 Privilege / service audit BAS-05 Container / orchestration config BAS-06 Web-service config BAS-07 Cloud-resource config

4–7 9 10 8

10 5–9 12 11

9 8 7 5,6,10,21

7,8 14,24 13

23

12 11,13,16 9,14

9,22 10,21

11–14 18,19 15,16 17,20

15

10,11,12 14,15,17,36 18 16 13

21 21,22

4,18,19 5,20

19,20

11

Vulnerability Risk (VUL) VUL-01 Java component vuln VUL-02 Web-app code vuln VUL-03 Container-escape vuln VUL-04 Node.js / front-end framework vuln Investigation and Response Quality (Q) Q-01 Entry-point localization Q-02 Attack-chain reconstruction Q-03 Attacker-info extraction Q-04 Honesty & confidence calibration Q-05 Investigation thoroughness Q-06 Cross-language/service tracing Q-07 Remediation-verification completeness Q-08 Business-impact assessment

1,2,7 4,6

RDP-Service-Abuse

26–28,35–37

6,11,20

35 18 23 21,22,24,35

18 16,19 16,17 18 15

20 18,21 18,19 20 17

17,18 19

21,22

22,24

20,22

18 16

C

Implementation Details

C.1

Task Prompt for Linux Operating System

15 16 19

25 29–32,34 32,33 25 23,24 40–45

17,18

16 17,19 18

25 26,28 27 32 33 29 34 37

28,29 31,33 30,34 29 26 27 39 32

4 1,2,7 3

# Detection Agent Workspace ## Your Role You are a security incident response analyst working on a forensic analysis. A compromised server's disk snapshot and security platform alert data have been provided. Your job is to analyze them and produce a comprehensive incident response report. ## Data Sources - **Forensic disk snapshot**: `DISK_PATH/` (read-only copy of the server filesystem) - **SAS alerts**: `SAS_PATH/alerts.json` - **SAS vulnerability scan**: `SAS_PATH/vulns.json` - **SAS baseline check**: `SAS_PATH/baselines.json` ## Output Write all outputs to: `OUTPUT_PATH/` Required files: 1. `progress.md` — Investigation timeline and checklist 2. `intrusion-report.md` — Full intrusion analysis with attack chain 3. `vuln-report.md` — Vulnerability verification results 4. `baseline-report.md` — Configuration baseline assessment 5. `remediation-plan.md` — Structured remediation plan (natural language, prioritized P0-P3)

24

## Constraints - You are running on an analysis machine, NOT the compromised server - The disk snapshot at `DISK_PATH/` is a read-only forensic copy - Do NOT SSH into any machine during analysis - All findings must be based on disk evidence and SAS data - Do NOT fabricate CVEs or vulnerabilities — if unsure, say so with confidence levels - Preserve forensic evidence: recommend backup/quarantine over deletion in remediation-plan.md

C.2

Task Prompt for Windows Operating System

# Detection Agent Workspace ## Your Role You are a security incident response analyst working on a forensic analysis. A compromised server's disk snapshot and security platform alert data have been provided. Your job is to analyze them and produce a comprehensive incident response report. ## Data Sources - **Forensic disk snapshot**: `DISK_PATH/` — Windows filesystem dumped as a read-only directory tree on this Linux analysis machine - **SAS alerts**: `SAS_PATH/alerts.json` - **SAS vulnerability scan**: `SAS_PATH/vulns.json` - **SAS baseline check**: `SAS_PATH/baselines.json` ### Forensic Access The Windows disk is at `DISK_PATH/` as a directory tree on this Linux machine. Use `find`, `grep`, ` cat`, `stat`, `strings` directly against this path. Linux equivalents of the Windows-forensic artifact classes: - `cat` / `grep` — scheduled tasks (XML, Action, RunAs, Trigger; decode any base64 `powershell.exe -enc` payloads) - `strings -e l <hive> | grep` — Windows services (path + start account) - `strings -e l <hive> | grep` — Registry Run / RunOnce autostart - `strings -e l <evtx> | grep` and `strings <evtx> | grep` — full Event Log (focus EventID 1102 log clear, 4720 / 4722 account changes, 8224 VSS shadow delete) - `strings -e l <hive> | grep` — any hive key (Run, IFEO Debugger, AppInit, etc.) - `find` / `cat` / `grep` / `strings` — plugin-style artifacts (amcache.*, defender.*, firewall.* ...) - `strings <CIM repo> | grep` — WMI permanent subscriptions (`__EventFilter` / ` ActiveScriptEventConsumer` / `__FilterToConsumerBinding`) ## Evaluation / Operations Infrastructure Inventory (for forensic dedup) If any of the following IPs / keys / actions appear in the victim host's logs or forensic evidence, **they are traces left by the evaluation platform itself, not attacker activity**. You must dedup them out when identifying the attacker, and **must not mistake infrastructure activity for attacker lateral movement**: - Keys in `authorized_keys` carrying the `BENCHMARK-CONTROL-DO-NOT-REMOVE` comment, and their associated login traces The attacker is the activity on disk that lies **outside this inventory**. Do NOT let `remediationplan.md` recommend blocking this IP or deleting any key marked `BENCHMARK-CONTROL-DO-NOT-REMOVE`. ## Execution-Risk Responsibility When Writing remediation-plan.md A downstream remediation agent executes the plan you write as authoritative instructions—**every command you write is equivalent to authorizing an unattended agent to run it immediately on a production server**. Each remediation item must carry a `risk_class` (auth / network / account / service / data / runtime / none) and an `automation` level (auto / needs_approval / blocked).

25

Any command that would sever the current control plane (immediately expiring a password, stopping sshd, deleting a legitimate administrator key, etc.) **must NOT be written as a concrete command **—instead, write it as a "risk description + recommended manual-intervention procedure". ## Output Write all outputs to: `OUTPUT_PATH/` Required files: 1. `progress.md` — Investigation timeline and checklist 2. `intrusion-report.md` — Full intrusion analysis with attack chain 3. `vuln-report.md` — Vulnerability verification results 4. `baseline-report.md` — Configuration baseline assessment 5. `remediation-plan.md` — Structured remediation plan (natural language, prioritized P0-P3) ## Constraints - You are running on an analysis machine, NOT the compromised server - The disk snapshot at `DISK_PATH/` is a read-only forensic copy - Do NOT SSH into any machine during analysis - All findings must be based on disk evidence and SAS data - Do NOT fabricate CVEs or vulnerabilities — if unsure, say so with confidence levels - Preserve forensic evidence: recommend backup/quarantine over deletion in remediation-plan.md

C.3

Evaluation Prompt

# Evaluation Agent Workspace ## Your Role You are a security benchmark evaluation agent for a **single range**. Score the detection agent's analysis reports against the ground-truth checklist for this range. ## Input - **Checklist** (scoring rubric): `CHECKLIST_PATH` - **Detection reports**: `DETECTION_REPORTS_PATH/` ## Methodology Follow the evaluation SKILL at: `EVALUATION_SKILL_PATH` ## How to Work ### Step 1: Read the Checklist Read the checklist thoroughly. Each CHK item defines: - **Detection score** (max 3): Discovery(1) + Evidence(1) + Attribution(1) -- scored by you (LLM-as-a -Judge) - **Plan score** (max 2): Correctness(1) + Completeness(1) -- scored by you (LLM-as-a-Judge) Items marked "detection only" have N/A for plan score. ### Step 2: Read All Reports Read all detection agent outputs. 1. `progress.md` — Investigation timeline and checklist 2. `intrusion-report.md` — Full intrusion analysis with attack chain 3. `vuln-report.md` — Vulnerability verification results 4. `baseline-report.md` — Configuration baseline assessment 5. `remediation-plan.md` — Structured remediation plan ### Step 3: Score Each CHK Item **Detection score (0-3):** (skip for plan-only items) - Discovery (0/1): Did the agent identify this issue? - Evidence (0/1): Did it cite specific evidence (paths, log lines, config values)? - Attribution (0/1): Did it correctly attribute the finding? **Plan score (0-2):** (skip for detection-only items) - Correctness (0/1): Does remediation-plan.md contain correct remediation steps for this issue?

26

- Completeness (0/1): Does the plan include verification, edge cases, side effects? ### Step 4: Write Output Write to: `OUTPUT_PATH/` 1. **evaluation-report.md**: Per-CHK scoring with rationale and evidence quotes. End with strengths/ weaknesses analysis. 2. **scores.json**: Structured JSON with per-checkpoint, scores and overall summary. ## Scoring Constraints - Be strict and objective. Only give credit for what is explicitly present in the reports. - Quote specific evidence when justifying scores. - Do NOT give credit for things the agent "probably checked but didn't report." - Note environment limitations (e.g., Docker lacking NET_ADMIN) in the report but still score as 0.

C.4

Evaluation Skill

# Security Evaluation SKILL ## Overview This SKILL drives the evaluation Agent to quantitatively assess the outputs of the detection Agent and the remediation Agent. Core mechanism: — Assess the quality of the analysis report and the remediation plan (judged item by item by the evaluation Agent) Each checkpoint (CHK) is split into two independent scoring dimensions, so that a problem can be pinpointed to either the analysis stage or the plan stage: | Dimension | What is scored | Scoring source | Scoring method | |------|--------|---------|---------| | **Detection score** | Whether the issue was correctly identified, backed by sufficient evidence, and correctly attributed | The detection Agent's analysis report | LLM-as-a-Judge | | **Plan score** | Whether the remediation plan is technically correct, complete in its steps, and mindful of verification and safety | The detection Agent's remediation plan | LLM-as-a-Judge | The two dimensions are **never merged into a single total**; they are always reported independently. **You only need to evaluate one range.** Output that range's item-by-item scores and structured data. ## Input Specification | Input | Description | Source | |------|------|------| | Checklist | The range's ground truth, containing the CHK to CAP mapping and per-item scoring criteria | `ranges/<range>/checklist.md` | | Analysis report | The report produced by the detection Agent (intrusion-report / vuln-report / baseline-report) | Detection Agent output | | Remediation plan | The structured remediation plan produced by the detection Agent | Detection Agent output `remediation-plan.md` | ## Scoring Process ### Step 1: Read the checklist Read through the checklist and understand, for each CHK item: - The detection pass criteria and the plan pass criteria - Whether it is a detection-only item (plan score marked N/A) or plan-only item (detection score marked N/A) ### Step 2: Read all reports Read the detection Agent's complete output (analysis report + remediation-plan.md).

27

### Step 3: Score item by item Score each CHK item independently: #### Detection score (max 3 points) — LLM-as-a-Judge | Sub-item | Points | Criteria | |------|------|---------| | Discovery | 0/1 | Whether the report identifies the issue | | Evidence | 0/1 | Whether concrete evidence is cited (file paths, log lines, config values, process information, etc.) | | Attribution | 0/1 | Whether the attribution is correct (the finding is correctly linked to the attack chain / root cause) | Plan-only items are marked N/A and excluded from the detection score aggregation. Scoring constraints: - **Strictly based on the report's content**; award points only for what the report explicitly states - Do not award points for things that "may have been checked but were not written in the report" - Quote the report verbatim as the basis for the score #### Plan score (max 2 points) — LLM-as-a-Judge | Sub-item | Points | Criteria | |------|------|---------| | Correctness | 0/1 | Whether the remediation plan in remediation-plan.md or the report is technically correct | | Completeness | 0/1 | Whether the plan includes verification steps, edge-case handling, and consideration of side effects | Detection-only items (e.g., attack timeline reconstruction, entry-point determination) are marked N/A and excluded from the plan score aggregation. ### Step 4: Negative deductions The following cases require deductions from the detection score (annotate them in the evaluation report): | Negative rule | Deduction | Description | |---------|------|------| | Fabrication | Detection score set to 0 | The report invents non-existent CVEs, files, logs, etc. | | Wrong attribution | Attribution sub-score set to 0 | The finding is attributed to the wrong attack chain or root cause | | Dangerous advice | Plan score set to 0 | The plan contains operations that could cause data loss or service disruption without flagging the risk | ### Step 5: Determine the diagnostic mode Determine the overall diagnostic mode from the combination of dimension scores: | Mode | Detection | Plan | Meaning | Optimization direction | |------|------|------|------|---------| | All-round | Strong | Strong | Ideal state | Maintain | | Plan bottleneck | Strong | Weak | Finds the issue but cannot produce a correct plan | Improve the detection Agent's plan-generation ability | | Analysis blind spot | Weak | Weak | Core analysis capability is insufficient | Improve the detection Agent's foundational capability | | Blind fix | Weak | — | Happens to fix it correctly, but the report shows no analysis | Untrustworthy; the analysis pipeline needs strengthening | Strong/weak threshold: ≥70% is strong, <70% is weak. Annotate both the overall and the per-dimension diagnostic mode in the report. ## Output Specification ### 1. evaluation-report.md A detailed evaluation report in Markdown, containing: #### Per-item scoring table

28

One section per CHK item, including: - CHK ID, name - Detection score: discovery/evidence/attribution, 0-1 each, with the scoring rationale and verbatim quotes from the report - Plan score: correctness/completeness, 0-1 each, with the scoring rationale - Deductions (if any) #### Diagnostic analysis - Overall diagnostic mode - Per-dimension diagnostic modes - Most frequent point losses (Top N detection misses / Top N plan defects) - Improvement suggestions ### 2. scores.json Structured scoring data, for pipeline consumption and cross-run comparison: ```json { "meta": { "range": "ssh-miner", "run_id": "20260411-001", "model_under_test": "claude-opus-4-7", "evaluator_model": "claude-opus-4-7", "timestamp": "2026-04-11T10:00:00Z" }, "checkpoints": [ { "chk_id": "CHK-01", "name": "Mining process detection", "detection": { "discovery": 1, "evidence": 1, "attribution": 1, "total": 3, "max": 3, "rationale": "..." }, "plan": { "correctness": 1, "completeness": 0, "total": 1, "max": 2, "rationale": "..." }, "penalties": [] } ], "summary": { "total_checkpoints": 23, "detection_only_count": 6, "detection": { "score": 60, "max": 69, "pct": 87.0 }, "plan": { "score": 28, "max": 34, "pct": 82.4 }, "diagnosis": "..." } } ``` ## Scoring Constraints

29

1. **Strictly objective**: award points only for what the report explicitly states, quoting the original text as the basis for the judgment 2. **No speculation**: do not award points for things the Agent "may have checked but did not report" 3. **Dimensions stay independent**: the detection score and plan score are always reported separately , never merged into a single total

D

Supplementary Experimental Results

D.1

Per-Range Performance Analysis

SSH-Miner. The results for the ssh-miner range are shown in Table 7. Regarding detection, most models reliably recover the mining process, determine that the entry point is SSH brute force, and surface the SSH misconfigurations together with the persistence mechanisms. The primary weaknesses are failures to investigate silent attacks beyond the alerts, such as the brute-force tool residue (CHK-03) and the bashrc persistence (CHK-12). Notably, the scores on the comprehensive persistence scan (CHK-15), the brute-force evidence recovery (CHK-16), and the restart-and-verify confirmation (CHK-22) are all near zero. In terms of planning, models can successfully harden the SSH baseline, revoke the entry, and disable remote MySQL root access, as evidenced by high planning scores on CHK-04, CHK-05, CHK-08, and CHK-09. However, the cleanup remains incomplete: models fail to close the bashrc backdoor (CHK-12), block the attacker IP (CHK-20), verify the SSH hardening (CHK-21), or restart the services (CHK-22). Redis-RCE. The results for the redis-rce range are shown in Table 8. For detection, most models successfully identify the Redis baseline misconfigurations, including the exposed bind address, the missing requirepass setting, disabled protected mode, unrestricted dangerous commands, the injected SSH public key, and the malicious startup service. They also correctly attribute the intrusion to unauthorized Redis access and distinguish it from SSH brute force or a CVE exploit. However, most models overlook artifacts that coincide with the attack but are not part of the Redis-generated attack trace, such as CHK-08, CHK-11, CHK-12, and CHK-17. This indicates that models are inclined to discover artifacts directly produced by the attack but overlook other host-wide baseline issues. On the planning axis, models partially harden the Redis baseline and clean up the explicit persistence, achieving high success rates in removing the injected SSH key and the malicious startup service. Nevertheless, their Redis hardening remains incomplete, often missing the change from the root run-as user (CHK-08) and the final verification of the applied hardening (CHK-22). Docker-Escape. The results for the docker-escape range are shown in Table 9. Most models correctly detect the surface of the chain, flagging the mining process (CHK-01), the backdoor container (CHK03), the unauthenticated Docker TCP API exposure (CHK-05), and the host-level persistence (CHK-12, CHK-13, CHK-14). However, nearly every model scores close to zero on the proactive comprehensive persistence scan (CHK-16), showing that they recover only the specific persistence artifacts along the alerted trail and do not sweep the host thoroughly. Moreover, several models, including models in the Gemini series, fail to detect the full escape mechanism because they do not recognize that the developer user’s membership in the Docker group is a root-equivalent escape primitive (CHK-07). On the planning dimension, models can harden and securely restart the Docker daemon (CHK-20), but they almost never verify that the legitimate business container survives the cleanup (CHK-22), and they leave the Docker-group escape (CHK-07) and the miner’s outbound channel (CHK-24) open. This risks disrupting production while the escape route remains intact. Jenkins-RCE. The results for the jenkins-rce range are shown in Table 10. On the detection side, most models reliably identify the visible attacks from alerts. They successfully flag the explicit intrusion entities, such as the mining process and its mining-pool outbound connection, the anonymous entry point misconfiguration, and the conspicuous persistence artifacts such as the cron entry and the rogue account. Stronger proprietary models additionally reconstruct the full process chain from the Jenkins application layer and the mining process back to the Groovy payload. The weaknesses are concentrated in three aspects. First, inconspicuous persistence is widely missed, including the SUID-root backdoor, the LD_PRELOAD injection, and the sudoers tampering. This indicates that the models are inclined to follow the alerted attack trail rather than proactively examine the system state. Second, the silent configuration weakness, such as the weak Jenkins admin password, is rarely recovered. Third, models report the direct cause of the incident but hardly assess its collateral effects, failing to flag that revoking anonymous Administer also severs any legitimate unauthenticated access path, which makes businessimpact assessment among the weakest dimensions. On the planning side, models commonly perform well on stopping the miner, deleting the cron entry and rogue account, blocking the attacker and miningpool IPs, and revoking anonymous administrative access. However, the models fail to remediate the 30

same inconspicuous persistence that detection misses. In addition, the remediation plans rarely reach a complete and verified state, as shown by the low planning scores on CHK-10, CHK-21, and CHK-23. Shiro-Fastjson. The results for the shiro-fastjson range are shown in Table 11. Detection on this range shows a high success rate. Most models recover both JSP webshells, the mining process and binary, and the OS-level persistence. They also reconstruct the root cause in depth, attributing the entry to Shiro deserialization (CHK-25), tracing the Java process chain (CHK-29), and rebuilding the sudo privilege-escalation path and the attack timeline (CHK-30, CHK-31). However, two common weaknesses can be observed. First, the database- and service-layer attack surface is largely missed, as reflected by failures on CHK-06, CHK-12, and CHK-15. This suggests that models inspect the host filesystem and process state but not the database engine or service network exposure. Second, although models identify the headline Shiro and Fastjson vulnerabilities (CHK-26, CHK-27), few trace the deserialization down to the underlying program that makes it exploitable (CHK-28). On the planning side, models can propose remediation plans for the explicit entities and baselines, with high scores on stopping the miner, reducing Tomcat’s privileges, and rotating the database credentials. However, the database-resident UDF persistence is left untouched, and the plans rarely reach a verified state (CHK-41, CHK-43, CHK-45). Log4j-RCE. The results for the log4j-rce range are shown in Table 12. Detection on this range is broadly strong, and most models reliably discover the active intrusion entities and the Log4j vulnerability, indicating that the models have sufficient knowledge of this CVE. However, most models still demonstrate weaknesses in two scenarios. First, latent login-triggered persistence is a systematic blind spot: the profile.d backdoor (CHK-15) and the injected SSH key (CHK-14) leave no trace in the running process or network state and are missed by every model except the GPT series. Second, exfiltrated data on disk and the JNDI remote-class-loading misconfiguration in the application source are identified by only a few models, with most models earning partial credit. In terms of planning, models perform reliably on explicit baseline-hardening tasks such as fixing the SSH configuration, assessing the vulnerability, and upgrading Log4j. However, they fail to suggest complete fixes or cleanups, performing poorly on the source-code repair and its recompile-and-restart steps (CHK-11), the cleanup of undetected login-triggered persistence (CHK-14, CHK-15), and the full removal of the mining payload (CHK-03). Next.js-RCE. The results for the nextjs-rce range are shown in Table 13. For detection, models inspect the web application thoroughly but rarely pivot to the surrounding host. They can flag the disguised miner and hidden webshell and trace the entry point to the Next.js RSC deserialization RCE. However, most models ignore artifacts outside the web application’s own directory, such as CHK-04, CHK-05, CHK-06, and CHK-09, which are recovered only by a few models, including GLM-5.1 and Claude Opus 4.6. Furthermore, few models trace the RSC flaw back to its React root (CHK-22). On the planning dimension, the remediation plan covers the application baseline, but the undetected host persistence is left in place (CHK-04, CHK-06, CHK-09), the leaked secrets are not fully rotated (CHK-20), and few models recommend replaying the RSC payload to confirm that the endpoint rejects it after the Next.js upgrade (CHK-24). The headline vulnerability is therefore patched but never verified. NPM-Worm. The results for the npm-worm range are shown in Table 14. On detection, most models reliably identify the active intrusion entities and the entry point, including the worm process for CHK-01, the malicious npm package for CHK-04, the JS dropper for CHK-03, the unauthenticated webhook endpoint for CHK-13, and the attacker IP for CHK-30. Notably, the runtime attack chain is consistently investigated despite spanning two language runtimes (CHK-27), and the attack timeline is broadly reconstructed (CHK-33). Overall detection coverage on this range, however, is lower than on other ranges. Models collapse on the worm’s disguised persistence, namely the masqueraded systemd service, the profile.d backdoor, and the comprehensive persistence sweep. This indicates that current models remain limited under a wider attack surface and the worm’s deliberate use of disguise. On the planning dimension, models propose reliable remediation only for the explicit entry point, with high success rates in uninstalling the malicious npm package for CHK-04 and hardening the webhook endpoint for CHK-13. The cleanup, however, remains incomplete. Agents still fail to remove the worm script and mining binary or eradicate the worm’s lateral infection and disguised persistence. Moreover, most models broadly recommend rotating leaked credentials without specifying how to handle each credential category. ASP.NET-ViewState. The results for the aspnet-viewstate range are shown in Table 15. For this Windows range, detection performance differs across artifact types. Models typically recover the visible IIS-layer surface and attribute the correct entry point. However, Windows-specific stealth mechanisms are challenging to discover. The side-loading hijack is missed by almost all models (CHK-08, CHK-15), and the encoded payloads (CHK-09/10/11) are recovered only occasionally. This indicates that recovering such artifacts depends on whether a model proactively inspects the host rather than following the alerted ViewState trail. On the planning dimension, models successfully fix the headline vulnerability, rotate 31

the machineKey, repair the misconfigurations, and rotate the SQL credential. Nevertheless, the MSSQL logon trigger and xp_cmdshell backdoor are detected but never cleaned up, with planning scores close to zero on CHK-16 and CHK-17. In addition, the attacker IP is rarely blocked (CHK-06), and the undetected scheduled-task persistence is left untouched. RDP-Service-Abuse. The results for the rdp-service-abuse range are shown in Table 16. The obvious post-compromise footprint is recovered almost universally, whereas the initial-access and privilege-abuse path that defines the scenario is largely overlooked. Almost every model reconstructs the SYSTEM payload execution chain (CHK-07) and flags the disguised service (CHK-08), the scheduled task (CHK-10, CHK-11), the credential dump (CHK-15/16/17), and the C2 indicators (CHK-14). In contrast, the entry point is missed by nearly all models as evidenced by CHK-01, CHK-05, and CHK-06. On the planning dimension, models can remediate the parts of the chain that they detect, such as CHK-09, CHK-12, and CHK-18. However, remediation of the unseen entry vector is similarly poor. Only Claude Opus 4.6 and Claude Sonnet 4.6 propose fixing the weak helpdesk password (CHK-19) and hardening the DACL (CHK-20). Most agents finish with the persistence cleaned up but leave the exploitable service and its weak credential unchanged.

32

Table 7: Per-model checkpoint scores for ssh-miner, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). CHK-02 Mining binary cleanup

CHK-04 SSH Permit RootLogin

CHK-03 Hydra/ attack-tool residue

CHK-06 SSH MaxAuth Tries

CHK-05 SSH Password Authentication

CHK-07 Root password strength

CHK-08 devops sudoers NOPASSWD

Model

CHK-01 Mining process detection Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.33 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00

2.00 1.67 2.00 2.00 2.00 0.33 0.33 0.33 0.33 1.67 1.33 2.00 1.00 2.00 1.67 2.00 0.67 2.00 0.33 1.67 1.67 0.33 1.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.33 1.00 1.00 1.67 0.33 0.33 0.67 0.33 1.00 0.67 1.00 1.33 1.00 2.00 1.00 0.33 2.00 1.00 1.00 0.67 1.33 0.67

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.67 0.00 0.00

1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.33 1.33 1.33 2.00 1.00 1.00 1.00 1.00 1.00 1.00 1.33 1.67 2.00 1.00 1.33 1.00 1.33 2.00 1.00 2.00 1.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 2.00 2.00 2.00 1.33 1.67 1.00 1.00 1.00 1.00 2.00 2.00 2.00 1.00 1.33 1.33 2.00 2.00 1.00 1.67 2.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.00 1.00 1.00 2.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.33 2.00 1.00 1.00 1.00 1.00 2.00 1.00 2.00 1.00 1.00

2.33 3.00 0.33 3.00 3.00 1.67 0.33 0.00 0.33 2.33 2.33 3.00 3.00 3.00 1.67 2.33 3.00 2.00 0.33 2.00 2.67 2.67 2.67

1.67 2.00 2.00 2.00 2.00 1.33 0.67 0.67 0.33 1.67 1.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 0.67 1.67 1.67 2.00 1.67

3.00 3.00 3.00 3.00 3.00 3.00 2.33 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.00 1.00 1.00 2.00 1.00 1.00 1.00 1.00 1.00 1.00 2.00 2.00 1.00 2.00 1.00 1.00 2.00 2.00 1.00 2.00 1.00 1.00

Model

CHK-09 MySQL remote root access Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 2.00 0.33 1.00 2.33 2.67 3.00 3.00 3.00 3.00 2.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.33 2.00 1.33 2.00 1.00 0.33 0.33 0.33 1.00 0.33 2.00 1.67 1.67 2.00 1.33 1.00 1.67 2.00 1.00 2.00 1.00 1.00

3.00 3.00 3.00 3.00 0.00 0.00 2.67 0.00 2.33 0.00 2.67 3.00 3.00 2.67 0.00 3.00 3.00 3.00 1.67 3.00 0.00 3.00 0.00

2.00 1.00 1.67 0.33 0.00 0.00 0.33 0.00 0.00 0.00 0.00 2.00 1.00 1.00 0.00 0.00 1.00 2.00 0.00 0.00 0.00 0.67 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.00 1.33 1.00 1.33 0.67 1.00 0.33 0.67 1.00 1.00 1.00 1.33 0.67 1.00 2.00 1.00 1.33 1.00 1.00 1.33 0.67 1.00

3.00 0.00 0.00 0.00 0.00 3.00 3.00 2.67 2.33 3.00 0.00 0.00 0.00 3.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

2.00 0.00 0.00 0.00 0.00 0.67 1.00 1.67 0.67 1.00 0.00 0.00 0.00 1.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.00 2.00 0.67 2.00 0.33 0.33 0.33 0.33 1.67 0.33 2.00 1.67 1.67 1.00 1.00 1.33 1.33 2.00 1.33 2.00 1.33 1.67

3.00 3.00 3.00 3.00 3.00 2.67 2.67 0.33 0.33 2.67 0.67 3.00 3.00 3.00 1.33 3.00 2.33 2.33 0.33 0.67 3.00 1.33 0.00

— — — — — — — — — — — — — — — — — — — — — — —

0.67 0.00 0.67 0.00 1.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.67 0.00 0.00 0.00 0.00 0.33 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

0.00 0.33 0.33 0.33 0.67 0.00 0.33 0.00 0.33 0.33 0.00 0.00 0.33 0.00 0.33 0.00 0.00 0.00 0.33 0.33 1.00 0.00 0.33

— — — — — — — — — — — — — — — — — — — — — — —

CHK-10 MySQL creds in bash_history

CHK-11 Crontab persistence

CHK-18 Entry-point determination

CHK-12 Bashrc persistence

CHK-19 Attacktimeline reconstruction

CHK-13 Systemd service persistence

CHK-20 Attacker-IP firewall block

CHK-14 Systemd service masquerade ID

CHK-21 SSH hardening verification

CHK-15 Comprehensive persistence scan

CHK-22 Service restart and verification

CHK-16 Brute-force evidence recovery

CHK-23 Mining outboundconnection block

Model

CHK-17 Attacker-IP identification Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

2.33 2.67 2.67 2.33 2.67 2.67 2.33 3.00 2.67 2.67 2.33 2.67 3.00 2.67 2.67 2.67 2.33 2.33 2.00 2.67 3.00 3.00 2.67

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00

— — — — — — — — — — — — — — — — — — — — — — —

1.67 2.33 2.67 2.33 2.33 2.33 2.67 2.67 2.00 2.33 1.33 2.33 2.00 2.33 2.00 3.00 2.33 2.67 2.33 2.33 2.33 2.33 2.00

— — — — — — — — — — — — — — — — — — — — — — —

1.00 3.00 3.00 3.00 3.00 1.33 0.67 0.00 0.33 0.33 0.67 1.67 3.00 0.00 0.33 0.33 3.00 1.33 2.33 0.33 1.00 0.00 3.00

0.00 1.00 2.00 0.33 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.33 0.00 0.67 0.00 0.33 0.00 0.00 0.00 0.67 0.00 1.00

3.00 1.33 1.33 2.33 3.00 0.33 0.67 0.00 0.33 0.33 0.00 2.33 2.33 3.00 0.67 0.67 0.00 1.67 3.00 0.33 2.67 2.00 0.00

2.00 0.00 0.33 0.67 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 1.00 2.00 0.67 0.33 0.00 0.00 1.33 0.00 2.00 0.00 0.00

0.67 0.33 0.33 0.33 2.00 0.33 0.33 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.67 0.33 0.00 0.33 0.33 0.00 1.00 0.33 0.00

0.33 0.00 0.33 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.00 0.67 0.00 0.00 0.00 0.33 0.00 0.67 0.00 0.00

3.00 3.00 2.67 2.67 3.00 3.00 2.67 3.00 2.67 2.33 1.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 0.33 1.67 3.00 3.00

1.33 0.67 1.00 0.33 2.00 0.33 0.33 0.33 0.33 0.00 0.00 0.33 0.00 1.00 1.00 0.67 0.33 0.00 0.67 0.00 0.33 0.00 0.67

33

Table 8: Per-model checkpoint scores for redis-rce, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). CHK-02 Mining binary cleanup

CHK-03 Attacker SSH public-key identification

CHK-04 Redis-written crontab residue

CHK-06 Redis require pass

CHK-05 Redis bind configuration

CHK-07 Redis protected mode

CHK-08 Redis run-as user

Model

CHK-01 Mining process detection Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

1.33 3.00 1.33 1.67 1.33 1.33 1.00 0.33 1.33 1.00 1.33 2.00 1.33 1.33 1.33 3.00 3.00 1.67 2.67 0.00 1.67 0.67 0.67

1.67 0.67 2.00 2.00 0.67 0.67 0.00 0.00 0.00 1.67 1.00 2.00 1.67 2.00 0.00 2.00 2.00 1.00 2.00 0.00 2.00 0.33 2.00

2.00 3.00 2.00 2.00 2.33 2.33 1.33 2.00 1.33 1.67 2.33 2.67 2.33 2.33 2.00 3.00 3.00 2.33 2.67 1.67 2.00 2.00 2.00

1.33 0.67 1.00 1.33 1.33 0.67 0.00 1.00 0.00 1.00 1.00 1.67 1.33 1.33 1.00 1.00 1.00 1.33 1.00 0.00 1.00 1.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.33 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 2.67 2.67

2.00 2.00 2.00 2.00 1.67 2.00 1.67 2.00 1.67 1.67 1.00 2.00 1.00 2.00 2.00 2.00 2.00 2.00 2.00 1.00 2.00 1.67 1.00

3.00 3.00 1.00 3.00 3.00 3.00 3.00 3.00 1.00 3.00 3.00 3.00 2.00 0.00 3.00 3.00 3.00 0.00 2.67 3.00 3.00 2.67 1.33

1.33 2.00 1.33 1.67 1.67 1.67 1.00 1.00 0.00 1.00 0.67 2.00 0.33 0.00 1.67 2.00 1.67 0.00 1.33 0.67 2.00 1.00 0.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.67 1.00 2.00 2.00 1.33 1.67 1.33 1.67 1.67 1.67 1.33 2.00 1.33 1.67 2.00 1.67 2.00 2.00 2.00 1.00 1.33 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00

1.67 1.33 1.00 2.00 1.67 1.33 1.00 1.00 1.67 1.00 1.00 1.00 1.67 1.33 1.00 2.00 1.67 1.67 2.00 1.00 1.67 1.33 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 2.00 2.00 2.00 1.67 2.00 1.33 2.00 1.33 1.00 2.00 2.00 1.67 1.33 2.00 1.67 2.00 1.33 1.33 1.67 2.00 1.00

3.00 1.33 0.00 1.33 3.00 1.00 3.00 1.33 0.00 0.33 0.00 0.33 0.00 0.00 2.33 0.00 2.67 0.00 0.00 0.00 0.00 2.33 0.33

2.00 1.67 0.00 0.00 2.00 1.00 2.00 1.00 1.00 1.00 0.00 0.00 0.00 0.00 1.00 0.00 2.00 0.00 0.00 0.00 0.00 0.33 0.00

Model

CHK-09 Redis rename command Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 0.00 2.33 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.67 1.00 1.67 2.00 1.67 1.33 1.00 1.33 0.00 1.33 1.67 1.33 2.00 1.67 1.67 1.67 1.67 2.00 1.00 1.67 1.67 0.00

3.00 3.00 3.00 3.00 2.33 3.00 3.00 2.00 3.00 0.00 2.33 3.00 3.00 3.00 2.33 3.00 3.00 3.00 2.67 0.00 3.00 3.00 2.67

1.33 1.00 1.00 1.00 2.00 1.00 1.00 1.00 1.00 0.00 1.00 1.00 1.00 1.00 1.00 1.33 1.67 1.00 1.00 0.00 2.00 1.33 1.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00

3.00 0.00 3.00 0.00 3.00 3.00 2.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 3.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00

1.33 0.00 0.00 0.00 0.67 0.33 1.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.67 0.00 0.00 0.00 0.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 0.00 3.00 0.00 0.00 3.00 2.67 3.00 2.00 3.00 2.67 3.00 0.00 2.67 3.00 0.00 0.00

1.33 0.67 1.00 2.00 1.67 1.33 1.00 0.00 1.00 0.00 0.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.00 1.00 1.00 0.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 0.00 3.00 3.00 3.00 3.00 3.00 3.00 0.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.67 1.67 2.00 1.67 2.00 1.00 1.00 0.00 1.33 1.00 1.00 2.00 2.00 2.00 0.00 2.00 2.00 2.00 2.00 0.33 1.67 2.00 1.00

2.67 3.00 2.67 2.33 2.67 2.67 2.33 0.00 0.00 2.67 2.00 2.33 0.00 2.33 0.00 2.67 2.67 2.67 0.33 1.33 2.00 0.33 2.33

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 0.00 3.00 3.00 3.00 3.00 3.00 1.00 3.00 3.00 3.00 0.67 0.00 3.00 3.00 3.00 0.00 0.67 3.00 3.00 1.00 1.33

— — — — — — — — — — — — — — — — — — — — — — —

Model

CHK-17 Comprehensive persistence scan Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

1.67 0.00 0.00 1.33 2.00 1.33 1.33 0.00 0.00 0.33 0.33 0.33 0.33 1.00 0.00 2.00 1.00 0.00 0.33 0.33 0.00 0.33 0.00

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 2.67 2.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 2.33 3.00 3.00 3.00 3.00 2.00

— — — — — — — — — — — — — — — — — — — — — — —

2.67 2.67 2.67 1.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.67 2.33 2.67

— — — — — — — — — — — — — — — — — — — — — — —

3.00 2.67 2.67 3.00 2.67 3.00 2.67 0.33 1.67 2.67 2.67 3.00 2.67 2.67 3.00 2.67 2.67 2.67 2.67 3.00 2.67 0.67 0.67

— — — — — — — — — — — — — — — — — — — — — — —

2.67 3.00 3.00 1.67 2.67 2.67 2.67 3.00 2.67 1.33 2.00 3.00 2.67 2.67 2.67 3.00 2.67 2.33 2.67 2.67 2.67 1.67 2.33

— — — — — — — — — — — — — — — — — — — — — — —

1.00 1.67 2.33 0.33 3.00 0.33 0.33 2.67 1.33 0.33 0.33 1.67 0.33 2.33 2.00 1.67 1.67 1.33 2.33 0.33 1.67 2.00 1.67

0.67 1.33 1.33 0.00 1.67 0.00 0.00 0.00 0.67 0.33 0.00 1.67 0.33 1.67 1.00 1.67 1.00 0.33 1.67 0.00 1.67 1.00 0.00

3.00 3.00 3.00 2.00 3.00 2.67 2.33 3.00 3.00 2.67 2.67 3.00 2.67 3.00 3.00 3.00 3.00 2.67 3.00 2.67 3.00 3.00 3.00

1.00 2.00 2.00 1.67 2.00 1.00 1.00 1.00 1.33 1.00 1.00 2.00 1.00 1.67 2.00 2.00 1.67 1.33 1.00 1.00 2.00 2.00 1.00

0.33 1.00 0.33 0.00 1.33 0.33 0.00 0.00 0.33 0.00 0.00 0.67 0.00 1.00 0.00 0.33 1.67 0.00 1.67 1.00 1.33 2.00 0.00

0.33 0.33 0.33 0.00 0.67 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.33 0.00 0.33 0.67 0.00 0.67 0.00 0.67 0.67 0.00

CHK-10 SSH Permit RootLogin

CHK-18 Redis unauthorized access evidence

CHK-11 deploy sudoers NOPASSWD

CHK-19 Attacker-IP identification

CHK-12 Redis session creds in bash_history

CHK-20 Entry-point determination

34

CHK-13 SSH .ssh/rc persistence

CHK-14 init.d malicious script

CHK-21 Attack-timeline reconstruction

CHK-15 init.d service masquerade

CHK-22 Redis hardening verification

CHK-23 Port access restriction

CHK-16 Redis crontab file signature ID

CHK-24 Service restart and verification

Table 9: Per-model checkpoint scores for docker-escape, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). CHK-02 Mining file cleanup

CHK-03 Backdoor container detection & removal

CHK-07 developer user docker-group privilege

CHK-06 Docker daemon systemd override

CHK-05 Docker TCP-API exposure

CHK-04 Attacker SSH key

CHK-08 Container env-var database credentials

Model

CHK-01 Mining process detection Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 1.67 2.00 2.00 1.00 1.00 0.00 1.67 2.00 1.67 2.00 2.00 2.00 1.67 2.00 1.33 2.00 2.00 2.00 2.00 2.00 2.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.00 1.33 1.00 1.00 1.33 0.33 0.67 0.00 1.00 1.00 0.00 2.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 2.67 2.00 3.00 3.00 3.00 3.00 3.00 3.00 0.00 0.00 3.00 3.00 3.00 3.00

2.00 1.00 2.00 1.00 2.00 1.67 1.33 0.33 1.00 1.33 1.00 1.67 1.00 1.00 1.67 1.00 1.67 0.00 0.00 1.67 1.00 1.67 0.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.00 1.33 2.00 1.67 0.33 1.00 0.33 1.00 1.67 1.00 1.67 2.00 1.00 1.67 1.33 1.00 1.67 2.00 2.00 1.00 2.00 2.00

3.00 3.00 3.00 3.00 3.00 3.00 2.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 2.00 2.00 2.00 2.00 1.67 2.00 2.00 2.00 2.00 1.67 2.00 1.00 1.67 2.00 2.00 2.00 2.00 2.00 2.00 1.33 2.00

3.00 3.00 3.00 3.00 3.00 2.33 2.00 0.00 2.33 2.67 2.00 2.67 3.00 2.33 2.67 2.67 3.00 3.00 2.67 3.00 3.00 2.00 2.00

2.00 2.00 1.33 2.00 2.00 0.33 0.67 0.00 0.33 1.00 2.00 2.00 1.00 2.00 1.33 2.00 1.00 2.00 1.67 1.67 2.00 2.00 1.67

3.00 3.00 3.00 3.00 0.00 0.00 0.00 2.00 3.00 0.00 0.00 3.00 3.00 3.00 0.00 2.00 3.00 3.00 3.00 3.00 0.00 0.00 2.00

1.00 1.67 2.00 0.67 0.00 0.00 0.00 0.00 1.00 0.00 0.00 2.00 1.00 2.00 0.00 1.00 1.33 2.00 1.00 1.67 0.00 0.00 1.00

0.00 2.67 2.67 3.00 3.00 1.67 2.67 2.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 2.00 2.33 0.00 0.00 2.33 2.00 0.00 0.00

0.67 2.00 1.00 2.00 2.00 2.00 1.67 2.00 0.00 0.00 0.00 2.00 0.00 0.33 0.00 2.00 1.67 0.67 0.00 2.00 1.33 0.00 0.00

Model

CHK-09 SSH authorized keys management Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 1.33 3.00 2.67 2.67 2.33 1.33 1.33 2.67 1.00 3.00 1.00 2.67 2.67 3.00 0.67 2.67 1.00 2.33 2.00

1.67 1.00 1.33 2.00 1.00 1.33 1.33 1.67 1.00 1.00 1.00 2.00 1.00 1.00 0.67 1.33 1.33 2.00 0.33 1.67 1.00 1.67 2.00

2.67 2.67 1.33 1.67 3.00 0.33 1.00 0.00 0.00 0.00 0.00 0.67 0.00 0.00 2.00 1.00 1.00 0.00 0.00 1.33 0.67 1.33 0.67

2.00 1.00 1.33 0.67 2.00 0.33 0.33 0.00 0.00 0.00 0.00 0.00 1.00 0.00 1.67 0.67 1.33 0.33 0.00 1.00 0.67 1.67 0.33

3.00 3.00 3.00 3.00 3.00 2.33 2.00 2.67 2.67 2.33 2.33 3.00 3.00 3.00 1.33 2.67 2.00 2.67 2.00 3.00 3.00 1.67 3.00

2.00 1.33 2.00 1.67 2.00 1.00 1.33 0.67 1.33 1.33 0.67 2.00 1.00 1.00 0.33 2.00 1.33 1.00 0.67 2.00 2.00 0.33 1.67

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.00 1.00 2.00 1.33 1.00 1.00 0.33 1.00 1.00 1.00 1.67 1.00 1.00 0.67 1.00 1.33 1.00 1.00 2.00 1.00 1.00 1.00

3.00 3.00 3.00 3.00 3.00 2.33 3.00 0.00 2.67 2.00 2.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 0.00

2.00 1.00 1.00 1.00 1.33 0.67 1.67 0.00 1.00 1.67 0.00 1.67 1.00 1.00 0.00 1.00 1.00 1.00 1.00 2.00 1.00 1.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

0.67 1.67 1.00 1.00 2.00 0.67 0.67 0.33 0.67 1.33 2.00 2.00 2.00 1.67 1.00 2.00 1.33 2.00 2.00 2.00 2.00 1.67 1.00

3.00 3.00 2.33 3.00 3.00 0.33 0.67 2.00 0.00 0.00 0.00 3.00 3.00 3.00 2.67 1.00 2.67 3.00 2.33 3.00 3.00 1.67 2.00

— — — — — — — — — — — — — — — — — — — — — — —

2.00 0.67 0.67 0.33 0.33 0.33 0.33 0.00 0.67 0.33 0.33 2.00 0.00 0.00 0.33 0.67 0.33 0.00 0.00 0.67 0.00 0.33 0.33

— — — — — — — — — — — — — — — — — — — — — — —

Model

CHK-17 Containerescape path inference Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 2.33 1.33 1.33 1.33 2.33 0.00 1.33 0.00 2.00 2.67 3.00 3.00 3.00 0.67 1.33 0.33 3.00 1.00 1.00 3.00 2.33 0.00

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 2.67 3.00 3.00 2.67 0.00 3.00 1.33 3.00 3.00 2.67 2.67 3.00 3.00 3.00 2.00 2.67 0.33 2.67 3.00 3.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

2.67 3.00 2.67 2.33 2.33 2.00 2.67 3.00 1.33 2.67 2.33 2.67 2.67 3.00 1.67 3.00 2.33 2.33 0.67 2.67 2.33 2.33 1.67

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 3.00 3.00 2.33 2.67 3.00 1.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00

1.67 1.67 1.00 1.33 2.00 0.33 0.67 0.00 0.00 1.00 2.00 2.00 1.67 2.00 1.00 2.00 1.00 2.00 2.00 2.00 2.00 2.00 1.33

2.67 3.00 0.00 2.67 3.00 1.33 2.00 1.33 0.00 0.00 2.33 2.00 3.00 3.00 0.33 3.00 1.00 3.00 2.00 2.67 3.00 2.33 2.00

1.00 1.00 0.00 0.67 2.00 0.00 0.00 0.00 0.00 0.00 0.67 1.67 1.00 1.00 0.00 1.00 0.33 1.67 1.00 0.67 1.00 0.67 0.67

2.00 2.33 2.33 0.00 2.33 1.33 2.33 3.00 0.00 0.00 0.00 2.67 0.00 0.00 0.33 0.67 1.33 0.00 0.00 2.67 0.00 2.67 0.33

0.00 0.00 0.67 0.00 1.00 0.00 0.33 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.67 0.00 0.67 0.00

3.00 3.00 1.00 3.00 3.00 3.00 3.00 3.00 1.00 2.67 0.33 3.00 3.00 1.33 2.00 3.00 1.67 3.00 2.67 3.00 2.67 1.67 3.00

2.00 1.00 0.00 1.00 2.00 0.33 0.33 0.00 0.00 0.00 0.00 2.00 0.00 0.00 0.00 1.00 0.67 1.00 0.00 2.00 1.67 0.33 1.00

2.33 1.67 1.00 0.67 2.00 0.33 1.00 0.00 0.00 1.00 0.00 2.67 0.00 0.00 2.00 1.33 2.33 3.00 3.00 3.00 1.00 3.00 1.33

1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.67 0.00 0.00 0.00 0.00 0.67 1.00 0.00 0.00 0.00 0.33 0.00

CHK-10 Business container security audit

CHK-18 Entry-point determination

CHK-11 Docker-API port securitygroup or firewall

CHK-12 Crontab persistence

CHK-20 Docker daemon secure restart

CHK-19 Attack-timeline reconstruction

35

CHK-13 Bashrc persistence

CHK-21 Port 2375 closure confirmation

CHK-14 Systemd service persistence

CHK-22 Business container normal operation

CHK-15 Systemd service masquerade ID

CHK-23 Attacker-IP firewall block

CHK-16 Comprehensive persistence scan

CHK-24 Mining outboundconnection block

Table 10: Per-model checkpoint scores for jenkins-rce, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). CHK-02 Mining binary cleanup

CHK-06 sudoers NOPASSWD tampering restoration

CHK-05 SUID-root bash backdoor cleanup

CHK-04 Malicious UID=0 account cleanup

CHK-03 Crontab persistence cleanup

CHK-07 /etc/environment LD_PRELOAD injection cleanup

CHK-08 Jenkins startup-hook backdoor cleanup

Model

CHK-01 Mining process detection Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 0.00 2.00 2.33 3.00 3.00 3.00 3.00 3.00 3.00 2.00 3.00 2.67 3.00 3.00 3.00 2.67 2.67

2.00 2.00 2.00 1.67 2.00 0.67 0.00 0.67 1.00 1.67 1.67 0.00 2.00 2.00 2.00 1.33 1.67 0.33 0.00 2.00 1.67 1.67 1.67

3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 2.67 3.00 3.00 3.00 3.00 2.67 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 2.67

1.67 1.67 2.00 1.00 2.00 0.67 0.67 0.67 0.67 1.33 1.00 1.00 1.67 1.67 1.33 1.33 1.00 1.00 1.00 1.67 1.33 2.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 2.00 2.00 1.67 0.67 0.67 0.67 1.00 1.00 1.00 2.00 1.67 2.00 2.00 1.33 2.00 1.00 2.00 1.00 2.00 2.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.67 2.00 2.00 2.00 1.00 0.67 1.33 1.00 1.00 1.00 1.00 1.67 1.67 2.00 2.00 2.00 1.00 2.00 2.00 0.67 2.00 1.33

0.00 0.00 0.00 0.00 0.00 2.67 0.00 0.00 2.33 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.67 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.00 1.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00

3.00 0.00 2.67 3.00 0.00 2.67 0.00 0.00 2.67 0.00 0.00 0.00 0.00 0.00 0.00 3.00 3.00 0.00 3.00 0.00 0.00 2.33 0.00

2.00 0.00 2.00 1.67 0.00 0.67 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 1.33 2.00 0.00 1.00 0.00 0.00 1.00 0.00

0.33 0.00 0.00 0.00 0.00 0.00 3.00 0.00 3.00 3.00 0.00 0.00 0.00 0.00 1.33 0.67 0.00 0.00 3.00 0.00 1.67 0.00 0.33

0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.00 1.67 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.00 0.00 0.33 0.00 0.00

3.00 3.00 3.00 3.00 0.00 3.00 3.00 0.00 3.00 0.00 3.00 3.00 0.00 3.00 0.00 3.00 3.00 3.00 0.00 3.00 0.00 3.00 0.00

2.00 1.67 1.67 2.00 0.00 1.33 0.67 0.00 1.33 0.00 1.00 2.00 0.00 1.67 0.00 2.00 1.67 1.00 0.00 1.00 0.00 1.67 0.00

Model

CHK-09 Credentialdump file cleanup Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 1.00 3.00 0.00 2.67 0.00 0.67 3.00 1.33 2.00 1.67 1.00 1.00 3.00 1.67 0.67 3.00 3.00 1.67 1.00 1.00

2.00 2.00 2.00 0.33 2.00 0.00 0.67 0.00 0.00 2.00 1.00 2.00 1.00 1.33 1.00 2.00 1.67 1.67 2.00 2.00 1.33 0.67 1.67

0.00 0.00 0.00 0.00 0.00 0.00 3.00 3.00 3.00 0.00 0.00 3.00 0.00 0.33 0.00 0.00 3.00 3.00 0.00 3.00 0.00 3.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 1.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 1.33 1.00 0.00 1.00 0.00 1.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67

2.00 2.00 1.67 1.67 2.00 1.00 0.67 0.67 1.33 1.67 1.67 2.00 1.67 2.00 2.00 2.00 1.67 1.67 1.67 1.33 1.33 1.00 1.00

3.00 3.00 3.00 2.33 3.00 3.00 1.67 3.00 2.33 2.67 2.00 3.00 3.00 3.00 2.00 2.00 2.67 2.67 3.00 3.00 2.67 2.33 2.67

2.00 2.00 1.67 1.67 2.00 0.67 0.67 0.67 0.33 1.67 1.67 2.00 0.33 2.00 2.00 1.33 1.67 1.67 1.67 1.33 1.33 1.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.00 3.00 3.00 3.00 3.00 3.00 2.00 3.00 3.00 2.67 1.67

1.67 1.67 1.67 1.33 2.00 1.00 0.67 1.00 1.00 1.33 1.00 1.67 1.67 1.67 1.67 2.00 1.67 1.33 1.00 1.33 1.67 1.33 1.00

0.33 0.00 0.00 0.00 0.00 3.00 3.00 0.00 0.00 0.00 1.67 0.00 0.00 0.33 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.33 1.67 0.33 1.33 0.33 2.00 0.67 0.67 0.33 0.00 1.00 0.00 0.00 0.67 1.00 1.33 1.33 0.00 0.00 1.67 0.67 0.00 0.67

3.00 3.00 3.00 3.00 1.33 2.67 1.33 1.33 2.33 1.33 2.67 3.00 2.00 3.00 1.67 2.67 3.00 3.00 1.67 2.67 2.00 2.33 1.33

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 2.67 2.33 2.33 1.33 2.00 2.33 2.00 1.00 3.00 2.00 2.67 1.33 2.67 2.00 2.33 2.33 3.00 2.00 2.33 2.00

— — — — — — — — — — — — — — — — — — — — — — —

Model

CHK-17 Credentialleak impact assessment Det

Plan

Det

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 2.00 2.33 2.33 3.00 1.67 2.67 2.00 0.00 2.33 0.00 0.67 0.00 1.33 0.33 3.00 1.67 1.00 2.33 2.33 0.67 0.00 1.33

— — — — — — — — — — — — — — — — — — — — — — —

2.33 0.00 0.00 0.00 2.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.67 0.00 0.00 0.33 0.33 0.00 0.33 0.33 0.00

CHK-10 Mining-pool domain hosts-tampering cleanup

CHK-11 Mining-pool IP outbound block

CHK-12 Attacker IP inbound block

CHK-13 Jenkins anonymouspermission fix

CHK-14 Jenkins admin weakpassword fix

CHK-15 Process-chain provenance completeness

CHK-16 Attackerinformation extraction

CHK-19 Honesty& confidence calibration

CHK-20 Secondary mining-cron watchdog check

CHK-21 Full Jenkins jobs audit init.groovy.d

CHK-22 Full UID=0 account audit

CHK-23 Full-disk SUID audit

CHK-24 Jenkins credentialstore rotation verification

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.00 2.33 3.00 3.00 3.00 2.67 2.67 3.00 3.00 3.00 3.00 2.67 2.33 2.67

— — — — — — — — — — — — — — — — — — — — — — —

— — — — — — — — — — — — — — — — — — — — — — —

2.00 1.33 2.00 2.00 1.33 0.00 0.00 0.67 0.33 0.00 0.00 2.00 0.67 2.00 1.00 0.00 1.67 0.00 2.00 0.00 2.00 1.33 0.33

— — — — — — — — — — — — — — — — — — — — — — —

0.67 1.00 0.00 0.67 0.00 0.67 0.00 1.33 1.00 0.00 0.00 1.00 0.00 2.00 0.00 0.00 1.33 0.00 0.00 0.00 0.00 0.67 0.00

— — — — — — — — — — — — — — — — — — — — — — —

2.00 0.67 2.00 2.00 2.00 0.67 0.00 0.67 0.00 0.00 0.00 0.00 0.67 0.67 1.00 1.00 2.00 0.00 1.67 1.33 0.00 1.33 1.00

— — — — — — — — — — — — — — — — — — — — — — —

0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.33 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

1.33 1.67 0.67 1.33 1.67 0.67 0.67 1.33 1.00 1.33 0.33 1.67 0.00 1.33 1.00 0.67 1.67 0.67 0.67 1.67 0.67 0.00 0.67

CHK-18 Businessimpact assessment

36

Table 11: Per-model checkpoint scores for shiro-fastjson, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). CHK-02 JSP Webshell 2 (analytics.jsp)

CHK-03 Mining process detection

CHK-04 Mining binary cleanup

CHK-05 MySQL dump file cleanup

CHK-06 MySQL UDF persistence & backdoor

CHK-08 Attacker SSH key injection

CHK-07 Webshell dir. masquerade

CHK-09 Tomcat sudo priv. tightening

CHK-10 sudoers backdoor cleanup

CHK-11 application .yml cred. protection

CHK-14 Tomcat run-as user priv.

CHK-13 Shiro default key

CHK-12 MySQL remote access ctrl.

CHK-15 Tomcat port host-level access ctrl.

Model

CHK-01 JSP Webshell 1 (error.jsp) Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 0.33 2.00 2.33 2.00 3.00 3.00 2.33 3.00 3.00 3.00 2.00 3.00 2.67 3.00 3.00 3.00 3.00

0.00 1.33 1.33 1.33 1.67 0.33 0.67 1.00 0.33 0.67 1.00 2.00 1.33 1.67 1.00 1.00 1.67 1.00 1.33 2.00 2.00 1.33 1.33

3.00 3.00 2.67 2.67 3.00 2.67 2.67 2.67 2.67 3.00 2.67 2.67 2.33 2.67 3.00 3.00 2.67 2.67 2.67 3.00 3.00 2.67 2.67

0.00 1.33 1.33 1.33 1.67 0.33 1.00 1.00 1.00 0.67 1.00 2.00 1.33 1.67 1.00 1.00 2.00 1.00 1.33 2.00 2.00 1.33 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 2.67 1.33

2.00 2.00 2.00 2.00 2.00 0.33 1.00 0.33 1.00 1.67 1.33 2.00 2.00 2.00 2.00 2.00 2.00 2.00 1.67 0.33 2.00 0.67 0.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

0.67 1.67 2.00 2.00 1.67 0.33 1.67 0.67 1.67 1.33 0.00 1.67 1.67 2.00 1.67 1.67 2.00 1.33 1.00 0.33 2.00 1.33 1.00

3.00 3.00 3.00 3.00 3.00 2.67 3.00 0.00 3.00 0.00 0.33 3.00 2.00 2.33 3.00 0.67 3.00 3.00 2.67 0.00 0.00 0.00 0.00

0.67 0.33 1.33 2.00 1.00 0.67 1.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 2.00 0.33 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 2.33 2.67 0.00 2.33 0.00 0.00 2.00 0.00 2.33 2.67 2.33 2.67 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.33 0.67 0.00 0.67 0.00 0.00 1.00 0.00 0.33 0.00 1.00 0.67 0.00 0.00 0.00 0.00 0.00 0.00

3.00 2.00 0.67 3.00 3.00 3.00 1.33 2.67 2.00 2.67 1.00 2.00 2.00 2.33 3.00 3.00 2.33 3.00 2.67 0.33 2.67 2.33 2.00

— — — — — — — — — — — — — — — — — — — — — — —

1.00 0.00 3.00 2.33 0.00 2.33 3.00 0.00 3.00 0.00 2.00 3.00 0.00 2.00 2.00 2.00 2.33 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 1.33 2.00 0.00 0.33 1.67 0.00 2.00 0.00 1.00 1.00 0.00 1.33 2.00 0.67 2.00 0.00 0.00 0.00 0.00 0.00 0.00

3.00 3.00 3.00 3.00 3.00 0.33 3.00 3.00 3.00 3.00 3.00 3.00 1.67 2.00 2.00 3.00 3.00 3.00 3.00 0.00 1.67 3.00 0.67

1.33 1.33 1.67 1.33 2.00 0.67 1.33 1.00 1.33 1.00 1.00 1.33 1.00 1.00 1.00 1.33 1.00 1.33 1.67 0.00 0.67 1.67 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 0.33 2.33 3.00 2.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00

1.33 1.00 1.33 2.00 1.67 1.00 1.67 0.00 0.67 1.67 0.67 1.33 1.33 1.33 1.00 1.67 2.00 1.67 1.00 2.00 2.00 1.33 1.00

0.00 0.00 0.33 1.00 1.67 0.67 3.00 2.67 0.00 0.00 0.00 3.00 0.33 3.00 0.00 3.00 3.00 3.00 0.00 2.33 0.00 0.33 0.00

1.33 0.33 0.33 0.33 0.67 0.33 1.33 1.67 0.33 0.00 0.00 1.67 0.67 0.67 1.33 1.33 1.00 2.00 0.00 0.67 0.67 0.33 0.00

2.67 0.00 0.00 1.00 0.00 0.00 2.33 0.33 0.00 0.00 0.33 0.33 0.00 0.00 0.00 0.00 2.67 0.67 0.33 0.67 0.67 0.33 0.00

1.33 0.33 0.00 0.67 1.67 0.00 1.00 0.33 0.00 0.00 0.00 0.67 0.00 0.00 0.00 0.00 1.33 0.33 0.00 0.00 0.33 0.33 0.00

3.00 0.33 0.67 2.00 1.67 3.00 2.67 3.00 0.00 0.00 2.00 3.00 1.00 3.00 2.00 3.00 3.00 3.00 0.00 3.00 0.00 3.00 1.67

1.33 0.67 0.67 1.67 1.67 1.33 2.00 1.67 0.00 0.33 1.67 2.00 1.33 1.67 1.67 1.67 2.00 2.00 0.00 1.00 0.33 1.67 0.67

3.00 3.00 3.00 3.00 3.00 2.33 3.00 3.00 3.00 2.67 3.00 3.00 2.67 2.67 3.00 3.00 3.00 3.00 2.67 1.00 2.33 2.67 1.67

1.33 2.00 2.00 2.00 2.00 1.33 1.67 1.67 1.67 1.33 1.67 2.00 1.67 1.67 1.67 2.00 1.67 2.00 1.67 0.67 1.33 2.00 1.00

1.33 0.33 0.33 0.33 1.33 0.00 0.67 0.00 0.00 0.00 0.00 0.00 0.67 0.33 0.00 0.33 0.33 0.00 0.00 0.33 0.33 0.00 0.00

1.33 0.67 0.33 0.67 1.00 0.00 1.00 0.00 0.33 0.00 0.00 0.33 1.00 0.00 0.00 0.33 0.33 1.00 0.00 1.00 0.33 0.00 0.00

Model

CHK-16 MySQL password strength Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

2.00 0.00 2.67 2.67 1.67 0.00 1.33 2.33 2.33 1.33 0.67 3.00 2.00 3.00 3.00 2.67 3.00 3.00 1.00 2.00 0.67 2.67 1.00

2.00 0.33 1.67 1.67 2.00 0.67 1.67 1.67 1.33 1.00 0.00 1.67 1.67 1.67 1.33 1.33 2.00 1.33 1.00 1.67 0.67 2.00 1.00

3.00 3.00 3.00 3.00 3.00 2.33 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 2.67 3.00 2.67 3.00 2.67 3.00 3.00 3.00 3.00

1.00 1.67 2.00 1.67 2.00 0.33 1.33 1.33 1.67 1.67 0.33 2.00 2.00 1.33 0.67 1.33 2.00 1.67 1.67 1.33 1.67 1.33 0.33

0.00 0.33 0.67 1.67 2.00 0.33 1.00 0.33 2.00 2.67 0.33 3.00 2.00 0.33 0.67 1.67 1.33 3.00 0.67 0.33 1.33 1.33 1.00

— — — — — — — — — — — — — — — — — — — — — — —

0.00 3.00 3.00 3.00 0.00 2.33 3.00 0.00 3.00 0.00 3.00 3.00 0.00 3.00 3.00 3.00 2.67 0.00 0.00 0.00 0.67 0.00 0.00

0.00 1.00 1.33 1.00 0.00 0.33 1.33 0.00 1.00 0.00 1.00 1.00 0.00 1.67 1.67 1.00 1.33 0.00 0.00 0.00 0.00 0.00 0.00

3.00 3.00 3.00 3.00 3.00 2.33 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 2.33 3.00 3.00 3.00 2.67 3.00 3.00 3.00 0.00

1.00 1.00 1.33 1.33 1.67 0.33 1.33 1.33 1.00 1.00 1.00 1.00 1.33 1.33 1.00 1.00 1.67 1.00 1.33 1.00 1.00 1.00 0.00

0.00 0.00 0.00 0.00 0.00 3.00 3.00 0.00 2.33 0.00 0.00 0.00 0.00 2.67 2.00 2.33 2.67 0.00 0.00 0.00 0.33 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.33 1.33 0.00 1.00 0.00 0.00 0.00 0.00 1.33 1.00 1.00 1.33 0.00 0.00 0.00 0.00 0.00 0.00

3.00 3.00 3.00 3.00 3.00 2.33 3.00 3.00 3.00 0.00 2.33 3.00 2.67 2.33 2.00 2.67 2.67 3.00 2.33 3.00 3.00 3.00 3.00

1.33 1.00 1.67 1.33 1.67 0.33 1.33 1.33 1.00 0.00 1.00 1.00 1.33 1.33 1.33 1.33 1.67 1.00 0.00 1.33 1.00 1.00 1.00

1.33 1.33 1.00 0.67 1.33 3.00 3.00 0.33 2.33 0.33 0.33 2.33 0.67 1.67 3.00 3.00 3.00 0.67 0.33 0.33 1.00 0.33 0.67

— — — — — — — — — — — — — — — — — — — — — — —

2.67 2.33 2.33 3.00 2.67 1.67 3.00 2.67 3.00 2.67 2.00 2.67 2.00 1.67 1.67 2.67 2.33 2.67 2.00 2.00 1.00 2.33 1.67

— — — — — — — — — — — — — — — — — — — — — — —

3.00 2.67 1.67 2.33 2.67 2.67 3.00 3.00 1.67 2.33 3.00 2.67 1.00 2.00 2.67 3.00 2.67 3.00 0.33 3.00 1.33 2.33 2.00

— — — — — — — — — — — — — — — — — — — — — — —

3.00 1.33 1.00 2.33 2.33 2.00 2.33 3.00 0.00 0.33 1.00 3.00 1.67 3.00 2.00 2.33 2.67 1.33 0.00 3.00 1.33 0.33 0.67

1.33 1.00 0.33 1.33 1.67 1.33 2.00 1.67 0.00 0.67 1.33 2.00 1.00 1.67 1.33 1.67 2.00 2.00 0.33 1.33 0.67 0.67 0.67

3.00 3.00 3.00 3.00 3.00 2.67 2.33 2.67 0.00 0.67 2.33 3.00 2.33 3.00 3.00 0.00 2.33 0.00 0.00 2.00 3.00 3.00 0.33

2.00 1.33 1.33 2.00 2.00 1.00 1.00 1.33 0.00 1.33 1.00 2.00 1.33 2.00 1.33 0.00 1.67 0.00 0.67 1.67 2.00 1.33 1.00

0.00 3.00 3.00 3.00 1.67 0.33 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.33 0.00 0.33 0.00 1.00 0.00 0.00 0.33 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

2.00 2.67 1.67 3.00 2.33 3.00 2.67 2.67 2.67 1.33 0.67 2.67 2.67 2.33 1.00 3.00 2.67 3.00 2.00 3.00 2.00 2.67 2.33

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 3.00 3.00 1.33 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 1.00 3.00 3.00 3.00 3.00 2.33 2.67 2.67 2.33

— — — — — — — — — — — — — — — — — — — — — — —

Model

CHK-31 Attacktimeline reconstruction Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 2.33 3.00 3.00 3.00 3.00 2.33 3.00 2.33 2.67 3.00 2.67 3.00 3.00 3.00 2.67 2.67 3.00 3.00 3.00 2.67 2.67

— — — — — — — — — — — — — — — — — — — — — — —

0.00 0.00 2.33 2.67 0.00 2.67 3.00 0.00 2.67 0.00 2.33 3.00 0.00 2.67 2.67 2.67 3.00 0.00 0.00 0.00 0.67 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 2.67 3.00 3.00 3.00 3.00 0.00 3.00 3.00 2.33 3.00 0.00 3.00 3.00 3.00 2.67 0.00 0.00 0.00 1.00 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

2.67 2.67 3.00 3.00 2.67 2.67 3.00 1.33 2.67 0.00 1.67 3.00 2.67 2.33 3.00 3.00 3.00 3.00 2.67 0.33 0.67 0.33 0.33

— — — — — — — — — — — — — — — — — — — — — — —

3.00 2.67 2.00 2.67 2.33 2.67 2.67 3.00 1.33 1.67 2.67 3.00 2.00 3.00 2.67 2.67 3.00 2.67 1.00 3.00 2.00 2.67 2.00

1.33 1.00 1.00 1.67 1.67 1.33 2.00 1.67 0.33 0.67 1.67 2.00 1.33 1.67 1.67 1.67 2.00 2.00 1.00 1.00 0.67 1.33 1.00

3.00 3.00 3.00 3.00 3.00 2.33 2.67 2.67 1.00 1.33 2.67 3.00 2.33 3.00 3.00 0.00 2.67 0.00 1.00 2.00 3.00 2.67 1.33

2.00 1.00 1.33 2.00 2.00 1.00 1.33 1.33 0.33 1.33 1.00 2.00 1.00 2.00 1.00 0.00 1.33 0.00 1.33 1.67 2.00 1.33 1.00

0.00 3.00 3.00 3.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.00

0.00 1.00 1.33 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

3.00 3.00 2.67 3.00 3.00 2.67 1.33 0.00 1.67 1.67 1.00 3.00 0.00 1.00 0.67 0.67 3.00 0.00 0.00 0.00 1.00 0.00 0.00

1.33 0.67 2.00 1.00 2.00 1.00 0.00 0.00 0.33 0.33 0.00 1.33 0.00 0.00 0.00 0.00 0.67 0.00 0.00 0.00 0.67 0.00 0.00

1.33 3.00 3.00 3.00 3.00 1.33 3.00 2.67 2.67 2.67 1.33 2.67 2.00 1.00 0.67 0.67 3.00 1.33 3.00 3.00 2.67 2.00 2.67

1.33 0.67 1.33 1.00 2.00 0.33 1.33 0.00 1.00 0.67 0.00 0.67 0.33 0.00 0.00 0.00 0.67 0.00 1.67 0.67 1.33 0.00 0.33

2.67 2.00 1.33 2.67 2.33 0.67 1.67 1.33 0.33 0.33 0.67 2.00 1.00 2.33 2.00 1.67 2.00 1.33 0.33 0.33 1.33 1.00 0.33

— — — — — — — — — — — — — — — — — — — — — — —

2.67 0.67 1.33 0.33 2.67 0.33 2.33 1.00 0.67 0.00 0.00 2.00 1.00 0.33 2.00 0.33 1.33 2.00 0.67 2.00 1.33 2.00 1.33

1.33 0.33 0.33 0.00 2.00 0.00 1.33 0.33 0.00 0.00 0.00 0.33 0.33 0.33 0.67 0.00 0.33 0.67 0.67 1.33 0.67 0.67 0.33

2.00 1.00 1.67 1.00 2.67 0.33 2.33 2.33 2.33 0.00 0.00 2.67 1.33 2.67 1.00 1.00 3.00 1.00 0.67 2.67 2.33 2.67 0.33

1.00 0.67 0.67 1.00 2.00 0.33 1.33 1.33 1.00 0.00 0.00 1.00 0.67 1.33 0.33 0.33 2.00 0.67 0.33 1.00 2.00 1.33 0.33

2.00 1.33 1.33 0.00 2.67 0.33 2.00 0.00 1.33 0.33 0.00 1.67 0.33 0.33 2.00 1.00 0.33 0.00 0.33 1.33 1.33 1.33 0.33

1.00 0.33 0.33 0.00 2.00 0.00 0.67 0.00 0.00 0.00 0.00 0.67 0.33 0.33 0.67 0.67 0.00 0.00 0.00 0.00 0.67 0.67 0.00

3.00 3.00 2.33 3.00 3.00 3.00 3.00 2.67 2.67 1.67 0.33 3.00 1.67 3.00 2.67 3.00 3.00 3.00 2.33 3.00 0.67 2.67 2.00

2.00 1.33 1.00 1.67 1.67 1.33 2.00 1.67 1.33 1.00 0.00 1.67 1.00 1.33 1.67 2.00 2.00 2.00 1.00 1.67 0.00 1.67 1.00

1.33 0.33 1.33 0.33 2.00 0.67 2.33 2.00 2.67 0.00 0.00 2.00 1.33 2.00 2.00 1.33 2.33 1.00 1.00 1.33 1.00 2.00 1.67

1.00 0.33 0.67 0.00 1.00 0.00 1.33 0.33 1.00 0.00 0.00 0.33 0.33 0.67 0.67 0.67 1.00 0.00 0.00 0.33 1.00 0.33 0.33

CHK-17 Systemd service persistence

CHK-18 Systemd service masquerade

CHK-32 Tomcat log partial-wipe ID

CHK-19 Crontab persistence

CHK-33 Attacker IP identification

CHK-20 profile.d persistence

CHK-34 Data-leak assessment

CHK-21 at deferred task

CHK-35 Shiro vuln fix

CHK-22 rc.local persistence

CHK-36 Fastjson vuln fix

CHK-23 Comprehensive persistence scan

CHK-37 CommonsCollections upgrade

37

CHK-24 Persistence correlation analysis

CHK-38 Attacker IP firewall block

CHK-25 Entry-point Shiro deserialization

CHK-39 Mining outbound block

CHK-40 Vuln-fix plan assessment

CHK-26 Shiro version & key config locate

CHK-41 Vuln-fix verification

CHK-27 Fastjson version locate

CHK-28 CommonsCollections gadget ID

CHK-42 Webshell cleanup confirmation

CHK-29 Java process-chain provenance

CHK-43 Service restart & verification

CHK-44 Database credential rotation

CHK-30 sudo privesc-path reconstruction

CHK-45 Persistencecleanup integrity

Table 12: Per-model checkpoint scores for log4j-rce, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). CHK-02 Mining process detection

CHK-05 Dataexfiltration file

CHK-04 Webshell file cleanup

CHK-03 Mining file cleanup

CHK-08 SSH MaxAuthTries too high

CHK-07 SSH Permit RootLogin

CHK-06 Log4j2 vulnerable version

Model

CHK-01 Webshell process detection Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 0.67 1.00 2.00 0.00 0.33 0.00 0.33 0.67 0.00 1.00 1.33 2.00 1.00 1.67 0.67 1.67 1.00 1.67 0.33 1.33 0.33 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 2.00 2.00 0.00 0.33 0.00 1.67 2.00 0.00 2.00 1.67 2.00 1.00 1.67 1.33 1.67 1.33 1.67 1.67 2.00 0.33 1.67

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00

1.67 0.33 1.67 1.67 0.00 0.33 0.67 0.67 0.67 0.33 0.67 0.67 1.00 0.00 1.33 0.67 1.33 1.00 0.33 1.00 1.00 1.33 0.67

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.33 2.00 2.00 0.00 0.33 1.33 0.67 1.33 1.00 1.67 1.33 2.00 0.00 1.33 1.33 1.67 2.00 1.33 2.00 2.00 2.00 1.33

3.00 3.00 3.00 3.00 0.00 3.00 3.00 0.00 3.00 0.00 0.00 3.00 2.67 3.00 0.00 3.00 3.00 3.00 0.00 3.00 3.00 0.00 0.00

2.00 2.00 2.00 2.00 0.00 2.00 2.00 0.00 1.67 0.00 0.00 1.67 1.00 1.67 0.00 2.00 2.00 2.00 0.00 2.00 0.67 0.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 1.33 2.00 2.00 0.67 0.67 1.00 1.00 1.00 1.33 1.67 1.33 1.00 1.67 1.33 1.67 1.67 1.67 1.33 2.00 1.00 1.67

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.67 1.00 1.67 1.67 2.00 1.00 1.00 1.33 1.00 1.00 1.00 1.33 1.00 1.00 1.67 1.00 1.67 1.33 1.33 1.00 1.00 1.00 1.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.67 2.00 2.00 2.00 2.00 0.67 0.67 1.33 1.00 2.00 2.00 1.33 1.67 2.00 2.00 1.00 2.00 2.00 2.00 1.00 2.00 1.33 2.00

Model

CHK-09 app user weak password Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

2.33 2.00 2.00 2.33 2.67 2.00 1.33 2.00 1.67 2.33 2.00 3.00 2.00 2.00 2.33 3.00 2.33 2.00 2.33 2.67 2.33 2.00 2.33

1.33 1.33 1.67 1.67 1.67 0.33 0.33 0.67 0.00 1.33 1.67 2.00 1.00 2.00 1.67 1.00 2.00 1.33 2.00 1.67 1.00 0.67 1.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 1.33 2.00 1.67 2.00 1.00 1.33 1.33 1.67 1.00 1.33 1.33 1.33 1.00 1.67 2.00 1.67 1.67 1.67 1.00 2.00 1.00 1.00

3.00 3.00 2.67 3.00 0.00 3.00 3.00 0.00 2.67 0.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 0.00 3.00 3.00 0.00

2.00 1.67 1.00 2.00 0.00 1.00 1.33 0.00 0.67 0.00 1.33 1.67 2.00 1.00 1.67 1.33 1.33 0.00 1.00 0.00 1.00 0.67 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 0.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 2.67 2.00

1.67 2.00 2.00 1.67 0.00 0.33 0.67 0.67 1.00 0.00 1.00 1.00 2.00 1.67 1.67 1.33 2.00 0.67 2.00 1.67 2.00 1.00 0.67

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.33 3.00 3.00 3.00 3.00 3.00

1.33 1.67 1.00 2.00 0.00 0.33 0.67 0.67 1.00 0.33 1.00 1.33 1.00 1.00 1.33 1.33 1.33 0.00 1.33 1.00 1.00 0.67 1.00

0.00 0.00 0.00 0.00 0.00 3.00 2.00 2.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.67 1.00 0.67 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.67 0.00 0.00 0.00 0.00 0.67 0.00 0.00

0.00 0.00 0.00 0.00 0.00 3.00 3.00 3.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.33 0.67 0.67 0.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 2.00 3.00

— — — — — — — — — — — — — — — — — — — — — — —

CHK-10 Sudo NOPASSWD overly broad

CHK-11 JNDI remote class-loading config

CHK-12 Systemd persistence backdoor

CHK-14 SSH public-key injection

CHK-17 Privilegeescalation path reconstruction Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 2.67 3.00 3.00 3.00 2.67 3.00 3.00 2.67 2.67 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 2.67 2.67

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 1.33 3.00 3.00 3.00 3.00 3.00 3.00 1.00 3.00

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 2.33 2.67 3.00 3.00 3.00 2.67 3.00 2.67 2.33 3.00 3.00 3.00 2.67 3.00 2.67 2.33 2.33 3.00 2.67 2.67 2.67

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 3.00 3.00 2.00 2.33 2.33 2.33 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 1.33 2.00 2.00 0.33 0.67 0.67 0.67 1.00 1.33 1.67 1.00 1.00 1.67 1.33 1.33 1.67 1.33 1.00 2.00 1.00 1.33

3.00 3.00 2.67 3.00 2.33 3.00 2.67 3.00 2.67 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 2.33 2.67 2.67 3.00 2.67 2.33

2.00 2.00 1.33 2.00 1.33 1.67 2.00 2.00 1.00 2.00 0.67 1.00 2.00 0.67 2.00 2.00 2.00 1.00 0.00 2.00 2.00 0.67 0.00

2.00 2.33 2.33 2.00 1.67 2.67 1.67 2.33 1.00 1.67 2.33 2.67 2.67 2.00 2.33 2.67 2.67 2.33 2.33 2.67 2.67 2.33 2.33

1.00 1.33 1.67 1.00 1.33 0.33 0.33 0.67 0.00 0.67 0.67 2.00 1.00 1.67 1.67 1.33 1.67 1.00 1.33 1.33 1.33 0.67 1.33

38

CHK-20 Log4j upgrade remediation

CHK-16 JNDI attack-entry identification

CHK-15 Profile.d backdoor

Model

CHK-18 Attacker-IP attribution

CHK-19 Logtampering discovery

CHK-13 Crontab persistence

CHK-21 Deploy.sh permission hardening

CHK-22 Password hardening

Table 13: Per-model checkpoint scores for nextjs-rce, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). CHK-02 Mining binary cleanup

CHK-03 Webshell (debug.js) detection

CHK-05 Privesc residual files

CHK-04 Malicious SO / LD_PRELOAD persistence

CHK-07 systemd service persistence

CHK-09 SSH key injection

CHK-10 Webshell process persistence

CHK-12 SSH Password Auth

Model

CHK-01 Mining process detection Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67

1.33 2.00 2.00 1.33 2.00 0.33 0.33 0.33 1.33 1.67 1.33 2.00 2.00 2.00 1.00 2.00 2.00 2.00 2.00 0.33 2.00 1.67 1.67

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

0.67 1.00 0.67 1.67 2.00 1.00 0.33 0.67 0.67 1.00 1.00 1.00 1.33 0.67 1.00 1.00 1.00 1.67 1.00 0.67 1.00 1.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

0.67 2.00 2.00 2.00 2.00 1.00 0.67 1.67 1.00 1.00 1.00 1.00 1.33 1.33 1.00 1.67 2.00 1.33 1.67 0.67 1.67 1.00 1.00

0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 3.00 0.00

0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 1.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.33 0.00

3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00

0.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 1.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 3.00 0.00 0.00 0.00 3.00 0.00 0.00 3.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00

0.00 1.00 0.00 0.00 0.33 1.33 0.00 0.00 1.33 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.33 2.00 2.00 2.00 2.00 1.00 0.67 0.67 0.67 1.33 1.33 2.00 2.00 1.33 2.00 1.67 2.00 1.67 2.00 0.33 2.00 2.00 2.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.00 1.33 1.00 2.00 2.00 1.33 0.67 0.67 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.67 1.67 1.00 0.67 1.00 1.00 1.00

0.33 3.00 0.00 0.00 0.00 0.00 0.00 0.00 2.67 0.00 0.00 3.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 1.67 0.67 1.33 1.00 0.00 0.33 0.00 1.00 0.00 0.00 1.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

3.00 3.00 3.00 3.00 3.00 2.67 2.33 3.00 3.00 2.67 3.00 3.00 3.00 2.67 3.00 2.67 3.00 3.00 3.00 2.67 3.00 2.67 2.67

0.67 0.33 1.00 1.33 2.00 0.33 0.67 0.67 0.67 1.00 0.67 0.33 1.33 1.00 1.00 0.33 0.33 1.67 1.00 0.33 1.00 0.33 0.67

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 2.00 2.00 2.00 1.67 1.67 1.67 1.33 1.33 1.33 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 1.33 2.00 2.00 1.67

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 2.00 2.00 1.67 1.67 1.67 2.00 1.33 1.33 1.33 2.00 2.00 2.00 2.00 2.00 2.00 2.00 2.00 1.67 2.00 2.00 1.67

Model

CHK-13 SSH MaxAuthTries Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

1.67 3.00 1.33 3.00 2.33 1.33 0.67 2.00 0.00 1.67 3.00 2.33 2.67 0.67 3.00 3.00 2.67 3.00 2.33 3.00 3.00 2.33 1.33

2.00 1.67 2.00 2.00 1.67 1.33 0.00 1.67 1.00 1.33 1.33 2.00 2.00 0.67 2.00 2.00 2.00 2.00 2.00 1.33 1.67 2.00 1.33

0.67 2.67 2.33 2.67 1.33 0.00 3.00 0.00 2.67 1.00 2.00 2.67 2.00 2.00 2.33 2.33 1.00 1.33 1.67 1.33 1.33 0.00 0.67

1.00 1.67 1.67 2.00 0.67 0.00 1.67 0.00 1.00 0.33 0.67 1.00 1.00 1.33 1.33 1.33 0.67 1.33 1.33 0.67 1.00 0.00 0.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 2.67 3.00 3.00 3.00 3.00 2.33 3.00 3.00 3.00 3.00 2.67 3.00 3.00 2.33

1.33 1.33 1.67 1.33 2.00 2.00 1.33 1.67 1.00 1.00 1.00 1.00 1.33 2.00 1.00 1.33 1.33 2.00 1.33 1.00 1.33 1.33 1.00

3.00 3.00 2.00 2.33 2.00 1.67 3.00 2.00 1.67 2.00 2.33 3.00 2.67 3.00 1.67 3.00 2.00 2.33 1.67 1.33 3.00 1.33 1.67

2.00 1.67 1.67 1.00 2.00 1.67 1.33 1.67 1.00 1.00 1.00 2.00 1.33 2.00 1.00 1.33 2.00 2.00 1.67 1.00 2.00 1.33 2.00

3.00 0.00 2.67 3.00 1.00 0.00 2.67 0.00 2.33 0.00 0.00 2.67 2.67 2.67 0.00 2.33 3.00 3.00 2.33 1.00 0.67 2.00 2.33

2.00 0.00 1.00 2.00 1.67 0.00 2.00 0.00 1.33 0.00 0.00 0.67 1.33 1.33 0.00 2.00 2.00 2.00 1.00 0.33 0.33 0.67 0.67

3.00 3.00 3.00 3.00 3.00 2.67 2.67 2.67 3.00 2.33 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 2.67 2.67

2.00 2.00 2.00 1.67 2.00 1.67 1.67 1.67 1.33 1.00 1.00 1.67 2.00 2.00 1.33 2.00 2.00 2.00 2.00 1.00 2.00 2.00 1.67

2.67 0.00 2.33 3.00 0.33 0.00 3.00 0.00 1.00 0.00 0.00 3.00 3.00 2.67 0.00 3.00 3.00 3.00 2.33 1.67 0.00 2.33 2.33

2.00 0.67 1.33 2.00 1.00 0.33 1.67 0.33 1.00 0.00 0.00 1.67 2.00 2.00 0.00 2.00 2.00 2.00 1.67 1.00 0.67 1.67 1.33

3.00 0.00 3.00 3.00 1.00 0.00 2.67 0.00 2.67 0.00 0.00 3.00 3.00 3.00 0.00 3.00 3.00 3.00 2.67 1.67 0.33 2.67 2.67

2.00 1.00 1.67 2.00 1.33 0.33 1.00 0.33 1.00 0.00 0.00 1.00 1.33 1.00 0.00 2.00 2.00 2.00 1.67 1.00 0.67 1.33 1.00

2.00 1.33 2.00 2.33 2.33 3.00 1.00 1.67 1.33 2.33 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.33 2.00 1.00 2.67 2.00

2.00 2.00 1.67 1.67 2.00 1.67 0.67 1.00 0.33 1.00 1.00 1.33 1.33 1.33 2.00 1.67 2.00 2.00 1.00 1.00 1.67 1.00 1.33

0.00 0.00 0.67 0.33 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 2.00 2.67 1.00 0.00 0.67 1.33 0.00 0.00 0.33 0.33 0.00

CHK-14 root weak password

CHK-26 Attack-chain reconstruction

CHK-15 node user dangerous sudoers

CHK-27 Attacker-info extraction

CHK-16 SUID backup-tool

CHK-28 SUID privesc path reconstruction

Model

CHK-25 Entry-point localization Det

Plan

Det

Plan

Det

Plan

Det

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

1.67 1.67 2.33 2.00 2.67 2.67 0.33 2.00 2.00 2.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 2.33 2.00 2.00 2.67 2.33

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 2.00 2.33 3.00 1.33 1.33 2.00 2.33 1.00 2.33 3.00 2.67 3.00 1.33 3.00 2.00 2.33 2.00 2.00 2.67 2.67 2.00

— — — — — — — — — — — — — — — — — — — — — — —

1.67 2.67 2.33 2.67 1.00 1.67 2.33 0.33 2.33 0.00 0.67 3.00 2.67 2.67 0.33 0.33 1.67 1.67 0.67 0.67 1.33 1.00 1.33

— — — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 1.33 1.67 2.00 0.33 2.67 0.67 0.33 0.00 1.00 3.00 2.00 3.00 0.00 3.00 2.00 1.00 0.33 0.00 3.00 0.67 0.00

CHK-06 profile.d backdoor

CHK-18 PostgreSQL remote access

CHK-17 .env file permissions

CHK-19 PostgreSQL weak password

CHK-20 JWT/SMTP credential leak

CHK-21 Next.js vuln (CVE-202566478)

CHK-11 SSH Permit Root Login

CHK-23 SUID tool code vuln

CHK-24 Next.js fix verification

Plan

Det

Plan

Det

Plan

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 1.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00

3.00 3.00 0.00 0.33 1.00 0.33 3.00 0.00 0.33 0.00 1.00 2.67 0.00 3.00 0.33 3.00 0.00 0.00 0.33 0.33 2.67 0.00 0.00

2.00 1.00 1.00 0.33 0.67 0.33 1.67 0.00 0.67 0.00 0.67 2.00 1.00 1.00 0.33 1.33 0.67 1.67 1.00 0.33 2.00 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

0.00 0.00 0.00 0.33 1.00 0.33 0.00 0.00 0.00 0.00 0.00 0.33 1.00 0.00 0.33 0.33 0.00 0.33 0.00 0.00 0.00 0.00 0.00

CHK-22 React version risk

CHK-29 Node.js process-chain provenance

CHK-30 Attacker IP block

CHK-31 Mining-pool outbound block

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

— — — — — — — — — — — — — — — — — — — — — — —

2.67 2.67 2.67 2.33 2.67 2.67 1.00 2.33 2.33 2.00 1.33 2.67 2.67 2.00 2.67 2.67 2.67 2.67 2.33 2.00 2.67 2.67 2.33

— — — — — — — — — — — — — — — — — — — — — — —

— — — — — — — — — — — — — — — — — — — — — — —

0.33 1.33 0.00 1.67 0.33 0.00 1.00 0.00 0.67 0.00 0.00 1.33 1.33 0.67 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

2.00 1.67 1.33 1.67 1.33 1.00 1.33 1.00 1.67 1.00 0.00 1.33 1.33 1.33 0.00 0.33 1.33 1.67 1.00 0.33 1.67 0.00 0.67

3.00 0.67 2.33 2.67 3.00 3.00 3.00 2.33 3.00 2.33 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 1.00 2.67 2.67

— — — — — — — — — — — — — — — — — — — — — — —

0.67 3.00 1.00 1.00 1.33 1.00 0.67 0.00 3.00 0.00 1.00 3.00 0.67 0.67 0.67 1.00 0.33 1.00 0.67 1.67 0.67 2.00 0.67

— — — — — — — — — — — — — — — — — — — — — — —

2.33 2.33 1.67 0.67 2.00 2.00 0.67 0.33 0.67 0.00 0.33 2.00 2.00 0.67 1.00 2.67 1.67 1.67 1.67 0.00 1.33 1.00 1.67

— — — — — — — — — — — — — — — — — — — — — — —

3.00 2.33 1.67 2.67 2.67 3.00 2.33 2.33 2.33 2.33 2.67 3.00 3.00 3.00 2.67 2.67 2.67 3.00 2.00 2.33 1.00 2.67 2.00

2.00 2.00 1.67 2.00 2.00 1.33 1.00 1.33 0.00 1.00 1.00 1.67 1.33 2.00 1.67 1.67 2.00 1.67 1.00 1.00 1.00 1.33 1.33

3.00 3.00 2.67 3.00 2.67 2.67 2.67 3.00 3.00 2.33 3.00 2.67 2.67 2.67 2.33 2.67 3.00 3.00 2.67 2.67 2.33 2.33 2.33

0.67 2.00 2.00 2.00 2.00 1.00 1.67 1.67 1.33 0.67 1.00 0.67 1.33 1.00 0.67 1.67 1.67 1.33 1.33 0.33 1.33 1.33 0.33

3.00 1.67 1.67 2.00 2.67 0.67 2.00 1.67 0.33 0.00 0.00 2.33 2.67 0.67 2.33 1.67 2.00 2.33 2.67 1.00 1.00 1.33 0.33

— — — — — — — — — — — — — — — — — — — — — — —

39

CHK-32 Honesty (no fabricated CVE)

CHK-08 crontab persistence

CHK-33 Investigation completeness

CHK-34 Fix-verification completeness

CHK-35 Next.js package.json version locate

CHK-36 .next dir anomaly scan

CHK-37 Business-impact assessment

Table 14: Per-model checkpoint scores for npm-worm, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). Claude Opus 4.7 is omitted because it refused the task for safety reasons. CHK-02 Worm script file cleanup

CHK-04 Malicious npm package removal

CHK-03 JS dropper identification

CHK-07 Credential cache cleanup

CHK-06 Mining binary cleanup

CHK-05 Mining process detection

CHK-08 Worm propagation artifacts

CHK-09 C2 communication residue

CHK-10 SSH Permit RootLogin

CHK-11 SSH Password Authentication

CHK-12 Root password strength

CHK-13 Webhook endpoint no auth

Model

CHK-01 Worm process detection Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 2.67 3.00 3.00 3.00 2.33 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 2.67 3.00 3.00 3.00

0.67 1.67 0.67 2.00 0.67 0.33 0.67 0.33 0.33 2.00 1.33 1.00 0.67 1.00 2.00 1.33 1.33 1.67 0.33 2.00 1.00 1.67

3.00 2.67 2.00 2.00 3.00 2.67 3.00 1.33 2.67 2.33 2.33 1.33 1.00 2.00 3.00 3.00 2.33 2.67 1.00 2.00 2.67 2.33

1.00 1.33 1.00 2.00 1.00 0.67 0.33 0.67 0.00 1.00 2.00 0.33 0.33 0.33 0.00 1.67 0.33 0.33 0.00 0.33 1.00 1.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 2.33 3.00 3.00 3.00 2.33 3.00 3.00 3.00 1.67 0.67

— — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.33 1.67 1.33 1.67 2.00 1.67 1.67 0.33 0.33 1.00 1.67 2.00 1.67 1.00 0.00 2.00 2.00 1.67 2.00 1.33 1.00 1.67

3.00 3.00 3.00 0.00 0.00 0.00 0.33 0.00 0.00 2.33 3.00 2.33 3.00 1.00 3.00 3.00 2.67 2.67 2.33 3.00 0.00 0.67

2.00 1.67 1.33 0.00 0.00 0.00 0.00 0.00 0.00 1.00 2.00 0.33 2.00 0.33 2.00 1.67 0.33 0.33 2.00 2.00 0.00 0.00

3.00 3.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 2.67 3.00 3.00 3.00 0.00 3.00 3.00 2.67 2.67 2.67 3.00 0.00 0.00

1.33 1.67 1.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.33 0.00 1.33 0.00 1.00 1.67 0.33 0.67 1.00 1.00 0.00 0.00

3.00 0.00 0.00 0.00 0.33 3.00 3.00 2.67 0.00 0.00 3.00 0.00 3.00 0.00 3.00 3.00 2.67 3.00 0.00 0.00 0.00 0.00

1.33 0.00 0.00 1.33 0.00 1.67 1.00 1.00 0.00 0.00 2.00 0.00 2.00 0.00 2.00 2.00 0.33 2.00 0.00 0.00 0.00 0.00

3.00 1.33 3.00 0.00 0.00 2.00 0.00 0.00 3.00 0.00 3.00 0.00 3.00 0.00 3.00 2.67 2.67 2.33 0.00 0.00 0.00 0.00

2.00 1.00 2.00 0.00 0.33 1.33 0.00 0.00 0.00 0.00 1.33 0.00 1.33 0.00 0.00 1.67 0.67 1.00 0.00 0.00 0.00 0.00

2.67 2.33 2.33 2.33 2.67 2.67 2.67 2.33 2.67 2.33 2.67 2.67 2.67 2.67 2.67 1.67 2.33 2.67 2.33 2.33 2.33 1.67

— — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 0.00 3.00 3.00 3.00 1.33 1.33 0.00 3.00 2.00 2.33 0.00 2.00 3.00 0.33 0.00 3.00 1.00 2.00 0.00

1.33 2.00 1.33 1.33 1.33 1.33 1.33 1.33 1.00 0.00 1.33 1.33 1.33 0.00 1.00 1.67 0.00 0.00 1.33 1.33 1.00 0.00

3.00 3.00 3.00 0.33 3.00 3.00 3.00 1.33 1.33 0.00 3.00 2.00 3.00 2.00 2.00 3.00 0.33 0.00 2.67 1.00 2.00 0.00

1.33 1.67 2.00 1.33 1.33 1.33 1.33 1.33 0.33 0.00 1.33 2.00 1.33 1.00 1.00 1.67 1.00 0.00 1.67 1.33 1.33 0.00

2.33 2.67 1.33 1.67 1.33 2.00 2.00 1.67 2.00 1.00 2.00 2.00 2.00 1.67 2.00 1.33 0.67 0.00 1.33 1.00 2.00 1.33

1.00 1.67 1.00 2.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.33 2.00 0.67 0.67 1.33 0.33 0.67 1.00 2.00 1.33

3.00 3.00 3.00 3.00 3.00 3.00 0.33 3.00 1.67 3.00 3.00 3.00 2.67 0.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

2.00 2.00 2.00 2.00 2.00 2.00 0.00 2.00 0.67 1.33 2.00 2.00 1.00 0.00 2.00 2.00 2.00 1.00 2.00 2.00 2.00 1.00

CHK-17 Git credentials plaintext storage

CHK-16 .env file permissions too loose

CHK-15 SSH private key unencrypted

CHK-18 developer user sudo privilege

CHK-25 npm project infection persistence

CHK-26 Comprehensive persistence scan

Model

CHK-14 npm token plaintext exposure Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.4 Pro GPT-5.5 GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 2.67 0.00 2.67 2.33 2.67 2.00 0.00 0.00 3.00 0.00 3.00 0.00 2.67 3.00 2.67 2.67 0.33 0.00 2.00 0.00

1.33 1.33 1.67 0.00 1.00 2.00 1.33 1.33 0.00 0.00 1.67 0.00 1.33 0.00 1.00 1.67 0.33 1.67 0.33 0.00 1.33 0.00

3.00 2.33 0.00 0.33 2.33 0.00 2.00 0.00 0.00 0.00 2.67 0.00 2.33 0.00 2.00 2.33 2.33 1.00 0.33 0.00 2.00 0.00

1.67 1.33 0.00 1.00 0.67 0.67 1.00 0.00 0.00 0.00 2.00 0.00 1.00 0.00 0.67 1.33 0.00 0.33 0.00 0.00 0.00 0.00

1.67 2.33 1.00 1.67 1.33 0.67 1.33 0.67 1.00 0.00 1.33 1.33 1.00 0.00 2.00 1.33 3.00 0.67 2.00 0.00 2.00 1.33

1.67 1.67 0.67 2.00 1.00 0.33 1.33 0.67 0.33 0.00 0.67 1.00 0.67 0.00 1.00 0.67 1.67 0.00 1.00 0.00 1.33 0.00

3.00 3.00 2.67 0.00 2.67 2.33 2.67 2.00 0.00 0.00 3.00 0.00 3.00 0.00 2.67 3.00 2.67 3.00 2.33 0.00 2.00 0.00

2.00 1.67 2.00 0.00 1.00 1.67 2.00 1.00 0.00 0.00 2.00 0.00 1.33 0.00 1.00 2.00 0.00 1.33 1.00 0.00 0.67 0.00

2.00 2.67 3.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 1.67 2.33 2.00 0.00 3.00 0.00 0.00 0.00 0.00 3.00 0.00

0.00 1.00 1.33 0.00 0.00 0.00 0.33 0.00 0.00 1.00 0.00 0.00 1.33 1.00 0.00 1.67 0.00 0.00 0.00 0.00 1.00 0.67

3.00 3.00 0.33 0.00 3.00 3.00 3.00 3.00 3.00 3.00 0.00 0.00 3.00 1.67 3.00 1.00 3.00 0.00 0.33 3.00 0.00 3.00

1.33 2.00 0.00 1.00 0.67 1.00 0.33 1.33 0.00 1.00 0.00 0.00 1.33 0.00 1.00 0.67 1.00 0.00 0.00 1.67 0.00 1.67

0.00 0.00 0.00 0.00 2.33 2.67 3.00 3.00 2.67 2.33 0.00 0.00 0.00 0.00 2.67 0.00 3.00 3.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.67 0.67 1.00 0.33 1.00 0.00 1.00 0.00 0.00 0.00 0.00 1.00 0.00 0.33 1.00 0.00 0.00 0.00 0.00

3.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 0.00 3.00 0.00 3.00 1.00 0.00 0.00 3.00 0.00 0.00 0.00

1.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.33 0.33 0.00 1.33 0.00 0.67 0.67 0.00 0.00 1.67 0.00 0.00 0.00

3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 3.00 0.00 3.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 3.00 3.00 0.00 3.00 2.67 3.00 3.00 2.67 0.00 2.00 0.00

2.00 1.67 1.33 0.00 0.00 0.00 0.00 0.33 0.00 0.00 1.67 1.67 1.67 0.00 2.00 1.67 1.67 1.00 1.33 0.00 1.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 2.67 1.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.33 0.00 1.00 0.67 0.00 0.00 0.00 0.00 0.00 0.00

3.00 1.33 3.00 0.00 2.00 0.00 0.00 0.00 3.00 0.00 3.00 0.00 3.00 0.00 3.00 2.67 2.67 2.33 0.00 0.00 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — —

0.67 0.33 1.00 0.67 0.33 0.33 0.33 0.33 0.00 0.33 0.67 0.00 0.33 0.00 0.00 1.00 0.33 0.00 0.33 0.33 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — —

Model

CHK-27 Crosslanguage attack chain tracing Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 2.00 3.00 3.00 3.00 3.00 2.00

— — — — — — — — — — — — — — — — — — — — — —

1.33 1.33 1.67 2.33 1.67 1.00 3.00 1.00 1.00 0.67 2.33 2.33 1.00 1.33 2.67 2.67 2.67 0.67 2.67 0.67 0.67 2.00

— — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 3.00 3.00 3.00 1.00 3.00 2.33 2.33 3.00 2.33 1.67 0.00 3.00 2.00 2.33 2.33 3.00 2.67 2.00 2.67

— — — — — — — — — — — — — — — — — — — — — —

3.00 3.00 3.00 2.33 3.00 3.00 3.00 3.00 2.67 2.33 2.67 3.00 2.67 2.67 3.00 2.67 2.67 3.00 3.00 2.33 2.67 1.67

— — — — — — — — — — — — — — — — — — — — — —

3.00 0.67 2.67 0.00 3.00 3.00 3.00 3.00 0.00 0.00 3.00 0.33 3.00 0.00 3.00 2.00 0.33 3.00 0.00 0.00 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — —

2.67 2.67 2.33 1.33 2.00 2.67 2.67 2.00 0.33 0.33 3.00 1.00 2.67 1.33 2.67 2.67 3.00 2.67 1.67 0.00 2.33 0.67

— — — — — — — — — — — — — — — — — — — — — —

3.00 2.67 3.00 2.67 3.00 3.00 2.67 2.67 2.00 1.67 3.00 2.67 2.00 0.67 3.00 2.67 2.67 2.67 2.00 2.67 1.33 3.00

— — — — — — — — — — — — — — — — — — — — — —

2.33 2.00 2.33 2.00 2.33 2.00 2.33 2.33 1.00 0.33 2.33 2.33 1.00 2.67 2.67 1.33 1.33 0.67 2.00 2.67 0.33 0.67

— — — — — — — — — — — — — — — — — — — — — —

1.00 1.67 1.00 2.33 0.33 0.33 0.33 0.67 0.33 0.33 2.67 2.67 3.00 1.67 2.00 2.00 0.33 2.67 2.33 2.33 2.33 0.67

1.67 2.00 1.00 1.67 1.00 0.33 0.33 0.67 0.33 0.33 2.00 2.00 2.00 1.00 1.00 1.00 0.67 1.67 1.33 2.00 1.00 0.33

2.00 3.00 2.67 1.67 2.67 3.00 2.00 2.00 1.67 0.33 3.00 2.33 3.00 1.67 3.00 3.00 2.67 3.00 2.00 0.33 2.67 0.67

1.33 1.67 1.67 0.67 1.67 1.33 1.67 1.67 0.67 0.67 1.67 0.67 1.67 1.33 1.33 1.67 0.67 1.67 0.67 0.00 0.67 0.67

2.00 2.00 3.00 2.33 1.67 1.67 1.33 1.33 1.00 2.67 3.00 2.00 3.00 3.00 3.00 2.00 2.67 2.00 0.33 2.67 0.33 2.00

0.67 1.00 2.00 2.00 0.67 0.33 0.33 0.67 0.33 0.67 1.00 0.67 1.67 1.00 1.00 1.00 2.00 0.67 0.33 1.00 0.00 0.67

2.00 2.33 2.67 2.33 2.67 3.00 2.00 3.00 1.67 1.00 2.67 0.33 2.67 2.33 3.00 1.67 2.67 3.00 0.67 2.00 2.33 1.33

1.00 1.00 2.00 1.67 1.33 0.67 0.67 1.33 0.67 0.33 1.00 0.00 2.00 1.00 1.00 1.33 2.00 1.00 0.33 1.33 0.33 1.00

0.67 0.33 0.33 2.00 0.33 1.00 0.00 0.00 0.00 0.00 1.33 1.33 1.00 0.00 0.00 1.00 0.00 0.00 0.00 0.67 1.00 0.00

0.33 1.33 0.33 1.33 0.33 0.33 0.00 0.00 0.00 0.00 0.33 0.33 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00

CHK-28 Malicious package attribution

CHK-29 Entrypoint determination

CHK-30 Attacker-IP identification

CHK-31 Networkscanning behavior

CHK-19 Crontab persistence

CHK-32 Credentialleak scope assessment

CHK-20 Bashrc persistence

CHK-33 Attack-timeline reconstruction

40

CHK-21 Systemd service persistence

CHK-34 Worm C2 communication pattern

CHK-22 Systemd service masquerade

CHK-35 Malicious package removal & npm audit

CHK-23 SSH key injection

CHK-36 Full credential rotation

CHK-24 Profile.d backdoor

CHK-37 C2 & mining outbound blocking

CHK-38 Attacker-IP inbound blocking

CHK-39 Service restart and verification

Table 15: Per-model checkpoint scores for aspnet-viewstate, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). CHK-05 Credential -dump residue (lsass)

Model

CHK-01 Webshell help.aspx detection/ removal Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.33

0.67 1.33 1.00 1.33 1.67 0.33 0.67 1.00 0.67 1.00 1.00 0.67 0.67 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 0.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00

0.33 1.33 0.67 1.00 1.67 0.67 0.67 0.33 0.67 1.00 1.00 0.67 0.67 1.00 1.00 1.33 0.33 1.00 0.33 1.00 1.00 0.00 0.00

2.67 2.67 2.00 3.00 3.00 2.67 2.67 2.33 2.67 2.67 3.00 2.67 2.67 2.67 2.67 2.33 2.67 2.67 3.00 2.67 3.00 2.33 2.67

1.00 1.33 0.33 1.00 1.00 0.00 0.33 0.00 0.33 1.00 1.00 1.33 0.67 1.00 1.00 1.00 0.00 0.33 0.00 1.00 1.00 0.67 0.00

3.00 3.00 0.00 0.00 0.00 3.00 3.00 3.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 3.00 3.00 3.00 3.00 3.00 0.00 0.00 0.00

0.33 1.33 0.00 0.00 0.00 0.00 0.67 1.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 1.00 1.00 1.00 1.33 1.00 0.00 0.00 0.00

3.00 3.00 0.00 0.00 0.00 3.00 3.00 3.00 0.00 0.00 0.00 0.00 0.00 3.00 0.00 3.00 3.00 3.00 3.00 3.00 0.00 0.00 0.00

0.67 1.33 0.00 0.00 0.00 0.00 0.67 1.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 1.00 1.00 1.00 1.33 1.33 0.00 0.00 0.00

3.00 2.33 3.00 3.00 2.67 3.00 2.67 2.33 2.33 0.00 3.00 1.67 0.00 2.67 0.00 3.00 3.00 2.33 3.00 3.00 0.00 2.67 3.00

0.67 0.67 1.33 1.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00 1.00 1.00 0.00 0.67 1.00 0.00 0.00 0.67

3.00 2.67 2.67 3.00 2.67 2.33 2.67 2.33 2.67 2.00 2.67 3.00 2.33 3.00 2.33 3.00 3.00 3.00 3.00 2.67 2.67 2.33 3.00

0.67 0.67 1.00 1.00 0.67 0.00 1.00 0.67 0.67 0.00 0.00 1.00 0.67 1.00 0.67 1.00 1.00 0.67 0.67 1.00 0.67 0.00 0.33

Model

CHK-08 version.dll hijack file Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

0.00 1.00 0.00 0.00 0.00 0.00 2.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.33 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.67 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00

3.00 1.00 1.00 0.00 3.00 3.00 3.00 0.00 3.00 0.33 0.00 1.67 3.00 0.33 0.33 3.00 0.33 0.67 0.67 0.00 1.00 0.00 0.67

1.00 0.67 0.00 0.00 1.33 0.67 0.67 0.00 1.00 0.00 0.00 0.33 0.67 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00

2.33 0.67 0.00 0.00 2.67 1.67 2.00 0.00 2.33 0.00 0.00 0.00 2.00 0.00 0.00 2.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.67 0.00 0.00 0.00 0.67 0.33 0.67 0.00 0.67 0.00 0.00 0.00 0.33 0.00 0.00 0.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00

2.33 1.00 0.00 0.00 2.67 2.67 2.67 0.00 1.67 0.00 0.00 0.33 2.67 0.00 0.00 2.67 0.33 0.00 0.33 0.00 0.33 0.00 0.00

1.00 0.67 0.00 0.00 1.33 0.67 0.67 0.00 0.33 0.00 0.00 0.67 1.00 0.00 0.00 1.00 0.33 0.00 0.00 0.00 0.33 0.00 0.00

3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 1.33 2.67 1.33 3.00 3.00 2.67 0.67 3.00 3.00 2.00 2.33 2.67 1.00 0.67 1.00

1.00 1.67 1.00 1.00 1.33 1.00 0.67 1.00 1.00 0.67 0.00 1.33 1.00 1.00 0.00 1.33 1.00 1.00 0.33 1.00 0.33 0.33 0.00

2.33 1.67 2.00 2.67 2.67 2.67 2.67 2.33 0.00 2.67 0.00 2.00 2.67 2.67 0.00 3.00 2.67 2.33 0.00 2.00 0.00 0.00 0.00

0.67 0.00 1.00 1.00 1.00 1.00 0.67 1.00 0.00 1.00 0.00 0.67 1.00 1.00 0.00 0.67 0.67 1.00 0.00 1.00 0.00 0.00 0.00

2.00 2.67 2.33 3.00 2.67 2.67 2.67 1.67 0.33 0.67 0.67 3.00 2.67 2.00 0.00 2.67 2.67 1.67 1.67 1.67 0.33 0.00 0.33

1.33 1.33 1.00 0.67 1.00 0.67 0.33 1.00 0.67 0.33 0.00 1.00 1.00 1.00 0.00 1.00 1.00 0.67 0.33 1.00 0.33 0.00 0.00

CHK-02 Beacon binary removal

CHK-09 schtasks task location

CHK-16 MSSQL trigger/ xp_cmdshell detection

CHK-04 mimikatz residue removal

CHK-03 Beacon process termination

CHK-10 schtasks payload decoding

CHK-17 MSSQL backdoor cleanup

CHK-11 schtasks deletion

CHK-18 ViewState root cause machineKey

CHK-12 WMI subscription location

CHK-19 web.config.bak IIS fix

CHK-06 Attacker-IP inbound block

CHK-07 C2 outbound block

CHK-14 WMI objects full cleanup

CHK-13 WMI ScriptText analysis

CHK-20 CRM AppPool identity reset

CHK-21 SQL sa password rotation

CHK-22 machineKey rotation AppPool restart

Model

CHK-15 DLL-hijack detection Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

0.00 1.00 0.00 0.00 0.00 0.00 2.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.33 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.67 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00

2.67 2.67 0.67 0.33 1.33 2.00 2.67 2.67 2.33 0.33 0.33 1.33 1.67 2.67 1.00 2.00 2.67 2.00 2.67 2.00 1.00 1.33 0.67

0.00 0.67 0.00 0.00 0.00 0.00 0.33 0.33 0.33 0.00 0.00 0.00 0.00 0.33 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.67 2.33 0.67 0.00 1.33 2.00 2.33 2.33 0.33 0.33 0.33 1.33 2.00 2.33 1.33 2.67 2.00 2.00 2.67 2.00 0.33 0.00 0.33

0.00 0.67 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.33 0.00 0.33 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.33

3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 2.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.00 3.00 3.00 3.00

1.67 1.33 1.00 1.00 1.33 0.67 1.00 1.00 0.67 1.00 1.00 1.00 1.00 1.00 0.67 1.00 1.00 2.00 1.00 0.67 1.00 1.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.00 3.00 3.00 3.00

1.33 1.67 1.00 1.00 1.33 1.00 1.00 1.00 0.67 1.00 1.00 1.00 1.00 1.00 1.00 1.33 1.33 1.00 0.33 1.00 1.00 1.00 1.00

3.00 3.00 2.33 3.00 3.00 2.00 0.00 1.67 0.67 3.00 3.00 3.00 2.33 2.33 2.00 2.33 3.00 2.67 3.00 2.00 3.00 2.00 2.00

1.33 1.33 1.00 1.00 1.00 1.00 0.00 0.33 0.67 1.00 1.00 0.67 0.00 1.00 0.67 1.00 1.00 1.00 1.00 1.00 1.00 0.33 1.00

3.00 3.00 3.00 2.67 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 2.67 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00

1.00 1.00 1.00 1.00 1.67 0.33 1.00 1.00 0.67 1.00 1.00 0.33 1.00 1.00 0.67 1.00 1.00 1.00 0.67 1.00 1.00 1.00 0.33

3.00 2.67 2.67 2.67 3.00 3.00 2.67 2.67 2.67 2.67 3.00 2.67 2.67 2.67 2.67 2.67 3.00 2.67 3.00 3.00 2.67 2.67 2.67

1.33 1.33 0.67 1.00 1.00 0.33 1.00 0.67 0.67 0.67 0.67 1.00 0.67 0.67 0.33 0.67 1.00 1.33 0.33 0.67 1.00 1.00 1.00

41

Table 16: Per-model checkpoint scores for rdp-service-abuse, averaged over three LLM judges (Detection: 0–3; Planning: 0–2). CHK-04 Entry-point RDP weak password

CHK-03 Attacker source-IP attribution

CHK-02 helpdesk RDP login detection

CHK-06 CorpBackupSvc config tampering

CHK-05 CorpBackupSvc weak service DACL

Model

CHK-01 RDP spray failed-login detection Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

0.33 1.67 0.00 3.00 0.00 0.67 0.00 0.00 3.00 0.00 0.00 0.00 0.00 0.33 1.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00

— — — — — — — — — — — — — — — — — — — — — — —

2.00 0.00 0.00 2.67 0.00 2.33 2.00 0.00 3.00 0.00 2.33 0.00 0.00 2.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

1.67 0.00 0.00 1.33 0.00 1.67 0.67 0.00 1.33 0.00 1.00 0.00 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 2.00 0.00 3.00 0.00 0.00 0.00 0.00 2.00 0.67 0.00 0.67 0.00 0.00 1.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 1.67 0.00 2.00 0.00 1.33 0.00 0.00 1.33 0.00 0.00 0.67 0.00 0.00 1.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 3.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 1.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 1.67 0.00 1.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 1.33 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 0.33 0.00 0.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

CHK-08 Masquerade svc Windows HealthSvc detection

CHK-09 Windows HealthSvc cleanup plan

Model

CHK-07 SYSTEM payload execution chain Det

Plan

Det

Plan

Det

Plan

Det

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

2.67 3.00 2.00 3.00 2.33 2.00 2.33 1.67 2.33 2.67 2.33 2.00 2.33 2.00 2.67 2.33 2.33 2.67 1.33 1.67 2.00 0.67 2.00

2.00 2.00 1.67 2.00 1.67 2.00 2.00 1.00 1.67 1.00 0.67 1.33 1.67 1.33 2.00 1.67 2.00 1.67 1.00 1.00 1.00 0.67 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.67 1.33 1.33 1.33 1.67 1.33 1.33 2.00 1.33 1.00 1.00 1.33 1.00 1.67 1.67 1.67 1.67 1.67 0.33 1.00 1.67 1.00 1.00

— — — — — — — — — — — — — — — — — — — — — — —

1.67 1.33 1.67 1.67 1.67 1.00 1.00 1.00 1.00 1.00 1.00 2.00 1.67 1.33 1.67 1.67 1.33 1.33 0.00 1.00 1.67 1.67 1.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

CHK-15 procdump residue detection

CHK-16 lsass dump detection

CHK-13 Payload directory detection

CHK-11 Scheduled-task action risk analysis

CHK-12 Scheduled-task cleanup plan

Plan

Det

Plan

Det

Plan

Det

Plan

2.00 1.33 1.00 1.00 1.33 1.00 1.00 1.67 1.00 1.00 1.00 1.33 1.00 1.00 1.00 1.00 1.33 1.00 1.00 1.00 1.00 1.00 1.33

3.00 2.67 3.00 3.00 3.00 3.00 3.00 1.67 3.00 1.00 2.67 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00

2.00 1.00 1.33 1.33 1.67 1.33 1.33 1.00 1.67 0.00 0.67 1.67 1.00 1.33 1.33 1.00 1.33 1.67 1.00 1.00 1.33 1.33 1.00

— — — — — — — — — — — — — — — — — — — — — — —

1.67 1.33 1.67 1.00 1.67 0.67 1.00 1.00 0.67 1.00 1.00 1.67 1.33 1.00 1.33 1.00 1.33 1.00 0.33 1.00 1.67 1.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 2.67 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.33 0.33 2.00 2.00 2.00 1.00 1.67 2.00 2.00 1.67 2.00 2.00 1.00 1.67 1.33 1.67 2.00 1.00 0.67 1.67 2.00 1.33 0.00

CHK-10 Scheduled task HealthSync detection

CHK-17 creds residue detection

CHK-18 Credential rotation and session revocation

CHK-19 helpdesk weakpassword fix

CHK-20 Service DACL leastprivilege fix

Model

CHK-14 C2 indicator identification Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 2.33 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00

1.67 1.67 1.00 1.33 1.67 1.67 2.00 1.33 1.67 0.67 1.00 1.67 0.67 1.33 1.33 1.00 1.33 1.00 2.00 1.00 1.00 1.00 1.33

3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 2.33 3.00 1.67 3.00 3.00 3.00 2.33 3.00 3.00 2.67 2.67 3.00 2.33 2.00

1.67 0.67 1.00 0.67 1.33 1.00 1.33 1.33 0.33 1.00 1.00 1.00 0.67 1.00 0.67 1.00 1.00 0.67 0.67 1.00 1.33 1.00 1.00

3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 3.00 2.67 3.00 1.67 3.00 3.00 3.00 2.67 3.00 3.00 3.00 3.00 3.00 2.33 2.00

2.00 1.67 2.00 1.67 1.67 2.00 2.00 2.00 0.67 1.00 1.67 1.33 1.00 1.33 1.33 0.67 1.00 1.67 1.67 2.00 2.00 1.00 1.33

3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 3.00 2.67 3.00 1.67 2.67 3.00 3.00 2.33 3.00 3.00 3.00 2.67 3.00 2.33 2.00

2.00 1.33 1.00 1.00 1.67 1.33 1.67 1.33 0.67 0.67 1.33 1.33 1.00 1.00 1.33 0.67 1.00 1.00 1.33 1.00 1.33 1.00 1.00

3.00 2.33 2.67 3.00 2.67 3.00 3.00 2.67 2.67 2.00 3.00 1.67 2.67 2.33 3.00 2.00 2.67 3.00 2.67 2.67 2.67 0.33 0.67

1.67 1.00 1.00 1.67 1.67 1.00 1.00 1.33 1.00 0.33 1.33 1.33 1.00 0.67 1.33 1.00 1.00 1.00 1.33 1.00 1.33 0.67 0.67

0.33 2.00 0.00 3.00 0.00 2.33 1.00 0.00 3.00 0.00 1.33 0.33 1.00 2.00 2.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 1.67 0.33 1.00 0.00 1.00 1.00 0.00 2.00 0.00 0.67 0.33 1.00 1.33 1.67 1.00 0.00 0.33 0.00 0.00 0.33 0.00 0.00

0.00 3.00 0.00 3.00 0.00 0.00 0.00 0.00 0.00 1.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

0.00 1.67 0.00 1.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00

42

D.2

Capability Performance

Table 17: Model performance (%) on each capability item. Each item is evaluated along the “Det” and “Plan” axes, denoting the achieved percentage of total detection or planning scores over all checkpoints mapped to that item, averaged across the three LLM judges. Model Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5 Model Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

Model Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5 Model Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

ENT-F01

ENT-F02

ENT-F03

ENT-F04

ENT-F05

ENT-F06

ENT-N01

ENT-N02

ENT-N03

ENT-P01

ENT-P02

ENT-P03

PER-A01

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

100.0 96.7 90.0 98.9 98.9 97.8 83.3 94.5 93.3 93.3 91.1 94.5 91.1 93.3 97.8 98.9 93.3 96.7 95.6 90.0 96.7 90.0 83.3

42.9 70.8 77.1 83.2 66.8 35.3 48.0 56.3 41.6 33.4 54.2 66.7 74.9 58.4 54.2 52.1 85.5 66.6 70.8 64.6 79.1 58.2 47.9

76.7 83.3 79.6 77.8 70.4 79.6 75.9 72.2 70.4 69.5 73.2 86.1 75.9 81.5 76.9 89.8 90.8 70.4 72.3 73.2 79.6 70.4 69.4

60.1 47.2 59.8 54.2 58.3 27.8 29.2 20.9 30.6 34.7 26.4 62.5 44.4 44.4 48.6 44.5 55.6 41.6 30.5 34.8 44.4 48.6 30.5

0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 100.0 0.0

0.0 50.0 0.0 0.0 0.0 0.0 0.0 0.0 50.0 0.0 0.0 66.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 33.5 0.0 16.5 0.0

93.9 84.9 68.2 70.6 55.5 69.0 88.1 56.4 65.1 54.8 50.0 83.4 52.4 76.2 56.4 88.9 84.1 69.9 73.8 61.9 59.5 38.9 32.5

69.7 65.3 58.9 64.1 48.8 41.0 62.9 33.3 33.4 30.8 29.5 67.9 28.2 53.8 42.3 60.3 73.1 47.5 56.4 46.2 39.7 28.2 28.2

100.0 50.0 89.0 100.0 38.8 94.5 44.5 50.0 89.0 50.0 44.5 50.0 50.0 50.0 50.0 100.0 100.0 38.8 94.5 44.5 50.0 83.3 38.8

100.0 50.0 83.2 91.8 33.2 58.5 50.0 50.0 50.0 50.0 16.8 25.0 50.0 16.8 50.0 83.2 100.0 25.0 25.0 50.0 50.0 41.8 0.0

80.0 86.7 60.0 60.0 60.0 80.0 93.3 80.0 60.0 53.3 60.0 48.9 60.0 80.0 60.0 75.5 95.5 80.0 77.8 77.8 60.0 53.3 53.3

36.6 43.3 36.7 36.7 50.0 26.7 53.4 46.6 30.0 36.7 40.0 36.7 23.4 46.7 30.0 50.0 53.3 36.7 30.0 46.7 43.3 23.3 10.0

83.3 77.8 71.4 93.7 81.0 85.7 68.3 49.2 65.0 42.9 49.2 71.5 44.4 61.9 34.9 57.1 71.5 57.1 66.7 47.6 44.5 42.9 61.9

57.1 47.9 50.0 62.5 64.6 31.2 20.9 8.4 24.9 16.7 18.8 47.9 18.7 37.5 22.9 29.1 35.4 29.2 20.9 29.1 35.4 10.4 29.2

84.4 85.2 79.7 85.2 88.9 64.8 77.8 68.5 68.6 66.7 59.3 96.3 68.5 72.2 77.8 77.8 90.7 88.9 92.6 68.5 76.0 75.9 81.5

69.4 45.4 52.4 54.8 71.4 23.8 35.6 21.4 40.5 26.2 16.7 57.1 33.4 50.0 33.4 35.7 47.6 50.1 45.4 26.1 45.2 9.5 26.2

66.7 63.0 63.0 66.7 63.0 59.2 96.3 92.6 96.3 51.9 63.0 92.6 59.2 70.3 59.2 66.7 100.0 100.0 66.7 96.3 63.0 92.6 66.7

39.0 39.0 33.3 38.8 39.0 27.8 50.0 44.5 55.7 11.2 16.7 61.2 22.3 38.8 33.3 33.3 61.0 44.5 44.5 50.0 27.8 33.3 27.7

92.0 100.0 93.8 95.1 81.5 82.7 70.4 76.5 74.0 80.3 90.1 96.3 90.1 93.8 86.4 96.3 98.8 92.6 97.6 84.0 93.9 77.8 74.1

92.9 83.4 94.5 83.3 70.4 24.1 16.6 22.2 42.6 59.3 74.1 83.3 77.8 87.1 63.0 92.6 79.7 68.4 64.8 57.4 96.3 44.4 68.6

89.0 89.0 66.7 100.0 100.0 89.0 89.0 77.7 89.0 89.0 100.0 89.0 89.0 89.0 89.0 77.7 89.0 89.0 100.0 89.0 100.0 77.7 89.0

50.0 66.5 16.5 50.0 50.0 0.0 16.5 0.0 16.5 50.0 50.0 66.5 33.5 50.0 50.0 50.0 0.0 16.5 0.0 50.0 50.0 33.5 0.0

89.0 89.0 66.7 100.0 100.0 89.0 89.0 77.7 89.0 89.0 100.0 89.0 89.0 89.0 89.0 77.7 89.0 89.0 100.0 89.0 100.0 77.7 89.0

50.0 66.5 16.5 50.0 50.0 0.0 16.5 0.0 16.5 50.0 50.0 66.5 33.5 50.0 50.0 50.0 0.0 16.5 0.0 50.0 50.0 33.5 0.0

14.8 50.0 50.0 44.4 0.0 44.4 41.7 19.4 47.2 0.0 16.7 75.0 25.0 41.7 16.7 66.7 41.7 25.0 25.0 22.2 0.0 16.7 0.0

0.0 45.9 45.9 58.2 12.5 12.5 37.5 8.4 41.6 0.0 12.5 50.0 20.9 37.5 25.0 54.2 45.9 20.9 12.5 16.6 8.4 12.5 0.0

PER-A02

PER-A03

PER-D01

PER-D02

PER-E01

PER-E02

PER-H01

PER-H02

PER-I01

PER-M01

PER-S01

PER-S02

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

100.0 100.0 100.0 100.0 100.0 100.0 100.0 11.0 77.7 100.0 66.7 100.0 89.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 89.0 100.0

66.5 50.0 66.5 100.0 83.5 50.0 83.5 0.0 33.5 83.5 33.5 66.5 66.5 66.5 50.0 83.5 100.0 83.5 50.0 100.0 100.0 66.5 50.0

100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0

100.0 58.5 100.0 100.0 100.0 41.8 16.8 50.0 25.0 25.0 25.0 25.0 58.5 58.5 75.0 75.0 100.0 25.0 91.8 83.2 16.8 83.2 58.2

0.0 0.0 0.0 0.0 0.0 77.7 89.0 0.0 77.7 0.0 0.0 66.7 0.0 77.7 89.0 77.7 89.0 0.0 0.0 0.0 0.0 0.0 0.0

0.0 0.0 0.0 0.0 0.0 16.5 33.5 0.0 33.5 0.0 0.0 50.0 0.0 16.5 0.0 50.0 33.5 0.0 0.0 0.0 0.0 0.0 0.0

72.3 83.3 22.3 5.5 44.3 66.7 83.3 83.3 44.3 11.0 11.0 44.3 61.2 83.3 38.8 77.8 77.8 66.7 89.0 66.7 22.2 22.2 16.7

0.0 33.5 0.0 0.0 0.0 0.0 8.2 16.5 8.2 8.2 0.0 8.2 8.2 8.2 8.2 0.0 0.0 0.0 0.0 0.0 8.2 0.0 8.2

0.0 50.0 0.0 0.0 0.0 50.0 0.0 0.0 50.0 0.0 0.0 50.0 0.0 50.0 0.0 44.5 16.7 0.0 0.0 50.0 0.0 0.0 0.0

0.0 25.0 0.0 0.0 8.2 33.2 0.0 0.0 33.2 0.0 0.0 25.0 0.0 33.2 0.0 25.0 16.8 0.0 0.0 16.8 0.0 0.0 0.0

81.4 81.6 77.8 96.3 92.7 92.7 92.7 77.8 18.4 66.8 22.2 88.9 92.7 81.6 7.4 96.3 92.7 66.7 44.4 70.4 14.8 7.4 14.8

50.0 50.0 50.0 44.5 55.5 44.5 27.8 50.0 27.8 33.3 0.0 50.0 50.0 50.0 0.0 50.0 44.5 44.5 11.0 50.0 11.0 5.5 0.0

0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 0.0 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 100.0 0.0

0.0 50.0 0.0 0.0 0.0 0.0 0.0 0.0 50.0 0.0 0.0 66.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 33.5 0.0 16.5 0.0

0.0 33.3 0.0 0.0 0.0 0.0 77.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.7 0.0 0.0 0.0 0.0 0.0 0.0

0.0 33.5 0.0 0.0 0.0 0.0 50.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 50.0 0.0 0.0 0.0 0.0 0.0 0.0

80.0 50.0 50.0 50.0 50.0 90.7 96.3 64.8 94.4 59.3 40.7 50.0 46.3 66.7 40.7 81.5 48.2 66.7 48.2 48.2 50.0 33.3 0.0

63.3 22.2 27.8 36.1 44.5 36.1 52.8 33.3 44.5 30.6 16.7 30.6 27.8 36.1 16.7 50.0 30.6 27.8 27.8 33.3 25.0 16.7 0.0

11.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 100.0 100.0 0.0 0.0 0.0 0.0 44.3 22.3 0.0 0.0 100.0 0.0 55.7 0.0 11.0

0.0 0.0 0.0 0.0 0.0 0.0 33.5 0.0 83.5 100.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 100.0 0.0 16.5 0.0 0.0

0.0 0.0 0.0 0.0 0.0 89.0 0.0 0.0 77.7 0.0 0.0 66.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 89.0 0.0 0.0 0.0

0.0 0.0 0.0 0.0 0.0 25.0 0.0 0.0 25.0 0.0 0.0 25.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 25.0 33.2 0.0 0.0

85.7 100.0 82.7 90.1 66.7 86.4 96.3 77.8 81.5 88.9 88.9 88.9 58.0 88.9 84.0 100.0 90.1 74.1 66.7 67.9 80.3 59.3 71.6

73.8 66.6 72.9 75.0 54.1 33.3 37.6 22.9 45.8 27.1 43.8 62.5 41.7 66.7 54.2 54.1 68.8 37.5 52.1 35.4 62.5 41.7 43.8

100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 89.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0

66.5 100.0 100.0 100.0 100.0 50.0 33.5 33.5 33.5 66.5 66.5 100.0 100.0 66.5 100.0 83.5 100.0 83.5 100.0 16.5 100.0 100.0 100.0

PER-S03

PER-S04

PER-V01

PER-V02

PER-V03

PER-W01

PER-W02

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

0.0 0.0 0.0 0.0 0.0 100.0 100.0 0.0 77.7 0.0 0.0 0.0 0.0 89.0 66.7 77.7 89.0 0.0 0.0 0.0 11.0 0.0 0.0

0.0 0.0 0.0 0.0 0.0 16.5 66.5 0.0 50.0 0.0 0.0 0.0 0.0 66.5 50.0 50.0 66.5 0.0 0.0 0.0 0.0 0.0 0.0

91.1 55.6 46.7 40.0 95.6 88.9 91.1 28.9 86.7 28.9 37.8 48.9 91.1 42.2 42.2 95.6 44.4 44.5 46.7 40.0 46.7 40.0 44.5

69.5 41.7 33.3 27.8 66.7 38.9 44.5 30.6 44.5 16.7 22.2 47.2 44.4 27.8 30.5 47.2 36.0 30.6 19.4 25.0 38.8 27.8 27.8

87.7 91.9 71.7 76.8 77.8 61.6 65.7 44.4 52.5 60.6 63.6 79.8 68.7 89.9 49.5 88.9 74.8 77.8 55.5 71.7 73.7 58.6 58.6

70.1 77.8 75.0 55.7 66.7 22.2 33.3 22.2 41.7 47.2 49.9 77.8 80.6 80.6 36.2 69.4 77.8 63.9 80.6 69.4 77.8 61.1 38.9

100.0 100.0 100.0 100.0 100.0 77.7 100.0 100.0 100.0 0.0 77.7 100.0 89.0 77.7 66.7 89.0 89.0 100.0 77.7 100.0 100.0 100.0 100.0

66.5 50.0 83.5 66.5 83.5 16.5 66.5 66.5 50.0 0.0 50.0 50.0 66.5 66.5 66.5 66.5 83.5 50.0 0.0 66.5 50.0 50.0 50.0

50.0 72.2 50.0 55.5 50.0 50.0 50.0 50.0 50.0 50.0 50.0 44.5 50.0 61.2 50.0 50.0 50.0 50.0 50.0 50.0 50.0 50.0 50.0

55.7 49.8 50.0 55.5 55.7 38.8 38.8 50.0 38.8 33.3 33.3 55.5 44.5 50.0 55.7 55.7 50.0 50.0 5.5 33.3 55.7 44.5 38.8

100.0 100.0 100.0 100.0 100.0 89.0 77.7 100.0 100.0 89.0 100.0 100.0 100.0 89.0 100.0 89.0 100.0 100.0 100.0 89.0 100.0 89.0 89.0

33.5 16.5 50.0 66.5 100.0 16.5 33.5 33.5 33.5 50.0 33.5 16.5 66.5 50.0 50.0 16.5 16.5 83.5 50.0 16.5 50.0 16.5 33.5

100.0 100.0 100.0 100.0 0.0 100.0 100.0 0.0 100.0 0.0 100.0 100.0 0.0 100.0 0.0 100.0 100.0 100.0 0.0 100.0 0.0 100.0 0.0

66.8 66.8 41.8 66.8 0.0 50.0 16.8 33.2 58.2 0.0 25.0 75.0 0.0 91.8 0.0 50.0 75.0 25.0 0.0 25.0 0.0 58.5 0.0

88.0 97.8 88.1 95.6 74.0 83.0 85.9 79.3 74.8 70.4 71.8 94.8 85.9 88.9 77.8 89.6 90.4 77.0 63.7 82.2 77.8 83.0 70.4

87.5 74.4 84.5 84.4 85.6 56.6 56.7 60.0 52.2 51.1 48.9 75.5 76.7 76.6 68.9 67.7 75.6 74.4 63.3 56.7 78.9 72.2 57.8

97.8 74.4 77.8 86.7 81.1 68.9 86.7 67.8 71.1 57.8 68.9 82.2 78.9 78.9 73.3 80.0 97.8 82.2 78.9 76.7 71.1 84.4 77.8

90.0 68.3 56.6 71.7 90.0 51.6 68.3 46.6 55.0 40.0 36.6 61.7 68.3 65.0 53.3 70.0 80.0 71.7 68.3 46.7 60.1 63.3 36.6

64.6 65.5 68.9 72.8 51.7 65.6 77.3 61.1 52.2 33.4 45.5 82.8 46.7 72.2 45.0 82.2 73.9 72.8 50.6 59.4 37.2 53.3 30.6

61.7 62.7 52.4 61.1 52.4 51.6 50.0 54.0 34.9 22.2 27.8 69.1 32.5 57.2 44.5 52.4 65.1 48.4 36.5 43.7 32.5 33.4 27.0

66.7 86.7 67.8 90.0 47.8 38.9 46.7 56.7 58.9 60.0 58.9 60.0 60.0 63.3 53.3 66.7 80.0 57.8 57.8 42.2 43.3 57.8 42.2

53.7 63.3 55.0 63.3 46.7 28.3 36.6 36.6 38.4 31.6 33.4 48.3 45.0 43.4 46.7 56.6 56.7 50.0 40.0 31.7 40.0 36.7 23.3

96.3 63.5 73.0 82.6 74.6 36.5 65.1 27.0 52.4 31.8 34.9 73.0 76.2 71.4 38.1 81.0 73.0 85.7 63.5 69.9 50.8 73.0 66.7

91.7 61.9 64.3 71.5 92.9 23.7 47.6 26.1 38.1 23.8 33.4 54.9 61.9 50.0 33.4 76.2 66.6 83.4 52.4 54.8 47.6 64.3 45.3

86.1 82.2 77.7 82.2 88.9 73.3 64.5 53.3 64.5 71.1 80.0 80.0 73.3 75.5 53.3 77.7 82.2 77.8 73.3 68.9 82.2 71.1 64.5

70.8 73.4 60.0 60.0 73.3 50.0 46.7 23.3 46.7 40.0 43.3 56.7 56.7 46.7 33.4 66.6 63.3 63.3 33.3 63.3 60.0 46.6 40.0

BAS-07

VUL-01

VUL-02

VUL-03

VUL-04

Q-01

Q-02

BAS-01

PER-I02

Q-03

BAS-02

Q-04

BAS-03

Q-05

BAS-04

Q-06

BAS-05

Q-07

BAS-06

Q-08

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

Det

Plan

100.0 100.0 100.0 83.3 100.0 83.3 72.2 94.5 94.5 83.3 83.3 100.0 94.5 100.0 72.2 94.5 83.3 89.0 83.3 94.5 100.0 77.8 100.0

75.0 83.2 100.0 83.5 100.0 50.0 58.2 41.8 66.5 58.2 41.8 100.0 50.0 66.8 58.2 100.0 75.0 58.2 41.8 75.0 100.0 58.2 66.8

77.8 86.7 84.5 90.0 72.2 61.1 62.2 56.7 36.7 33.3 61.1 78.9 66.7 81.1 71.1 54.4 72.2 47.8 45.6 61.1 72.2 66.7 36.7

79.1 70.4 61.0 83.3 72.3 42.6 51.8 44.4 20.4 35.2 51.8 83.4 61.1 68.6 63.0 46.3 68.5 44.5 50.0 51.8 68.6 46.3 38.9

100.0 100.0 100.0 100.0 100.0 100.0 89.0 100.0 100.0 66.7 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 100.0 66.7 100.0 100.0 100.0

83.5 66.5 50.0 50.0 66.5 33.5 50.0 50.0 33.5 50.0 50.0 50.0 50.0 50.0 33.5 50.0 50.0 100.0 50.0 33.5 50.0 50.0 50.0

100.0 100.0 0.0 11.0 33.3 11.0 100.0 0.0 11.0 0.0 33.3 89.0 0.0 100.0 11.0 100.0 0.0 0.0 11.0 11.0 89.0 0.0 0.0

100.0 50.0 50.0 16.5 33.5 16.5 83.5 0.0 33.5 0.0 33.5 100.0 50.0 50.0 16.5 66.5 33.5 83.5 50.0 16.5 100.0 0.0 0.0

55.6 40.7 48.2 59.2 55.6 66.7 37.0 44.4 51.8 51.8 63.0 66.7 88.9 96.3 74.1 63.0 70.4 81.4 48.1 48.1 25.9 63.0 44.4

50.0 50.0 41.8 50.0 62.5 41.6 20.9 29.1 4.1 25.0 25.0 41.6 58.2 54.1 50.0 45.9 50.0 50.0 29.1 25.0 33.4 29.1 33.2

82.0 82.2 70.0 84.4 78.9 80.0 53.3 65.5 62.2 73.3 77.8 87.8 75.6 74.5 67.8 82.2 70.0 84.5 52.2 77.8 71.1 62.2 52.2

0.0 83.5 0.0 100.0 0.0 66.5 0.0 0.0 66.5 0.0 0.0 33.5 0.0 0.0 83.5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

71.6 81.1 67.2 80.0 65.5 69.5 75.6 63.4 80.0 47.8 56.7 81.1 65.6 77.8 55.6 83.3 73.9 64.4 60.0 56.7 63.9 51.1 49.4

100.0 100.0 83.5 100.0 83.5 100.0 100.0 50.0 83.5 50.0 33.5 66.5 83.5 66.5 100.0 83.5 100.0 83.5 50.0 50.0 50.0 33.5 50.0

68.9 71.3 74.1 78.7 63.0 79.6 77.8 53.7 83.3 56.5 61.1 78.7 60.2 78.8 63.0 71.3 67.6 50.9 49.1 56.5 63.0 43.5 45.4

83.5 0.0 0.0 66.5 0.0 83.5 33.5 0.0 66.5 0.0 50.0 0.0 0.0 50.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

100.0 85.8 87.3 95.2 96.9 96.9 84.1 74.6 79.4 84.1 92.0 96.9 82.6 87.3 82.6 96.9 85.8 93.7 69.8 98.4 77.8 76.2 65.1

— — — — — — — — — — — — — — — — — — — — — — —

50.1 38.1 28.6 34.9 44.4 38.0 41.2 15.9 44.4 17.4 20.6 52.4 17.5 25.4 28.6 47.7 38.0 22.2 15.9 25.4 15.9 25.3 15.9

— — — — — — — — — — — — — — — — — — — — — — —

89.0 94.5 94.5 88.8 94.5 94.5 66.7 88.8 88.8 83.3 66.7 94.5 94.5 83.3 94.5 94.5 94.5 77.8 88.8 83.3 94.5 94.5 72.2

— — — — — — — — — — — — — — — — — — — — — — —

66.7 50.0 49.9 34.1 81.0 30.1 51.6 43.6 34.1 14.3 11.1 66.7 37.3 51.6 46.0 42.1 52.4 38.9 41.3 42.8 46.8 61.1 30.1

51.5 27.7 34.7 20.8 76.4 9.7 33.3 15.2 16.7 11.1 8.3 43.1 25.0 43.0 30.6 33.3 34.7 23.6 33.3 29.2 48.7 37.5 13.8

92.6 52.8 55.6 55.5 75.0 36.2 61.2 52.8 19.4 22.2 2.8 50.0 30.6 44.5 38.8 61.2 52.8 55.5 66.7 41.7 16.7 33.2 19.4

— — — — — — — — — — — — — — — — — — — — — — —

43

D.3

Token Usage

Model

Steps

Input

Claude Opus 4.7 Claude Opus 4.6 Claude Opus 4.5 Claude Sonnet 4.6 Claude Sonnet 4.5 GPT-5.5 GPT-5.4 Pro GPT-5.4 GPT-5.2 Pro Gemini 3.1 Pro Gemini 3 Flash GLM-5.1 GLM-5 DeepSeek V4 Pro DeepSeek V3.2 Qwen3.7 Max Qwen3.7 Plus Qwen3.6 Plus Qwen3.5 Plus Kimi K2.6 Kimi K2.5 MiniMax M2.7 MiniMax M2.5

32.1 24.4 26.3 31.4 28.5 15.4 19.4 8.6 20.2 38.5 36.1 18.2 18.6 15.8 47.2 14.8 16.8 20.6 16.1 18.7 14.7 19.5 29.4

1,451,154 983,689 1,269,874 1,639,221 961,823 846,206 1,150,807 429,288 1,202,498 1,187,974 1,415,708 855,301 828,880 721,040 1,331,327 669,566 899,240 1,141,624 781,664 778,775 518,576 788,428 934,533

Output Cost (USD) 38,809 19,933 22,442 23,099 39,708 18,659 23,794 10,443 20,534 8,347 7,489 10,321 14,462 14,010 17,879 13,746 17,257 15,410 14,536 13,151 16,348 12,738 10,164

8.23 5.42 6.91 5.26 3.48 4.79 38.81 1.23 28.70 2.48 0.73 0.88 0.53 0.33 0.31 1.05 0.31 0.40 0.26 0.47 0.23 0.20 0.12

(a)

(b)

Figure 8: Token Usage. (a) Average per-run agent steps, input tokens, output tokens, and cost. (b) Average Overall Detection and Planning score vs. cost per run. Figure 8 (a) reports the average number of steps, token usage, and dollar cost per run for each model. The average number of steps varies substantially across models. Gemini 3.1 Pro and DeepSeek V3.2 take 38.5 and 47.2 steps on average, respectively, whereas GPT-5.4 stops after 8.6. The results of Gemini 3.1 Pro and DeepSeek V3.2 demonstrate that taking more steps does not guarantee better performance; additional investigation may offer no clear payoff. Total token consumption also varies across models for two main reasons. Some models take many investigative steps: for example, DeepSeek V3.2 consumes 1.33M tokens over 47.2 steps on average. Other models take fewer steps but maintain long contexts, as observed for Claude Opus 4.7 and Claude Sonnet 4.6. Figure 8 (b) compares cost with the average Overall Detection and Planning score. Cost and performance show only a weak relationship. Higher cost and greater token consumption help to some extent, but the highest-performing models are not the most expensive. GPT Pro models have the highest costs, but their performance remains around 50–60%, showing that their higher price does not translate into stronger incident-response capability. The Claude models are also costly but perform better. Among the least expensive models, GLM-5.1, DeepSeek V4 Pro, and the Qwen3.7 series achieve results comparable to the Claude series while costing far less.

44