MazeRunner: Nonlinear Task and Clue Orchestration for LLM-driven Black-Box Automated Penetration Testing

. RESEARCH PAPER . MazeRunner: Nonlinear Task and Clue Orchestration for LLM-driven Black-Box Automated Penetration Testing Zhenyuan Li1* , Yi Jiang1 , Junjie Cheng1 , Yaokun Li1 , Jing Qiu2 & Shouling Ji1 1 Zhejiang University, Hangzhou 310058, China Guangzhou University, Guangzhou 510006, China

arXiv:2608.14216v1 [cs.CR] 14 Aug 2026

2

Abstract Penetration testing constitutes a resource-intensive yet indispensable component of modern network security. Although large language models (LLMs) have demonstrated substantial potential in automating security auditing, existing LLM-driven agents remain largely confined to executing end-to-end workflows within simplified, linear scenarios. Real-world black-box engagements are fundamentally different: the underlying attack graph is initially unknown and must be incrementally constructed from environmental feedback. Each observation may expose multiple plausible attack branches, failures are often semantically ambiguous, and decisive clues may be separated by vast action horizons. Consequently, current autonomous agents repeatedly fall into depth-first traps, misattribute failures, and forget previously discovered evidence. We present MazeRunner, an autonomous penetration testing system based on a three-agent task and clue orchestration framework. MazeRunner separates global task orchestration, context-intensive execution, and failure-oriented review, while maintaining task states and environmental evidence in a persistent Task & Clue Cache. This design enables the system to revise failed actions, recover missing prerequisites, switch between attack branches, and correlate clues across distant execution stages. We evaluate MazeRunner on 10 recently released Hack The Box targets under a common budget of 20 million LLM tokens per system–target run and a protocol designed to prevent target-specific solution leakage. With Claude Sonnet 4.5, MazeRunner completes 47.7% of the annotated subtasks, compared with 36.2% for PentestGPT-V2 and 34.2% for Claude Code. MazeRunner obtains user-level or higher access on 6 targets, including root access on 2, whereas each same-model baseline obtains user-level access on 2 targets and fails to reach root access. Analysis of the complete execution traces further shows that MazeRunner explores a broader range of attack branches and achieves higher shell-acquisition efficiency per token than the same-model baselines. Keywords Citation

1

Automated Pentesting, LLM Agents, Multi-Agent Systems, Task Orchestration, Failure Recovery MazeRunner: Nonlinear Task and Clue Orchestration. Sci China Inf Sci, for review

Introduction

As a proactive and rigorous security assessment methodology, penetration testing (pentesting) plays a pivotal role in both industrial and academic domains. According to Fortune Business Insights, the global annual expenditure on pentesting reached $2.74 billion in 2025 [1]. Nevertheless, traditional pentesting relies heavily on human expertise, suffering from prohibitive labor costs, low efficiency, and poor scalability, thereby fundamentally limiting its widespread deployment [2]. Therefore, automated pentesting and threat assessment solutions have garnered extensive attention from both researchers and practitioners. Although often conceptualized as a sequence of standard phases, black-box pentesting initiates with only a target and an ultimate objective, lacking a known execution path. The underlying attack graph is initially opaque and must be incrementally synthesized as new environmental clues, such as unknown services or credentials, are discovered. This renders the exploitation process inherently dynamic, requiring frequent task generation, lateral pivoting, and strategic recalibration based on emerging evidence [3]. Moreover, environmental feedback is highly ambiguous: execution failures do not necessarily invalidate a chosen path, while apparent successes or promising clues can easily trigger deceptive “rabbit holes.” Because critical evidence required for late-stage decisions may be acquired dozens of steps earlier [4], relying on an ever-growing LLM context window is fundamentally inadequate. Instead, effective autonomous pentesting mandates the integration of a persistent and verifiable external state architecture. Early automation efforts in pentesting mainly focused on vulnerability scanning and attack surface analysis [13–15]. Due to the absence of robust analytical reasoning and dynamic decision-making capabilities required in unconstrained scenarios, the automation of subsequent, more complex exploitation phases has remained largely stagnant. The advent of Large Language Models (LLMs) has catalyzed new opportunities, inspiring recent works * Corresponding author (email: lizhenyuan@zju.edu.cn)

Table 1

Comparison of MazeRunner with existing LLM-based automated pentesting systems

Automatic

Scenario Scope

Independent Task/Clue

Failure/Fault Handling

AutoPT [5]

×

Vuln. Exploiting

Vuln. only

Vuln. Switching

Single

CLI

PentestAgent [2]

×

Vuln. Exploiting

×

×

“Recon-Plan-Execute”

CLI + Manual

System

Multi-Agent Architecture

Tool/Target Interaction

CAI [6]

×

Open

×

×

Specialist Agents

CLI + Manual

PentestGPT [7]

×

Open

Task only

Manual

Single

Manual

TermiAgent [8]

✓

Open

Task only

×

Single

CLI

Incalmo [9]

Multi-host

Task + Clue

×

“Plan-Execute”

CLI + Scripts

PentestMCP [10]

#

✓

Vuln. Exploiting

Task only

Plan Switching

✓

Multi-host

Clue only

Self-Healing Retry

“Recon-Enum-Exploit” “Plan-GuardExecute-Observer”

MCP + CLI

Trinity [11] Cochise [3]

✓

AD-specific

Task only

Branch Switching

“Plan-Execute”

CLI

PentestGPT-V2 [12]

✓

Open

Task only

Review + Restart

Dynamic Sub-agents

CLI

MazeRunner

✓

Open

Task + Clue

Review + Branch Selection

“Plan-Execute-Review”

CLI

CLI

to leverage LLMs for automating these advanced stages [2, 6, 7, 9]. As summarized in Table 1, these pioneering frameworks have successfully demonstrated the feasibility of utilizing LLMs to interpret raw security tool outputs, map recognized vulnerabilities to corresponding exploits, and autonomously generate actionable payloads. Operationally, the majority of these systems interface with target environments by executing standard security tools through Command Line Interfaces (CLI). By leveraging the vast security knowledge embedded within their foundational models and employing basic heuristic planning, these systems excel at executing atomic operations. Consequently, they have achieved notable success in automating relatively straightforward, linear attack paths, particularly within standardized or isolated Capture-The-Flag (CTF) environments, as illustrated in Figure 1 and Figure 2. These unique domain characteristics manifest in ex(c) XBEN-003-24 (a) HTB-Interpreter (b) Vulhub-Drupal isting LLM-driven pentesting frameworks as three reTarget Target Target External Surface curring operational bottlenecks: 1) Deficient OrchestraPort 80 Port 443 Port 22 tion & The “Depth-First Trap”: Absent rigorous superPort 8080 Port 80 Jetty Jetty SSH vision, the stochastic nature of LLMs leads to unpreHTTP HTTPS Service Web Service Web Service dictable task orchestration. When a task fails, agents frequently hyper-focus on that local branch. As execuMirth Connect Service 4.4.0 Drupal 7.31 CMS Auth Web App tion context accumulates, this disproportionate anchorPre-Auth RCE ing drives repeated, unproductive attempts, inevitably Auth Bypass SQL Injection User-1 Shell User-2 Shell culminating in goal drift, hallucinated tasks, and extra overhead. 2) Contextual Amnesia: The finite context Port 3306 Port 54321 Port 6661 Flag.txt Shell window of LLM agents cannot accommodate the masFlask MariaDB Mirth HL7 sive volume of feedback generated during iterative enService Service Service Attack Goals Rabbit Hole vironmental interactions, causing tasks and clues to be Unsafe XML Credential System Entities Attack Path Input Injection Disclosure ignored or forgotten in long-term engagements. 3) ErInternal Vulnerabilities Clue Correlation Root Shell ror Misattribution: LLMs struggle with accurate state Surface tracking and root-cause analysis. Following a failure, Figure 1 Complexity comparison of diverse pentesting targets their decision-making degrades; they often erroneously link the failure to unrelated tasks, leading to the premature abandonment of viable attack trajectories. As evidenced by our study (§2 and §5), these bottlenecks constrain both the depth and breadth of LLM-driven automated pentesting. To address these limitations, we propose MazeRunner, a multi-agent automated pentesting framework architected specifically for task and clue orchestration within complex, black-box targets. At its core, the framework formalizes tasks and clues as fundamental primitives managed within a persistent, independent cache. This ensures strict state consistency to mitigate contextual amnesia and task redundancy. MazeRunner operates through a continuous, collaborative “Strategist-Executor-Reviewer” agent cycle. Specifically, we introduce a dedicated Reviewer Agent tasked with diagnosing execution failures to provide actionable, corrective feedback; this mechanism allows the system to escape the “Depth-First Trap” and enhances the precision of subsequent task selection/correction. Simultaneously, we expand the operational scope of the Strategist Agent beyond localized planning, empowering it to orchestrate global penetration strategies and dynamically recalibrate the task graph in response to execution anomalies. The Executor Agent is responsible for operationalizing these strategies into concrete environmental 2

MazeRunner Sonnet 4.5

5/5

6/6

6/6

Vulhub 5/5

6/6

6/6

6/6

6/6

HTB 6/6

6/6

7/13

Claude Code Sonnet 4.5

5/5

6/6

6/6

5/5

6/6

6/6

6/6

6/6

6/6

6/6

7/13

PentestGPT-V2 Sonnet 4.5

5/5

6/6

6/6

5/5

6/6

6/6

6/6

6/6

6/6

6/6

7/13

CAI Sonnet 4.5

5/5

6/6

6/6

5/5

6/6

2/6

1/6

2/6

2/6

1/6

2/13

78 04 60 ph 014 005 W-0 BOW- BOW- BOW-0 BOW-0 hugegra O B X X X X X

ofbi

mq

ion

Figure 2

z

ve acti

fus cold

al

drup

U

U

U

8/14 8/14

U

U

13/13

R

7/13

4/14

13/13

2/14

1/13

R

U: user shell (dashed border) R: root shell (double border) U

1.0

4/15

10/21

4/15

4/21

0.6

4/15

4/21

0.4

1/15

2/21

al ta er ea gDa terpret Princip DevAr Win In

te

0.8

Completion Depth

XBOW

0.2 0.0

Pira

Preliminary performance comparison across various targets

interactions while harvesting critical runtime clues. Ultimately, this closed-loop system synthesizes LLM-driven reasoning with deterministic procedural logic to enable resilient, adaptive, and zero-intervention pentesting. To evaluate MazeRunner, we conducted experiments across 10 Hack The Box (HTB) targets released between February and April 20261) . The targets cover Linux and Windows systems and span easy, medium, and hard difficulty levels. To enable a controlled comparison, we impose a common maximum budget of 20 million LLM tokens on each system–target run and prohibit the use of target-specific online write-ups. With Claude Sonnet 4.5, MazeRunner achieves a subtask completion rate of 47.7%, compared with 36.2% for PentestGPT-V2 and 34.2% for Claude Code. It achieves strictly greater subtask progress than each same-model baseline on 6 of the 10 targets. MazeRunner obtains user-level or higher access on 6 targets and reaches root access on 2 of them, whereas each baseline obtains user-level access on 2 targets and reaches root access on none. Analysis of the execution traces indicates that the baselines frequently expend their budgets on repeated local attempts or terminate before exploring viable alternative branches. By maintaining persistent task and clue states and introducing an independent failure-review stage, MazeRunner reduces such execution loops and improves shell-acquisition efficiency. In one representative run, it obtains root access using 79 commands, of which 7 are classified as invalid. The main contributions of this work are as follows: • We propose MazeRunner, an autonomous pentesting system built upon a multi-agent task and clue orchestration framework. By separating strategic task orchestration, context-intensive execution, and failure-oriented review, MazeRunner supports adaptive navigation of nonlinear attack paths in complex black-box environments. • We design a persistent Task & Clue Cache that maintains task dependencies, execution states, environmental evidence, and cross-stage clue associations. Combined with an independent Reviewer Agent, this design supports failure diagnosis, prerequisite recovery, branch switching, and long-horizon evidence reuse. • We evaluate MazeRunner on 10 recently released HTB targets under a common resource budget. With Claude Sonnet 4.5, MazeRunner achieves a 47.7% subtask completion rate and obtains user-level or higher access on 6 targets, including root access on 2. The corresponding baselines reach 36.2% and 34.2% completion and obtain user-level access on 2 targets without reaching root. MazeRunner also achieves broader attack-path exploration and higher shell-acquisition efficiency per token in the evaluated runs.

2

Background and motivation

2.1

Automated pentesting

penetration testing (pentesting) is an authorized simulated attack designed to evaluate the security of systems, networks, or applications. Traditionally, this process has relied heavily on human experts. However, this manual paradigm is severely constrained by prohibitive costs, a chronic shortage of skilled professionals [16], and systemic risks introduced by human oversight within complex workflows [17]. Furthermore, periodic manual testing is increasingly inadequate for securing rapidly evolving attack surfaces, driving the critical need for automated pentesting to enable continuous security validation. Despite this urgent demand, automating pentesting remains a formidable challenge [18, 19]. In practice, pentesting necessitates invoking diverse tools through an iterative exploration cycle of “task orchestration, execution, information analysis, and knowledge/tool retrieval” under highly uncertain blackbox conditions. Consequently, effective automation demands not merely the mechanical execution of attacks, but rather sophisticated, dynamic attack planning [20]. 1) HTB, https://www.hackthebox.com/

3

Table 1 summarizes representative LLM-powered pentesting systems, all of which aim to enhance efficiency and automation. However, the majority of these frameworks fall short of achieving end-to-end automation. For instance, tools such as CAI [6] and PentestGPT [7] function merely as conversational assistants, providing heuristic advisory recommendations rather than executing actions. Conversely, systems like PentestAgent [2] and AutoPT [5] focus primarily on vulnerability exploitation, lacking the comprehensive capability to handle other essential phases of the pentesting lifecycle. TermiAgent [8]improves memory management and exploit execution, but provides limited support for global task and clue organization. Furthermore, frameworks such as Cochise [3] and Incalmo [9] achieve automation only within constrained scenarios. Specifically, Incalmo addresses task planning in multi-host environments but remains partially dependent on pre-defined scripts, whereas Cochise is exclusively tailored for Active Directory targets. PentestMCP [10] primarily emphasizes standardized tool integration, while Trinity [11] introduces stateful multi-agent reasoning for network pentesting. Notably, general-purpose coding agents (e.g., Claude Code) have recently demonstrated a considerable degree of automated penetration capability. Building upon these paradigms, PentestGPT-V2 [12] has largely realized a fully autonomous, zero-intervention pentesting workflow. 2.2

Preliminary Study on Pentest Targets and Baseline

Recent studies [2, 5, 7, 12] demonstrate the potential of LLM agents in automated pentesting, particularly within simplified, “CTF-like” environments. As illustrated in Figure 1 (b, c), these tasks typically feature well-defined objectives and strictly linear progression paths, often reducing to a predictable sequence: a single open port, a specific service, an isolated vulnerability, a direct exploit, and the final target. Due to this Figure 3 Complexity of Pentesting Benchmarks structural simplicity, LLMs can maintain task focus without being diverted by environmental noise. Figure 3 further quantifies the complexity of various benchmarks, including HTB, Vulhub2) , XBOW3) , CVE-Bench4) , and MHBench5) , by comparing the average number of CWEs per target and the measured task depth. Task depth (d) CWE Count Depth Comparison quantifies the cumulative number of requisite sub-tasks (e.g., port scanning, information gathering, andand vulnerability exploitation) necessary to successfully compromise the target (a detailed breakdown is provided in Appendix Table A3). The results indicate that HTB targets exhibit significantly higher complexity in both breadth and depth compared to other datasets. Consequently, when confronted with the non-linear trajectories of complex targets such as “HTB-Interpreter” (Figure 1 (a)), which present multiple potential paths and misleading rabbit holes, existing LLMs struggle. Figure 2 summarizes the penetration efficacy of three representative baseline systems alongside our proposed MazeRunner across three distinct environments: HTB (non-linear), Vulhub (linear), and XBOW(linear). For the underlying foundational model, we selected Claude Sonnet 4.5, a recent iteration widely adopted and highly performant across the aforementioned works. As the results indicate, with the exception of CAI, which pauses for human instruction following initial reconnaissance, all evaluated systems, including Claude Code, successfully achieved full compromise on the Vulhub and XBOW targets. This high success rate is attributable to two factors: the relative structural simplicity of Vulhub and XBOW tasks, and the fact that many benchmark targets and their corresponding solutions significantly predate the model’s training data cutoff. During execution, we observed that the LLMs readily recognized these environments and rapidly retrieved known solutions from their parametric memory. However, this inherent advantage introduces severe evaluation bias and fails to generalize to zero-day or non-linear scenarios. Conversely, when evaluated against more complex and novel HTB challenges (specifically, new targets released in 2026), the performance of the baseline systems declined precipitously, yielding two shells on two targets. In contrast, MazeRunner enhanced overall penetration efficacy, successfully securing 1 root shell and 3 user shells. Comprehensive experimental details and further metrics are deferred to the Evaluation section. It is crucial to note that the PentestGPT-V2 results reported in our study diverge significantly from the performance metrics presented in their original paper. This discrepancy arises because their evaluation methodology only accounted for 2) Vulhub, https://vulhub.org/ 3) XBOW, https://xbow.com/ 4) CVE-Bench, https://github.com/uiuc-kang-lab/cve-bench 5) MHBench, https://arxiv.org/abs/2501.16466

4

Target

(a) MazeRunner + GPT 5.2

Target

1-Perform Nmap Scan 2-Identify Exposed SSH Port(22)

3-Identify Exposed Web Port(80, 443)

15-Attempt SSH Login

4-Discover API Endpoint

Get User-2 Shell

5-Identify Mirth Connect Version (4.4.0)

Target

1-Perform Nmap Scan

7-Match CVE-202343208

6-Search Config Files

9-Exploit Mirth Connect RCE (Metasploit)

8-Download exploit.py from Github

Get User-1 Shell

Attempt Failed

3-Identify Exposed Web Port(80, 443)

2-Identify Exposed SSH Port(22)

3-Identify Exposed Web Port(80, 443)

9-Attempt SSH Login

4-Attempt smbclient Login

6-Attempt SSH Login

4-Discover API Endpoint

Repeated Login Attempts (Failed)

5- Check Whether Port 1443 Is Open

11-Identify Exposed Port(6661)

13-Read mirth.properties

18-Generate Base64 Encoded Reverse Shell Payload

14-Recover DB Credentials (mirthdb / MirthPass123!)

19-Embed Payload into Malicious XML

12-Identify Exposed Port(54324)

7-Match CVE-202343208

7- Attempt FTP Login

8-Search Exploit Database (searchsploit)

8-Search Writeup Online

Bypass Exploration via Writeup Guidance

11-Generate Base64 Encoded Reverse Shell Payload

20-POST XML to /addPatient

12-Embed Payload into Malicious XML

Get Root Shell

Manual Password Brute-force employed

Target

1-Perform Nmap Scan 2-Identify Exposed SSH Port(22)

3-Identify Exposed Web Port(80, 443)

4-Attempt SSH Login

5-Identify Web Services

13-POST XML to /addPatient (b) Claude Code + Sonnet 4.5

Get Root Shell

Fail to Make Further Progress

(c) ClaudeCode + GPT 5.2

Get User Shell

16-Enumerate Database 17-Recover Internal Service Link

Repeated Login Attempts 5-Identify Mirth Connect Version (4.4.0) (Failed)

6- Check Whether Port 21 Is Open

10-Download exploit.py from Github 10-Identify Exposed Port(3306)

1-Perform Nmap Scan

2-Identify Exposed SSH Port(22)

Fail to Make Further Progress (d) PentestGPT + GPT 5.2

Figure 4 Illustrative diagnostic trajectories on HTB-Interpreter. These runs are excluded from the formal evaluation in Evaluation. Panel (a) additionally includes a human-assisted password-cracking step.

the publication dates of official write-ups; however, their underlying Claude Code agent actively leverages internet search capabilities to retrieve third-party write-ups and exploit scripts, resulting in severe data contamination. To ensure a rigorous and fair evaluation, our experimental design explicitly instructed the baseline agents via system prompts to strictly avoid referencing external write-ups, and we meticulously verified the integrity of the resulting execution traces. 2.3

Motivating Example

To further illustrate the limitations of current systems when navigating complex, non-linear targets, we present a detailed case study of the “HTB-Interpreter” machine. As depicted in the comparative example in Figure 4, we contrast the progression of existing baseline agents against our proposed MazeRunner framework within this identical complex pentesting environment. Overall, current automated pentesting systems exhibit significant limitations in task orchestration, state tracking, and failure recovery when handling long-horizon, black-box targets. As shown in Figure 4 (b), (c), and (d), agents relying on standard LLM toolchains (e.g., Claude Code or PentestGPT-V2) are easily trapped in “rabbit holes” during exploration. This results in a multitude of redundant or even hallucinated tasks, as detailed in Figure 5 (“HTB-VulnEscape”). For instance, these agents expend critical resources on repeated, unsuccessful SSH login attempts or engage in irrelevant probing that diverges from the critical attack path (e.g., blindly testing unassociated FTP or SMB ports). Furthermore, agents such as Claude Code (Figure 4 (b)) frequently resort to retrieving online write-ups to bypass the genuine exploration phase. In scenarios devoid of readily available solutions, or upon encountering tool execution failures (Figure 4 (c) and (d)), these baseline systems halt completely and fail to make further progress. This exposes a severe deficiency in error recovery and adaptability within complex, non-linear benchmark environments. In contrast, as depicted in Figure 4 (a), MazeRunner demonstrates exceptional capability in dynamic decisionmaking, error-aware task backtracking, and cross-stage clue correlation. Following an initial failed exploitation attempt (Step 8: downloading a faulty exploit script), the system avoids falling into an infinite retry loop. Instead, it successfully backtracks and pivots to a viable alternative strategy (Step 9: deploying Metasploit). Crucially, after acquiring the initial user shell, the system exhibits robust long-term state maintenance. It systematically reads configuration files, recovers database credentials, and enumerates internal services. It is worth noting that steps 16 and 17 require executing a password brute-force attack. Since relying on the LLM to automate this process incurs prohibitive token overhead and temporal latency, we employed manual intervention for this specific task. This allowed us to bypass the mechanical bottleneck and continue evaluating the framework’s overall end-to-end efficacy. Following this manual phase, the autonomous agent seamlessly resumes control, synthesizing the accumulated, 5

fragmented cross-stage clues to construct the precise malicious XML payload required to ultimately secure Root access. This contrast highlights that relying solely on rudimentary vulnerability matching and naive tool execution is profoundly insufficient for dynamic, non-linear penetration scenarios. Endowing autonomous agents with the dual capacity to recognize errors and backtrack within uncertain environments, coupled with the ability to persistently maintain and integrate fragmented clues across multiple attack stages, is imperative for overcoming current bottlenecks in automated pentesting. Addressing this fundamental gap serves as the core motivation driving this research.

3

MazeRunner design

3.1

System Overview

Automated black-box pentesting is inherently challengClaude Code Blocked Trajectory MazeRunner Adjust and Continue ing because the target’s internal topology and the unTarget Target derlying attack graph are initially opaque. Furthermore, Initial Reconnaissance Initial Reconnaissance dynamically emerging clues, competing attack branches, and semantically ambiguous execution failures profoundly Strategist RDP Service RDP Service Redundant complicate persistent state tracking and strategic replanScanning Adjust ning. Despite recent advancements, current LLM-driven pentesting frameworks fundamentally struggle to navigate Executor Probe Port 445 these non-linear environments, where complex branching Replan frequently precipitates erratic exploration. Consequently, Observed Failure Reviewer Credential Discovery (Cannot Connect) existing solutions remain structurally constrained by their Authentication Retryies reliance on continuous human intervention or rigid, predeUser: KioskUser0 Password: Null fined execution pipelines, severely impeding their autonomy and real-world generalizability. No New Evidence Connect via RDP Failed to MazeRunner strives for full autonomy in black-box Progress pentesting across complex, multi-stage scenarios. It delibCredential Discovery User shell erately eschews reliance on static strategies or predefined scripts [12] that inherently constrain generalizability and Figure 5 Redundant, hallucinated, inconsistent commands practical applicability. Instead, the framework leverages LLMs for adaptive orchestration, decomposing high-level penetration objectives into dynamic tasks, such as code generation for environmental discovery and targeted intelligence retrieval. By designating tasks and clues as core primitives and managing them through an independent cache, MazeRunner maintains a verifiable investigation state while mitigating task redundancy, contextual forgetting, and hallucinations. The cache comprises a Task Graph for task dependencies and execution states, and a Clue Graph for structured environmental evidence. Finally, the architecture employs three collaborative agents, as illustrated in Figure 6, to implement a “strategist-executor-reviewer” cycle, effectively balancing open-ended discovery with proactive error correction, thereby preventing the squandering of resources on unproductive trajectories. Specifically, the Strategist decomposes the user-provided objective and maintains the evolving task structure. The Executor receives the selected task, translates it into CLI operations, and records execution evidence and newly discovered clues in the shared cache. When an execution is unsuccessful or inconclusive, the Reviewer analyzes the resulting trace and provides structured guidance to the Strategist. The Strategist then revises the current plan, adds missing prerequisite tasks, or switches to an alternative branch. Deterministic state transitions, such as dependency resolution and downstream blocking, are handled programmatically. Together, these components form a closed-loop workflow covering task creation, execution, failure analysis, and replanning. (Create new task and adjust existing task)

·nmap -sV -p 21,22,80,88,135,139, 389,443, 445,1433 … ·nmap -sV -p 1-10000 –open … ·masscan -p1-65535 10.129.234.51 --rate=1000 …

Selected Task

Task Status

(Generate commands and interact via CLI)

Diagnosis & Guidance

Execution Trace

(Analyze failure trace and guide replanning)

·enum4linux -a …(timeout) ·smbmap -H … (timeout) ·rpcclient -U '' -N … (cannot connect) ·smbclient -L //… (timeout)

3.2

Task Creation (Strategist)

The MazeRunner workflow commences with a user-defined high-level objective, such as “obtaining user.txt from 10.0.2.5”. Upon receiving the objective, the Strategist interprets the goal and constructs an initial task set based on the currently available target information. Subsequent graph expansion driven by execution feedback and newly discovered clues is described in Section 3.6. To model relations among tasks, MazeRunner organizes the overall objective as a directed acyclic graph (DAG), where nodes represent trackable and schedulable task units, and directed edges encode prerequisite dependencies. 6

CLI Interface

Unknown Target Topology

(Create new task and adjust existing task)

Web Service 80/443

Schedule Task

Persistent Task & Clue Cache Read

Strategist

Target Host

SSH Service 22

Three-Agent Coordination

Write

Initially Task Clue-1 Service-1

Task Status

Task-?

Executor

Commands

(Generate commands and interact via CLI)

Results

Task-0

Clue-3 Service-3 Switch

Task-?

Diagnosis & Guidance

Clue-?

Revise

…

Task-?

Unrevealed Tasks

…

Prune

Task-? Unknown Clues

Execution Trace

Goal: Root Shell Figure 6

Reviewer

Read

(Analyze failure trace and guide replanning)

Write

MazeRunner’s architecture

Clues are stored separately in the Clue Graph and are associated with their producer and potential consumer tasks through explicit mappings.This structure allows the system to preserve multiple potential attack branches and, when one branch becomes invalid, backtrack to a valid upstream node and activate other dependency-satisfied candidates. Critically, MazeRunner maintains a clear distinction between planning granularity and execution-level actions. The Strategist Agent generates semantically complete tasks rather than atomic command-line inputs. Once a task is sufficiently concrete for the Executor Agent to operationalize, planning-level decomposition ceases, and the task is treated as an atomic planning unit. While a single planning unit may translate into multiple consecutive execution steps, the boundary of decomposition is governed by the requirement for high-level reasoning rather than the number of underlying commands. Furthermore, each task is initialized with comprehensive metadata, including task descriptions, prerequisites, relevant clues and expected outputs, to support robust dynamic scheduling and post-mortem analysis. 3.3

Task and Clue Orchestration

Upon task creation, MazeRunner manages the task lifecycle Task-1: Port & Service Attempt-1 S-1 S-2 through a dedicated Task & Clue Cache within the database. This Scan [Completed] cache serves as the foundational infrastructure for task manageAttempt-1 S-1 S-2 Task-2: Web Enum ment and cross-stage reasoning, recording the evolution of task [Completed] Attempt-2 S-1 S-2 states while accumulating critical clues gathered during execution. For each task, the cache maintains a unique identifier, its topology, Task-3: Exploit Reset Attempt-1 S-1 Pass [Completed] dependencies, current state, and associated execution records. As illustrated in Figure 7, task states are strictly categorized into six Task-4: SSH Login Attempt-1 S-1 S-2 types: pending, ready, running, completed, failed, and blocked. [Failed] Specifically, pending denotes a task awaiting its dependencies, (Awaiting Task-5: Get Shell Completed whereas ready indicates that all execution conditions have been pruning) [Blocked] satisfied. A newly generated task is first registered as pending. Failed Task Set Deterministic control logic immediately promotes it to ready when Blocked Task-6:Web Login all prerequisite dependencies have been satisfied. As shown in Attempt-1 [Ready] Ready Figure 7, “Task 6: Web Login” is ready because its dependency, Running Task-7: File Enum “Task 2: Web Enumeration,” has been completed. Conversely, [Pending] “Task 7: File Enumeration” remains pending until Task 6 is finPending ished. The running state denotes a task currently under execution. Figure 7 State transition diagram of penetration tasks To prevent concurrency anomalies caused by simultaneous context writes, MazeRunner employs a blocking mechanism to ensure that only one task is in the running state at any given time. The completed and failed states are updated based on the outcomes provided by the Executor Agent. To enhance the success rate, the agent may refine and retry failed tasks multiple times. If a task is ultimately deemed a failure, dependent tasks are marked as blocked, indicating they have become invalid and have been pruned from the active task graph. For instance, the failure of “Task 4: SSH Login” triggers the transition of “Task 5: Get Shell” to blocked, resulting in its subsequent pruning. While the Reviewer Agent does not manage tasks directly, it provides analytical feedback to the Strategist Agent, which subsequently creates new tasks or updates existing 7

states. Finally, to maintain state consistency and minimize LLM computational overhead, certain transitions, such as the progression from pending to ready, are handled programmatically rather than via model inference. Concurrently, the Executor Agent systematically ingests attack-critical telemetry as structured clues. These include open-port fingerprints, DOM structures, configuration files, credential fragments, privilege escalations, error logs, and historical failure data. Rather than naively concatenating this raw data into an LLM prompt, MazeRunner organizes these fragmented observations into structured evidence that can be explicitly referenced by subsequent tasks. Through this decoupled design, separating task states from evidentiary clues, MazeRunner can maintain a persistent shared memory across planning sessions, thereby ensuring the continuity of long-horizon reasoning. Consequently, the Task & Clue Cache functions as more than a historical record; it actively informs future decision-making, allowing MazeRunner to transcend the context window constraints inherent in single-turn LLM interactions. 3.4

Task Execution (Executor)

Once a task reaches the ready state, it could be dispatched to the Executor Agent. The Executor translates the structured task into specific actions by synthesizing its description, prerequisites, and cached clues to dynamically construct a sequence of execution steps. These steps are subsequently mapped to concrete command-line invocations or remote interactions. The Executor Agent interacts flexibly with the CLI and the underlying operating system, leveraging standard pentesting utilities, such as Nmap, Gobuster, and ffuf, without reliance on predefined Model Context Protocol (MCP) handlers or hardcoded skill sets. This design ensures robust coverage for diverse scenarios while enhancing system flexibility and extensibility. Furthermore, step-level execution traces and session states are recorded as metadata attached to their respective task nodes; For a single planning-level task, the Executor may perform multiple consecutive command-level attempts, including parameter adjustment, credential substitution, or interaction repair. These attempts remain internal to the task and are aggregated into a single execution trace. However, these granular logs are not promoted to planning-level nodes to maintain a clean abstraction between low-level execution and high-level strategy. To address the inherent complexity of non-linear pentesting, MazeRunner supports two distinct execution modes: 1) Non-interactive execution tailored for state-independent operations such as port scanning, directory enumeration, and page retrieval. 2) Interactive execution utilizes a pseudo-terminal (PTY) mechanism to support tasks requiring persistent session states. This includes command chaining following a successful shell exploit, context preservation after remote login, and multi-turn interactions necessitating real-time feedback. By leveraging the PTY mechanism, the Executor Agent can issue commands and process outputs within a single terminal session while preserving local context — including the current working directory, user identity, and environment variables. This prevents the fragmentation of context-dependent operations into isolated processes. When the task terminates, the Executor aggregates stdout, stderr, exit codes, network logs, session metadata, and newly observed clues. It then performs a preliminary completion assessment against the task’s expected output. The complete evidence package is written to the cache, while root-cause attribution and cross-task interpretation of unsuccessful or inconclusive outcomes are deferred to the Reviewer. 3.5

Result Analysis (Reviewer)

In automated pentesting, task failure does not necessarily imply that the current attack path is entirely invalid; it may also arise from improper tool usage, incorrect parameter settings, unsatisfied prerequisites, or a failure to correlate critical clues. Therefore, we introduce the Reviewer Agent, an independent component designed to analyze task failures and translate them into structured explanations usable for subsequent decision-making. During review, the Reviewer examines the failed task description, its expected output, the complete execution trace, and the contextual clues retrieved from the Task & Clue Cache. The analysis proceeds in two stages. First, it determines whether the failure is caused by: (1) an incorrect high-level attack path, (2) a localized execution issue involving tool usage, command construction, or interaction flow, or (3) an unsatisfied prerequisite. Second, independently of the failure category, it inspects the execution output for newly exposed evidence, such as usernames, endpoints, file paths, service metadata, or configuration fragments. Notably, while the Strategist Agent is designed to encourage broad exploration and divergent heuristic attempts, the Reviewer Agent helps converge the search space by identifying unviable tasks and recommending branch termination or switching when necessary. If the Reviewer determines that a failure is irrecoverable, the verdict is passed to the Strategist and the programmatic state-control logic. The original failed task and its execution evidence are retained for future reference, while downstream tasks that strictly depend on that node are marked as blocked and 8

removed from the active scheduling set. Completed upstream tasks remain unchanged, and their accumulated clues remain available to other branches. Three Typical Workflows 3.6

Task Re-Planning (Strategist)

Following the review phase, MazeRunner initiates the re(a) HTB-Principal-Replanning and Retry after Encryption Mismatch planning stage to generate subsequent tasks. At this point, Reviewer Target Executor Strategist Send forged token with default the system synthesizes the latest task states, dependency PoC claims Initial Plan Run the PoC with Err: Invalid Token structures, failure diagnoses, and newly discovered clues to Run CVE-2026- RSA-OAEP-256 + Token (iss=principal-platform) 29000 PoC A256GCM locally expand and adjust the active task graph, thereby Rejected by API determining the optimal next step. That is, the true auRetry Auth (Admin Token) Authentication failed tonomy of MazeRunner lies not merely in automated task Report failure Recommendation execution, but in its capacity to dynamically adapt attack Replan and retry Change enc from A256GCM to A128GCM trajectories based on real-time feedback. Retry the PoC with Updated Plan and retry the PoC A128GCM Execute the Success Candidate tasks for subsequent execution originate prirevised PoC marily from three sources: (1) local revisions and re(b) HTB-Kobold-Task Supplementation after Missing VHost Discovery executions of failed tasks with repairable causes; (2) entirely Executor Reviewer Target Strategist new tasks spawned by newly discovered clues; and (3) alterRun the original PoC against kobold.htb Initial Plan native branches accessed by backtracking to a valid upstream Use the CVE-2026Run the PoC on Exploit failed 23520 PoC against the discovered Modify PoC API path and node after an invalid branch is pruned. Throughout this Arcane Docker request format web target Management v1.13.0 process, the Strategist Agent dynamically schedules tasks Exploit failed Modify PoC payload parameters by evaluating dependency satisfaction, the heuristic value of and headers Exploit failed the current path, clue relevance, historical failure context, Report failure Recommendation and the execution state. Crucially, the recommendations Extend the plan A required vhost, such as an MCP-related provided by the Reviewer Agent are not strictly negative or Updated Plan endpoint, may still be Run vhost discovery Found Supplement the plan undiscovered mcp.kobold.htb punitive; rather, they frequently offer constructive insights with vhost discovery that enable the system to perform incremental adjustments (c) HTB-Interpreter-Branch Switching after Privilege-Escalation Failure Reviewer Target Strategist Executor around localized failures or newly revealed clues. For instance, following an unsuccessful web exploitation attempt, Check sudo privilege “sudo –l” Initial Plan Execute Branch 1 Branch 1: PrivEsc User cannot run sudo the system does not prematurely abandon the target. InBranch 2: Crack Hash from user shell Search SUID binaries stead, it generates supplementary tasks — such as inspectNo exploitable binary ing backup files, testing default credentials, or enumerating Check writable privileged paths No useful writable privileged path API endpoints. Similarly, when the Reviewer Agent extracts Report failure Switch branch novel usernames, file paths, or configuration details from erRecommendation The stored <password Run hash-cracking Updated Plan ror logs, this new evidence can trigger subsequent authenhash> is still unused; branch Recall stored Success switch to the hashpassword hash cracking branch tication attempts, local file reading, or privilege escalation sequences. Figure 8 Running examples Ultimately, MazeRunner forms a closed loop in which the Strategist maintains and revises the Task Graph, the Executor interacts with the target and records evidence, the Reviewer diagnoses unsuccessful executions, and the persistent cache preserves tasks and clues across iterations. This loop enables the system to revise local actions, supplement missing prerequisites, and switch branches without discarding valid upstream progress. 3.7

Running Examples

Next, we present three representative running examples, curated from approximately tens of thousands of lines of interaction logs across all target machines, to intuitively illustrate our architectural design choices. These specificscenarios manifest frequently throughout the execution traces. Because pentesting is fundamentally a black-box process, iterative trial and error is inevitable. For a well-defined task, such as “Run the PoC with RSA-OAEP-256 + A256GCM ”, the Executor Agent naturally translates the objective into concrete scripts, systematically iterating over various parameters or credentials. However, LLMs are highly susceptible to a depth-first trap. Intuitively, as an agent accumulates execution context from repeated failures within a single task, its attention becomes disproportionately anchored to that specific local branch. This hyper-focus inevitably leads to goal drift and the generation of hallucinated, redundant tasks, a critical vulnerability frequently observed in simple “Planner-Executor” baselines such as PentestGPT-V2 and Claude Code. To mitigate this limitation, the Reviewer Agent intervenes at critical junctures to analyze the root causes of failures and furnish actionable recommendations. As illustrated in Figure 8, these interventions primarily occur 9

in three ways: 1) Local Task Adjustment (Figure 8 (a)): Refining the parameters of the current task, such as correcting the chosen encryption algorithm. 2) Prerequisite Identification (Figure 8 (b)): Diagnosing missing execution conditions and directing the Strategist Agent to formulate the necessary prerequisite tasks. 3) Task Switching (Figure 8 (c)): If local adjustments repeatedly fail, the system leverages the shared Task & Clue Cache to abandon the dead-end branch and proactively pivot to an alternative attack path. Through this synergistic division of labor, the three agents effectively collaborate to systematically categorize and resolve execution anomalies, thereby accomplishing penetration tasks with significantly enhanced speed and overall efficacy.

4

Implementation

The implementation and runtime environment of MazeRunner is architected as a cohesive ecosystem comprising a collaborative multi-agent framework, a specialized execution backend, and a persistent storage layer. At its core, the system utilizes Kimi Code CLI6) to orchestrate a Strategist-Executor-Reviewer architecture, which facilitates a natural-language interface analogous to conventional CLI agents for defining attack objectives and environmental constraints. While this architecture supports human-in-the-loop intervention for future research, all evaluations in this study were initiated through a single initial task specification to rigorously assess the system’s operational autonomy. Complementing this reasoning layer, all low-level operations are routed to a Kali Linux-based execution backend to leverage industry-standard utilities such as Nmap, Gobuster, ffuf, curl, wget, ssh, ftp, and smbclient. Within this backend, the Executor Agent employs two distinct shell invocation modes, including a non-interactive mode for rapid scanning and enumeration as well as a PTY-based interactive mode specifically designed for tasks requiring persistent sessions, such as handling authentication prompts or stabilizing reverse shells. To mitigate the blocking behavior common in long-running security tasks, the framework incorporates a dataflow-driven control mechanism that monitors process idle time rather than relying on rigid, fixed timeouts. Furthermore, to mitigate hallucinations and ensure long-horizon persistence beyond the inherent constraints of the LLM context window, MazeRunner externalizes the global investigation state into a MySQL-backed persistent storage layer. This storage is partitioned into two specialized, interdependent structures: a Task Graph that maintains the global attack topology, inter-task dependencies, and execution states, and a Clue Graph that serves as a repository for distilled intelligence, such as discovered credentials and vulnerability fragments. Crucially, the Clue Graph maintains explicit mappings between producer tasks, those that generate specific clues, and potential consumer tasks that leverage or derive from those findings, thereby enabling the efficient cross-stage reasoning necessary for complex, non-linear pentesting scenarios.

5

Evaluation

In this section, we focus on evaluating the performance of MazeRunner by answering the following research questions: • RQ1: How effective is MazeRunner at autonomously completing end-to-end penetration tasks? (§5.2) • RQ2: What are the primary root causes and operational bottlenecks that lead to task failures? (§5.3) • RQ3: What is the computational and temporal overhead of MazeRunner? (§5.4) • RQ4: How do the individual architectural components contribute to the overall performance? (§5.5) 5.1

Evaluation Setup

All experiments were conducted on three dedicated servers running Kali Linux to ensure the seamless invocation of pre-installed pentesting utilities. Regarding LLM access, all models were queried via standard pay-as-you-go (token-based) APIs, with the sole exception of the Claude Code framework powered by Sonnet 4.5, which utilized a flat-rate monthly subscription. 5.1.1

Baselines, Targets and Foundational Model Selection

The selection of evaluation targets was governed by three primary criteria. First, the dataset must support comprehensive scenario coverage, encompassing diverse operating systems, varying complexity levels, and prevalent 6) Kimi Code, https://www.kimi.com/code/docs/en/

10

Common Weakness Enumerations (CWEs). Second, the targets must necessitate non-linear, multi-step exploitation trajectories to adequately benchmark the systems’ advanced reasoning and orchestration capabilities within complex environments. Third, to rigorously maintain evaluation integrity, the publication dates of all selected targets must strictly postdate the training data cutoffs of the evaluated LLMs. Consequently, we curated a dataset comprising all 10 HTB machines from Season 10 (4 Easy, 4 Medium, and 2 Hard targets) available as of our data collection cutoff in April 2026. These machines were officially released between February 2026 and April 2026. This timeline fully satisfies the knowledge cutoff constraints of all selected LLMs, effectively eliminating the risk of evaluation bias stemming from data contamination, as previously demonstrated in §2. Regarding baseline systems, we selected PentestGPT-V2, an open-source, specialized framework capable of autonomously executing the end-to-end pentesting lifecycle, as illustrated in Table 1. Furthermore, we incorporated Claude Code, a general-purpose coding agent that has recently demonstrated remarkable efficacy in automated offensive security tasks. Due to compatibility constraints inherent to Claude Code, the aforementioned baseline systems were evaluated using two foundational models: Claude Sonnet 4.5 and DeepSeek-V3.2. In contrast, our proposed MazeRunner framework underwent extended evaluation utilizing GPT-5.2 and GPT-5.2-Codex, in addition to the models tested with the baselines. 5.1.2

Experimental Protocol

pentesting is inherently a complex, open-ended challenge. In real-world engagements, the precise locations of vulnerabilities and exploitable clues, alongside the requisite steps to construct a viable attack path, remain completely unknown prior to execution. While the selected HTB targets possess canonical solution paths, this does not negate the value of exploring alternative, non-standard attack vectors. On the contrary, if an automated system immediately isolates the correct execution branch and achieves the objective in a single, flawless attempt, it is highly indicative of data contamination (i.e., knowledge leakage from training corpora) rather than genuine reasoning and exploration capabilities. Furthermore, due to the inherent stochasticity of LLM inference, the generated task plans for a given scenario will inevitably vary across different iterations, exhibiting natural fluctuations in both execution sequencing and task granularity. Consequently, achieving a quantified, strictly objective, and perfectly precise evaluation of pentesting efficacy remains highly challenging, rendering manual review indispensable for data annotation and result aggregation. Prior works typically evaluate performance based on the binary success rate of obtaining a Flag or Shell [9], an approach that obscures critical nuances. Alternatively, some studies rely on the “number of completed sub-tasks” as a metric [12]; however, this overlooks the system’s exploration of incorrect yet meaningful attack branches. In this paper, we propose quantifying the absolute number of explored task branches, such as the enumeration of distinct ports or the execution of diverse exploit vectors targeting a single service, to measure the system’s capacity for lateral exploration. Furthermore, existing literature severely lacks in-depth failure analysis. Our empirical results demonstrate that in open-ended environments devoid of knowledge leakage, automated pentesting exhibits substantial stochasticity and a notable probability of failure. The root causes of these failures often stem from specific limitations in LLM capabilities (e.g., information retrieval errors or flawed payload generation) or from highly repetitive tasks (e.g., brute-force password cracking) that are fundamentally ill-suited for LLMs. A rigorous analysis of these failure modes is crucial for understanding current bottlenecks and guiding future research. Automated penetration testing frameworks may continue generating new tasks even after useful progress has stalled, making natural termination unsuitable for controlled comparison. Wall-clock limits are also unstable because scanning, connection attempts, and network timeouts introduce substantial environmental variance. We therefore use cumulative LLM token consumption as the primary resource constraint and impose a maximum budget of 20 million tokens on each system–target run. Token consumption includes the input and output tokens reported by the corresponding model provider. Because tokenization differs across model families, direct effectiveness comparisons are made primarily between frameworks using the same backbone model. The 20-million-token threshold provides sufficient budget for sustained exploration while preventing a small number of repetitive runs from dominating the evaluation cost. If a framework terminates naturally before reaching the budget, we retain its last verified penetration state. If a run exceeds the budget, only actions completed within the first 20 million tokens are included in the effectiveness results. Section 5.4 separately analyzes the complete execution traces to characterize the practical resource consumption of sustained execution. To reduce target-specific solution leakage, all agents are prohibited from accessing machine-specific walkthroughs, write-ups, flag disclosures, or search results that explicitly identify the target’s canonical attack path. Access to general vulnerability databases, vendor advisories, exploit repositories, and documentation is permitted because such resources are available in realistic penetration testing. We manually inspect the browsing and command traces for 11

Table 2 Model

End-to-end progress and shell acquisition under a common maximum budget of 20M LLM tokens per target run Target-wise Progress ⩽20M

Framework

WingSilenCCTV Kobold Data tium MazeRunner 7/13U 5/11 12/12R 9/14U Claude Code 7/13U 4/11 5/12 7/14 PentestGPT-V2 7/13U 4/11 5/12 7/14

Ptero- Inter- Princidactyl preter pal 8/14U 8/14U 13/13R 5/14 8/14U 7/13

7/13U 3/11 Claude Code 7/13U 4/11 PentestGPT-V2 7/13U 3/11

9/12U 7/12U 5/12

8/14

MazeRunner

7/13U 5/11

7/12U

GPT-5.2-Codex MazeRunner

7/13U 5/11

7/12U

Sonnet 4.5

MazeRunner

DeepSeek-V3.2

GPT-5.2

Overall Results ⩽20M DevGar- Completion User Root Pirate Area field Rate Shell Shell 4/15 2/21

3/22

47.65%

6

4/15 2/21

2/22

34.23%

2

0

5/14

12/13U 4/15 2/21

3/22

36.24%

2

0

8/14

9/14U 3/14

13/13R 4/15 2/21

1/22

39.60%

4

1

8/14

5/14 4/14 9/14U 3/14

7/13

4/15 2/21

1/22

32.89%

2

0

13/13R 4/15 2/21

1/22

36.91%

3

1

10/14U 5/14

8/14U 13/13R 9/15 4/21

4/22

48.32%

5

1

9/14U

8/14U 13/13R 8/15 4/21

4/22

46.98%

5

1

5/14

5/14

2

target-specific leakage. If a framework accesses a prohibited write-up, all actions after that access are discarded, and the last independently verified penetration state before the access is retained for the formal effectiveness analysis. We therefore describe the setting as target-write-up-free rather than zero-knowledge, because the agents retain access to general vulnerability knowledge and public exploit resources. 5.2

End-to-End Effectiveness

First, we evaluate the end-to-end task completion capabilities of various system-LLM combinations (combination for short) based on the volume of resolved sub-tasks and the successful acquisition of user and root shells. As summarized in Table 2, we report the completed and total number of sub-tasks for each target, with ground-truth task counts manually established by synthesizing target specifications with the task-partitioning logic of various automated frameworks (refer to Appendix Table A3 for a representative example). Superscripts U and R denote the acquisition of user and root shells, respectively. The results demonstrate that MazeRunner achieves higher capped completion rates with both shared backbone models. Specifically, when utilizing the Claude Sonnet 4.5 model, MazeRunner successfully secures two additional root shells and four additional user shells compared with both Claude Code and PentestGPT-V2. We observe comparable performance advantages when evaluating the systems with the DeepSeek-V3.2 model. It is critical to note that Claude Code and its derivative, PentestGPT-V2, employ a dynamic sub-agent mechanism. These sub-agents occasionally bypassed our experimental constraints regarding the use of online write-ups; we identified clear evidence of knowledge leakage in at least two instances. In contrast, MazeRunner maintains strict adherence to zero-leakage constraints. When these contaminated cases are excluded, the performance advantage of our proposed framework becomes P even more pronounced. To evaluate lateral exploration, we define “Exploration Breadth” as v∈V max(outdeg(v) − 1, 0) across the pentesting decision graph. This metric explicitly quantifies a system’s capacity for multi-branch decision-making: rigid, linear execution (e.g., A → B → C) scores 0, whereas spawning parallel attack vectors from a single pivot point (e.g., simultaneous directory fuzzing and SSH brute-forcing) positively increases the score. As shown in Figure 9, MazeRunner outperforms both baselines, achieving the highest median breadth and demonstrating highly adaptive scaling on complex targets. In contrast, Claude Code’s consistently low scores quantitatively reflect its tendency for premature abandonment, while PentestGPT-V2’s narrow interquartile range indicates a reliance on redundant, single-threaded execution loops rather than strategic lateral expansion. Crucially, this robust multibranch exploration directly facilitates greater penetration depth. The aforementioned results were observed within the relatively constrained bounds of experimental target machines; this capacity for extensive lateral exploration becomes indispensable when navigating the complexities of real-world enterprise environments. 5.3

Failure Reason Analysis

We manually analyze the 56 target-level runs that do not obtain root access across the three frameworks, two shared backbone models, and 10 targets. Each run is assigned one or more failure categories: Planning Error, Clue Forgetting, Premature Abandonment, Code/Payload Failure, and Information Retrieval Failure. Because failure causes may compound within a run, the categories are treated as non-exclusive. Definitions and representative cases are provided in Appendix B. As shown in Table 3, MazeRunner exhibits lower observed rates of Planning Error and Clue Forgetting than both baseline frameworks. At the same time, Code/Payload Failures account for a larger proportion of its unsuccessful runs. One possible explanation is that improved task orchestration allows more MazeRunner runs to progress beyond reconnaissance and planning into execution-intensive stages, where 12

Exploration Breadth

30

Table 3

Distribution of failure causes across frameworks and backbone LLMs Unsuccessful Planning Clue Execution Retrieval Abandonment Traces Error Forgetting Failure Failure

20 10 MazeRunner ClaudeCode PentestGPT-V2 Figure 9

Panel A: By Framework MazeRunner 17 Claude Code 20 PentestGPT-V2 19

29.4% 55.0% 68.4%

5.9% 25.0% 21.1%

23.5% 30.0% 10.5%

52.9% 25.0% 36.8%

5.9% 5.0% 0.0%

Panel B: By Backbone LLM Sonnet 4.5 28 DeepSeek-V3.2 28

50.0% 53.6%

21.4% 14.3%

32.1% 10.7%

25.0% 50.0%

0.0% 7.1%

Exploration breadth by system

exploit compatibility and payload correctness become the dominant bottlenecks. This interpretation is based on the observed traces and should not be treated as a controlled causal result. Comparing the foundational models reveals that Claude Sonnet 4.5 exhibits lower observed rates of code/payload execution and information retrieval failures than DeepSeek-V3.2. This complements MazeRunner’s orchestration capability: while the framework mitigates early-stage planning and state-management bottlenecks, Sonnet provides stronger support for the execution-intensive stages that follow. Conversely, baseline systems frequently stall in the early exploration phases, rendering Sonnet’s coding prowess underutilized. Finally, we highlight a notable anomaly regarding GPT-5.2-Codex, a model explicitly fine-tuned for coding tasks. Surprisingly, it underperformed compared to the standard GPT-5.2 model. We hypothesize that its built-in task-orchestration behavior interferes with MazeRunner’s external orchestration, introducing additional token overhead without commensurate performance gains. Furthermore, among the remaining instances of planning failure within our system, the inherent stochasticity of LLM orchestration plays a substantial role. Because LLMs are fundamentally probabilistic, this randomness is unavoidable. In our specific pentesting context, it primarily manifests as a statistical bias during task prioritization: when multiple viable attack vectors exist, the model gravitates toward the most common solutions. For simple tasks or scenarios where the correct exploit aligns with historically frequent vulnerabilities, this bias has minimal impact, as the system can reliably iterate through feasible options utilizing the cached task list. However, in complex scenarios, particularly when the correct attack vector is obscure or statistically rare, this prioritization bias severely degrades problem-solving efficiency. For instance, during the evaluation of the “HTB-Garfield” target, MazeRunner failed to access the critical “ACL/scriptPath” chain. Instead, it disproportionately expended resources executing prevalent, zero-prerequisite attacks, such as Zerologon, administrator credential stuffing, PetitPotam, Kerberoasting, and RDP brute-forcing. Consequently, it entirely overlooked the correct, yet unconventional, strategy of exploiting a Kerberos initial account backdoor. Fundamentally, this dilemma represents a classic trade-off between penetration capabilities and computational overhead. While exhaustively enumerating all obscure attack paths might eventually yield success, it requires a prohibitive expenditure of resources. Moreover, such exhaustive probing would inevitably trigger context window saturation, transforming the LLM’s finite memory into a new operational bottleneck. Finally, other observed execution failures stem from the system attempting highly repetitive, token-intensive tasks, such as autonomously generating scripts for password brute-forcing. To optimize both efficiency and token economy, future iterations of MazeRunner should explicitly delegate these deterministic, brute-force operations to specialized external tools rather than relying on LLM-driven code generation. 5.4

Overhead

While the effectiveness analysis in Section 5.2 is restricted to a common 20M-token budget for fair comparison, the overhead analysis uses the complete execution traces. These two settings provide complementary perspectives: the former compares framework effectiveness under the same resource budget, while the latter characterizes practical resource consumption and efficiency during sustained pentesting. As illustrated in Figure 10, we measure shellacquisition efficiency as the number of successful shell acquisitions per 100M tokens, per $10, and per 1000 minutes of execution time. MazeRunner achieves the highest token and cost efficiency under both Claude Sonnet 4.5 and DeepSeek-V3.2, while its time efficiency varies across models. The differences in financial overhead are also affected by substantial disparities in model pricing; the unit cost of Claude Sonnet 4.5 is over tenfold that of DeepSeekV3.2. Consequently, this leads to a substantial gap in final financial overhead. Overall, DeepSeek-V3.2 achieves competitive penetration performance while maintaining high cost-efficiency. Comparing MazeRunner against the baseline frameworks further demonstrates its operational efficiency. Claude Code exhibits a tendency for premature task termination; while this results in lower overall overhead, it severely degrades its sub-task completion rate. Conversely, PentestGPT-V2, much like our system, engages in continuous 13

Sonnet 4.5 Cost Efficiency

Sonnet 4.5 Token Efficiency

2

1.58

Shells / 1000 min

3.19

3

1.24 1

0.73 0.4

1

6 5

4.35

4

3.16

3 2 1

0

DeepSeek-V3.2 Token Efficiency 5

0 MazeRunner PentestGPT-V2 Claude Code

MazeRunner PentestGPT-V2 Claude Code

9

8.83

MazeRunner PentestGPT-V2 Claude Code

DeepSeek-V3.2 Cost Efficiency

DeepSeek-V3.2 Time Efficiency 7

7

Shells / $10

3.39

6.01

6

3 2.06

5.05

5

2.32

Shells / 1000 min

8 4

6.8

7

4

0

Shells / 100M Tokens

Sonnet 4.5 Time Efficiency

2

Shells / $10

Shells / 100M Tokens

5

4.85

4

2

3

2

1 0

MazeRunner PentestGPT-V2 Claude Code

Figure 10

6 5

4.15

4 2.92

3 2

1

1

0

0

MazeRunner PentestGPT-V2 Claude Code

5.49

MazeRunner PentestGPT-V2 Claude Code

Token, cost, and time efficiency over complete execution traces

exploration. However, owing to its propensity for generating redundant operations and hallucinatory task loops, PentestGPT-V2 consumes approximately twice the tokens and financial budget before its progress stagnates, yet ultimately achieves less penetration depth than MazeRunner (detailed command-level execution statistics are provided in Appendix A.3). Finally, temporal expenditure across all combinations exhibits high variance and stochasticity. This randomness is primarily because many pentesting procedures (e.g., port scanning and connection attempts) involve inherent network timeouts and wait periods, which account for a substantial portion of the total execution time. Given the varying depths and breadths of exploration across different tasks, pure execution time provides limited comparative benchmarking value in this context and should therefore be interpreted cautiously. Furthermore, we analyzed the token consumption distribution across the system’s three primary agents. As depicted in Figure A1, the Executor Agent accounts for the overwhelming majority, ranging from 75% to 95%, of the total token expenditure. The bulk of computational resources is dedicated to low-level payload and script generation, alongside the subsequent parsing of environmental feedback. Executing irrelevant or hallucinatory tasks wastes substantial token budgets without yielding any strategic benefit. Consequently, investing a relatively minor fraction of the token budget in the Strategist and Reviewer agents for high-level planning and proactive error correction helps prevent costly execution loops and reduce unnecessary computational expenditure, ultimately driving down the overall financial and computational overhead of the pentesting lifecycle. 5.5

Effectiveness of Individual Components 80

Full system

Subtask completion rate (%)

We conducted an ablation study on a subset of 5 targets, where 73.5%, 44.0M 75 the framework successfully secured at least a user shell utilizing 70 GPT-5.2, to empirically evaluate the individual contributions of −22.0 pp completion MazeRunner’s core modules. Our proposed framework relies 65 +80.7% tokens −14.7 pp completion on three primary agents, the Strategist, Executor, and Reviewer, +27.7% tokens 60 augmented by an independent Task & Clue Cache. Because the w/o Task & Clue Cache 55 Strategist and Executor are indispensable for executing the fun58.8%, 56.2M damental pentesting loop, our ablation experiments specifically 50 w/o Reviewer isolate and remove the Reviewer Agent and the independent 51.5%, 79.5M 45 cache to quantify their respective impacts. 35 45 55 65 75 85 Removing the independent cache drops the completion rate Total tokens (M) Figure 11 Ablation study results (defined as the ratio of successfully resolved sub-tasks to total sub-tasks across all 5 target machines) to 58.8% and inflates overhead by roughly 27%. This illustrates“Contextual Amnesia”: without persistent memory, agents overwrite critical historical intelligence as their context windows fill, resulting in the redundant re-exploration of previously scanned environments. Ablating the Reviewer causes the most severe performance collapse. The completion rate 14

plunges from 73.5% to 51.5%, while token consumption surges by over 80% (to 79.5M). This directly corroborates the “depth-first trap” phenomenon: without proactive error correction, the system cannot recover from failures, forcing the Executor to blindly waste resources on dead-end paths and hallucinated tasks. All in all, this study confirms that while the Strategist and Executor drive the attack’s momentum, the Reviewer and Cache are indispensable for maintaining trajectory, recovering from errors, and ensuring cost-efficiency.

6

Related Work

Planning is a fundamental prerequisite for automated pentesting, requiring agents to continuously evaluate attack vectors and recalibrate paths under severe uncertainty [21]. Existing approaches to attack orchestration [19] broadly bifurcate into two primary paradigms: traditional deterministic planning and LLM-driven planning. The deterministic paradigm encompasses classical planning [22], attack graphs [23], POMDPs [24], and reinforcement learning [25]. While these methods excel in logical consistency by explicitly modeling states and transitions, their reliance on rigidly structured representations and idealized environments renders them inherently brittle against real-world, black-box targets. Conversely, LLM-driven planning offers a highly adaptable alternative. Instead of requiring pre-computed states, LLMs dynamically decompose high-level objectives into executable sub-tasks via natural language reasoning and intermediate feedback. Recent scholarship has extensively explored varied reasoning structures, including oneshot [26], step-wise [27], hierarchical [28], tree-structured [7,29], and reflective planning [30], providing a compelling foundation for autonomous security decision-making. Despite these advancements, current LLM planners exhibit critical deficiencies in live pentesting. Unlike generalized domains, real-world targets feature profound black-box uncertainty, deceptive “rabbit holes,” and fragmented cross-stage clues. Consequently, effective planning necessitates robust long-term state persistence, the capacity to synthesize chronologically distant evidence, and the agility to backtrack from errors. While both paradigms offer foundational insights, they remain fundamentally inadequate for addressing the core challenge resolved in this work: resilient, long-horizon attack orchestration and error-aware backtracking within highly uncertain environments.

7

Discussion & Conclusion

7.1

Limitations & Future Work

Despite the advanced orchestration and memory capabilities introduced by MazeRunner, there are certain pentesting scenarios that cannot achieve full automation. First, a major challenge arises from limitations in multimodal information processing; specifically, the current system is unequipped to handle tasks requiring Graphical User Interface (GUI) interactions. Future work could integrate multimodal perception and GUI interaction capabilities to support browser- and desktop-based security operations. Second, defining strict security boundaries remains a practical hurdle. To avoid disrupting normal business operations during live testing, potentially destructive or intrusive actions still necessitate human expert review. Third, because this research focuses primarily on task orchestration rather than vulnerability discovery, the system struggles to progress in the absence of known vulnerability disclosures or readily available exploit code. Furthermore, because LLMs are inherently probabilistic models, they exhibit a statistical bias toward common attack methodologies when presented with multiple potential exploitation paths. Consequently, in scenarios that necessitate rare or unconventional attack vectors, the system may expend significant iterations before attempting the correct method, or worse, prematurely abandon the active clue to pursue alternative branches. Fundamentally, this issue reduces to the classic search-space dilemma of balancing penetration depth against exploration breadth, a challenge that lacks a definitive algorithmic resolution and frequently confounds even expert human analysts. Future work should improve failure diagnosis, recovery, and long-horizon context management. Although the Reviewer corrects many execution failures, ambiguous tool outputs, network timeouts, and repeated exploit failures may still trigger incorrect task transitions. More structured error representations, adaptive retry policies, and rollback mechanisms could improve robustness. Historical trajectory retrieval and experience-based adaptation may further help the system prioritize less common but viable attack paths. In addition, context compression, clue deduplication, and relevance-aware memory retrieval could reduce prompt noise and preserve salient evidence. Visualizing the evolving task graph and clue dependencies may also improve interpretability and facilitate human supervision. 15

Recent agentic model Kimi K3 with Kimi Code reveals an important shift in penetration success rate by identifing exploitable behaviors or attack paths that are not included in canonical solution paths. These observations suggest that sufficiently capable models with long-horizon reasoning, code generation, and tool-use abilities can sometimes compensate for the absence of elaborate external task planning by continuously interpreting environmental feedback and revising their actions. Nevertheless, such capability does not eliminate the need for explicit orchestration. Its success often depends on large and highly variable token and command budgets, while unconstrained exploration may reduce cost predictability, execution reproducibility, state auditability, and operational safety. MazeRunner is therefore complementary to frontier agents rather than a replacement for them. Its Task & Clue Cache and independent review mechanism provide a model-agnostic control layer for persistent state management, failure attribution, branch switching, and resource governance. Kimi K3 and Kimi Code can, in turn, serve as a stronger reasoning and execution backbone within MazeRunner, combining their ability to discover non-predefined attack paths with explicit state consistency, controllable execution, and traceable decision making. 7.2

Conclusion

This paper addresses a central challenge of black-box automated pentesting: the attack graph must be discovered online, multiple plausible branches compete for a limited execution budget, failures require attribution rather than blind retry, and decisive clues must persist across long action horizons. MazeRunner meets these requirements through an incremental Strategist-Executor-Reviewer loop and a persistent Task & Clue Cache, enabling dynamic graph expansion, error-aware correction, branch backtracking, and cross-stage evidence reuse. Across 10 recently released HTB targets, MazeRunner achieves higher shell acquisition rates, over 26% greater aggregate exploration breadth than the baselines, and substantially lower computational overhead. These results establish MazeRunner as a robust and cost-effective foundation for automated security auditing in open-ended environments. Acknowledgements This work was supported by the National Natural Science Foundation of China (Grant No. 62402419), the “Pioneer and Leading Goose” R&D Program of Zhejiang (Grant No. 2025C02263), the Natural Science Foundation of Ningbo (Grant No. 2025J027), the Ningbo Yongjiang Talent Programme, the CCF-Tencent Rhino-Bird Open Research Fund, and the National Key Research & Development Project of China (Grant No. 2023YFB3106800). References 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24

Fortune Business Insights. Penetration testing market size, share & growth report [2034], 2025. Accessed: 2026-03-16 Shen X, Wang L, Li Z, et al. Pentestagent: Incorporating llm agents to automated penetration testing. In: Proceedings of Proceedings of the 20th ACM Asia Conference on Computer and Communications Security, 2025. 375–391 Happe A, Cito J. Can llms hack enterprise networks? autonomous assumed breach penetration-testing active directory networks. ACM Trans. Softw. Eng. Methodol., 2025 Chen A, Guo Z, Jin Z, et al. Breaking the bulkhead: Demystifying cross-namespace reference vulnerabilities in kubernetes operators. arXiv preprint arXiv:2507.03387, 2025 Wu B, Chen G, Chen K, et al. Autopt: How far are we from the fully automated web penetration testing? IEEE Transactions on Information Forensics and Security, 2025 Mayoral-Vilches V, Navarrete-Lozano L J, Sanz-Gómez M, et al. Cai: An open, bug bounty-ready cybersecurity ai. arXiv preprint arXiv:2504.06017, 2025 Deng G, Liu Y, Mayoral-Vilches V, et al. Pentestgpt: Evaluating and harnessing large language models for automated penetration testing. In: Proceedings of 33rd USENIX Security Symposium (USENIX Security 24), 2024. 847–864 Mai W, Hong G, Liu Q, et al. Shell or nothing: Real-world benchmarks and memory-activated agents for automated penetration testing. arXiv preprint arXiv:2509.09207, 2025 Singer B, Lucas K, Adiga L, et al. Incalmo: An autonomous llm-assisted system for red teaming multi-host networks. arXiv preprint arXiv:2501.16466, 2025 Zhai J, Zhou X, Miao H, et al. Pentestmcp: Llm and mcp based multi-agent framework for automated penetration testing. 2025 Para R, Dharini N, Prasanth C, et al. Trinity: A neuro-symbolic multi-agent framework for stateful network penetration testing. In: Proceedings of 2026 6th International Conference on Trends in Material Science and Inventive Materials (ICTMIM). IEEE, 2026. 1875–1881 Deng G, Liu Y, Li Y, et al. What makes a good llm agent for real-world penetration testing? arXiv preprint arXiv:2602.17622, 2026 Metasploit. https://www.metasploit.com/, 2026. Accessed: 2026-04-04 Jing P, Cai Z, Cao Y, et al. Revisiting automotive attack surfaces: A practitioners’ perspective. In: Proceedings of 2024 IEEE Symposium on Security and Privacy (SP). IEEE, 2024. 2348–2365 Searchsploit. https://hackviser.com/tactics/tools/searchsploit, 2026. Accessed: 2026-04-04 2025 isc2 cybersecurity workforce study. https://www.isc2.org/insights/2025/12/2025-ISC2-Cybersecurity-Workforce-Study, 2025. Accessed: 2026-04-04 Uptime Institute. Uptime announces annual outage analysis report 2025. https://uptimeinstitute.com/about-ui/press-releases/ uptime-announces-annual-outage-analysis-report-2025, May 2025. Accessed: 2026-04-04 Simon R, Mees W. Sok: a comparison of autonomous penetration testing agents. In: Proceedings of Proceedings of the 19th International Conference on Availability, Reliability and Security, 2024. 1–10 Skandylas C, Asplund M. Automated penetration testing: Formalization and realization. Computers & Security, 2025, 155: 104454 Wang Y, Liu S, Wang W, et al. A unified modeling framework for automated penetration testing. Computers & Security, 2025, page 104787 Xi Z, Chen W, Guo X, et al. The rise and potential of large language model based agents: A survey. Science China Information Sciences, 2025, 68: 121101 De Pasquale G, Grishchenko I, Iesari R, et al. Chainreactor: Automated privilege escalation chain discovery via ai planning. In: Proceedings of 33rd USENIX Security Symposium (USENIX Security 24), 2024. 5913–5929 Obes J L, Sarraute C, Richarte G. Attack planning in the real world. arXiv preprint arXiv:1306.4044, 2013 Schwartz J, Kurniawati H, El-Mahassni E. Pomdp+ information-decay: Incorporating defender’s behaviour in autonomous penetration testing. In: Proceedings of Proceedings of the International Conference on Automated Planning and Scheduling, 2020. 235–243

16

25 26 27 28 29 30

Chen J, Hu S, Zheng H, et al. Gail-pt: An intelligent penetration testing framework with generative adversarial imitation learning. Computers & Security, 2023, 126: 103055 Zhou D, Schärli N, Hou L, et al. Least-to-most prompting enables complex reasoning in large language models. arXiv preprint arXiv:2205.10625, 2022 Wei J, Wang X, Schuurmans D, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 2022, 35: 24824–24837 Lin B Y, Fu Y, Yang K, et al. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. Advances in Neural Information Processing Systems, 2023, 36: 23813–23825 Yao S, Yu D, Zhao J, et al. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 2023, 36: 11809–11822 Madaan A, Tandon N, Gupta P, et al. Self-refine: Iterative refinement with self-feedback, 2023. URL https://arxiv. org/abs/2303.17651, 2023, 2303

17

Appendix A Appendix A.1

More Evaluation Setup Information and Results Targets and Ground Truth

Table A1 details the 10 HTB targets utilized for evaluation in this paper. To mitigate the risk of data contamination (i.e., knowledge leakage from training corpora), we exclusively selected recently released machines. Furthermore, Table A3 delineates the sub-tasks and corresponding environmental clues along the primary attack chain, demonstrating that task progression is clue-driven.

Appendix A.2

Cost of MazeRunner’s Agents

Figure A1 illustrates the computational cost distribution among MazeRunner’s three core agents across various foundational LLMs.

Appendix A.3

Command Execution Efficiency and Validity

To rigorously investigate the operational behaviors of the evaluated systems, we first systematically define the taxonomy of an “invalid command.” Specifically, a command is classified as invalid if it exhibits any of the following characteristics: 1) redundant repetition (e.g., cyclical scanning or homogeneous parameter brute-forcing); 2) logical contradiction (e.g., persistently invoking sudo after confirmed privilege denial); 3) persistent exploit failure (e.g., continuing to mutate and execute payloads after three consecutive failures); 4) contextual amnesia (e.g., re-initiating reconnaissance on previously discovered assets); 5) intuition deficits (e.g., attempting blind file writes without verifying permissions, or scanning out-of-scope subnets); 6) explicit tool execution errors; 7) severe trajectory deviation from the ground-truth attack chain; or 8) execution hallucinations (e.g., proceeding under the false assumption of task completion without affirmative environmental feedback). Based on this taxonomy, Table A2 details the ratio of invalid commands to total executed operations across the 10 target machines (where orange and red highlights denote the successful acquisition of User and Root shells, respectively). This fine-grained, commandlevel data empirically corroborates our prior observations regarding systemic execution bottlenecks. Notably, PentestGPT-V2 exhibits severe execution redundancy. Across complex targets such as Kobold and Pirate, it generates an overwhelming volume of commands (frequently exceeding 400 operations) coupled with a disproportionately high invalid command ratio (e.g., 281/390, or 72% invalid, on Pirate using Sonnet). This empirically confirms its susceptibility to the “depth-first trap,” wherein the agent blindly iterates over hallucinated or flawed payloads without strategic course correction. Conversely, while Claude Code generally issues far fewer total commands, it also secures fewer shells, quantitatively substantiating its tendency toward premature abandonment when confronting execution roadblocks. In contrast, MazeRunner demonstrates a superior balance between exploratory depth and execution precision. Architecturally, the Executor Agent generates and executes scripts before the Reviewer Agent intervenes to correct course. Consequently, while the statistical ratio of invalid commands may appear comparable to baselines, leveraging the Reviewer to swiftly prune flawed paths minimizes the absolute volume of total commands. This prevents the system from spiraling into resource-draining execution loops while simultaneously Table A1 ID

Box Name

OS

Difficulty

Overview of the selected evaluation dataset and ground-truth Released

Problem Domain

Date

# of subtasks

Brief Description

1

WingData

Linux

Easy

2026/2/14

Web, Wing FTP Server RCE, Credential Recovery, Tarfile PrivEsc

13

Wing FTP (CVE-2025-47812) foothold, credential recovery, and privesc via vulnerable Python tar script.

2

CCTV

Linux

Easy

2026/3/7

Web, IoT/CCTV, ZoneMinder SQLi, motionEye PrivEsc

11

ZoneMinder SQLi (CVE-2024-51482) for SSH access; privesc via root-running motionEye interface.

3

Kobold

Linux

Easy

2026/3/21

AI Tooling, MCP Inspector RCE, Docker Escape

12

MCP Inspector unauthenticated RCE foothold; Docker group abuse for container escape.

4

Silentium

Linux

Easy

2026/4/11

AI Workflow Platform, Flowise ATO/RCE, Credential Leakage, Gogs RCE

14

Flowise ATO/RCE and Docker credential leak, leading to root via local Gogs RCE.

5

Pterodactyl

Linux

Medium

2026/2/7

Web, Pterodactyl Panel, Path Traversal/LFI, PEAR-assisted RCE

14

Pterodactyl Panel LFI (CVE-202549132) chained with PHP PEAR for RCE.

6

Interpreter

Linux

Medium

2026/2/21

Web, Mirth Connect RCE, SSTI/Python eval, Internal Service Abuse

14

Mirth Connect (CVE-2023-43208) foothold; root RCE via internal XML service and Python SSTI.

7

Principal

Linux

Medium

2026/3/12

Cryptographic Trust, JWT/JWE Auth Bypass, SSH CA Misconfig

13

JWT auth bypass (CVE-2026-29000) for admin tokens; SSH CA misconfig for root certificate forgery.

8

DevArea

Linux

Medium

2026/3/28

DevOps, Apache CXF LFI, Hoverfly API RCE, Sudo/Binary PrivEsc

15

Apache CXF LFI leading to Hoverfly API RCE; root via writable binary and sudo abuse.

9

Pirate

Windows

Hard

2026/3/28

Active Directory, gMSA Abuse, Kerberoasting, NTLM Relay

21

AD chain: BloodHound, gMSA abuse, NTLM relay, and SPN hijacking to compromise DC.

10

Garfield

Windows

Hard

2026/4/4

Active Directory, ACL Abuse, RODC, RBCD, Ticket Forgery

22

AD escalation via ACL abuse, lateral movement, RBCD against RODC, and Golden Ticket attacks.

18

Table A2

MazeRunner

ID

Claude Code

Sonnet 4.5

13.07M 0.31M

0.30M

PentestGPT-V2

DeepSeek-V3.2

11.20M 0.41M

0.25M

DeepSeek

Sonnet

DeepSeek

Sonnet

DeepSeek

Sonnet

WingData

86/165

43/155

64/210

69/132

90/194

48/148

CCTV

29/132

48/123

32/119

42/147

87/218

67/211

Kobold

23/133

13/125

71/222

55/287

73/231

209/578

Silentium

39/205

28/163

18/123

36/125

19/150

86/386

Pterodactyl

48/237

45/132

35/130

26/143

159/486

121/473

Interpreter

27/95

42/175

22/126

25/135

86/308

73/242

DeepSeek-V3.2 $0.42

Principal

80/351

7/79

22/82

19/108

28/86

49/303

GPT-5.2

DevArea

12/128

28/210

49/210

35/177

71/333

58/345

GPT-5.2-Codex $0.74

Pirate

62/178

214/459

174/452

119/238

348/526

281/390

Garfield

60/100

98/188

125/216

136/244

129/205

94/167

Table A3

0.60M

0.78M

Statistics of invalid/total commands

GPT-5.2

11.21M 1.21M

GPT-5.2-Codex 1.50M 0.0

14.11M

2.5

5.0

7.5

10.0

12.5

Average Token Consumption (M)

15.0

17.5 $0.25

Sonnet 4.5 $0.46

$4.95

$0.02 $0.01 $0.30

$0.15

$3.15

Strategist Executor Reviewer

$0.26

0

Figure A1

$3.16

1

2

3

4

5

6

Average Cost Consumption (USD) Average cost of MazeRunner’s agents

Sub-tasks and clues along the primary penetration path of “HTB-Interpreter” Driven By

#

Task

Clues

1

Port Scanning & Service Identification

←

IP Address

2

Web App Fingerprinting & Version Discovery

←

Port 80/443

3

Vulnerability Intelligence & Attack Surface Confirmation

←

Mirth Connect 4.4.0

4

Exploit Known Vulnerability for RCE

←

CVE-2023-43208 /api/users Deserialization Chain

5

Obtain Stable mirth User Shell (user-1 shell)

←

CVE-2023-43208 Exploit Script

6

Internal Port Scanning & Service Discovery

←

mirth User Session

7

Extract Database Connection Info

←

mirth.properties

8

Login to Database & Extract Key Business Data

←

Cleartext DB Credentials

9

Offline Cracking of User Credentials

←

Username & Hash from PERSON / PERSON PASSWORD Tables

10

Obtain sedric User Shell via SSH (user-flag shell)

←

Valid Username + Cracked Cleartext Password

11

Analyze Business Forwarding Logic

←

XML Config in CHANNEL Table / Local Service Info

12

Identify Controllable Input Points & Unsafe Fields

←

/addPatient Endpoint + Controllable XML Message Body

13

Craft Python Code Execution Payload

←

Controllable XML Content + Python Execution Context

14

Obtain Root via Internal notif Service (root-flag shell)

←

Root-Privileged notif.py Service

achieving the highest shell acquisition rate. A prime example is the Principal machine: MazeRunner paired with Sonnet 4.5 acquired a root shell utilizing only 79 total commands, with a mere 7 invalid executions, showcasing exceptional payload accuracy. Finally, comparing the foundational models within our framework reveals that Sonnet consistently requires fewer total commands and generates fewer invalid actions than DeepSeek, highlighting how a superior coding model, when properly orchestrated, drastically streamlines the penetration lifecycle.

Appendix B

Classification of Pentesting Failure Reasons

To systematically analyze the limitations of the evaluated frameworks, we categorize the observed execution failures into the following five distinct taxonomy classes: • Planning Error: Despite possessing sufficient clues or environmental feedback, the agent pursues erroneous or hallucinated attack vectors based on misidentified target assets or flawed vulnerability paths. This causes the overall penetration workflow to deviate significantly from the ground truth without subsequent recovery. • Clue Forgetting (Contextual Amnesia): The agent fails to execute the correct penetration sequence due to the loss or overwriting of previously acquired critical intelligence within its context window (e.g., forgetting user credentials, passwords, open ports, or vulnerability disclosures). • Premature Abandonment: The agent correctly identifies a viable attack trajectory but prematurely halts execution or switches contexts before exploiting the vulnerability. • Code/Payload Failure: Downloaded exploit scripts or autonomously generated payloads fail to execute successfully or achieve the intended objective (e.g., failing to establish Remote Code Execution) due to syntax errors, misconfigurations, or environmental incompatibilities. • Information Retrieval Failure: The agent is unable to acquire the necessary, correct information via web searches or external reconnaissance tools (e.g., failing to locate the ground-truth CVE using SearchSploit).

19

Related documents

• NEUROTESTGEN: Neuro-Symbolic Guided Test Generation with Large Language Models#1078249
• LLMs Improving LLMs: Agentic Discovery for Test-Time Scaling#172745
• Enhancing Automated Unit Test Generation for NLP Libraries Using Large Language Models#1084706
• Enhancing Automated Unit Test Generation for NLP Libraries Using Large Language Models#919499
• Testing Neural Networks via Bayesian-Guided Exploration of Decision Landscapes#259573