EvoX: Rethinking the Agent Harness as a Runtime Policy

EvoX: Rethinking the Agent Harness as a Runtime Policy

arXiv:2611.15006v1 [cs.AI] 11 Nov 2026

Haoyang Zhang†

Qumeng Sun Xiao Zhang Zhidong Fan Junjie Wang

Xiang Liu

Infinite Evolution Lab, EvoMap https://evomap.ai/evox

Abstract Large language model (LLM) agents increasingly interact with tools and environments, yet plausible model decisions do not always translate into successful execution due to tool-call mismatches, interrupted workflows, and unsupported completion claims. We present EvoX, a Rust-based, multi-model agent runtime guided by The Second Policy, which views the harness as an Operational Policy complementing the foundation model’s Semantic Policy. EvoX combines modelfacing guidance with deterministic runtime control through three complementary mechanisms: Action Realization, Execution Continuity, and Outcome Grounding. To evaluate harnesses without suppressing their native behaviors, we introduce Contrast, a controlled and auditable framework integrating explicit fairness constraints, provider-bound request verification, and evidence-based adjudication. Across five benchmarks, six foundation models, and six agent harnesses, EvoX achieves the highest numerical success rate of 56.2% over 691 common task–model pairs, exceeding the average of the other five harnesses by approximately 3.0 percentage points. Its larger advantages on GLM-5.3 (+10.7 points) and Kimi K3 (+7.9 points) highlight the model-dependent effectiveness of harness design and the potential for cross-provider adaptability. Achieved without memory, evolution, or extensions, these results establish EvoX as a competitive multi-model agent runtime and underscore the importance of harness-level operational control in realizing foundation-model capabilities.

1

Introduction

Recent advances in large language models (LLMs) have enabled AI agents to move beyond code generation toward autonomous tool use and multi-step execution in real-world environments [1–4]. However, plausible model decisions do not guarantee successful execution, as tool-call mismatches, recoverable failures, and premature termination can prevent agents from completing their tasks. Agent harnesses mediate this transition by connecting model decisions with tools, execution states, and environmental feedback. Existing systems adopt different designs: SWE-agent emphasizes task-oriented agent–computer interfaces [5], OpenHands provides a general-purpose development environment [6], Claude Code and Codex support tool-driven coding workflows [7, 8], while Pi adopts a minimal, extensible agent loop [9]. Despite these differences, harnesses influence not only tool access but also how models process feedback, recover from failures, and manage execution. Recent studies further demonstrate that harness design can substantially affect task success and efficiency, even when the underlying model remains unchanged [10–12]. This raises a fundamental question: What role should the harness play in turning model decisions into reliable execution? †

Corresponding Author. 17@evomap.ai

To make this role explicit, we introduce EvoX, a Rust-based, multi-model agent system guided by a design principle termed The Second Policy. This principle distinguishes two complementary policies: a Semantic Policy, embodied by the foundation model, that interprets tasks, plans actions, and reasons over feedback; and an Operational Policy, embodied by the harness, that governs action execution, runtime states, and recovery. Rather than introducing a separately trained model, the operational policy combines model-facing guidance with deterministic runtime controls to mediate execution without modifying foundation model parameters. EvoX realizes this design through three complementary mechanisms. Action Realization detects malformed or text-leaked tool calls, validates tool schemas, and normalizes compatible arguments to mitigate mismatches between model outputs and executable tools. Execution Continuity combines failure-recovery guidance, repeated-error detection, runtime circuit breakers, and context management to support recovery while limiting unproductive execution loops. Outcome Grounding guides models to align reported outcomes with observed actions and tool feedback. Beyond its core harness, EvoX supports diverse models, tools, and multi-agent workflows, alongside optional long-term memory and Gene Evolution Protocol (GEP)-based experience evolution [13]. Evaluating these harness designs introduces a related challenge: ensuring comparable execution conditions without suppressing the native behaviors that distinguish individual harnesses. We therefore develop Contrast, a controlled and auditable framework for cross-harness evaluation. Contrast distinguishes locked, shared, and harness-native conditions, integrating task qualification, model-request verification, execution evidence, and failure-aware adjudication to support systematic and traceable comparisons. Using Contrast, we evaluate EvoX against five agent harnesses across five benchmarks and six foundation models under controlled conditions. Across 691 common task–model pairs, EvoX achieves the highest numerical success rate of 56.2%, followed by Pi (55.9%), Claude Code (55.3%), and Hermes (54.4%). It exceeds the average of the other five harnesses by approximately 3.0 percentage points, with larger advantages of 10.7 and 7.9 points on GLM-5.3 and Kimi K3, respectively. The results reveal model-dependent harness effectiveness: Claude Code and Codex lead on their affiliated Opus 5 and GPT-6 Astra models, whereas EvoX leads on GLM-5.3 and Kimi K3, demonstrating competitive cross-provider adaptability. Achieved without memory, evolution, or extensions, these findings position EvoX as a competitive multi-model agent runtime and highlight the importance of harness-level Operational Policy in realizing foundation-model capabilities.

2

Related Work

2.1

Agent Harnesses and Runtime Control

Agent harnesses shape how model-generated decisions are translated into executable actions through their choices of interfaces, tools, and runtime control. SWE-agent [5] introduces specialized agent– computer interfaces for repository navigation, code editing, and testing, while CodeAct [14] uses executable code as a compositional action space. OpenHands [6] provides sandboxed execution and multi-agent coordination, whereas Claude Code and Codex integrate tool orchestration and execution controls into development workflows [7, 8]. Other harnesses explore different runtime architectures: Pi emphasizes a minimal, extensible agent loop [9], Hermes Agent incorporates persistent memory and reusable skills [15], and Prime Agent employs a persistent execution environment with recursive subagents and continual harness state [16]. Complementary approaches improve reliability through feedback and structured workflows: Reflexion [17] uses verbal feedback for subsequent attempts, CRITIC [18] performs tool-assisted self-correction, and Agentless [19] structures software repair into localization, patch generation, and validation. These approaches demonstrate the importance of execution design beyond model reasoning, while addressing different aspects of agent behavior. Building on these insights, EvoX introduces “The Second Policy” to organize action realization, execution continuity, and outcome grounding as complementary responsibilities of harness-level operational control. 2.2

Controlled Evaluation of Agent Harnesses

Reliable agent evaluation requires both executable task verification and comparable execution conditions. SWE-bench [20] evaluates repository-level issue resolution, SWE-bench Live [21] addresses 2

Figure 1: Architecture and runtime design of EvoX. The system integrates a model-driven Semantic Policy with a harness-level Operational Policy to coordinate decisions and environment interactions. The operational policy comprises three complementary mechanisms: Action Realization mediates executable tool calls, Execution Continuity coordinates recovery and runtime control, and Outcome Grounding aligns reported outcomes with observed execution. The runtime supports diverse tools and optional persistent adaptation through long-term memory and GEP-based experience evolution. The right panel illustrates the conceptual distinction between a minimal execution loop and explicit harness-level operational control. benchmark freshness and contamination, and AgentBoard [22] introduces fine-grained progress measurements beyond final success. More directly, Harness-Bench [10] and The Scaffold Effect [11] investigate harness-dependent performance, while Same Model, Different Harness [12] examines context management and stalled-execution handling under fixed model weights. However, identical tasks and nominal model configurations do not necessarily ensure comparable execution when request parameters, runtime conditions, and failure attribution differ. To address these challenges, Contrast distinguishes locked, shared, and harness-native conditions, integrating task qualification, model-request verification, and evidence-based adjudication to support controlled comparisons.

3

EvoX: Architecture and Runtime Design

3.1

System Overview and Execution Flow

System Architecture. EvoX is a Rust-based, multi-model agent runtime that connects foundation models with tools and interactive environments through a unified execution framework. As illustrated in Figure 1, the system integrates user-facing interfaces, model invocation, context management, tool execution, and runtime state management. Its architecture distinguishes model-driven semantic decisions from harness-level operational control, establishing the foundation for The Second Policy. The runtime accommodates heterogeneous models, tools, and interaction interfaces, with additional capabilities for multi-agent coordination and persistent adaptation. Execution Flow. Given a user task x, EvoX maintains an interaction history ht and a runtime state st at each interaction step t. The history ht records model outputs and environmental feedback, while st tracks execution conditions, including task status and failure information. The system assembles the model context from task instructions, interaction history, available tool interfaces Tt , and relevant runtime information. The model then proposes an action, which the harness interprets and dispatches when executable under the applicable controls. Tool results update the interaction history and runtime state, forming an iterative decision–execution–feedback loop. Crucially, the realized execution depends not only on the model’s proposal but also on how the harness processes it. 3

3.2

The Second Policy

The distinction between proposed actions and realized execution reveals a fundamental role of the agent harness. A semantically appropriate action may fail to satisfy an executable interface, recoverable errors may interrupt task progress, and final responses may overstate what execution has established. Rather than treating these issues as isolated execution exceptions, EvoX organizes their handling through a unified design principle termed The Second Policy. This perspective recognizes the harness as an explicit source of operational control that complements, rather than replaces, model-driven decision-making. Complementary Decision Policies. The Semantic Policy, represented by the foundation model, interprets task objectives, proposes actions, and revises decisions according to interaction feedback. The Operational Policy, implemented by the harness, governs how these proposals are interpreted, executed, and incorporated into subsequent interactions. While semantic decisions determine what the agent attempts, operational control manages how those decisions are realized under runtime constraints. Importantly, the harness influences both the context in which actions are proposed and the conditions under which they are executed. We formalize this interaction by representing the semantic policy as πθ and the operational policy as H = (Γ, ∆), where Γ denotes model-facing guidance and ∆ denotes deterministic runtime control. At each interaction step, the two policies cooperate as gt = Γ(ht , st ), ct = Ctx(x, ht , gt , Tt ), ut ∼ πθ (· | ct ), (ât , dt ) = ∆(ut , st , Tt ).

(1)

Here, Γ supplies runtime guidance gt , which is incorporated into the model context ct before generating an output ut . The runtime controller ∆ subsequently interprets this output, producing an executable action ât when applicable and a disposition dt indicating how execution should proceed. Depending on the proposal and runtime conditions, the disposition may permit execution, return a diagnostic, or terminate the interaction. The resulting tool feedback or runtime diagnostic ot updates ht+1 and st+1 , informing subsequent model decisions and runtime control. This formulation describes complementary responsibilities within the agent system rather than introducing an additional trained policy network. Hybrid Operational Control. The two components of H address different aspects of execution reliability. Model-facing guidance Γ incorporates behavioral expectations and observed feedback into subsequent model interactions, encouraging error recovery and evidence-grounded reporting while preserving flexibility in semantic decisions. In contrast, runtime control ∆ applies explicit rules to action representations and execution states, including tool validation, bounded interface correction, and repeated-error detection. Such interventions may reject incompatible operations or terminate unproductive execution paths without requiring additional model decisions. Accordingly, guidance shapes how the model responds to execution feedback, while deterministic control governs specific runtime conditions where applicable. Together, they establish operational control across the interaction loop rather than at a single execution stage. 3.3

Harness-Level Operational Mechanisms

The operational policy is realized through three complementary mechanisms that address action execution, interaction continuity, and outcome reporting. Each mechanism operates at a distinct point in the execution loop while sharing the control formulation in Equation (1). Action Realization. A model may select an appropriate tool yet produce an invocation that the execution interface cannot directly process. Given a model output ut , Action Realization uses runtime control ∆ to examine its compatibility with registered tool interfaces through tool-call detection, schema validation, and argument normalization. Where supported, this includes identifying malformed structured calls or recognizable tool intents expressed in nonstandard textual forms. When a compatible representation can be recovered, the runtime applies bounded normalization to obtain an executable action ât . Otherwise, the invocation is rejected or returned for model-side correction rather than being arbitrarily reinterpreted. This mechanism separates semantic action intent from 4

executable representation, reducing avoidable interface-level failures while preserving the model’s intended operation. Execution Continuity. Recoverable execution failures, such as invalid paths, missing dependencies, or unsuccessful commands, need not imply that the underlying task is infeasible. After receiving tool feedback ot , the harness updates the interaction history ht+1 and runtime state st+1 to reflect the observed execution. Model-facing guidance Γ then encourages the semantic policy to diagnose failures and revise subsequent actions rather than prematurely terminating the task. Meanwhile, runtime control ∆ monitors recurring tool-error patterns and, when enabled, applies circuit breakers to terminate repeated, unproductive attempts. Context management and automatic compaction further support extended interactions by maintaining a manageable history as context limits are approached. Execution Continuity therefore balances flexible, model-guided recovery with statedependent stopping decisions, allowing execution to proceed when correction remains possible while limiting ineffective repetition. Outcome Grounding. A model-generated completion claim does not necessarily establish that the corresponding operations were successfully executed. When producing a final response, the semantic policy is guided by Γ to distinguish intended actions from completed operations and to ground its claims in the observations recorded in ht . EvoX incorporates grounded-action guidance that encourages the model to report task outcomes according to observed tool results rather than unsupported assumptions of success. Unlike deterministic action validation or circuit breaking, this mechanism primarily operates through model-facing instructions rather than independent correctness verification. Outcome Grounding therefore targets consistency between observed execution and model-generated reports, without treating those reports as proof of task completion. Together, these mechanisms operationalize The Second Policy across action interpretation, execution continuity, and outcome reporting. They preserve the foundation model’s responsibility for semantic reasoning while making selected aspects of execution explicitly manageable by the harness. 3.4

Extensibility and Persistent Adaptation

Beyond its core operational mechanisms, EvoX provides extensible execution capabilities and persistent experience reuse across tasks. Extensible Agent Infrastructure. EvoX supports multiple foundation model providers and diverse execution tools, including file and shell operations, web retrieval, browser interaction, and desktop control. Its modular interfaces enable external integrations through the Model Context Protocol (MCP), reusable skills, and native extensions. For complex tasks, the runtime supports subagent delegation, coordinated workflows, and optional swarm-based multi-agent collaboration. These capabilities extend the same runtime architecture across heterogeneous environments and collaborative execution settings without modifying foundation model parameters. Persistent Adaptation. Beyond individual tasks, EvoX incorporates long-term memory and Gene Evolution Protocol (GEP)-based experience evolution [13]. Long-term memory extracts reusable information from interaction histories and retrieves task-relevant memories to enrich subsequent model contexts. GEP organizes execution experiences into Capsules, from which recurring successful patterns can be consolidated into reusable Genes. Selected Genes provide task-specific guidance and may restrict available tools or influence model routing, allowing accumulated experience to inform future execution. These capabilities complement within-task operational control with persistent, experience-informed adaptation.

4

Contrast: Controlled and Auditable Harness Evaluation

4.1

Evaluation Principles and Fairness Contract

Evaluating agent harnesses presents a fundamental challenge: external execution conditions must remain comparable without eliminating the native behaviors that distinguish individual harnesses. Standardizing internal prompts, tools, and control loops would obscure the mechanisms under study, whereas leaving external conditions uncontrolled could confound performance comparisons. Inspired by Harbor’s unified framework for evaluating heterogeneous agents [23], we introduce Contrast, a controlled and auditable evaluation framework built around an explicit fairness contract. As illustrated 5

Figure 2: Overview of Contrast for controlled and auditable harness evaluation. A fairness contract distinguishes locked, shared, and harness-native conditions, preserving native execution behaviors while controlling external evaluation factors. Versioned tasks undergo qualification and integrity checks before execution through native harnesses in controlled environments. A model bridge mediates model requests, enforces applicable configurations, and audits provider-bound request parameters. Execution traces, model-call records, verifier results, and runtime diagnostics provide evidence for failure attribution, validity decisions, and auditable result selection. Only valid, scored task–model combinations enter paired statistical comparisons, while excluded or quarantined records remain available for audit and disclosure. in Figure 2, this contract governs task qualification, harness execution, model-call verification, and subsequent result comparison. Evaluation Objective. Let τi , mk , and aj denote an evaluation task, a foundation model, and an agent harness, respectively. We represent the resulting task–model–harness execution as rijk = Run (τi , mk , aj ; ϕik , ν ijk ) ,

(2)

where ϕik specifies externally controlled evaluation conditions and ν ijk represents harness-native configurations and behaviors. The outcome rijk includes the execution result and associated runtime evidence, which are subsequently examined for validity and comparability. The objective is to compare harnesses over matched task–model combinations while preserving their intrinsic execution strategies, rather than enforcing identical internal trajectories. Explicit Fairness Contract. Contrast classifies evaluation conditions into three complementary categories: locked, shared, and native. Locked conditions follow predefined rules, including modelcompatible reasoning configurations, output limits, and network access policies. Shared conditions maintain consistent values within each comparison, such as model identity, provider routing, and execution budgets. In contrast, native conditions preserve harness-specific properties, including system prompts, tool interfaces, context management, and internal recovery strategies. These native differences are recorded rather than homogenized, allowing harness design to remain the primary variable of interest. The fairness contract therefore establishes comparability through explicit control of external factors without suppressing the operational characteristics being evaluated. Cross-Stage Verification. Importantly, comparable configurations do not necessarily imply equivalent execution conditions. Contrast consequently verifies applicable constraints at multiple stages, from pre-execution configuration checks to inspection of model requests forwarded to providers. For supported model-call routes, controlled parameters are enforced and audited against the fairness contract, while harness-native behaviors remain observable through execution records. This approach distinguishes declared configurations from realized model-call conditions and supports subsequent evidence-based validity assessment. The framework connects controlled experimental design with auditable execution, providing a consistent basis for comparing heterogeneous agent harnesses. 6

4.2

Task Qualification and Integrity

Reliable harness comparison requires consistent task instances and verifiable evaluation outcomes. As shown in Figure 2(A), Contrast qualifies benchmark tasks before execution to reduce inconsistencies in task artifacts, verifier behavior, and answer exposure. Task Standardization and Identity. Contrast materializes heterogeneous benchmarks into a common task specification comprising instructions, execution environments, and task-specific verifiers. Each task τi is associated with versioned artifacts and content-based identities that link qualification evidence to subsequent execution. Task contents, relevant file permissions, and environment identities are checked against prepared configurations to detect unintended changes. Benchmark-specific capability requirements are also considered to identify incompatible execution interfaces without altering harness-native behaviors. Qualification and Integrity Checks. Contrast employs complementary positive and negative controls to assess basic verifier reliability. An Oracle applies the reference solution to confirm that successful execution is recognized, while a No-op checks that an unchanged task state is not incorrectly rewarded. Qualification evidence is bound to task identities, preventing modified instances from inheriting previous validation. Additional integrity checks examine task environments, cached artifacts, and accessible resources for potential exposure of reference solutions or hidden tests. Tasks with unresolved concerns are withheld pending review, while post-run evidence is further examined for contamination during adjudication (Section 4.4). 4.3

Controlled Native Execution and Model-Call Verification

Establishing comparable evaluation conditions requires more than aligning declared configurations, as heterogeneous harnesses differ in execution interfaces, model protocols, and runtime behavior. As illustrated in Figure 2(B), Contrast combines native harness execution with controlled environments and provider-bound model-call verification to operationalize the fairness contract. Harness-Native Execution. For each task–model–harness combination (τi , mk , aj ), Contrast launches the corresponding agent through a standardized adapter while preserving its native system prompts, tool interfaces, execution loops, and recovery strategies. The Cove execution substrate provides task-specific containers, controlled resources, and environment configurations, ensuring that participating harnesses operate against consistent evaluation targets. Harness versions, task identities, and runtime configurations are recorded to establish execution provenance. This design preserves the harness-specific behaviors represented by ν ijk while enforcing applicable external conditions specified by ϕik . Provider-Bound Request Verification. Identical model identifiers and nominal inference settings do not necessarily produce equivalent API requests across harnesses. To address this discrepancy, Contrast routes supported model calls through a shared Model Bridge that adapts heterogeneous API protocols and enforces applicable constraints on model identity, reasoning configurations, and output limits. Beyond pre-execution configuration checks, the framework inspects provider-bound requests to verify that controlled parameters are reflected in the actual transmitted payloads. This separates declared settings from realized model-call conditions, reducing configuration-induced discrepancies without modifying native agent decision logic. Request-level verification establishes compliance with observable protocol constraints but cannot guarantee identical provider-side execution semantics. Campaign Orchestration and Provenance. Contrast organizes task–model–harness combinations into controlled evaluation campaigns with explicit execution budgets, versioned configurations, and traceable trial identities. Each run produces execution records linking agent actions, model requests, verifier outputs, runtime diagnostics, and resource consumption. These records support subsequent validity checks and failure attribution rather than treating every unsuccessful run as an agent failure. The resulting evidence is collected for the adjudication process described in Section 4.4. 4.4

Evidence-Based Adjudication and Paired Comparison

Controlled execution alone does not guarantee reliable evaluation, as unsuccessful runs may arise from agent failures, infrastructure errors, or invalid assessment conditions. To distinguish these cases, Contrast evaluates execution outcomes against recorded evidence before incorporating them into 7

performance comparisons. As illustrated in Figure 2(C–F), this process connects execution records, failure attribution, result selection, and paired statistical analysis. Execution Evidence. For each trial rijk , Contrast collects execution evidence eijk comprising agent traces, provider-bound model requests, verifier outcomes, runtime diagnostics, and resource consumption. These records are associated with task, model, harness, and execution identities, allowing reported outcomes to be traced to their underlying observations. Evidence integrity and measurement completeness are examined to identify missing, inconsistent, or potentially contaminated records. This supports the distinction between task-level outcomes and evaluation artifacts. Failure Attribution and Adjudication. As shown in Figure 2(D), Contrast applies evidence-based rules to determine whether an execution constitutes a valid task outcome. Given an adjudication protocol ρ, we express this process as (yijk , qijk ) = Adjudicate(rijk , eijk ; ρ),

(3)

where yijk denotes the task score and qijk indicates whether the record is scored, excluded, or quarantined. Importantly, scored records include both successful executions and attributable agent failures, whereas verified evaluation-side faults may invalidate a trial. Records with unresolved evidence conflicts can be isolated for further review rather than automatically treated as successful or excluded. This distinction prevents infrastructure failures from being indiscriminately attributed to agents while retaining legitimate task failures in the evaluation. Auditable Result Selection. When multiple attempts exist for a task–model–harness combination, Contrast applies versioned selection and invalidation rules to determine its representative outcome. The selected record retains its execution provenance, while superseded or excluded attempts remain traceable through the corresponding evidence. This process separates result selection from observed task performance and supports consistent application of the evaluation protocol. Subsequent comparisons use the selected scores and their associated validity decisions. Paired Comparison and Reporting. Rather than aggregating different sets of valid executions for each harness, Contrast constructs a common task–model set in which all compared harnesses have scored outcomes, as illustrated in Figure 2(E). Formally, for the set of compared harnesses A, Icommon = {(i, k) | qijk = scored, ∀aj ∈ A} .

(4)

Harness performance and pairwise differences are evaluated over this matched set, with uncertainty estimated through paired, task-level resampling. The resulting reports include performance estimates, confidence intervals, and disclosure records for excluded or unresolved executions (Figure 2(F)). Together, these procedures establish an auditable measurement process in which performance comparisons are grounded in validated execution evidence rather than raw trial outcomes alone.

5

Experiments and Analysis

5.1

Evaluation Setup

We evaluate EvoX using Contrast to examine how different agent harnesses affect task outcomes under matched foundation-model and evaluation conditions. Evaluation Scope. We compare six agent harnesses: EvoX, Claude Code [7], Codex [8], Hermes [15], Pi [9], and Prime Agent [16]. The evaluation covers six foundation models: Opus 5 [24], Sonnet 5 [25], GPT-6 Astra [26], GLM-5.3 [27], Kimi K3 [28], and DeepSeek V4 Flash [29]. We consider five benchmarks covering multilingual programming, web-assisted reasoning, algorithmic problem solving, repository-level software repair, and terminal-based engineering. Given computational constraints, we aim to select approximately the first 28 eligible tasks from each benchmark’s predefined evaluation panel, subject to task qualification and availability. The resulting evaluation includes 28 tasks from Aider Polyglot [30], 21 from GAIA [31], 28 from Harbor Index [32], 27 from SWE-Live [21], and 28 from Terminal-Bench 2.1 [33]. Together, these 132 tasks yield 4,752 planned task–model–harness combinations. Controlled Harness Configurations. Following the fairness contract in Section 4.1, we align foundation-model identities, model-specific reasoning configurations, requested output limits, execution budgets, and applicable network restrictions across harnesses. Model requests are routed through a shared gateway, with controlled parameters checked against provider-bound requests. 8

Meanwhile, each harness retains its native system prompts, tool interfaces, context management, execution loops, and recovery strategies. EvoX is evaluated in its non-interactive execution mode using version v1.1.0-pre.93, with memory, evolution, and extensions disabled. This configuration focuses the comparison on baseline harness execution rather than optional persistent adaptation. Metrics. We measure task success using binary outcomes determined by benchmark-specific verifiers and predefined scoring rules. Attributable agent failures are counted as unsuccessful outcomes, while invalid trials and confirmed answer leakage are excluded. Our primary comparison uses 691 task–model pairs for which all six harnesses have valid scored outcomes. All reported success rates are computed over this common set. 5.2

Overall Harness Performance

Success rate (%)

60

56.2%

55.9%

55.3%

54.4%

50.4%

49.6%

Pi

Claude Code

Hermes

Prime

Codex

Others mean 53.1%

40

20

0

EvoX

Figure 3: Overall performance of six agent harnesses on 691 common task–model pairs across five benchmarks. Bars indicate task success rates, with EvoX highlighted in blue. The dashed line represents the mean success rate of the other five harnesses (53.1%). Competitive Overall Performance. As shown in Figure 3, EvoX achieves the highest success rate of 56.2%, followed by Pi (55.9%), Claude Code (55.3%), and Hermes (54.4%). Prime Agent and Codex achieve 50.4% and 49.6%, respectively. Compared with the mean performance of the other five harnesses (53.1%), EvoX maintains an advantage of approximately 3 percentage points. Notably, this result is achieved without enabling memory, evolution, or extensions, demonstrating the competitiveness of its baseline operational design. The Role of Harness-Level Control. Under matched tasks and foundation models, the observed performance differences highlight the role of harness-level execution in shaping agent outcomes. While the four leading harnesses achieve relatively close results, EvoX maintains a larger margin over Prime Agent and Codex. These findings support the importance of the Operational Policy as a complementary component to the model-driven Semantic Policy, rather than attributing task success solely to foundation-model capabilities. 5.3

Harness–Model Interactions

As shown in Figure 4, harness effectiveness varies substantially across foundation models. Model–Harness Affinity. Claude Code achieves the highest success rate on Opus 5 (74.8%), while Codex leads on GPT-6 Astra (80.2%), suggesting favorable alignment between these first-party harnesses and their associated model families. These observations indicate that harness effectiveness depends on the underlying model rather than following a consistent ranking across providers. Cross-Provider Adaptability. Unlike provider-specific harnesses, EvoX is designed to operate across heterogeneous foundation-model ecosystems. It achieves the highest success rates on GLM-5.3 (57.8%) and Kimi K3 (61.3%), while remaining competitive on GPT-6 Astra (78.5%) relative to Codex (80.2%). This demonstrates that a provider-independent operational design can effectively support models from different families without relying on a proprietary foundation model. Neverthe9

Opus 5

Sonnet 5

GPT-6 Astra

GLM-5.3

Kimi K3

DeepSeek V4 Flash

Overall

EvoX

68.2

45.4

78.5

57.8

61.3

22.1

56.2

Pi

70.1

56.3

77.7

44.8

58.9

24.0

55.9

Claude Code

74.8

55.5

76.9

46.6

54.0

21.2

55.3

Hermes

67.3

45.4

76.9

50.0

55.6

28.8

54.4

Prime

64.5

33.6

78.5

46.6

53.2

23.1

50.4

Codex

62.6

42.9

80.2

47.4

45.2

16.3

49.6

SR (%) 100

75

50

25

0

EvoX − Others mean (pp)

+0.4

−1.3

+0.5

+10.7

+7.9

−0.6

+3.0

EvoX advantage (pp) −12

0

+12

Figure 4: Harness–model performance across six foundation models. Each cell reports the task success rate (%) of a harness–model combination, with outlined cells indicating the highest performance for each model. The overall column summarizes performance across 691 common task–model pairs. The bottom row reports the performance difference between EvoX and the mean of the other five harnesses, in percentage points. less, EvoX does not consistently lead across all models, highlighting the importance of model-specific compatibility. Model-Conditional Advantages. Relative to the mean of the other five harnesses, EvoX improves success rates by 10.7 percentage points on GLM-5.3 and 7.9 points on Kimi K3, compared with an overall advantage of approximately 3.0 points. Its relative differences are considerably smaller on Opus 5 (+0.4) and GPT-6 Astra (+0.5), and slightly negative on Sonnet 5 (-1.3) and DeepSeek V4 Flash (-0.6). These variations suggest that the effectiveness of the Operational Policy depends in part on the foundation model underlying the Semantic Policy it serves. 5.4

Generalization Across Task Families

Table 1: Performance across five benchmark families. Each entry reports task success rate (%) on common task–model pairs, with n denoting the number of pairs per benchmark. ∆ indicates the difference between EvoX and the mean of the other five harnesses, in percentage points. Best results are highlighted in bold. Benchmark

EvoX

Pi

Claude Code

Hermes

Prime

Codex

∆ (pp)

Aider Polyglot GAIA Harbor Index SWE-Live Terminal-Bench 2.1

64.7 86.7 27.3 48.4 57.3

61.7 85.7 25.9 49.2 60.7

62.3 82.9 31.5 49.2 55.3

63.5 85.7 24.5 43.7 59.3

61.1 85.7 23.1 35.7 50.0

55.1 78.1 27.3 37.3 54.0

+4.0 +3.0 +0.8 +5.4 +1.5

Cross-Task Competitiveness. To examine whether the competitiveness of EvoX extends across different workloads, Table 1 compares six harnesses on five benchmark families. EvoX achieves the highest success rates on Aider Polyglot (64.7%) and GAIA (86.7%), covering multilingual programming and web-assisted reasoning. Moreover, EvoX exceeds the average performance of the other five harnesses across all evaluated benchmarks, with gains ranging from 0.8 to 5.4 percentage 10

points. These results indicate competitive performance across diverse task requirements rather than reliance on a single benchmark. Task-Dependent Performance. The relative advantage of EvoX is most pronounced on SWE-Live, where it exceeds the competing harness average by 5.4 percentage points. However, the strongest individual harness varies across benchmarks: Claude Code leads on Harbor Index, while Pi and Claude Code slightly outperform EvoX on SWE-Live, and Pi leads on Terminal-Bench 2.1. These differences highlight the absence of a universally dominant harness across task families. Compared with the pronounced model-dependent variations observed in Section 5.3, the benchmark-level results suggest that EvoX maintains relatively consistent competitiveness across the evaluated task types. 5.5

Execution Outcomes and Behavioral Evidence

Table 2: Execution outcome profiles across 691 common task–model pairs. Normal completion excludes nonzero exits, timeouts, and cost-cap interruptions. Exit denotes a nonzero process exit; TO and Cap denote timeout and cost-cap termination, respectively, with P/F indicating passing or failing verifier outcomes. A passing verifier result after a timeout is scored as unsuccessful under the evaluation protocol. The final columns report the normal-completion rate and verifier success rate conditional on normal completion. Normal

Other outcomes

Rate (%)

Harness

Pass Fail Exit TO-F TO-P Cap-P Cap-F Other Normal Pass|Normal

EvoX Pi Claude Code Hermes Prime Agent Codex

373 365 344 361 330 329

134 187 161 150 152 154

117 5 111 159 119 144

49 107 30 2 68 47

4 6 5 0 4 3

12 18 37 14 15 12

2 2 3 5 3 2

0 1 0 0 0 0

73.4 79.9 73.1 74.0 69.8 69.9

73.6 66.1 68.1 70.6 68.5 68.1

Execution Outcome Profiles. Beyond aggregate success rates, we examine how harnesses terminate their executions and whether completed runs satisfy task objectives. As shown in Table 2, EvoX completes 73.4% of trials normally, closely matching the 73.3% average of the other five harnesses. However, among normally completed trials, EvoX achieves a verifier success rate of 73.6%, compared with 68.3% for the competing harnesses on average. This suggests that its competitive performance is not simply attributable to a higher frequency of normal termination. Notably, timeout and exit-code distributions also differ across harnesses, reflecting their distinct execution and termination semantics. Evidence-Grounded Execution. To further characterize execution behavior, we examine 22 reviewed EvoX success trajectories alongside comparator runs under matched task–model conditions. Among 15 selected cases where EvoX succeeds while at least four competing harnesses fail, eight exhibit more extensive execution-grounded verification, including end-to-end testing, differential checks, and failure reproduction. For example, on cancel-async-tasks with GLM-5.3, EvoX reproduces repeated cancellation through a subprocess receiving real SIGINT signals. On capstone-2743, it uses GDB to locate an out-of-bounds error and validates the correction through differential checks over approximately 230,000 encodings. Other trajectories show continued local analysis following unsuccessful attempts to retrieve external information, illustrating how execution can proceed despite intermediate failures. Operational Implications. These observations are consistent with the intended roles of Execution Continuity and Outcome Grounding in the Operational Policy. Beyond producing a final response, EvoX exhibits behaviors that incorporate execution feedback and verification into task completion. 5.6

Evaluation Integrity and Sensitivity

Cross-Harness Model Sensitivity. We examine how the relative competitiveness of each harness changes when individual foundation models are excluded from evaluation. As shown in Figure 5, different harnesses exhibit distinct sensitivity patterns. Excluding Sonnet 5 reduces the relative advantage of Pi and Claude Code by 1.9 and 1.8 percentage points, respectively, while excluding GPT-6 Astra reduces that of Codex by 1.5 points. In contrast, Prime Agent gains 2.4 points when 11

Excluded model Harness

Full-set adv. (pp)

Opus 5

Sonnet 5

GPT-6 Astra

GLM-5.3

Kimi K3

DeepSeek V4 Flash

EvoX

+3.0

+0.5

+0.9

+0.5

−1.5

−1.1

+0.6

Pi

+2.7

0.0

−1.9

+0.7

+1.5

−0.5

+0.2

Claude Code

+2.0

−1.1

−1.8

+0.7

+1.0

+0.6

+0.7

Hermes

+1.0

+0.3

+0.5

+0.5

−0.1

0.0

−1.2

Prime

−3.9

0.0

+2.4

−0.9

−0.2

−0.5

−0.8

Codex

−4.8

+0.3

−0.1

−1.5

−0.6

+1.5

+0.5

Shared scale

Decrease

−2.5

0

+2.5

Increase

Change in relative advantage (pp)

Figure 5: Model-composition sensitivity across six agent harnesses. The first column reports each harness’s performance advantage over the mean of the other five harnesses on the full evaluation set. Each remaining entry shows the change in this relative advantage after excluding the indicated foundation model, measured in percentage points. Negative values indicate reduced relative competitiveness, while positive values indicate improvement. Sonnet 5 is excluded. These variations further illustrate that harness effectiveness depends in part on the foundation models it serves. EvoX Across Model Subsets. With all six models, EvoX exceeds the average of the other five harnesses by approximately 3.0 percentage points. Its advantage remains positive under every singlemodel exclusion, indicating competitive performance across the examined model subsets. However, excluding GLM-5.3 reduces its advantage by 1.5 points, while excluding Kimi K3 reduces it by 1.1 points. These results reinforce the importance of model–harness compatibility while highlighting the contribution of GLM-5.3 and Kimi K3 to EvoX’s aggregate performance. Evaluation Integrity and Scope. To ensure evaluation integrity, Contrast examines task environments and execution records for potential answer leakage, excluding cases confirmed through review. Our evaluation covers 691 common task–model pairs across five benchmarks, six foundation models, and six agent harnesses under matched evaluation conditions. For EvoX, the reported results characterize its baseline operational behavior, with memory, evolution, and extensions disabled.

6

Conclusion

In this work, we present EvoX and the perspective of The Second Policy, viewing the agent harness as an Operational Policy that complements the foundation model’s Semantic Policy. EvoX realizes this perspective through action realization, execution continuity, and outcome grounding, while Contrast enables controlled and auditable comparisons across heterogeneous harnesses. Across five benchmarks, six foundation models, and six harnesses, EvoX achieves the highest numerical success rate of 56.2%, exceeding the competing harness average by approximately 3.0 percentage points. Its competitive cross-provider performance, particularly on GLM-5.3 and Kimi K3, highlights the importance of harness–model compatibility. These results are achieved without memory, evolution, or extensions, establishing a competitive foundation for further operational adaptation. Looking ahead, we envision the harness as a promising substrate for agent self-evolution, where verified execution experience can be accumulated, refined, and reused to improve future behavior. Through experience sharing and coordination across agents, this direction could further extend toward collective evolution 12

and self-organizing agent swarms. Together, these findings and possibilities highlight the broader potential of harness design for building reliable, adaptive, and collaboratively evolving agent systems.

References [1] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. A survey on large language model based autonomous agents. Frontiers Comput. Sci., 18(6):186345, 2024. [2] Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. Large language model-based agents for software engineering: A survey. ACM Trans. Softw. Eng. Methodol., 2026. doi: 10.1145/3796507. [3] Xinzhe Li. A review of prominent paradigms for LLM-based agents: Tool use, planning (including RAG), and feedback learning. In Proceedings of the 31st International Conference on Computational Linguistics (COLING), pages 9760–9779, 2025. [4] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023. [5] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems (NeurIPS), 2024. [6] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations (ICLR), 2025. [7] Anthropic. How Claude Code works. https://code.claude.com/docs/en/ how-claude-code-works, 2026. Accessed: 2026-10-12. [8] OpenAI. Codex CLI. https://github.com/openai/codex, 2026. Accessed: 2026-10-12. [9] Earendil Works. Pi agent harness. https://github.com/earendil-works/pi, 2026. Accessed: 2026-10-12. [10] Yilun Yao, Xinyu Tan, Chao-Hsuan Liu, Yaoming Li, Zhengyang Wang, Wenhan Yu, Zhewen Tan, Yuxuan Tian, Guangxiang Zhao, Lin Sun, Xiangzheng Zhang, and Tong Yang. HarnessBench: Measuring harness effects across models in realistic agent workflows. CoRR, abs/2605.27922, 2026. [11] Naman Vats and Oleg Golev. The scaffold effect in coding agents: Harness choice as a hidden variable in coding-agent evaluation, 2026. URL https://arxiv.org/abs/2607.22585. [12] Sydney Lewis. Same model, different harness: Different coding-agent results, 2026. URL https://arxiv.org/abs/2608.26218. [13] Junjie Wang, Yiming Ren, and Haoyang Zhang. From procedural skills to strategy genes: Towards experience-driven test-time evolution. CoRR, abs/2604.15097, 2026. [14] Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In ICML, volume 235 of Proceedings of Machine Learning Research, pages 50208–50232. PMLR / OpenReview.net, 2024. [15] Nous Research. Hermes Agent Documentation. https://hermes-agent.nousresearch. com/docs/, 2026. [16] Seth Karten, Alex L. Zhang, Kevin Thomas, Sebastian Müller, Elie Bakouch, Daniel Auras, Mika Senghaas, Fares Obeid, Konstantin Dunas, Johannes Hagemann, and Sami Jaghouar. Prime agent: A self-improving RLM harness. CoRR, abs/2608.23552, 2026. 13

[17] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In NeurIPS, 2023. [18] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. CRITIC: Large language models can self-correct with tool-interactive critiquing. In ICLR, 2024. [19] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying llm-based software engineering agents. Proc. ACM Softw. Eng., 2(FSE):801–824, 2025. [20] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. SWE-bench: Can language models resolve real-world github issues? In ICLR, 2024. [21] Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, Elsie Nallipogu, Qingwei Lin, Yingnong Dang, Saravan Rajmohan, and Dongmei Zhang. SWE-bench goes live! In NeurIPS, 2025. [22] Chang Ma, Junlei Zhang, Zhihao Zhu, Cheng Yang, Yujiu Yang, Yaohui Jin, Zhenzhong Lan, Lingpeng Kong, and Junxian He. AgentBoard: An analytical evaluation board of multi-turn LLM agents. In NeurIPS, 2024. [23] Harbor Framework Team. Harbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments, 2026. URL https://doi.org/10.5281/zenodo. 20953922. [24] Anthropic. Introducing Claude Opus 5. claude-opus-5, 2026.

https://www.anthropic.com/news/

Introducing Claude Sonnet 5. [25] Anthropic. claude-sonnet-5, 2026.

https://www.anthropic.com/news/

[26] OpenAI. GPT-6 Astra: A new generation of intelligence. https://openai.com/index/ gpt-6-astra/, 2026. [27] Z.ai. GLM-5.3: Frontier coding with emergent cyber capabilities. https://z.ai/blog/glm-5. 3, 2026. [28] Kimi Team. Kimi K3: Open frontier intelligence. CoRR, abs/2607.24653, 2026. [29] DeepSeek-AI. DeepSeek-V4: Towards highly efficient million-token context intelligence. CoRR, abs/2606.19348, 2026. [30] Aider. o1 tops aider’s new polyglot leaderboard. polyglot.html, 2024.

https://aider.chat/2024/12/21/

[31] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general AI assistants. In ICLR. OpenReview.net, 2024. [32] Lin Shi, Haowei Lin, Zixuan Zhu, Xiaoyue Zhou, Xiang Li, Xiangning Lin, Yaxuan Deng, Han Xu, Yuangang Li, Shanda Li, Zizhao Chen, Hanwen Xing, Harsh Raj, Bo Chen, Quan Shi, Steven Dillmann, Yipeng Gao, Puneesh Khanna, Ruofan Lu, Chao Beyond Zhou, Michael Yang, Robert Zhang, Siyuan Chai, Jiayu Chang, Yizhao Chen, Xiaokun Chen, Yiwei Dai, Wenting Yang, Hange Liu, Minghao Liu, Zihan Wang, Adnan El Assadi, Benedikt Stroebl, E. Kelly Buchanan, Han Meng, Junwei He, Longxuan Yu, Radin Shayanfar, Yukyung Lee, Zhikang Dong, Allen G. Hart, Anjiang Wei, Anurag Kashyap, Arpandeep Khatua, Audrey Jixin Zheng, Chengrui Ma, David Heineman, Dubing Chen, Hai-Anh Trinh, Haishuo Fang, Hefan Zhang, Hui Shen, Issa Sugiura, Jiankai Sun, Jiechao Gao, Junhong Lin, Junnan Li, Kai Yang, Lei Hsiung, Maoyu Wang, Mengze Tang, Nabil Omi, Negin Raoof, Nicholas Edwards, Octavia Guo, Orfeas Menis Mastromichalakis, Pengliang Ji, Przemyslaw Hejman, Qi Qi, Qunshu Lin, Richard Zhuang, Rui Yang, Ruichen Zheng, Ryan Marten, Shaghayegh Fazliani, Shizheng Hou, Sicong Jiang, Sijie Li, Boqin Yuan, Michael Glass, Song Bian, Terry Yue Zhuo, Tianqing Wu, Tom Tang, Wanjia Zhao, Weihao Xuan, Wenhua Liang, Xian Liu, Xin Lan, Xuan Zhang, 14

Xuandong Zhao, Yanchuan Tang, Yifan Jiang, Yijiang Li, Yitong Guan, Yizhi Li, Yonghui Liu, Yuheng Tang, Yujun Mao, and et al. Harbor adapters and harbor-index: Infrastructure and a curated meta-dataset for large-scale agentic evaluation. CoRR, abs/2609.04298, 2026. [33] Terminal-Bench Team. Terminal-Bench 2.1. terminal-bench-2-1, 2026.

15

https://www.tbench.ai/news/

Related documents

• EvoX: Rethinking the Agent Harness as a Runtime Policy#1649036
• Harness Engineering in LLM Tool Use via Agent-Native Reusable Tool Primitives#822166
• OpenRE-Bench: A Benchmark Harness for Multi-Agent Requirements Engineering with Mined GitHub Issues#1647004
• OpenRE-Bench: A Benchmark Harness for Multi-Agent Requirements Engineering with Mined GitHub Issues#1616955
• PDEU-Bench: Benchmarking the Personalized Planning Lifecycle of Tool-Calling LLM Agents#1129540