The Last Mile to the Real World: Agent-Native Primitives as Reusable Building Blocks for Solving Real-World Tasks

The Last Mile to the Real World: Agent-Native Primitives as Reusable Building Blocks for Solving Real-World Tasks ♢ F OUNDATION · A DAPTATION · C OMPOSITION · E VOLUTION ♢ Haibo Jin1

Peng Kuang1

Xiaopeng Yuan1

Dehao Wu1

Runguo Li1

Colin Su1

arXiv:2611.07461v1 [cs.CL] 5 Nov 2026

1

Lucheng Fu2†

Jingxuan Sun1

University of Illinois Urbana-Champaign †

Work done at UIUC

2

Jerry Wang1

Xuyang Liu1

Xucheng Yu1

Haojing Luo3

Georgia Institute of Technology

#

Yaoning Yu1

Haohan Wang1# 3

heelo.dev

Corresponding Author

Website: https://heelo.dev

Abstract How can an agent handle a real-world task end to end? Despite recent progress in LLM agents, frontier models paired with the harnesses built for them still fall short on public benchmarks of real-world tasks: on Terminal-Bench 4.0 at maximum reasoning effort, GPT-5.6 Sol with Codex resolves 37.3% of the tasks and Claude Opus 5 with Claude Code resolves 51.8%. These tasks fail at the step from reaching a resource to applying it, which we call the last mile to the real world: a tool call must ground its arguments, often in an earlier call’s output, each result must reach the steps that consume it, and a failure must be traced back to the work it implicates. Existing approaches put the resource within reach but leave the competence to apply it with the caller, so each harness rebuilds that competence. To address this gap, we introduce agent-native primitives, reusable units that equip a resource with a resident LLM, which applies it in context and communicates with other primitives, and HEELO (Harness Engineering around Evolving AgentNative Primitives for the Last Mile to Real-WOrld Tasks), a harness engineering framework that builds each task’s harness from a library of such primitives. HEELO composes the primitives a task needs into a task-specific graph, verifies the result against evidence from the task’s environment and revises the primitives that a failure implicates, and admits new or improved primitives into the library after validation. Results proved in our companion studies bound activation by the task’s demand, however large the library, and identify when per-task composition beats the best fixed harness. Each task domain, however, places its own demand on the last mile, so we instantiate HEELO with eight primitive families, each built as the solution to one kind of real-world task, and evaluate them on more than twenty benchmarks. On each comparator’s backbone, Dev-Primitives raise Terminal-Bench 4.0 resolution from 37.3% under Codex to 55.5%. Over the mini-SWE-agent seed harness, Harness Primitives composed per task raise Pass@1 from 73.0% to 80.5% on 100 SWE-bench Verified tasks and from 60.0% to 72.2% on 45 Terminal-Bench 2.0 tasks, above the 79.0% and 56.7% of Codex. Tool Primitives raise ToolBench pass rate from 55.8% to 75.1%, Code Primitives raise repository delivery from 0.3180 to 0.5134, Agent Primitives raise mean accuracy over eight math, code, and QA benchmarks from 58.8% for a single agent to 75.3%, and Navigation Primitives raise GAIA-Text-103 accuracy from 45.6% to 58.3%. In formal theorem proving, Math Primitives place an annotated reference method in the top five for 95.0% of held-out Putnam problems, against 84.2% for the strongest reranker. Further analyses show that the benefit comes from applying each resource inside its primitive and that the task decides which primitives run. Beyond these benchmarks, we built heelo.dev, a platform of agent-native primitives for real-world actions. On its 46 real-world tasks, Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex complete 9 each on their own and 39 and 35 when connected to HEELO through our MCP server, with 1.9× and 2.3× less time and 1.55× lower cost per completed task. HEELO provides a unified framework for building agents that handle real-world tasks end to end. Keywords: LLM Agents, Agent-Native Primitives, Harness Engineering, Real-World Tasks, LLM Tool Use

1

The Last Mile to the Real World Public benchmarks across the eight primitive families vs Codex

SWE-bench Verified

2

vs Codex 3

SWE-bench Verified

75

4

vs Claude Code

Putnam hit@5

25

vs LatentMAS 21

AIME 2024 20 AIME 2025 19

8

vs LatentMAS

18

20

17

10

NarrativeQA

15

vs RAPTOR

14

AppWorld

vs HippoRAG 2

SWE-bench Verified* vs Codex

$0.1

70

MAS-GPT SelfConsistency

single 65 agent

5k

10k

20

20k

HEELO

HEELO, GPT-OSS-20B

40

Agentless

$1.0

$1.5

$2.0

$2.5

GAIA-Text-103

vs SWE-QA-Pro Agent

Tool

Harness · building a harness token cost in actor attempts LongAgent CoA HEELO

$0.3

from scratch one actor attempt

$0.1

HEELO

$1 2.6× cheaper

at 512K

32K

Navigation

64K 128K

512K

23.35× 17.23× 1× 1×

540–640× less

0.043× 0.027×

0.01

0.1

1

10

4

5

6

7

8

9

10

11

12

13

14

Math · formal theorem proving

Knowledge · same budget

85.8

49.9

52.1

56.7

79.0

70.0

73.4

40.0

45.6

77.6

33.2

Putnam dark, IMO light; 10 compilations

accuracy at equal evidence tokens

91.0

55.4

57.1

72.2

80.5

72.7

75.1

44.0

58.3

79.8

38.2

+5.2

+5.5

+5.0

+15.5

+1.5

+2.7

+1.7

+4.0

+12.7

+2.2

+5.0

3

Baseline 37.3 51.8 76.4 HEELO 55.5 53.0 85.6 Gain +18.2 +1.2 +9.2 Knowledge

75

cost per query, 32K to 512K tokens

vs GPT-5.4

Harness

2.5× fewer tokens than MAS-GPT

HEELO

Navigation · growing space

ToolBench

SWE-QA-Pro

Dev

$0.03

τ²-Bench

vs iterative retrieval

RepoProbe

vs iterative retrieval

$0.01

delivery vs cost per task 60 1.6× Claude Code at lower cost

NESTFUL

12

13

50 $0.003

$5k

Code · repository construction

vs Codex

vs Claude-4.6-Sonnet

11

60

accuracy vs tokens per query

cost per query

16

HEELO Claude-4.6-Sonnet

70

Agent · multi-agent systems

Terminal-Bench 2.0

vs Claude-4.6-Sonnet

vs RAPTOR

$1k

2k

9

vs AWM

Baseline HEELO Gain

vs Claude Code

7

2

40

$0.2k

DevOps-Gym

6

vs LatentMAS

1

60

vs Codex

5

accuracy (%)

50

GPQA-Diamond

MultiDoc2Dial

Claude Code's best at 1.8× lower cost

DevOps-Gym

vs LLM reranker 22

APIBench

Pass¹ vs cost; HEELO on Qwen3-8B 6.8× cheaper

vs Claude Code

1

23

resolved vs total cost, 37 setups

Terminal-Bench 4.0

24

IMO ProofBench hit@5

vs LLM reranker

26

25

Tool · τ²-Bench

resolved (%)

RepoZero

vs execution feedback

HEELO

other baseline

Pass¹ (%)

Terminal-Bench 4.0

Codex

delivery (×100)

RepoCraft

vs execution feedback

RepoGenesis

vs execution feedback

Claude Code

Dev · Terminal-Bench 4.0

Agent

Math

Code

15

16

17

18

19

20

21

22

23

24

25

26

77.4

68.1

71.2

62.8

53.3

56.7

45.5

84.2

88.3

53.8

41.0

43.6

81.5

80.2

77.6

70.4

73.3

76.7

59.6

95.0

91.7

62.9

48.2

50.1

+4.1

+12.1

+6.4

+7.6

+20.0 +20.0 +14.1 +10.8

+3.4

+9.1

+7.2

+6.5

0.8 1.7 2.5

pass@20 APOLLO AxProverBase HEELO

2.2× AxProverBase 6.7 5.8

10 12.5 15

0

5 10 15 proof success (%)

77 76.2 77.8 73.8 72.5 73.5

MultiDoc2Dial DMV MultiDoc2Dial VA SWE-QA Sphinx APIBench HF

20

56.7 52.9

40

60 80 accuracy (%)

100

* 100 sampled tasks, the Harness Primitives setting

(a) Score (%): HEELO leads all 26 comparisons

(b) Efficiency in every primitive family

heelo.dev: 46 real-world tasks in four loops Claude Code

ion

25

6

Scraping 20 compliance

Receipt 18 rendering 17 an t Invoice fields & checks 16 15 Cost 14 routing Mail & attachments Lead discovery 1

2

3

4

5

on

t sis As

9

3.4 min · $0.80

8.0 min · $0.65

3.5 min · $0.42

time and cost per completed task

Time per task (s)

Outreach pipeline Sequenced sending

Consent & compliance Contact finding Source coverage

6

12

7

8

9

10

11

12

13

14

6 20 3 20

25

26

All

Claude Code 0 0 0 0 0 50 33 0 0 0 100 + HEELO 100 100 100 100 100 100 100 100 57 0 100 Codex 0 0 0 0 0 33 67 0 100 100 0 + HEELO 100 100 100 0 80 100 100 100 100 67 0

42 75 29 29

28 90 25 83

16

17

18

19

20

21

22

23

24

1000 100

hollow: not every task done

Cost per task ($)

Share-card rendering

Claude Code 0 0 0 56 0 77 0 0 0 75 50 0 100 + HEELO 100 100 100 100 100 83 100 100 100 100 100 100 100 Codex 0 17 0 39 0 0 33 0 0 50 50 0 100 + HEELO 100 100 100 56 100 100 100 100 100 100 100 100 100 15

35/46 done

6.3 min · $1.24

iti 10 Deliverability uis Acq 11

13

Codex + HEELO

9/46 done

pipeline

8

Codex

39/46 done

7 Launch

Review & 19 approval

Claude Code + HEELO

9/46 done

filled: every task done

ut ib

Crawling & extraction 21

Va lid a

Page publishing End-to-end validation Click Funnel attribution math Safety & Lead 26 1 25 2 input checks hand-off 24 3 Multi-platform Problem Di mining 23 n str 4 posting 75 tio Request Budget audit log 22 5 control 50

alone → with HEELO

Time per task (s)

Cost per task ($)

1

DISTRIBUTION Page publishing

15

ASSISTANT Mail & attachments

2

Click attribution

16

Cost routing

3

Safety & input checks

17

Invoice fields & checks

4

Multi-platform posting

18

Receipt rendering

5

Budget control

19

Review & approval

6

Share-card rendering

7

Launch pipeline

20

VALIDATION Scraping compliance

ACQUISITION Outreach pipeline

21

Crawling & extraction

8

22

Request audit log

9

Sequenced sending

23

Problem mining

10

Deliverability

24

Lead hand-off

11

Consent & compliance

25

Funnel math

12

Contact finding

26

End-to-end validation

13

Source coverage

14

Lead discovery

10

0.01 0.1

← longer

(c) Task score (%): Claude Code 28 → 90, Codex 25 → 83 with HEELO

1

1000 100

costlier →

← longer

10

0.01 0.1

1

costlier →

(d) Time and cost per task: alone → with HEELO

Figure 1: HEELO on public benchmarks and on the real-world tasks of heelo.dev. (a) Each spoke compares HEELO with one baseline: Codex or Claude Code on the same backbone for Dev and Harness Primitives, and otherwise the strongest baseline of the setting. (b) Efficiency per family. (c, d) Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex run the 46 tasks on their own and connected to HEELO through MCP; in (d), time and cost average a capability’s tasks, and a marker is filled when every task of its capability is done.

2

The Last Mile to the Real World

Introduction

1. Introduction A central objective of LLM agents is to complete real-world tasks end to end: to act on the world through tools, code, proofs, documents, and other agents until the task’s own acceptance criteria are met [1–3]. Recent advances in language models and in the harnesses built around them have improved this ability, as GPT acts through Codex [4] and Claude through Claude Code [5]. Through these harnesses an agent reaches tools and APIs [6, 7], code in large repositories [8, 9], formal proof libraries [10, 11], documents and memories [12, 13], and other agents [14, 15]. Meanwhile, benchmarks increasingly judge agents by whether the real task succeeds: the tool calls fulfil the request, the repository’s tests pass, the answer is correct [6, 16–18]. Existing studies have made resources easier to reach and to reuse through tool learning [2, 6, 19, 20], retrieval and memory [12, 21–23], skill and code libraries [24–26], and multi-agent frameworks, workflow search, and harness generators [15, 27–30], providing an important foundation for agents that act in practical environments. Despite this progress, frontier models in the harnesses built for them still fall short on public benchmarks of realworld tasks. On Terminal-Bench 4.0 at maximum reasoning effort, GPT-5.6 Sol with Codex resolves 37.3% of the tasks and Claude Opus 5 with Claude Code resolves 51.8%, and on the four stages of DevOps-Gym the same two pairs average 49.9% and 52.1% success. In the airline domain of 𝜏 2 -Bench [31], Claude-4.6-Sonnet succeeds in all four trials on half of the tasks. Tasks as ordinary as booking a flight under an airline’s policy or filling in a form whose later fields depend on earlier answers fail at the step from reaching a resource to applying it, which we call the last mile to the real world. Its difficulty shows in three aspects. Application in context: a reached resource must be applied to the task’s own objects, as a tool call must ground every argument, often in an earlier call’s output, and respect the rules of its environment [32–34], retrieved code must be adapted to its target and to the components that depend on it [35, 36], a proof method applies only where its conditions hold [37, 38], and an answer drawn from documents must stay faithful to its source [39, 40]. Composition: what one step returns or requires must reach the steps that consume it, in a form they can use. Evidence-driven revision: a failure observed in the task’s environment must be traced back to the work it implicates, so that only this work is redone. Together, these aspects extend the last mile from a single call to the whole harness. However, existing approaches keep the resource but leave the competence to apply it with the caller, or fold it into one monolithic agent, so the harness rebuilds this competence for every task: what the caller must read grows with the library, what it learns ends with the task, and its results are hard to check. This raises a natural question: how can an agent handle a real-world task end to end? To address this gap, we argue that the competence to apply a resource should travel with the resource itself, and we introduce agent-native primitives, each a reusable resource equipped with a resident LLM. The primitive extends reusable software components, temporally extended options, and design by contract [41–43] to LLM-based units. The resident LLM is what makes the primitive agent-native: it gives the resource a natural-language interface, applies it to each request in context, and communicates requirements to other primitives. Passive artifacts thus become active participants that talk to one another, and the primitive’s interface lets a harness find and invoke it without reading the resource. Primitives alone do not determine which of them a task needs, how their results reach one another, which to revise when the task fails, or which new primitives to admit. We therefore introduce HEELO (Harness Engineering around Evolving Agent-Native Primitives for the Last Mile to Real-WOrld Tasks), a harness engineering framework that builds each task’s harness, everything the task needs around the model, from an evolving library of primitives through three mechanisms. First, library selection and composition finds candidates through their interfaces or their own relevance assessments and composes the few the task needs into a task-specific graph that carries each result and message to its consumer. Second, evidence-based execution and improvement runs the graph in rounds, verifies the result against evidence from the task’s environment, and traces each failure to the primitives it implicates, so that revision redoes their work and keeps the rest. Third, validation and publishing improves the library between tasks, admitting a new or improved primitive once evidence such as tests, a compiler, or development runs confirms

3

The Last Mile to the Real World

Introduction

it. Results proved in our companion studies show that need-driven activation stays bounded by the task’s demand, however large the library grows, and identify when per-task composition beats the best fixed harness. However, each task domain places its own demand on the last mile: cooperating agents must combine their work, a repository change must reach every component it affects, a search must follow what remains unresolved, and a harness must fit its task. We therefore instantiate HEELO with eight primitive families, each built as the solution to one kind of real-world task: Tool Primitives for LLM tool use [44], Code Primitives for repository construction [45], Math Primitives for formal theorem proving [46], Knowledge Primitives for knowledge-intensive tasks [47], Agent Primitives for multi-agent systems [48], Dev-Primitives for software engineering [49], Navigation Primitives for large information spaces [50], and Harness Primitives for agent harnesses [51]. Each family realizes the definition above and the mechanisms its task needs, so the eight companion studies become instances of one framework. For these families, we build 25,519 Tool Primitives, 16,464 of them wrapping live APIs; 1,424 Code Primitives, each validated against the tests it carries; and 82 Math Primitives whose Lean artifacts compile. We also introduce a benchmark of 522 repository-reconstruction tasks drawn from real Python projects in seven domains. Extensive experiments on more than twenty benchmarks demonstrate that, on the same backbones, Dev-Primitives outperform Codex by 18.2 points on Terminal-Bench 4.0, and Harness Primitives composed per task exceed the mini-SWE-agent seed harness by 7.5 and 12.2 points and Codex by 1.5 and 15.5 points on 100 SWE-bench Verified and 45 Terminal-Bench 2.0 tasks, for a composition cost of 2.7% to 4.3% of one actor attempt. Code Primitives surpass Claude Code on its backbone by 56.2% in repository delivery and keep 95.1% of their delivery at 24.0% lower cost with GPT-OSS-20B adaptation and diagnosis. Tool Primitives on Qwen3-8B lead every reference model on 𝜏 2 -Bench at up to 7.7× lower cost than GPT-5.4 and complete 2.6–3.8× as many of 50 curated real-world tasks as GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro calling the same tools. Agent Primitives on Qwen3-8B exceed the strongest prior multi-agent system by 7.5 points with ∼3–4× fewer tokens than a text-based one, Navigation Primitives lead the strongest GAIA-Text-103 baseline by 12.7 points, and with GPT-5.6 Luna, Knowledge Primitives gain 14.3 points over Hybrid RAG on average and Math Primitives lead the strongest reranker by 10.8 points in Putnam hit@5. Further analysis across families reveals that the gain lies inside the primitive and that the task decides which primitives run: Tool Primitives more than double the ToolBench pass rate of budget-matched selfrefinement over raw schemas, and active Navigation Primitives grow 1.23× while distractors grow the information space 16× around the same evidence. Beyond these research benchmarks, we build heelo.dev, a platform through which agents such as Claude Code, Codex, or Cursor invoke real-world actions as agent-native primitives, from publishing pages and sending outreach to reading invoices and scraping sources. Its 46 real-world tasks cover the 26 capabilities of four end-to-end loops, with the pass criteria of each task fixed before its first run. On their own, Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex, both at medium reasoning effort, complete 9 of the tasks each, with mean task scores of 28% and 25%. Connected to HEELO through our MCP server, Claude Code completes 39 and Codex 35, with scores of 90% and 83%. Per completed task, Claude Code with HEELO needs 3.4 minutes and $0.80 instead of 6.3 minutes and $1.24 on its own, and Codex with HEELO 3.5 minutes and $0.42 instead of 8.0 minutes and $0.65. HEELO thus provides a unified framework for handling real-world tasks end to end. Our main contributions are summarized as follows: • We introduce agent-native primitives under one definition that pairs a reusable resource, from tools, code components, and proof methods to source models, multi-agent patterns, and repository files, with a resident LLM that applies it in context and messages other primitives. We further build libraries of 25,519 Tool Primitives, 1,424 validated Code Primitives, and 82 Math Primitives with Lean-checked scaffolds. • We propose HEELO, one harness engineering framework that unifies the eight families: each task’s harness is a graph composed from an evolving library and revised where the task’s evidence implicates it, and the library grows between tasks with validated primitives, so that a new kind of task can join as a new primitive family.

4

The Last Mile to the Real World

Related work

• Experiments on more than twenty benchmarks across eight kinds of real-world tasks show that every family improves on its reference baseline: primitives lead Codex by 18.2 points on Terminal-Bench 4.0 and Claude Code by 56.2% in repository delivery, and cost up to 7.7× less than GPT-5.4 on 𝜏 2 -Bench. Cross-family analyses locate the gain inside the primitive and show that the task decides which primitives run. • We build heelo.dev, a platform that exposes real-world actions, from flights and calendars to email and browser automation, as agent-native primitives to agents such as Claude Code, Codex, or Cursor. On its 46 real-world tasks in four end-to-end loops, Claude Code and Codex complete 9 tasks each on their own and 39 and 35 when connected to HEELO through MCP, with 1.9× and 2.3× less time and 1.55× lower cost per completed task.

2. Related Work Tool Learning and Interfaces. LLM agents learn to call tools from self-annotated data [2], large collections of real-world APIs [6, 7], and synthesized single- and multi-turn function-calling data [19, 52, 53], while compact models make function calling robust on device [54]. Benchmarks span structured calls and tool-augmented dialogue [20, 55], large-scale, compositional, and nested call sequences [32, 56, 57], and stateful, conversational environments [31, 33, 34, 58], while AgentDojo evaluates tool-using agents under adversarial prompt injection [59]. OpenAPI, the Model Context Protocol, Agent2Agent, and Agent Skills make services, tools, agents, and packaged instructions discoverable [60–63]. Across these settings, the caller binds each call itself: it writes schema-compliant arguments and grounds each value in the intended object, often an earlier call’s output. Tool Primitives move schema binding into the tool, so that the caller states its need in natural language and a resident LLM conditioned on the tool’s schema produces the typed call. Code Reuse and Repositories. Software reuse made interfaces, dependencies, and adaptation cost central to building systems from components [41, 64]. Learned systems discover libraries and abstractions [65–67], make tools for later tasks [26, 68, 69], retrieve code and documentation during generation [35, 70, 71], and distill reusable procedures from agent experience [72, 73]. Repository-level agents inspect, edit, and execute code [3, 8, 9], follow explicit localization and repair stages [74, 75], or divide work among task graphs, roles, and coordinated developers [76–78]. Evaluation ranges from cross-file and repository-level generation [36, 79] and issue repair [16] to long-horizon evolution [80], repository generation and whole-repository migration [81, 82], and terminal and DevOps workflows [17, 83]. In these systems, the reused artifact stays fixed while a central agent decides how to use it, and the units of reasoning and their connections are designed in advance or kept inside one trajectory [84, 85]. Code Primitives make adaptation an operation of the reused component, and Dev-Primitives route requirements along the repository’s own dependencies. Retrieval, Memory, and Knowledge. Retrieval-augmented generation brings external content into model computation [12, 86, 87], adaptive methods decide when and what to retrieve [40, 88–91], and long-context models use evidence unevenly across positions [39]. Reading-comprehension benchmarks ask a model to combine what it reads, through discrete reasoning over paragraphs in DROP [92] and multi-hop reasoning in HotpotQA [93]. Beyond flat passages, structured representations organize a source into summary trees [21], graphs [94–96], associative or global memories [97–99], and task-conditioned structures [100]. Agent memories keep interactions [13, 101–104], feedback [22], insights [23], skills [24], and workflows or procedures [25, 105] across tasks. In large information spaces, retrieval chooses its next query from the remaining gaps [106–108], and multi-agent readers divide long inputs [109, 110] or retrieval, filtering, and planning [111–113] among workers. What persists in these methods is the corpus, a structure built from it, or experience from interaction, and when readers divide long inputs among workers, coordination grows with how the space is partitioned, regardless of what the query requires. By contrast, Knowledge Primitives keep a revisable model of one authoritative source and commit only the revisions that the source supports, and Navigation Primitives let unresolved information needs decide which primitives run. Formal Theorem Proving. Lean and Mathlib give provers a checked proof language and a large reusable library [10, 11]. LLM provers generate whole proofs or repair them from verifier feedback [114–116], and agentic 5

The Last Mile to the Real World

Related work

loops alternate generation with Lean-guided repair [117, 118]. Retrieval-augmented proving, premise selection, and declaration search supply formal facts [119–121], and lemma libraries grow across problems [122]. Such retrieval ranks a fact, a lemma, or a prior proof by relevance, and LLM rerankers that reason about fit report a score or an ordering [123–126], leaving the prover to decide how to use a candidate. Proof planning describes methods by their conditions and effects [37, 38], and adaptation-guided retrieval prefers cases that adapt to a new problem [127, 128]. Math Primitives bring this view to LLM provers: a Method Contract carries a method’s required conditions, intended action, and residual obligations, and each contract proposes its use on the goal before ranking. Multi-Agent Systems and Communication. Multi-agent frameworks organize LLMs by assigning predefined roles [15, 129], programming conversations among customizable agents [14], recruiting collaborators iteratively [130], or letting agents debate [131, 132]. Other methods design the team automatically: DyLAN selects agents by estimated value [133], GPTSwarm optimizes collaboration graphs [27], and MAS-GPT generates a configuration from the query [134]. Recent methods communicate below text by sharing a learned latent space [135], projecting or propagating key–value caches [136, 137], or aggregating experts through cross-attention [138]. For cache messages, rotary embeddings make a key’s position part of its meaning [139]. These systems take the agent or the team as the unit of design, and the latent methods mainly improve how information passes between agents. Agent Primitives make a recurring multi-agent computation pattern the reusable unit and implement its computation structure through latent KV cache communication, with all agents on one LLM so that their inputs and outputs stay aligned. Agent Harnesses and Workflows. The harness around a model decides how observations, memory, retries, and checks shape its actions: ReAct interleaves reasoning with actions [1], Reflexion and Self-Refine feed verbal feedback into later attempts [22, 140], and DSPy compiles declarative pipelines of model calls [141]. Codex and Claude Code, the platform harnesses built for GPT and Claude, supply the tools and execution loop around these frontier models [4, 5]. Design methods search over agent programs and workflows [28, 29] or edit and evaluate candidate harnesses to select one for all tasks [142, 143]. Others adapt the harness at test time, specializing it for each case from retrieved execution experience [144] or rewriting harness code and generating task-specific modules [30, 145]. These methods fix one harness across tasks, specialize it through instructions that the actor must follow, or write new control code at test time, which must then be implemented, debugged, and validated. Harness Primitives validate each control operation when it enters the library, so that a task’s harness is selected and compiled from implemented parts before the actor runs. Key Differences. Across these lines of work, the units (a function, a code component, a lemma, a passage or memory, a role or team, a harness) keep a resource but leave the competence to apply it with the caller or fix it in an agent design. An agent-native primitive keeps that competence with the resource by equipping it with a resident LLM: the primitive owns the resource, exposes an interface through which a harness finds and invokes it, applies the resource to a natural-language request in context, and communicates with other primitives; where a domain offers an exact procedure, a program replaces the resident LLM. HEELO selects and composes primitives per task, revises only what the evidence of a failure implicates, and publishes improved primitives through a validation path separate from task execution. Eight primitive families instantiate this design on eight kinds of real-world tasks. Overall, primitives let the harness of a real-world task be engineered around reusable, validated units.

3. Methodology Figure 2 gives an overview of HEELO. We first state the problem of building a harness for a real-world task and define the agent-native primitive from which the harness is built. We then describe the three mechanisms that build and run each task’s harness and improve the library between tasks, the eight primitive families that instantiate the method on eight kinds of real-world tasks, and three properties of the selection that decides which primitives run, each proved in a companion study.

6

The Last Mile to the Real World

Methodology

Agent-Native Primitive Library Tool

Code

Math

Schema–function pairs

Code components + tests

Method Contracts

Knowledge

Real-World Tasks

Source models

Agent

Dev

Multi-agent patterns

Repository components

Library Selection and Composition

Navigation

Harness

Territories

Control operations

Compositions from One Library Illustrative compositions

1

LLM tool use

Task request

Retrieve candidates through their interfaces, or read every interface at once

Context

Repository change

Find

2 Assess

3 Compose

Assess relevance:

Compose the plan: activated primitives + dependencies

Does it apply? How would it be used?

A

Edit + message

Activate

B

Constraints

Issue repair with Dev-Primitives

Execute

Diagnose

Formal proof with Math Primitives

Formal proof

Acceptance criteria

Activated primitives and their messages

Task-specific graph

Knowledge-intensive Math Knowledge Code

Dev

Method Contract

C

Prover

Large information space with Navigation Primitives

Harness Route need

Separate release path

Candidate primitive

Validate Tests / compile / source / matched runs

Add or replace in the library

Execute & Verify

Need resolved

Diagnose & Revise

Execute

Verify

Return

Diagnose

Revise

Resume

Run the plan in the task's environment

Verdict + feedback Failure Implicated primitives Guidance

Pass: return the result

Trace the failure to implicated primitives

Re-plan / reactivate / reroute

Continue execution

Primitive Library

Agent-Native Primitive

Territory worker

Improvement

Evidence-Based Execution

Validation and Publishing

Lean compiles the proof

Resource what the primitive owns (tool schema, code component, proof method, source model, ...)

Failed Check

Accept when the task's own checks pass

Resident LLM

Revise only impacted work (within a round budget)

Interface

Natural-language interface; applies the resource in context

How a harness finds and invokes it

Output Result + messages to other primitives

Figure 2: Overview of HEELO. Top: the agent-native primitive library, one box per family with the resource its primitives own. Middle: a real-world task enters as a task request with its context, constraints, and acceptance criteria; library selection and composition finds candidates through their interfaces, lets each assess its own relevance, and composes a task-specific graph whose activated primitives exchange messages, and three illustrative compositions are drawn on the right. Bottom: evidence-based execution runs and verifies the plan in the task’s environment, improvement traces a failure to the implicated primitives and revises only that work within a round budget, and validation and publishing adds a validated primitive to the library on a separate path. The bottom strip shows what an agent-native primitive holds: a resource, a resident LLM that serves as its natural-language interface, an interface through which a harness finds and invokes it, and an output of results and messages to other primitives.

3.1. Problem Formulation A real-world task 𝑞 arrives with a context 𝑐𝑞 : its inputs, its environment, its constraints, and the acceptance criteria by which it is judged (the task request of Figure 2). An agent acts on the task through a harness 𝐻, the control program between the model and the environment. The harness decides which operations run, with which inputs, in which order, and when the run stops. Running 𝐻 on 𝑞 produces a trajectory 𝜏 . The task’s own acceptance procedure, such as its tests, its proof checker, or its answer key, scores the trajectory as 𝑅𝑞 (𝜏 ) ∈ [0, 1], and we write 𝑅(𝑞, 𝐻) = E𝜏 [𝑅𝑞 (𝜏 ) | 𝐻]. A fixed harness, such as one general-purpose coding agent, uses the same 𝐻 for every task. Following our companion study on Harness Primitives [51], we build the harness of each task from a library ℒ of reusable units: 𝐻𝑞* ∈ arg max 𝑅(𝑞, 𝐻),

𝐻𝑞 = 𝐹 (𝑞, 𝑐𝑞 ; ℒ) ∈ ℋ(𝑐𝑞 ),

𝐻∈ℋ(𝑐𝑞 )

7

(1)

The Last Mile to the Real World

Methodology

where ℋ(𝑐𝑞 ) is the set of harnesses compatible with 𝑐𝑞 . The construction procedure 𝐹 approximates 𝐻𝑞* from the units of ℒ, within a budget on model calls or revision rounds. While 𝐹 builds and runs 𝐻𝑞 , it keeps a context 𝒞𝑡 for round 𝑡, which starts as 𝒞0 = 𝑐𝑞 and accumulates results and feedback. The last mile lies inside Eq. (1) at two levels. Each unit of ℒ must be applied to the task’s own objects, and 𝐹 must decide which units the task needs and how their results combine. HEELO answers the first level with agent-native primitives and the second with the mechanisms that realize 𝐹 .

3.2. Agent-Native Primitives A unit of the library helps a task only once it is applied in context. A tool call must ground its arguments, a retrieved component must be adapted to its target, a proof method must be instantiated on the goal, and an answer must stay faithful to its source. A caller that applies every unit itself rebuilds this competence for each task and must hold the details of every unit it uses in one context [44, 49]. An agent-native primitive keeps this competence with the unit by equipping the unit with a resident LLM that applies it. Definition 3.1 (Agent-native primitive). An agent-native primitive 𝑃𝑖 equips a reusable resource 𝑎𝑖 with a resident LLM ℳ and exposes an interface 𝑑𝑖 through which a harness finds and invokes it. Invoked with a natural-language request 𝑥𝑖 in a context 𝑐𝑖 , the resident LLM, conditioned on the resource, applies 𝑎𝑖 and returns a result 𝑟𝑖 together with the messages 𝑚𝑖 = {𝑚𝑖→𝑗 }𝑗 that the primitive addresses to other primitives 𝑃𝑗 : the requirements that 𝑟𝑖 places on them, or task-relevant information they need, with 𝑚𝑖 = ∅ when none is needed: (︀ )︀ (𝑟𝑖 , 𝑚𝑖 ) = 𝑃𝑖 (𝑥𝑖 ; 𝑐𝑖 ) = ℳ [ 𝑎𝑖 ; 𝑐𝑖 ; 𝑥𝑖 ] , (2) where [·] denotes the input context of the resident LLM. The resident LLM is what makes the primitive agent-native. It reads the resource and serves as its natural-language interface, sparing the caller from holding the resource’s details in its own context. The primitive’s messages carry requirements and task-relevant information to other primitives. Resources that were passive artifacts thus become active participants that reason locally and communicate with one another. The resource 𝑎𝑖 can be anything the task relies on, from a tool’s schema and function to a repository component, a proof method, or a source model, and the interface 𝑑𝑖 is what a harness reads in its place, such as a tool’s schema descriptor or the selection side of a Method Contract. One base model can serve as the resident LLM of every primitive of a library, conditioned on a different resource each time, so a primitive is defined by the resource it owns. Where a domain offers an exact procedure, as for the control operations of Harness Primitives, implemented code applies the resource and receives its request as parameters. A resource that a downstream prover or solver consumes, such as the execution side of a Method Contract or a source model, is applied by that model. Relevance Assessment. Similarity between a request and an interface does not show that a primitive applies. A primitive can therefore assess its own relevance before it is selected [45, 46]: (𝑠𝑖 , 𝑢𝑖 ) = ℳassess (𝑃𝑖 , 𝑥, 𝒞𝑡 ),

(3)

where 𝑠𝑖 judges whether 𝑃𝑖 applies to a request 𝑥 derived from the task, and 𝑢𝑖 proposes how 𝑃𝑖 would be used. Run over the whole library at once, this assessment removes the dependence on a pool that similarity retrieval chooses first (Proposition 3.3). Carried Validation. A resource may carry its own means of validation, such as tests or a schema. The primitive applies them before it returns and reports a structured failure when they fail. Carried validation shows that the primitive’s own step succeeded. Whether the result serves the task is for the harness to decide. Library. Primitives form a library ℒ = {𝑃𝑖 }𝑁 𝑖=1 . A library is either built ahead of the tasks that draw on it and reused across them, or induced from a task’s own data, such as a repository or an information space [49, 50]. A library built ahead of its tasks does not change while a task runs. An induced library follows the task’s data and can change as the task edits that data. 8

The Last Mile to the Real World

Methodology

3.3. Harness Engineering around Agent-Native Primitives Primitives determine how each resource is applied. They do not determine which primitives a task needs, how their results reach one another, or what to change when the task still fails. Two mechanisms realize the construction procedure 𝐹 of Eq. (1): library selection and composition builds the harness, and evidencebased execution and improvement runs and revises it. A third mechanism, validation and publishing, improves the library between tasks (Figure 2). Library Selection and Composition. A primitive sees only the request it is handed, so it cannot decide alone whether the task needs it. A library also usually holds more primitives than one task uses. COMPOSE therefore finds candidates through their interfaces, either by retrieval or by reading every interface at once. It reads their relevance assessments where primitives provide them and builds a plan )︀ (︀ (4) Π𝑡 = COMPOSE(𝑞, 𝒞𝑡 ; ℒ) = {(𝑖, 𝑥𝑖 )}𝑖∈𝒜𝑡 , ℰ𝑡 , where 𝒜𝑡 ⊆ {1, . . . , 𝑁 } is the activation set, 𝑥𝑖 is the local request of 𝑃𝑖 , and ℰ𝑡 ⊆ 𝒜𝑡 × 𝒜𝑡 holds the dependencies along which results and messages flow. The plan is the task-specific graph of Figure 2. Only the activated primitives run, so the cost of applying primitives follows |𝒜𝑡 |, however large 𝑁 is. A dependency carries an earlier result to the request that consumes it, a message to the primitive that must act on it, or internal state between models that share parameters. A plan may also hand a selected resource to the prover or solver that consumes it. A need that no primitive covers is left to the harness itself. Evidence-Based Execution and Improvement. A primitive can succeed at its own step while the task still fails, and the task’s environment reports such a failure without naming the primitive that should change. HEELO therefore runs the plan in rounds that end in evidence and revises what the evidence implicates. In round 𝑡, the activated primitives act, and verification judges the integrated result: (︀ )︀ (𝑟𝑖 , 𝑚𝑖 ) = 𝑃𝑖 𝑥𝑖 ; 𝑐𝑡𝑖 , 𝑖 ∈ 𝒜𝑡 , (︀ )︀ (5) 𝑜𝑡 = EXECUTE 𝑆𝑡 , (𝑣𝑡 , 𝜑𝑡 ) = VERIFY(𝑞, Π𝑡 , 𝑆𝑡 , 𝑜𝑡 ), 𝜑𝑡 = (𝑒𝑡 , ℐ𝑡 , 𝑢𝑡 ). Here 𝑐𝑡𝑖 holds what the plan routes to 𝑃𝑖 : the results of its predecessors in ℰ𝑡 , the messages 𝑚→𝑖 = {𝑚𝑗→𝑖 }𝑗∈𝒜𝑡 addressed to it, and the part of the task state it needs. Each primitive changes only what it owns. Messages reach their recipients before the round ends, so a requirement found while one primitive works can still be met in the same round by the primitives that depend on it. The task state 𝑆𝑡 integrates the round’s results into the task’s artifacts, such as a repository, a proof, or an answer. EXECUTE runs what the task can run and records evidence 𝑜𝑡 , such as build status, test outcomes, compiler messages, and traces. Where nothing runs, the results themselves are the evidence. VERIFY turns the integrated state and its evidence into a verdict 𝑣𝑡 ∈ {PASS, FAIL} and structured feedback 𝜑𝑡 . The feedback names the observed failures 𝑒𝑡 , the primitives ℐ𝑡 ⊆ {1, . . . , 𝑁 } that they implicate, and revision guidance 𝑢𝑡 . A failed build or test sets 𝑣𝑡 = FAIL, and no model judgment overrides it. Verification can also judge each step as soon as it completes, or run inside the plan as a primitive of its own. On a failure, the feedback enriches the context, and COMPOSE revises the plan: (︀ )︀ 𝒞𝑡+1 = 𝒞𝑡 ∪ {𝜑𝑡 }, Π𝑡+1 = COMPOSE 𝑞, 𝒞𝑡+1 ; ℒ, Π𝑡 , 𝜑𝑡 . (6) Revision changes only what 𝜑𝑡 implicates. It re-plans the affected steps, reactivates the implicated primitives or activates those the evidence newly implicates, or redirects an unresolved need. It keeps the work that the new evidence does not contradict. Revision can also precede execution: a plan that can be checked before it runs is corrected from its own diagnostics within a bounded number of attempts, and a fallback plan is kept if correction fails. The loop ends when 𝑣𝑡 = PASS or after 𝐵 revision rounds and returns the current result. Validation and Publishing. A library built ahead of its tasks is reused across them, so what enters it must be validated first, independently of the tasks it will serve. New and improved primitives may be mined from existing 9

The Last Mile to the Real World

Methodology

artifacts, harvested from runs whose results were verified in context, or proposed from failed trajectories. They enter the library only on the separate path of Figure 2: {︀ }︀ ℒ𝑘+1 = ℒ𝑘 ⊕ 𝑃˜ ∈ 𝒫˜𝑘 : VALIDATE(𝑃˜ ) = PASS , (7) where 𝒫˜𝑘 holds the candidates of stage 𝑘, and ⊕ adds a new primitive or replaces the one that 𝑃˜ revises. VALIDATE checks each candidate with the evidence that its domain affords, such as its tests, a compiler, its source, or matched runs on development tasks. An output produced inside a task, such as an adapted component, enters the library only through this path. Together, Eqs. (4)–(7) define the per-task harness and the evolving library that make up HEELO.

3.4. Eight Primitive Families The method above is shared across tasks, but each task domain places its own demand on the last mile: cooperating agents must exchange their intermediate state, a repository change must reach the components that depend on it, a search must follow the needs that remain unresolved, and a harness must fit its task. We therefore instantiate HEELO with eight primitive families, each built as the solution to one kind of real-world task, and the primitives of a family own the resource that its task applies, from the tools of LLM tool use and the components of repository construction to the proof methods of formal theorem proving and the control operations of agent harnesses. The families differ in how they specialize the method, as summarized below in each family’s own terms and notation, so that a symbol such as 𝐹 , 𝑀 , or 𝑅 keeps the meaning of its own study within its paragraph. We provide the full framework of each family, with its overview figure, in Appendix B. Tool Primitives. A Tool Primitive [44] wraps one ToolFace tool: its resource 𝑎𝑖 is the schema–function pair (𝑠𝑖 , 𝑓𝑖 ), and its interface 𝑑𝑖 is the schema descriptor. The shared model conditioned on the schema, 𝒫𝑖 (𝑥; 𝑐) = ℳ([𝑠𝑖 ; 𝑐; 𝑥]), maps the request 𝑥 to the arguments of 𝑠𝑖 and invokes 𝑓𝑖 , returning the result with a summary or a structured failure. HEART is the harness engineering framework around Tool Primitives, and its COMPOSE has three parts: request decomposition asks the user for missing details and writes an invocation plan, semantic search over schema descriptors retrieves candidate tools for each step, and value resolution grounds each step’s values in the query and earlier results; each step’s result is the context 𝑐 of the next. VERIFY is per-step result validation on four criteria, and targeted revision re-plans from its feedback within 𝐵 = 3 rounds. ToolFace holds 25,519 tools with manually authored schemas, each inducing a Tool Primitive. Code Primitives. A Code Primitive [45] owns an executable component 𝑎𝑖 = (𝐶𝑖 , 𝐼𝑖 , 𝐷𝑖 , 𝑉𝑖 , 𝑋𝑖 ) of implementation, contract, dependency closure, carried tests, and provenance; the contract 𝐼𝑖 is its interface 𝑑𝑖 , and a resident LLM 𝑀𝑖 assesses and adapts the component. LEGO constructs repositories from these primitives and task-specific code, with Retrieval and Activation over the CodeFace library as its COMPOSE: DECOMPOSE writes capability requirements, the top two candidates for each requirement assess their relevance (Eq. (3)), and MATCH activates the smallest mutually compatible set. In Primitive Adaptation and Collaboration, each activated primitive adapts its own component into a target-specific variant 𝑃̃︀𝑖 , reruns 𝑉𝑖 , and sends its consumers the requirements 𝜇𝑖→𝑗 the rewrite imposes. Execution Feedback extends the target’s tests with synthesized ones, DIAGNOSE realizes VERIFY, and revision reactivates the implicated primitives within five rounds per module. CodeFace, mined from repositories and validated in isolation, grows with implementations harvested from verified construction runs. Math Primitives. A Math Primitive [46] packages a reusable proof method as a Method Contract 𝑎𝑖 with parameters 𝑥𝑐 , required conditions Pre𝑐 , a target 𝐺𝑐 with its action, a Lean scaffold 𝑆𝑐 , and residual obligations Obl𝑐 . The contract’s selection side is the interface 𝑑𝑖 , and its execution side goes to the prover. COMPOSE is self-advertised method selection: in one batched call, each contract in the library proposes its use on the goal as a target, an action, and required facts (Eq. (3)). A commitment gate demotes a contract that claims the main role with a vague or unsupported proposal to a supporting role, a tiered ranking then orders the contracts, and an optional runoff re-ranks the top ten. The prover applies the top-ranked contract; Lean compiles each attempt, and its errors drive whole-proof repair within ten compilations. The 82 contracts are built once from Putnam 2000–2014 solutions and admitted after their Lean artifacts compile (Eq. (7)). 10

The Last Mile to the Real World

Methodology

Knowledge Primitives. Each authoritative source 𝐷 has one Knowledge Primitive [47], whose resource 𝑎𝑖 is the source model 𝑀 of 𝐷, organized around its entities with units citing 𝐷, and a task grounded in 𝐷 invokes it. SourceLearn constructs and progressively refines these source models, and at task time it realizes COMPOSE as source-model activation 𝐴𝐵 (𝑀, 𝑞), the whole model if it fits 𝐵 = 24k tokens and its task-relevant regions otherwise; (︀ )︀ the solver 𝐹 applies the activated model beside evidence retrieved from 𝐷, 𝑦ˆ = 𝐹 𝑞, 𝑅(𝐷, 𝑞), 𝐴𝐵 (𝑀, 𝑞) . The source model stays fixed during testing and improves on the publishing path of Eq. (7): Self-Directed Source Learning deepens unresolved aspects of the initial model and connects knowledge fragmented across entities, and Task-Guided Source Learning revises the regions implicated by incorrectly answered guidance tasks and recalibrates 𝑀 under a learned representation policy. Grounded source reconstruction realizes VALIDATE, rejecting a rewrite unless the source supports it and previously supported meaning is preserved. Agent Primitives. An Agent Primitive owns a recurring multi-agent computation pattern as its resource 𝑎𝑖 and encapsulates it as a reusable latent operator behind the interface of a standard LLM agent [48]; the three primitives are Review, Voting and Selection, and Planning and Execution. An LLM Organizer performs COMPOSE: guided by a Knowledge Pool of 45 query–MAS-structure pairs, it maps each query to a primitive composition plan in code, and executing the plan forms a primitive-based MAS. Internal agents follow task-agnostic role prompts on one shared LLM and exchange KV caches within and across primitives, re-indexing the sender’s keys for RoPE. Verification and revision happen inside the primitives, where a Critic’s feedback drives the Solver’s refinement in Review and a Selector judges candidates in Voting and Selection. The primitives come from decomposing existing MAS designs into recurring computation patterns, and the library stays fixed. Dev-Primitives. A Dev-Primitive [49] wraps a repository component 𝑎𝑖 , by default a file, with a resident LLM, and activation reads the repository in place of an interface 𝑑𝑖 . The primitive edits its own artifact where needed and sends directed messages to the components its reasoning implicates, (𝑎′𝑖 , 𝑚𝑖 ) = ℳ([𝑎𝑖 ; 𝑥𝑖 ; 𝒞𝑖 ]), where 𝒞𝑖 holds the messages it receives. HERMES, the harness engineering framework for software engineering via Dev-Primitives, instantiates them on demand: its COMPOSE, Dynamic Primitive Activation, localizes the relevant components, expands along their dependencies, and assigns each a local objective 𝑥𝑖 . The activated primitives collaborate in one round, and task-visible checks run on the modified repository. VERIFY is Bug Diagnosis, which traces a failure to the components it implicates, and activation re-plans from this feedback within 𝐵 = 3 revision rounds. The repository induces the library, and a persistent file created during the task becomes a new Dev-Primitive. Navigation Primitives. A Navigation Primitive [50] is a scope-specialized worker whose resource 𝑎𝑖 is one territory of the information space and whose interface 𝑑𝑖 is the territory’s WorkerCard. The library is induced per query: a deterministic partition splits the space into territories, bounded searchable regions fixed during execution. ANTMAN, an adaptive coordination framework for these workers, keeps the context 𝒞𝑡 as a revisable Need Graph 𝐺𝑡 holding each need’s status, evidence, attempts, and progress, and composes one need at a time: WorkerCard retrieval and an orchestrator decision over 𝐺𝑡 route an unresolved need to at most one worker, which searches its territory and reports evidence, progress, and uncertainty. Need-resolution assessment in the update 𝐺𝑡+1 = 𝑈 (𝐺𝑡 , 𝑟𝑡 ) realizes VERIFY, judging the need resolved, partial, or unresolved, and task-local recovery reframes a stalled need, reroutes it, or sends it to a fallback path, within 𝜌 routings per need. Harness Primitives. In a Harness Primitive [51], the resource 𝑎𝑖 is an implemented control operation, fixed at test time, and the interface 𝑑𝑖 combines an application scope, stating when it helps, with a composition contract of inputs, outputs, setup, and dependencies. The framework that composes these primitives is STITCH, and its COMPOSE acts before the harness sends its first request to the actor, the task-solving model: the harness composer 𝜋 reads the whole catalog and either keeps the seed graph or returns a composition(︀intent, whose primitives the de)︀ terministic compiler 𝐾 wires into 𝐺seed with their contract dependencies, Π0 = 𝐾 𝐺seed , 𝜋(𝑞, 𝑐𝑞 ; 𝐺seed , ℒ); ℒ, 𝑐𝑞 . An invalid candidate returns its diagnostics to the composer for bounded correction, and the seed harness is kept if correction fails. During the run, verification and repair primitives return failed checks to the actor. A development agent mines the library from failed trajectories, accepting candidates that pass mechanism tests and improve matched runs (Eq. (7)). 11

The Last Mile to the Real World

Methodology

3.5. Theoretical Analysis COMPOSE decides which primitives run, so it governs both what a harness costs and what it gains. The companion studies prove three properties of this choice, which we restate in the notation of this section: how many primitives a task activates, what choosing them per task gains over a fixed harness, and what limits a shortlist. We give the proofs, with the correspondence to each study’s notation, in Appendix A. Demand-Bounded Activation. Suppose that COMPOSE activates a primitive only by routing one unresolved need of the task to it, and routes each need at most 𝜌 times, reroutes included. Let 𝐷(𝑞) count the needs realized⋃︀while 𝑞 runs, with a reframed or newly introduced need counted as new. Let 𝐾(𝑞) count the routing events, 𝒜𝑞 = 𝑡 𝒜𝑡 be the set of primitives activated during the run, and ℒ𝑞 be the library available to 𝑞. Proposition 3.1 (Demand-bounded activation [50]). Under these conditions, |𝒜𝑞 | ≤ 𝐾(𝑞) ≤ 𝜌 𝐷(𝑞),

{︀ }︀ |𝒜𝑞 | ≤ min |ℒ𝑞 |, 𝜌 𝐷(𝑞) .

and consequently

(8)

The companion study on Navigation Primitives proves the bound for workers that each search one territory of an ¯ and 𝜌 does not depend on the library, then information space. If realized demand stays bounded, 𝐷(𝑞) ≤ 𝐷, ¯ growth of the library alone does not force the active part of such a harness to grow. |𝒜𝑞 | ≤ 𝜌𝐷: Task-Conditioned Composition. The companion study on Harness Primitives asks, under a simplified additive model, when choosing primitives per task beats one fixed harness. Tasks 𝑞 follow a distribution 𝒬, and each primitive 𝑃𝑖 helps on a scope Ω𝑖 of tasks with prevalence 𝜇𝑖 = Pr𝑞∼𝒬 (𝑞 ∈ Ω𝑖 ). Activating 𝑃𝑖 yields a gain 𝑔𝑖 (𝑞) > 0 on Ω𝑖 and a penalty 𝜅𝑖 (𝑞) > 0 off it, with conditional means ¯ 𝑖 . Under valid execution, the harness ∑︀𝑁 𝑔¯𝑖 and 𝜅 𝑁 with activation vector z ∈ {0, 1} attains 𝑅(𝑞, z) = 𝑅0 (𝑞) + 𝑖=1 𝑧𝑖 ∆𝑖 (𝑞), where 𝑅0 (𝑞) is the performance without primitives and ∆𝑖 (𝑞) is 𝑔𝑖 (𝑞) on Ω𝑖 and −𝜅𝑖 (𝑞) otherwise. All combinations are admissible, and 𝑅 lies in [0, 1]. A harness 𝐻 earns 𝐽(𝑞, 𝐻) = 1[𝐻 executes validly] · 𝑅(𝑞, 𝐻), with 𝑅(𝑞, 𝐻) as in Section 3.1. For a composer whose harnesses 𝐻comp execute validly, with final activations 𝑧𝑖 , the utility-weighted false-positive and false-negative rates are 𝛼𝑖 = E[𝑧𝑖 𝜅𝑖 (𝑞) | 𝑞 ∈ / Ω𝑖 ]/¯ 𝜅𝑖 and 𝛽𝑖 = E[(1 − 𝑧𝑖 )𝑔𝑖 (𝑞) | 𝑞 ∈ Ω𝑖 ]/¯ 𝑔𝑖 . Proposition 3.2 (Fixed versus task-conditioned composition [51]). Assume 𝜇𝑖 ∈ (0, 1), 𝑔¯𝑖 > 0, and 𝜅 ¯ 𝑖 > 0 for all 𝑖, and let 𝑧𝑖* (𝑞) = 1[𝑞 ∈ Ω𝑖 ] be the oracle task-conditioned activation. (i) The best task-independent activation z*fixed ∈ arg maxz E𝑞 [𝑅(𝑞, z)] falls short of z* (𝑞) by the positive gap Γfixed below. (ii) Task-conditioned composition * outperforms the best fixed harness 𝐻fixed , which activates z*fixed , if and only if its utility-weighted selection loss is smaller than this gap: 𝑁 [︀ ]︀ [︀ ]︀ ∑︁ (︀ )︀ * * Γfixed = E𝑞 𝑅(𝑞, z (𝑞)) − E𝑞 𝑅(𝑞, zfixed ) = min 𝜇𝑖 𝑔¯𝑖 , (1 − 𝜇𝑖 )¯ 𝜅𝑖 > 0, 𝑖=1

[︀ ]︀ [︀ ]︀ * E 𝐽(𝑞, 𝐻comp ) > E 𝐽(𝑞, 𝐻fixed ) ⇐⇒

𝑁 ∑︁

[︀

(9) ]︀

𝜇𝑖 𝛽𝑖 𝑔¯𝑖 + (1 − 𝜇𝑖 )𝛼𝑖 𝜅 ¯ 𝑖 < Γfixed .

𝑖=1

A fixed harness that includes 𝑃𝑖 pays its expected out-of-scope penalty (1 − 𝜇𝑖 )¯ 𝜅𝑖 , and one that excludes it forgoes its expected in-scope gain 𝜇𝑖 𝑔¯𝑖 ; Γfixed sums the smaller of the two. The composer need not be perfect: under the additive model and valid execution, building each task’s harness from the library beats the best fixed harness built from the same primitives if and only if its selection loss stays below this gap. Coverage and Retrieval Ceiling. Let 𝒰𝑞 ⊆ ℒ be the nonempty set of primitives that serve task 𝑞, which the study of self-advertised method selection takes to be the annotated reference methods of a goal. The set is random given the information 𝑋 that a selector reads: the task with its context and the interfaces [︀ of the library. With information ]︀ 𝑍, the best achievable expected hit@𝑘 of a size-𝑘 shortlist 𝑆 ⊆ ℒ is 𝑉𝑘 (𝑍) = E max|𝑆|=𝑘 Pr(𝑆 ∩ 𝒰𝑞 ̸= ∅ | 𝑍) . 12

The Last Mile to the Real World

Experiments

Proposition 3.3 (Coverage and retrieval ceiling [46]). (i) If 𝑇 holds the features that a similarity-only selector computes from 𝑋, then 𝑉𝑘 (𝑋) ≥ 𝑉𝑘 (𝑇 ); the inequality can be strict when distinct tasks share the same similarity representation but need different shortlists, and the full interfaces distinguish them. (ii) Consider a two-stage selector that takes the top 𝑚 primitives 𝒮𝑚 (𝑞) under a similarity-only selector and then orders them by any procedure, placing the remaining primitives after them. For every 𝑘 ≤ 𝑚 and every task 𝑞, [︀ ]︀ hit@𝑘 ≤ 1 𝒮𝑚 (𝑞) ∩ 𝒰𝑞 ̸= ∅ ,

recall@𝑘 ≤

|𝒮𝑚 (𝑞) ∩ 𝒰𝑞 | , |𝒰𝑞 |

(10)

for any model in the second stage. (iii) The expected hit@𝑘 objective 𝑆 ↦→ Pr(𝑆 ∩ 𝒰𝑞 ̸= ∅ | 𝑋) is monotone and submodular, and greedy selection with the true marginal gains attains at least 1 − (1 − 1/𝑘)𝑘 ≥ 1 − 1/𝑒 of the best size-𝑘 coverage. The study of self-advertised method selection proves these results for selecting proof methods. An assessment run over the whole library, as in Eq. (3), escapes the ceiling of Eq. (10); its coverage is then limited by the information in 𝑋 and by how accurately the model judges applicability. Expected hit@𝑘 is submodular: once a shortlist holds a primitive that is likely to serve the task, a near-duplicate adds little, so the best size-𝑘 shortlist need not consist of the 𝑘 individually most likely primitives.

4. Experiments We evaluate HEELO through its eight primitive families, each on the benchmarks of its own task and against the baselines of its companion study. Three families use the terms of their own studies: the resource of a Math Primitive is a Method Contract, that of a Knowledge Primitive is a source model, and a Navigation Primitive is a territorybacked worker. We order the families by the demand that their tasks place on the last mile, from calling a tool that needs no change to fitting the harness itself to the task. Each family keeps the notation of its paper; the budget 𝐵, for instance, counts revision rounds in HEART and HERMES, Lean compilations for Math Primitives, and tokens in SourceLearn. In every table, a shaded row marks the primitive system under study, and differences between rates are in percentage points (pp).

4.1. Experimental Setup Tasks and Datasets. The eight families cover the following tasks and benchmarks. (i) LLM tool use: ToolBench, which evaluates the use of a large catalogue of real APIs [6]; NESTFUL, whose call sequences feed one call’s output into the next [32]; 𝜏 2 -Bench, which extends the tool–agent–user setting of 𝜏 -bench to dual control [31, 34]; ACEBench, which targets robustness and fine-grained tool use across complex multi-tool workflows [58]; BFCLv4, which covers agentic web search and memory function calling [20]; and 50 end-to-end real-world tasks in five domains [44]. (ii) Repository construction: the 522 frozen tasks of LEGO-REPO, across seven domains and five difficulty bands [45], and the external repository benchmarks RepoZero, RepoGenesis, and RepoCraft under their published protocols. (iii) Formal theorem proving: 120 Putnam 2015–2025 problems and 60 IMO ProofBench problems in Lean 4 with Mathlib [10, 11], held out from the Putnam 2000–2014 solutions from which the contract library was built [46]. (iv) Knowledge-intensive tasks: MultiDoc2Dial, NarrativeQA, SWEQA, APIBench, and AppWorld [7, 146–149]. (v) Multi-agent systems: AIME25, AIME24, MATH, GSM8K, HumanEval+, MBPP+, MedQA, and GPQA-Diamond [150–156]. (vi) Software engineering: SWE-bench Verified [16], Terminal-Bench 4.0 [17], DevOps-Gym [83], and whole-repository migrations from SWE Refactor Bench [82]. (vii) Large information spaces: multi-document QA over HotpotQA [93], 2WikiMultiHopQA [157], and MuSiQue [158]; RepoProbe-Python, SWE-QA-Pro, and GAIA-Text-103 [18, 159, 160]; controlled spaces of 32K to 512K tokens built on LongAgent’s multi-needle setting [109]; and a demand intervention in a fixed 512K space. (viii) Agent harnesses: sampled subsets of 100 SWE-bench Verified and 45 Terminal-Bench 2.0 tasks, with 13

The Last Mile to the Real World

Experiments

two attempts per task and development tasks disjoint from evaluation [51]. Additional details on each study’s subsets, versions, and changes to the evaluation interface are in Appendix D. Baselines. We compare each family with established methods of its task and with controls that remove or replace its primitives while the rest of the pipeline stays fixed. (i) LLM tool use: ReAct [1], retrieval-only selection, a planner–executor, and DFSDT, ToolLLM’s depth-first search over a decision tree [6], all on the backbone and tool catalogue of the Tool Primitives, together with the reference systems of its benchmarks. (ii) Repository construction: construction paradigms from single-shot generation through execution feedback, diagnosis, and retrieval to retrieval with a generic code editor, and the independently developed repository agents OpenHands, Claude Code, SWE-agent, and Agentless, each under its own harness [5, 8, 9, 74]. (iii) Formal theorem proving: for method selection, random order, BM25 [161], TF-IDF [162], and three embedding retrievers, two from OpenAI [163] and Qwen3 Embedding [164], with each embedding retriever used alone and with listwise reranking of its top ten by the selector backbone [123]; for proving, one attempt without retrieval (Base), pass@20 with twice the compilation budget, APOLLO [117], and the same AxProverBase loop without a contract [118]. (iv) Knowledge-intensive tasks: Hybrid RAG, RAPTOR [21], HippoRAG 2 [98], and Agent Workflow Memory [25]. (v) Multi-agent systems: a single agent, TextMAS with natural-language communication, LatentMAS [137] in its sequential configuration, and each single primitive applied to every query, the last three with four agents each; on Llama-3-70B-Instruct, eight existing MAS methods, among them LLM-Debate [131] and MAS-GPT [134], and the prompting baselines Chain-ofThought [165] and Self-Consistency [166]. (vi) Software engineering: Codex [4], Claude Code [5], OpenHands [9], mini-SWE-agent [8], Grok Build, SageAgent, and Aider with the same backbones. (vii) Large information spaces: direct, iterative sparse, and dense retrieval, ChainRAG [106], S2G-RAG [108], ReAct, OWL [113], the repository agents ReAct with RepoGraph [167] and RepoDistill [168], the benchmark-specific SWE-QA-Pro Agent [160], and, for long contexts, LongAgent [109] and Chain of Agents [110]. (viii) Agent harnesses: the mini-SWE-agent seed harness, SWE-agent [8], Codex, MemoHarness [144], and MetaHarness [143]. Table 1: HEART and reference systems on the six ToolBench splits. Pass and win rates (%); reference rows use the backbone in the first column and HEART uses Qwen3-8B, whose matched-backbone controls are in Table 25 [44]. Green marks the best and amber the second-best value in each column. I1-Inst. Model

Method

I1-Tool

I1-Cat.

I2-Inst.

I2-Cat.

I3-Inst.

Avg.

Pass Win Pass Win Pass Win Pass Win Pass Win Pass Win Pass Win

Vicuna

ReACT & DFSDT

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

Alpaca

ReACT & DFSDT

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

0.0

ToolLLaMA

ReACT DFSDT DFSDT-Retriever

25.0 45.0 29.0 42.0 33.0 47.5 30.5 50.8 31.5 41.8 25.0 55.0 29.0 47.0 57.0 55.0 61.0 55.3 62.0 54.5 77.0 68.5 77.0 58.0 66.0 69.0 66.7 60.0 64.0 62.3 64.0 59.0 60.5 55.0 81.5 68.5 68.5 60.8 65.0 73.0 67.3 63.1

GPT-3.5-Turbo

ReACT DFSDT

41.5 – 44.0 – 44.5 – 42.5 – 46.5 – 22.0 – 40.2 – 54.5 60.5 65.0 62.0 60.5 57.3 75.0 72.0 71.5 64.8 62.0 69.0 64.8 64.3

Claude-2

ReACT DFSDT

5.5 31.0 3.5 27.8 5.5 33.8 6.0 35.0 6.0 31.5 14.0 47.5 6.8 34.4 20.5 38.0 31.0 44.3 18.5 43.3 17.0 36.8 20.5 33.5 28.0 65.0 22.6 43.5

Text-Davinci-003

ReACT DFSDT

12.0 28.5 20.0 35.3 20.0 31.0 8.5 29.8 14.5 29.8 24.0 45.0 16.5 33.2 43.5 40.3 44.0 43.8 46.0 46.8 37.0 40.5 42.0 43.3 46.0 63.0 43.1 46.3

GPT-4

ReACT DFSDT

53.5 60.0 50.0 58.8 53.5 63.5 67.0 65.8 72.0 60.3 47.0 78.0 57.2 64.4 60.0 67.5 71.5 67.8 67.0 66.5 79.5 73.3 77.5 63.3 71.0 84.0 71.1 70.4

GPT-5.4

ReACT DFSDT

54.5 62.0 52.8 58.5 54.2 63.5 65.4 65.8 71.5 62.0 46.5 76.5 57.5 64.7 61.2 67.5 64.5 65.2 63.8 64.2 76.5 71.2 76.2 65.8 68.4 81.2 68.8 69.2

Claude-4.6-Sonnet

ReACT DFSDT

57.4 65.2 55.2 61.5 57.8 67.5 70.2 68.4 75.4 64.5 51.2 81.5 61.2 68.1 65.4 70.8 70.5 69.5 68.4 70.2 81.5 75.6 80.5 69.4 73.8 86.5 73.4 73.8

Gemini-3.1-Pro

ReACT DFSDT

56.2 63.8 53.8 60.5 56.5 65.4 68.4 67.5 73.2 63.2 49.5 79.5 59.6 66.6 63.8 69.2 67.5 66.8 66.2 67.5 79.5 73.4 78.4 67.8 71.2 84.5 71.1 71.5

HEART

–

67.5 72.8 71.5 70.8 69.5 72.1 83.4 77.5 81.6 71.5 77.2 89.4 75.1 75.7

14

The Last Mile to the Real World

Experiments

Table 2: HEART and reference models on NESTFUL nested call sequences. F1 for function and parameter names, partial and full sequence-match accuracy, and win rate (every predicted API executable and leading to the gold answer) under one- and three-shot demonstrations; HEART runs Qwen3-8B in its three harness mechanisms and its Tool Primitives (8B × 4) [44]. Green marks the best and amber the second-best value in each column. One-shot ICL Model

#Params (B)

Three-shot ICL

F1 F1 Part Full Win F1 F1 Part Full Win Func. Param. Acc. Acc. Rate Func. Param. Acc. Acc. Rate

xLAM-1b-fc-r xLAM-2-1b-fc-r xLAM-7b-fc-r xLAM-2-8b-fc-r Hammer2.0-7b Hammer2.1-7b Llama-3.1-8B-Instruct ToolACE-8B ToolACE-2-Llama-3.1-8B Granite-20B-FunctionCalling Mixtral-8x7B-Instruct-v0.1 xLAM-8x7b-fc-r Llama-3.1-70B-Instruct Mixtral-8x22B-Instruct-v0.1 xLAM-8x22b-fc-r Llama-3.1-405B-Instruct-fp8 DeepSeek-V3

1 1 7 8 7 7 8 8 8 20 46.7 46.7 70 141 141 405 685

0.19 0.41 0.49 0.48 0.56 0.10 0.64 0.43 0.28 0.64 0.22 0.40 0.41 0.49 0.53 0.41 0.69

0.08 0.13 0.17 0.15 0.24 0.05 0.19 0.13 0.10 0.20 0.07 0.15 0.19 0.21 0.21 0.14 0.36

0.09 0.13 0.15 0.14 0.21 0.05 0.17 0.13 0.13 0.17 0.05 0.16 0.15 0.17 0.22 0.08 0.27

0.00 0.00 0.00 0.00 0.07 0.01 0.06 0.00 0.00 0.02 0.00 0.01 0.04 0.06 0.12 0.03 0.09

0.01 0.00 0.03 0.01 0.16 0.01 0.06 0.00 0.00 0.05 0.01 0.01 0.09 0.07 0.03 0.10 0.43

0.22 0.43 0.55 0.47 0.61 0.16 0.63 0.50 0.29 0.61 0.32 0.43 0.33 0.65 0.50 0.41 0.69

0.09 0.12 0.23 0.17 0.30 0.10 0.22 0.15 0.10 0.25 0.13 0.16 0.17 0.29 0.23 0.18 0.42

0.09 0.13 0.23 0.15 0.29 0.11 0.22 0.13 0.13 0.26 0.14 0.17 0.15 0.28 0.25 0.13 0.37

0.03 0.00 0.15 0.04 0.22 0.08 0.16 0.00 0.00 0.21 0.09 0.02 0.07 0.21 0.17 0.07 0.29

0.02 0.00 0.14 0.04 0.25 0.08 0.11 0.00 0.00 0.20 0.09 0.03 0.11 0.23 0.06 0.14 0.60

GPT-4o (2024-08-06) GPT-5.4 Claude-4.6-Sonnet Gemini-3.1-Pro

– – – –

0.73 0.78 0.77 0.76

0.41 0.48 0.47 0.46

0.38 0.50 0.49 0.48

0.28 0.40 0.39 0.38

0.59 0.66 0.65 0.63

0.74 0.80 0.79 0.78

0.41 0.50 0.49 0.48

0.38 0.52 0.51 0.50

0.28 0.42 0.41 0.40

0.60 0.68 0.67 0.65

8×4

0.82

0.52

0.55 0.44 0.75

0.84

0.56

0.58 0.47 0.79

HEART

Models. Each family keeps the backbones of its companion study, and its controls hold the backbone fixed across arms unless the control varies the model itself. Tool Primitives and HEART’s request decomposition, value resolution, and result validation all run Qwen3-8B [169]. Code Primitives adapt with GPT-5.6 Terra, which also fills the backbone and diagnosis positions, and a model-allocation study varies all three. Math Primitives are selected by GPT-5.6 Luna [170], GPT-5.6 Terra, or DeepSeek-V4.1-Flash [171], and GPT-5.6 Luna drives the prover. SourceLearn learns and applies Knowledge Primitives with GPT-5.6 Luna, GPT-OSS-120B [172], or DeepSeekV4.1-Flash, one backend per configuration performing every LLM step of every method. Agent Primitives run Qwen3-8B and two DeepSeek-R1-Distill models [173] as worker backbones, three of the study’s five, and Llama3-70B-Instruct [174] in the comparison with existing MAS methods; they are composed per query by a GPT-5.2 Organizer, which Claude-4 [175] replaces in an ablation. The controls of Dev-Primitives run GPT-5.6 Sol, and their harness comparisons run each baseline’s own backbone in both arms. Navigation Primitives and the rest of their system run on GPT-4.1, and ANTMAN-H keeps the GPT-4.1 orchestrator and runs every other component, the Navigation Primitives included, on Qwen3-8B. Harness Primitives are developed with GPT-5.6 Sol and composed for a GPT-5.6 Luna actor at medium effort, whose backbone the composer shares; GPT-5.6 Terra, DeepSeek-V4-Flash, and Claude-4.5-Haiku serve as actors not used in development.

4.2. Tool Primitives for LLM Tool Use LLM tool use places the smallest demand on the last mile: each tool does what the task needs, and what remains is to call it correctly. In function calling, the model writes schema-compliant arguments for each call [2, 7, 20] and maps one tool’s output into the next tool’s input schema, a step that nested and multi-turn calls make brittle, while a large catalogue in context degrades performance [39]. Tool Primitives reverse this burden: instead of requiring models to speak the language of APIs, they make tools speak the language of models [44]. Each Tool Primitive 15

The Last Mile to the Real World

Experiments

resolves the schema of its own tool, and HEART, a harness engineering framework, decomposes each query into an invocation plan, grounds each step’s arguments, and validates every returned result (Section 3.4). Setup. We evaluate HEART on the five benchmarks of its study and report three of them in full: the six single- and multi-tool splits of ToolBench [6], NESTFUL [32], and the retail, airline, and telecom domains of 𝜏 2 -Bench [31]. The baselines are each benchmark’s reference systems: on ToolBench, models from Vicuna [176] and Alpaca [177] to ToolLLaMA [6] and GPT-4 [178] under ReAct [1] and DFSDT; on NESTFUL, function-calling models such as xLAM [52], Hammer [54], ToolACE [19], and Granite [179] and general models from Llama-3.1 [174] and Mixtral [180] to DeepSeek-V3 [181] and GPT-4o [182]; and on 𝜏 2 -Bench, GPT-4.1, GPT-4.1-mini, GPT-o4-mini, and Claude-3.7-Sonnet. GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro appear on all five. HEART runs Qwen38B [169] in request decomposition, value resolution, result validation, and the Tool Primitives, with revision budget 𝐵 = 3 and retrieval over the 25,519 tools of ToolFace; results average three runs, and eight on 𝜏 2 -Bench. Main Results. On ToolBench (Table 1), HEART reaches an average pass rate of 75.1% and win rate of 75.7%. Without tree search, it leads Claude-4.6-Sonnet with DFSDT, the strongest commercial baseline, by 1.7 and 1.9 pp, and ToolLLaMA with DFSDT and its retriever, the strongest fine-tuned baseline, by 7.8 and 12.6 pp. The lead is largest on I3Inst., the hardest split, whose multi-tool instructions require cross-API dependencies: 77.2% pass and 89.4% win rates, 3.4 and 2.9 pp above Claude-4.6-Sonnet with DFSDT. On the same Qwen3-8B backbone and ToolFace catalogue, HEART leads DFSDT, the strongest of four scaffolds on ToolBench, by 19.3 pp pass rate, 75.1% against 55.8%, and by 0.31 NESTFUL full accuracy (Table 25).

Table 3: HEART and reference models on dual-control conversational tool use in 𝜏 2 -Bench. Pass𝑘 is the fraction of tasks whose 𝑘 independent trials all succeed, tokens, API cost (USD) and latency (s) are averages per task, and HEART runs its default Qwen3-8B backbone [44]. Green marks the best and amber the second-best value in each Pass𝑘 column within each domain. Method

Domain Pass1 Pass2 Pass3 Pass4 Tokens

Cost ($)

Latency (s)

GPT-4.1

Retail 0.75 Airline 0.56 Telecom 0.34

0.64 0.45 0.26

0.58 0.42 0.22

0.53 0.40 0.19

7,401 7,825 6,862

0.0592 0.0626 0.0549

14.28 15.62 34.41

GPT-o4-mini

Retail 0.71 Airline 0.59 Telecom 0.42

0.59 0.48 0.33

0.52 0.42 0.29

0.46 0.38 0.26

5,127 5,389 4,856

0.0031 0.0032 0.0029

4.24 4.68 26.05

GPT-4.1-mini

Retail 0.66 Airline 0.51 Telecom 0.44

0.52 0.39 0.30

0.44 0.32 0.22

0.39 0.26 0.18

5,214 5,563 5,021

0.0083 0.0089 0.0080

6.12 6.81 21.94

Retail 0.79 Claude-3.7-Sonnet Airline 0.50 Telecom 0.49

0.69 0.41 0.37

0.63 0.38 0.30

0.60 0.36 0.25

6,482 6,745 6,038

0.0972 0.1012 0.0906

11.45 12.32 30.88

Retail 0.70 Airline 0.64 Telecom 0.56

0.53 0.55 0.40

0.43 0.50 0.34

0.37 0.48 0.29

7,842 8,126 7,319

0.1176 0.1219 0.1098

16.24 18.42 35.11

Retail

0.72

0.66

0.61

7,153

0.1073

14.12

GPT-5.4

0.80

On NESTFUL (Table 2), HEART is Claude-4.6-Sonnet Airline 0.72 0.62 0.55 0.50 7,412 0.1112 16.52 best on all five metrics under both oneTelecom 0.58 0.47 0.39 0.33 6,728 0.1009 33.48 and three-shot demonstrations. Its fullRetail 0.68 0.54 0.46 0.40 6,914 0.0830 9.15 Gemini-3.1-Pro Airline 0.54 0.41 0.33 0.29 7,258 0.0871 10.82 sequence accuracy is 0.44 and 0.47, Telecom 0.55 0.42 0.34 0.30 6,492 0.0779 28.46 against 0.40 and 0.42 for GPT-5.4, the Retail 0.82 0.78 0.76 0.75 21,684 0.0152 18.72 strongest baseline, and its win rate 0.75 HEART Airline 0.73 0.68 0.65 0.63 24,531 0.0172 21.34 and 0.79, against 0.66 and 0.68. FineTelecom 0.63 0.57 0.53 0.50 20,887 0.0146 37.95 tuning on flat tool-use trajectories does not carry over to nested sequences: with oneshot demonstrations, the xLAM models up to 8B and both ToolACE models reach 0.00 full accuracy. Nor does scale close the gap, as DeepSeek-V3, with 685B parameters, reaches 0.09. On 𝜏 2 -Bench (Table 3), HEART has the highest Pass𝑘 in each domain for each 𝑘. Its Pass1 margins over Claude4.6-Sonnet, the strongest baseline model, are small, 0.82, 0.73, and 0.63 against 0.80, 0.72, and 0.58 on retail, airline, and telecom, and they widen with 𝑘: in the airline domain, whose tasks include booking a flight under the airline’s policy, HEART succeeds in all four trials on 0.63 of the tasks against 0.50, and in telecom on 0.50 against 0.33. Although each invocation step adds a Tool Primitive’s model call and HEART uses 20,887

16

The Last Mile to the Real World

Experiments

to 24,531 tokens per task, it costs $0.0146 to $0.0172 per task on Qwen3-8B, against $0.1098 to $0.1219 for GPT-5.4, up to 7.7× less. Table 4: Removing Tool Primitives or one of HEART’s harness mechanisms. Each row removes one component from the full configuration (Qwen3-8B, ToolFace): request decomposition, value resolution, result validation, or ToolFace with its Tool Primitives; metrics as in Table 25 [44]. Green marks the best and amber the second-best value in each column. ToolBench Variant

NESTFUL

Pass Win Full Acc. Win

w/o Decomposition w/o Resolution w/o ToolFace & Tool Primitives w/o Validation

57.2 62.4 16.1 47.6

64.1 67.3 27.9 49.6

0.26 0.28 0.06 0.15

0.51 0.52 0.06 0.33

HEART

75.1 75.7

0.44

0.75

Component Ablation. Removing one component at a time locates the gain in the natural-language interface (Table 4). Without ToolFace and its Tool Primitives, value resolution resolves arguments against raw schemas and dispatches calls itself; ToolBench pass rate then falls from 75.1% to 16.1% and NESTFUL full accuracy from 0.44 to 0.06. Removing result validation, which drives revision, gives the second-largest drop, to 47.6% and 0.15, and removing request decomposition or value resolution gives 57.2% and 62.4%. Removing the primitives also removes one model call per step; given that budget to self-refine its schema-compliant arguments at matched backbone, calls, and tokens, value resolution without primitives reaches 34.2% (Table 26), so the interface carries most of the gain. On the other two benchmarks, HEART has the highest ACEBench Overall score, 86.9, and the highest BFCLv4 Web Search and Memory Overall scores, 86.0 and 69.38 [20, 58]. On 50 end-to-end real-world tasks in five domains, it completes 84%, against 20% to 24% for GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro in their product interfaces and 22% to 32% when they call the ToolFace tools through native function calling. We provide the full framework of HEART, its overview figure, the matched-backbone and budget-matched controls, and further diagnostics in Appendix B.1. A Tool Primitive invokes its function as it is; a component that must itself change to fit its target calls for a primitive that adapts the resource it owns, the step a Code Primitive takes.

4.3. Code Primitives for Repository Construction Repository construction asks an agent to deliver a complete project whose modules, interfaces, configuration, tests, and dependencies work together [81]. When the project reuses code, the last mile lies in making that code fit its target. A component reused from another repository becomes part of the delivered source, so its signatures, data representations, and dependencies must be reconciled with the target. Retrieval-augmented code generation finds related code [35, 70, 71] but leaves this reconciliation to the caller. Code Primitives assign it to the component itself: each Code Primitive adapts the implementation it owns to the target and checks the result against the tests it carries, while LEGO retrieves, coordinates, and verifies these primitives during construction [45]. Setup. LEGO-REPO poses the task as reconstruction. Each of its 522 frozen tasks removes the implementation of an installable Python package, keeps the package’s interfaces, native tests, and environment, and grades the delivered repository with those tests. The tasks span seven domains and five difficulty bands, D1–D5, which combine module count with the number of native tests the original passes. Let Π(𝑋) denote the number of native tests passed using source set 𝑋. For task 𝑇 , the original sources 𝑂 define the identity ceiling 𝑐(𝑇 ) = Π(𝑂) and an empty reconstruction tree the floor 𝑓 (𝑇 ) = Π(∅), and generated sources 𝐺 score {︃ (︂ )︂ 522 𝑠(𝐺𝑇 ; 𝑇 ), admissible, Π(𝐺) − 𝑓 (𝑇 ) 1 ∑︁ 𝑠(𝐺; 𝑇 ) = clamp[0,1] , 𝑆delivery = 𝑠̃︀𝑇 , 𝑠̃︀𝑇 = (11) 𝑐(𝑇 ) − 𝑓 (𝑇 ) 522 𝑇 =1 0, otherwise, where a run is admissible when 𝑐(𝑇 ) − 𝑓 (𝑇 ) ≥ 5, execution completes, and provenance confirms that the tests exercised the generated target code. The zero-score rate is the fraction of tasks assigned 0. The baselines are 17

The Last Mile to the Real World

Experiments

construction paradigms, from single-shot generation to retrieval with a generic code editor, and independently developed repository agents, among them OpenHands and Claude Code [5, 9], each under its own harness with tasks, environment, and grading held fixed. Unless a configuration varies them, GPT-5.6 Terra fills the backbone, adaptation, and diagnosis positions, retrieval returns the top two of 1,424 CodeFace entries per requirement, and revision runs at most five rounds per module. Main Results. Construction from scratch Table 5: LEGO against construction paradigms on 522 repositories. Deweakens as repositories grow: under execu- livery score on the native [0, 1] scale for the smallest (D1) and largest (D5) tion feedback, GPT-5.6 Terra is the strongest difficulty bands and all tasks, zero-score rate and recorded-call cost per task (USD), with stages R retrieval, A adaptation, E generic editing, V execution of the study’s 13 backbones, and the mean de- feedback and D diagnosis; the two one-attempt rows are first-round prefixes of livery of the 13 falls from 0.393 on D1 to longer trajectories [45]. Green marks the best and amber the second-best value 0.038 on D5. Table 5 compares the construc- in each of the D1, D5, All, and Zero columns. Lower is better for Zero. tion paradigms on all 522 tasks with GPT-5.6 Configuration Stages D1 D5 All Zero (%) $/task Terra. LEGO reaches a delivery score of 0.5134 Single-shot ---– – 0.2172 57.1 0.86 against 0.3180 for execution feedback, a gain of Retrieval, one attempt R--– – 0.2314 58.6 0.87 Feedback ---V 0.525 0.129 0.3180 41.0 1.55 0.1954 or 61.4%, and lowers the zero-score rate Retrieval + Feedback R--V 0.544 0.141 0.3352 34.7 1.56 from 41.0% to 18.8%. Retrieval adds 0.0172 Feedback + Diagnosis --DV 0.569 0.135 0.3469 30.1 1.79 to execution feedback, whereas primitive adap- Retrieval + Diagnosis R-DV 0.649 0.127 0.3980 31.6 1.80 tation adds 0.1154 to retrieval with diagno- Retrieval + Adapt RA-V 0.653 0.275 0.4795 17.8 1.83 RAG + Edit RE DV 0.637 0.136 0.4093 24.3 2.01 sis; RAG with a generic editor, the strongest LEGO RADV 0.686 0.338 0.5134 18.8 2.08 paradigm without primitive adaptation, reaches same-repo donors excluded RADV 0.683 0.221 0.4743 24.9 2.08 0.4093. The strongest independent repository agents stay near the feedback baseline: OpenHands reaches 0.3263 with GPT-5.6 Terra and Claude Code 0.3170 with Claude Sonnet 5, while SWE-agent and Agentless, run with GPT-5.6 Terra through an issue-to-patch adapter, reach 0.2750 and 0.2181 [8, 74]. LEGO leads OpenHands and Claude Code on the same backbones by 57.3% and 56.2%, at 0.5134 and 0.4953. Across the 13 backbones of the study, with adaptation and diagnosis on GPT-5.6 Terra, LEGO improves all 13 and raises mean delivery from 0.1926 to 0.3400, a gain of 0.1474 [45]. Access versus Adaptation. Access to the retrieved com- Table 6: Access, invocation, and adaptation of the same reusable ponents adds little; the gain comes from adapting them. components. The reference --DV uses no library; the other rows The consumption controls of Table 6 keep LEGO’s re- share LEGO’s retrieval and diagnosis: RC DV shows the retrieved component read-only, RI DV vendors and calls it unchanged, and trieval and diagnosis and vary how the components are structured-schema adaptation sends the adaptation request as a used. As read-only context with their contracts and tests, typed field mapping instead of free text; delivery on the [0, 1] scale, they reach 0.3544, and vendored and called unchanged Δ from --DV [45]. Green marks the best and amber the secondthey reach 0.3430, against 0.3469 for diagnosis without best value in each of the D1, D5, and All columns. a library; on D5, unchanged invocation falls to 0.060 Configuration D1 D5 All Δ --DV against 0.135 for that reference. Structured-schema adapFeedback + Diagnosis (--DV) 0.569 0.135 0.3469 ref. tation keeps primitive adaptation but replaces the naturalContext-only (RC DV) 0.620 0.115 0.3544 +0.0075 Import + Call (RI DV) 0.627 0.060 0.3430 −0.0039 language request with a typed field mapping, and it Structured-schema adaptation 0.679 0.327 0.4974 +0.1505 scores 0.4974 against 0.5134. Most of LEGO’s 0.1041 LEGO (RADV) 0.686 0.338 0.5134 +0.1665 margin over generic editing lies in the primitive’s fields: given the carried tests, the interface contract and provenance, and the dependency closure, a generic editor over the same components recovers 0.0946 of it. Cost and Model Allocation. Spending LEGO’s recorded-call budget on more repair attempts does not close the gap: execution feedback with eight attempts per module costs $2.07 per task and reaches 0.3437, against 0.5134 at $2.08 for LEGO. The locally scoped models can also be small. With GPT-OSS-20B as the resident LLM 𝑀𝑖 of each primitive and as the diagnosis model, LEGO keeps 95.1% of its delivery, 0.4880, at 24.0% lower cost; replacing

18

The Last Mile to the Real World

Experiments

the backbone with DeepSeek-V3.2 keeps 48.1%. Delivery thus depends more on the backbone, which plans and integrates the repository, than on these models. The gain holds across repository sizes, library restrictions, and benchmarks. Over the five bands, the absolute gain stays between 0.161 and 0.209, so the relative gain rises from 30.7% on D1 to 162.0% on D5 as feedback falls. Excluding same-repository donors lowers delivery to 0.4743, and a library re-mined from a corpus disjoint from the benchmark repositories gains 0.1348 over feedback. Under the published protocols of three external repository benchmarks, LEGO with GPT-5.6 Terra raises the RepoZero (Python to Python) pass rate from 53.8% to 62.9%, the RepoGenesis resolved rate from 41.0% to 48.2%, and the RepoCraft normalized score from 0.436 to 0.501 over execution feedback, relative gains of 16.9%, 17.6%, and 14.9%. With the CodeFace entries traced to these targets removed, the library covers 29.6–34.8% of their target modules, against 61.4% on LEGO-REPO. We provide the full framework of LEGO, its overview figure, the field-by-field decomposition of the primitive, and the full model-allocation table in Appendix B.2, with breakdowns by band and domain, donor-provenance controls, attempt budgets, and seed uncertainty in Appendix C. A Code Primitive checks its adaptation against the tests it carries and leaves the integrated result to the repository’s build and tests. When an exact checker judges the final result, the open step moves earlier, to deciding which method to apply.

4.4. Math Primitives for Formal Theorem Proving In formal theorem proving with Lean 4 and Mathlib [10, 11], the kernel checks each completed proof, and LLM provers generate and revise proofs against this check [114, 115, 117], aided by premise selection, Mathlib search, and lemma libraries [119–122]. Here the last mile is deciding whether a method applies: part of the goal ∑︀ which∑︀ it targets, what action it takes there, and which conditions that action requires. For 𝑛 𝑖 𝑎2𝑖 ≥ ( 𝑖 𝑎𝑖 )2 over the reals, similarity retrieval proposes AM–GM, which needs 𝑎𝑖 ≥ 0, a condition the goal does not give, while Cauchy– Schwarz applies. Math Primitives let the library speak instead: each holds a proof method as a Method Contract, and under self-advertised method selection every contract proposes how it would be used on the goal before the contracts are ranked [46]. Setup. We evaluate on 120 Putnam problems from 2015–2025 and 60 IMO ProofBench problems, held out from the Putnam 2000–2014 solutions behind the 82 contracts. Each problem is labeled with the contracts whose method appears in its official solution; GPT-5.6 Sol proposed the labels, and a human checked each one before any selector ran. The baselines are random ordering, BM25, TF-IDF cosine, and three embedding retrievers, each also paired with a listwise LLM reranker over its top ten candidates. The LLM-based selectors within a block of Table 7 share one backbone, GPT-5.6 Terra, GPT-5.6 Luna, or DeepSeek-V4.1-Flash. Main Results. Self-advertisement gives the broadest shortlist coverage (Table 7). Lexical matching reaches at most 0.525 Putnam hit@5, dense embeddings 0.775, and LLM reranking of an embedding pool 0.858. Self-advertisement attains the highest recall@5 in all six backbone–benchmark settings, and on Putnam it also leads at hit@3, hit@5, and MRR under each backbone. With GPT-5.6 Luna, the prover’s backbone, it reaches 0.950 Putnam hit@5 against 0.842 for the strongest reranker, and 0.917 IMO ProofBench hit@5 against 0.883, a difference of two of the 60 problems. Its lead grows with the backbone while the rerankers plateau: from DeepSeek-V4.1-Flash to GPT-5.6 Luna to GPT-5.6 Terra, its Putnam hit@5 rises from 0.900 to 0.950 to 0.975, and the best reranker stays at 0.842, 0.842, and 0.858. The lead under DeepSeek-V4.1-Flash shows that it does not depend on the model family that drafted the contracts and labels. On IMO ProofBench, whose problems come from different competitions than the Putnam-derived library, self-advertisement keeps its recall@5 lead, at 0.494, 0.554, and 0.493 under GPT-5.6 Terra, GPT-5.6 Luna, and DeepSeek-V4.1-Flash, against at most 0.469, 0.468, and 0.449 for the rerankers. Rerankers remain more precise at rank one in most settings; under GPT-5.6 Luna, the best reaches 0.692 Putnam hit@1 against 0.658 for self-advertisement. The runoff targets this gap: with the same backbone, it re-ranks selfadvertisement’s top ten by comparing the leading proposals. Under GPT-5.6 Terra, it raises Putnam hit@1 from 0.625 to 0.725, above the strongest reranker’s 0.675, and IMO ProofBench hit@1 from 0.517 to 0.600, with the 19

The Last Mile to the Real World

Experiments

Table 7: Selecting Method Contracts for held-out Putnam and IMO ProofBench problems. hit@𝑘 is the fraction of problems whose top-𝑘 shortlist contains an annotated reference contract, R@𝑘 the fraction of annotated contracts in the top 𝑘, and MRR the mean reciprocal rank; rerankers reorder the top ten candidates of an embedding retriever, and self-advertisement ranks all 82 contracts of the library from their goal-specific proposals [46]. Green marks the best and amber the second-best value in each column within each selector model. Putnam 2015–2025 (𝑛 = 120) Selector

IMO ProofBench (𝑛 = 60)

hit@1 hit@3 hit@5 R@3 R@5 MRR hit@1 hit@3 hit@5 R@3 R@5 MRR

No LLM Random BM25 TF-IDF cosine OpenAI emb-3-small OpenAI emb-3-large Qwen3-Emb-0.6B

0.058 0.208 0.267 0.325 0.425 0.450

0.158 0.400 0.442 0.533 0.667 0.675

0.225 0.508 0.525 0.683 0.725 0.775

0.038 0.104 0.118 0.157 0.204 0.217

0.053 0.140 0.144 0.229 0.267 0.291

0.141 0.337 0.382 0.463 0.556 0.579

0.083 0.183 0.233 0.267 0.367 0.350

0.167 0.333 0.367 0.517 0.583 0.617

0.267 0.400 0.433 0.617 0.717 0.767

0.041 0.103 0.135 0.213 0.268 0.282

0.076 0.147 0.165 0.263 0.339 0.380

0.158 0.295 0.318 0.411 0.523 0.510

GPT-5.6 Terra OpenAI-small + rerank OpenAI-large + rerank Qwen3 + rerank Self-advertisement + runoff

0.642 0.667 0.675 0.625 0.725

0.767 0.800 0.808 0.908 0.950

0.775 0.842 0.858 0.975 0.983

0.285 0.315 0.338 0.342 0.374

0.324 0.373 0.407 0.472 0.503

0.704 0.742 0.751 0.778 0.831

0.483 0.600 0.633 0.517 0.600

0.717 0.783 0.783 0.867 0.833

0.767 0.900 0.900 0.883 0.933

0.336 0.378 0.373 0.422 0.409

0.397 0.455 0.469 0.494 0.512

0.607 0.712 0.724 0.694 0.736

GPT-5.6 Luna OpenAI-small + rerank OpenAI-large + rerank Qwen3 + rerank Self-advertisement + runoff

0.592 0.692 0.683 0.658 0.683

0.742 0.792 0.817 0.917 0.908

0.783 0.825 0.842 0.950 0.958

0.286 0.312 0.337 0.346 0.352

0.321 0.361 0.385 0.446 0.461

0.673 0.747 0.756 0.787 0.800

0.517 0.583 0.567 0.550 0.567

0.750 0.833 0.817 0.817 0.817

0.783 0.883 0.883 0.917 0.933

0.350 0.404 0.407 0.414 0.417

0.400 0.460 0.468 0.554 0.538

0.628 0.714 0.688 0.693 0.703

DeepSeek-V4.1-Flash OpenAI-small + rerank OpenAI-large + rerank Qwen3 + rerank Self-advertisement + runoff

0.550 0.567 0.575 0.600 0.608

0.750 0.775 0.783 0.808 0.842

0.775 0.825 0.842 0.900 0.892

0.262 0.293 0.307 0.293 0.308

0.305 0.367 0.371 0.393 0.399

0.643 0.673 0.687 0.719 0.735

0.550 0.550 0.617 0.517 0.517

0.767 0.833 0.783 0.817 0.833

0.767 0.850 0.850 0.883 0.917

0.354 0.383 0.388 0.395 0.412

0.388 0.446 0.449 0.493 0.523

0.640 0.685 0.705 0.672 0.684

best MRR on both benchmarks (0.831 and 0.736). Under DeepSeek-V4.1-Flash, it gives the best Putnam hit@1 and MRR (0.608 and 0.735), and under GPT-5.6 Luna it changes each hit metric by at most three problems. Downstream Proving. The selected contract Table 8: Downstream proof completion, a separate stage from shortlist also helps the prover (Table 8). AxProverBase is coverage (Table 7). Proof success (%) with GPT-5.6 Luna under ten Lean an Ax-Prover whole-proof repair loop driven by compilations per problem, except pass@20, which draws 20 samples and so has twice the budget; Base is one attempt without retrieval, and the conGPT-5.6 Luna with ten Lean compilations per tract arm adds the top-ranked contract to AxProverBase’s initial prompt [46]. problem [118], and the contract arm adds the Green marks the best and amber the second-best value in each column. execution side of the top-ranked contract to its Putnam 2015–2025 IMO ProofBench initial prompt. The contract more than doubles Method ALG ANA DISC All Basic Adv. All AxProverBase’s success on Putnam, from 5.8% Base 0.0 0.0 0.0 0.0 3.3 0.0 1.7 to 12.5%, and raises it from 10.0% to 15.0% on pass@20 2.3 0.0 0.0 0.8 3.3 0.0 1.7 IMO ProofBench. On Putnam the gains come APOLLO 4.5 0.0 2.2 2.5 13.3 0.0 6.7 from analysis (3.3% to 10.0%) and discrete probAxProverBase 9.1 3.3 4.3 5.8 20.0 0.0 10.0 lems (4.3% to 17.4%), while algebra stays at + self-advertised contract 9.1 10.0 17.4 12.5 26.7 3.3 15.0 9.1%; on IMO ProofBench the contract adds Basic problems (20.0% to 26.7%) and the one Advanced problem that any arm solves. Base and pass@20 solve almost no problems, although pass@20 has twice the budget, and APOLLO [117] trails both AxProverBase arms. Since the two arms share the prover, its backbone, and its budget, the gain reflects selection and contract content together. Applicability Beyond Similarity. The coverage gains of Table 7 come largely from methods that similarity overlooks. We place each problem by the rank of its highest-ranked annotated method under self-advertisement with

20

The Last Mile to the Real World

Experiments

GPT-5.6 Luna and by that method’s rank under BM25 and each embedding retriever (Figure 17 in Appendix C). Under all four retrievers, annotated methods that similarity ranks outside its top five enter the top five of selfadvertisement, and many of them lie outside the top ten that the rerankers receive. This is the candidate-pool ceiling of Proposition 3.3(ii) at work: a reranker reorders the pool that similarity retrieved, so no reranking backbone recovers a method that the pool omits. Self-advertisement judges all 82 contracts, and each method it brings into the shortlist arrives with a target, an action, and required conditions that can be checked against the goal. We provide the full method of self-advertised method selection, its overview figure, the theory behind the commitment gate and the runoff, and the experimental configuration in Appendix B.3. A Method Contract stays fixed once built, whereas a Knowledge Primitive’s source model changes as the agent studies its source.

4.5. Knowledge Primitives for Knowledge-Intensive Tasks When many tasks draw on one persistent, authoritative source, such as a document collection, a code repository, or API documentation, the last mile is understanding that source: how its knowledge is structured, interpreted, and applied, beyond the passages that one task retrieves. Retrieval brings such passages into each task [12, 86], structured indexes organize the source’s content [21, 94, 98], and agent memories keep experience from earlier tasks [22, 23, 25], yet repeated use of a source is still treated as repeated access. Knowledge Primitives turn this repeated use into source learning: the resource of a Knowledge Primitive is a persistent source model that holds reusable source-specific competence, while the source itself remains the authority for factual content. SourceLearn [47] develops this source model in two ways. Before future tasks are known, Self-Directed Source Learning studies what the source model does not yet explain well, and Task-Guided Source Learning uses guidance tasks to expose local gaps and recurring needs in how the source is represented. We provide the full framework of SourceLearn in Appendix B.4, together with its overview figure, the benchmark and baseline settings, and the comparison at a matched context budget. Setup. We evaluate SourceLearn on question answering over document collections, stories, and code repositories with MultiDoc2Dial [146], NarrativeQA [147], and SWE-QA [148]; tool use with APIBench [7]; and interactive tasks with AppWorld [149]. Each document collection, story, repository, or API library is one source. Guidance and test tasks are disjoint, with three fixed 30%/70% splits and AppWorld’s official training and test-normal splits. GPT-5.6 Luna judges QA answers against the references, and APIBench and AppWorld use their official evaluators. The baselines are task-local retrieval with Hybrid RAG, persistent source representations with RAPTOR [21] and HippoRAG 2 [98], and experience-based memory with AWM [25], which learns workflows from the same guidance tasks. Each configuration uses a single backbone, GPT-5.6 Luna [170], GPT-OSS-120B [172], or DeepSeek-V4.1Flash [171], for the LLM operations of all methods, which share the embedding model, the answer or agent prompt, and, on AppWorld, the ReAct-style code agent [1]. SourceLearn adds the activated source model to the Hybrid RAG evidence 𝑅(𝐷, 𝑞), which AWM also receives, so the comparison with Hybrid RAG isolates the contribution of the source model. Main Results. Across the 15 settings of Table 9, SourceLearn is the best method in 13 and the second best in one more. It improves on Hybrid RAG in 14 settings, and its average gain over the five benchmarks is 14.3 pp with GPT-5.6 Luna, 4.9 pp with GPT-OSS-120B, and 13.4 pp with DeepSeek-V4.1-Flash. The gains span document, code, and API sources and exceed 10 pp on four of the five benchmarks with GPT-5.6 Luna and with DeepSeekV4.1-Flash; with GPT-5.6 Luna, MultiDoc2Dial rises from 64.9% to 77.6% and APIBench from 47.8% to 70.4%. The two settings in which it is not the best both use GPT-OSS-120B: HippoRAG 2 leads on SWE-QA (56.7% against 53.4%), and on AppWorld SourceLearn falls below Hybrid RAG (27.4% against 30.4%). The baselines help unevenly. RAPTOR’s summary tree falls below Hybrid RAG on SWE-QA under all three backbones, where HippoRAG 2 is the strongest baseline, and AWM, the strongest baseline on APIBench, moves question-answering accuracy by at most 3.9 pp. Learning the source transfers across benchmarks more evenly than structuring its content or retaining task experience. SourceLearn also leads on three of the four sources tested with GPT-5.6 Luna

21

The Last Mile to the Real World

Experiments

Table 9: SourceLearn against retrieval and memory baselines on five benchmarks and three backbones. Accuracy or task success (%) for document QA, code QA, tool use, and interactive environments, with the change from Hybrid RAG in pp in parentheses [47]. Green marks the best and amber the second-best value in each column within each backbone. Document QA

Code QA

Tool Use

Interactive Env.

MultiDoc2Dial NarrativeQA

SWE-QA

APIBench

AppWorld

66.0 68.1 (+2.1) 66.6 (+0.6) 62.1 (−3.9) 80.2 (+14.2)

56.9 50.8 (−6.1) 62.7 (+5.8) 56.7 (−0.2) 69.5 (+12.6)

47.8 61.9 (+14.1) 55.2 (+7.4) 62.8 (+15.0) 70.4 (+22.6)

72.0 75.6 (+3.6) 77.4 (+5.4) 66.7 (−5.3) 81.5 (+9.5)

64.5 61.5 (−3.0) 66.7 (+2.2) 64.1 (−0.4) 68.7 (+4.2)

45.6 40.4 (−5.2) 45.7 (+0.1) 43.2 (−2.4) 56.7 (+11.1)

52.5 32.8 (−19.7) 56.7 (+4.2) 51.5 (−1.0) 53.4 (+0.9)

60.3 54.7 (−5.6) 56.0 (−4.3) 63.2 (+2.9) 71.8 (+11.5)

30.4 26.8 (−3.6) 33.9 (+3.5) 28.6 (−1.8) 27.4 (−3.0)

63.3 73.2 (+9.9) 67.3 (+4.0) 63.7 (+0.4) 81.3 (+18.0)

64.2 60.4 (−3.8) 62.0 (−2.2) 61.4 (−2.8) 76.2 (+12.0)

45.9 36.8 (−9.1) 55.7 (+9.8) 47.4 (+1.5) 61.6 (+15.7)

67.1 69.3 (+2.2) 69.6 (+2.5) 70.8 (+3.7) 82.2 (+15.1)

83.9 88.7 (+4.8) 86.9 (+3.0) 83.9 (0.0) 89.9 (+6.0)

Model

Method

GPT-5.6 Luna

Hybrid RAG RAPTOR HippoRAG 2 AWM SourceLearn

64.9 71.2 (+6.3) 66.3 (+1.4) 65.1 (+0.2) 77.6 (+12.7)

GPT-OSS-120B

Hybrid RAG RAPTOR HippoRAG 2 AWM SourceLearn

Hybrid RAG RAPTOR DeepSeek-V4.1-Flash HippoRAG 2 AWM SourceLearn

at a matched context budget, where each retrieval baseline fills SourceLearn’s per-question budget by reading further down its own ranking; HippoRAG 2 leads on Sphinx (73.5% against 72.5%; Table 29). Analysis. Both learning mechanisms contribute and complement each other. We remove each in turn with GPT5.6 Luna on two sources of each QA benchmark, with the splits and task-time source access of the main results (Figure 9). The full model 𝑀𝑇 is the most accurate on all three benchmarks, omitting Self-Directed or TaskGuided Source Learning lowers accuracy, and the initial model 𝑀0 is the lowest. Within Task-Guided Source Learning, removing failure-guided local refinement or cross-task representation learning lowers accuracy on all three benchmarks, and removing both, which recovers the self-directed model 𝑀𝑆 , lowers it further. The two pathways thus carry different information from the same guidance tasks: concrete failures repair local gaps, while recurring representation lessons change how the whole model represents the source. Learning also does more than compress the source (Figure 19). From 𝑀0 to 𝑀𝑇 , the model shifts from isolated look-up content toward rules and procedures, and the share of units with an explicit applicability condition rises from roughly 30% to 65–82% on the four MultiDoc2Dial sources. The share of the claims required by test questions that the model represents rises from 23.2% to 40.0%, and questions with all of their required claims represented are answered correctly 86.6% of the time, against 67.5% when none are. The Tool, Code, Math, and Knowledge Primitives each own a resource that a single model applies; Agent Primitives own a recurring multi-agent computation pattern, whose agents run on one LLM and exchange KV caches.

4.6. Agent Primitives for Multi-Agent Systems Where several agents share one problem, the last mile lies in how they combine their work. Multi-agent systems (MAS) let LLM agents critique, vote, and plan together, but they are usually designed task by task, with handcrafted roles and interaction prompts, so their complexity grows with the task, and no reusable abstraction amortizes it. Their agents also communicate in natural language, which degrades over long, multi-stage interactions. In two stress tests with Qwen3-8B, an instruction injected at the midpoint of a long context is followed in 15.6% of cases through text and in 73.3% through KV caches, and after ten unrelated sentences are injected into solved GSM8K traces, 47% of the problems stay solved through text and 93% through KV caches. Agent Primitives [48] address both problems with three reusable latent building blocks, Review, Voting and Selection, and Planning and Execution,

22

The Last Mile to the Real World

Experiments

whose internal agents communicate through KV caches and which an LLM Organizer composes into a primitivebased MAS for each query. Setup. The evaluation covers mathematical problem solving on AIME24, AIME25, MATH, and GSM8K [150– 153], code generation on HumanEval+ and MBPP+, which extend the tests of HumanEval and MBPP [154, 183, 184], and question answering on MedQA and GPQA-Diamond [155, 156]. The baselines are a single agent, a text-based MAS (TextMAS), LatentMAS [137] in its sequential configuration, and each primitive applied alone to all queries; the last three use four agents each, whereas the Organizer, GPT-5.2 by default, sets the number of agents per query. KV cache communication assumes that the communicating agents share parameters and positional encoding, so all agents of a run use one worker backbone: Qwen3-8B [169], DeepSeek-R1-Distill-Qwen-32B, or DeepSeek-R1-Distill-Llama-70B [173], abbreviated R1/Qwen-32B and R1/Llama-70B. Table 10: Primitives-based MAS against single agents, text and latent MAS, and each Agent Primitive used alone. Accuracy (%) on eight math, code and QA benchmarks for three worker backbones, where R1/ marks a DeepSeek-R1-Distill model; the Organizer sets the number of agents of a composed MAS per query, while a single primitive uses a fixed number [48]. Green marks the best and amber the second-best value in each column within each backbone. Math Problem Solving

Code Generation

Q&A

GPQAAvg. AIME25 AIME24 MATH GSM8K HumanEval+ MBPP+ MedQA Diamond

Model

Method

Qwen3-8B

Single TextMAS LatentMAS Review Voting Planning Primitives-based MAS

46.7 53.3 53.3 60.0 66.7 66.7 73.3

50.0 53.3 56.7 63.3 70.0 63.3 76.7

60.8 61.4 62.6 61.0 61.4 60.8 63.7

81.1 92.3 93.8 93.2 91.8 93.2 94.2

74.4 80.5 80.5 78.6 81.0 78.6 82.3

64.8 69.5 74.6 70.6 74.3 75.9 75.9

53.0 75.0 75.3 64.2 70.3 67.0 76.7

39.9 43.4 45.5 48.9 55.0 51.0 59.6

58.8 66.1 67.8 67.5 71.3 69.6 75.3

Single TextMAS LatentMAS R1/Qwen-32B Review Voting Planning Primitives-based MAS

53.3 53.3 56.7 53.3 56.7 53.3 63.3

73.3 73.3 73.3 70.0 70.0 66.7 73.3

63.4 68.0 78.2 71.3 74.7 74.6 79.8

93.8 93.8 95.2 94.6 95.0 93.8 95.0

80.5 82.3 83.5 81.1 86.6 85.3 86.6

73.8 74.6 75.7 73.8 74.6 74.3 75.7

68.0 79.6 81.2 73.0 79.3 75.0 82.7

62.1 62.6 63.6 62.1 63.1 62.6 64.6

71.0 73.4 75.9 72.4 75.0 73.2 77.6

Single TextMAS LatentMAS R1/Llama-70B Review Voting Planning Primitives-based MAS

50.0 53.3 40.0 50.0 56.7 50.0 56.7

70.0 70.0 43.3 70.0 73.3 63.3 76.7

69.6 72.8 78.6 75.9 77.2 75.6 79.3

92.4 93.2 78.6 91.8 93.8 92.4 93.8

82.3 82.3 73.2 82.3 82.9 82.3 85.3

66.4 68.8 55.6 67.2 68.8 66.7 70.6

64.5 77.8 68.6 72.9 77.8 74.0 81.9

65.2 65.7 41.9 65.2 65.7 65.2 66.7

70.1 73.0 60.0 71.9 74.5 71.2 76.4

Main Results. Composing primitives gives the highest mean accuracy with each backbone (Table 10). With Qwen38B, the primitive-based MAS reaches 75.3%, against 58.8% for a single agent, 66.1% for TextMAS, 67.8% for LatentMAS, and 71.3% for Voting and Selection, the strongest single primitive. With R1/Qwen-32B and R1/Llama70B, it improves on a single agent by 6.6 and 6.3 pp. Across the three backbones, it is best or tied for best on each benchmark except GSM8K with R1/Qwen-32B, where LatentMAS reaches 95.2% against 95.0%. Each primitive used alone also improves on the mean accuracy of a single agent, and composing them adds 4.0, 2.6, and 1.9 pp on the three backbones over the strongest one, Voting and Selection. TextMAS improves on a single agent by 2.4 to 7.3 pp. LatentMAS is the strongest baseline with R1/Qwen-32B, at 75.9%, but falls to 60.0% with R1/Llama-70B, 10.1 pp below a single agent, which the study attributes to its aggressive chunking of latent reasoning steps; the more conservative KV cache communication of Agent Primitives improves on a single agent with each backbone. Existing MAS Methods. Table 11 compares the primitive-based MAS on Llama-3-70B-Instruct [174] with eight MAS methods, LLM-Debate [131], Self-Refine [140], Quality-Diversity [185], SPP [186], AgentVerse [130], 23

The Last Mile to the Real World

Experiments

Table 11: Primitives-based MAS against independently designed multi-agent methods and two single-model prompting baselines. Accuracy (%) in the Agent Primitives study’s unified Llama-3-70B-Instruct setting, whose backbone differs from those of Table 10, with the change from its own single-agent reference in pp in parentheses; Chain-of-Thought and Self-Consistency are the prompting baselines [48]. Green marks the best and amber the second-best value in each column. Method

MATH

GSM8K

HumanEval+

GPQA

50.6 53.2 (+2.6) 61.6 (+11.0) 61.4 (+10.8) 58.5 (+7.9) 60.5 (+9.9) 51.7 (+1.1) 55.6 (+5.0) 55.4 (+4.8) 59.6 (+9.0) 68.7 (+18.1)

92.4 92.8 (+0.4) 95.0 (+2.6) 91.6 (−0.8) 90.8 (−1.6) 93.0 (+0.6) 92.8 (+0.4) 93.4 (+1.0) 93.2 (+0.8) 91.2 (−1.2) 93.4 (+1.0)

75.8 77.0 (+1.2) 75.8 (0.0) 74.5 (−1.3) 62.7 (−13.1) 70.2 (−5.6) 73.3 (−2.5) 73.9 (−1.9) 73.9 (−1.9) 75.8 (0.0) 78.9 (+3.1)

36.7 35.3 (−1.4) 37.2 (+0.5) 34.4 (−2.3) 38.3 (+1.6) 33.6 (−3.1) 35.1 (−1.6) 40.2 (+3.5) 36.5 (−0.2) 36.0 (−0.7) 37.6 (+0.9)

Primitives-based MAS 72.4 (+21.8) 93.8 (+1.4)

82.3 (+6.5)

53.2 (+16.5)

Single Chain-of-Thought Self-Consistency LLM-Debate Self-Refine Quality-Diversity SPP AgentVerse GPTSwarm DyLAN MAS-GPT

GPTSwarm [27], DyLAN [133], and MAS-GPT [134], and with two prompting baselines, Chain-of-Thought [165] and Self-Consistency [166]. It is the most accurate method on MATH, HumanEval+, and GPQA, with 72.4%, 82.3%, and 53.2%. On MATH and HumanEval+, it leads MAS-GPT, the strongest prior method, by 3.7 and 3.4 pp, and on GPQA the other methods reach 33.6 to 40.2%. Self-Consistency leads on GSM8K, with 95.0% against 93.8%. These gains come at moderate cost. Across the study’s five worker backbones, the primitive-based MAS reduces token usage and inference latency by about 3–4× relative to TextMAS, with 1.3–1.6× overhead relative to single-agent inference, and on Llama-3-70B-Instruct it uses fewer output tokens than each MAS baseline, the Organizer’s 200–400 tokens per query included. Analysis. The ablations attribute the gain to per-query composition and to KV cache communication. Replacing the Organizer’s choice with a random structure from the Knowledge Pool lowers accuracy by 4.9 to 6.6 pp with Qwen38B and by up to 12.4 pp with R1/Llama-70B, and removing the pool costs up to 11.9 pp. Replacing GPT-5.2 with Claude-4 [175] as the Organizer keeps accuracy within 1.8 pp, so the main benefit comes from problem-aware structure selection and not from a particular Organizer model (Tables 46 and 47). KV cache communication in turn depends on re-indexing the sender’s keys for RoPE. Without the re-indexing, the primitive-based MAS on R1/Llama70B falls from 85.3% to 31.1% on HumanEval+ and from 81.9% to 36.6% on MedQA, and each primitive alone falls to between 16.7% and 34.7%, so the loss arises within the primitives; with Qwen3-8B, the largest drop is 13.3 pp (Table 30). With text communication, the three primitives reach 50.0, 56.7, and 53.3% on Qwen3-8B AIME25, above the 46.7% of a single agent, and KV caches raise them to 60.0, 66.7, and 66.7%, so the primitive abstraction and KV cache communication each contribute. We provide the full framework of Agent Primitives, its overview figure, and additional results in Appendix B.5. The agents of an Agent Primitive share one LLM inside the primitive; when the units that must coordinate are the components of a repository, Dev-Primitives make each component a primitive of its own.

4.7. Dev-Primitives for Software Engineering The last mile of software engineering runs through the repository. The program state that a change must respect is spread over interdependent sources, configurations, and tests, and each edit changes the repository that the next step reads, so a single agent repeatedly reconstructs this distributed state within one trajectory, whose growing history leads to context explosion and semantic drift. Dev-Primitives [49] let individual software components reason about their own responsibilities and communicate relevant information directly to one another, and HERMES instantiates them through dynamic activation, localized collaboration, environment execution, and diagnosis-driven revision.

24

The Last Mile to the Real World

Experiments

Table 12: HERMES against same-backbone harnesses on SWE-bench Verified. Resolved rate (%, mean ± standard error over 500 instances), cost per instance and latency for each harness and backbone, with Δ the resolved-rate change of HERMES in pp. HERMES is run once per backbone, so a backbone compared with several harnesses repeats its HERMES results, and Qwen3-8B is run with HERMES only [49]. Green marks the best and amber the second-best value in each Resolved, Cost/Test, and Latency column. Lower is better for cost and latency. Baseline Results Backbone

HERMES Results

Resolved (%) Cost/Test ($) Latency (s) Resolved (%) Cost/Test ($) Latency (s)

Δ Res.

Baseline harness: mini-SWE-agent GPT-5.6 Sol 96.20 ± 0.86 GPT-5.6 Terra 95.40 ± 0.94 GPT-5.6 Luna 93.00 ± 1.14 Claude Fable 5 95.00 ± 0.98 Claude Opus 5 97.00 ± 0.76 Claude Sonnet 5 79.60 ± 1.80 DeepSeek-V4 77.40 ± 1.87 GPT-5.5 82.60 ± 1.70 GPT-5.4-mini 73.00 ± 1.99 GPT-5-mini 60.80 ± 2.19 Claude Opus 4.8 88.60 ± 1.42 Claude Opus 4.7 82.00 ± 1.72 Gemini-3.5-Flash 78.80 ± 1.83 Gemini-3.1-Pro-Preview 78.80 ± 1.83 GPT-5.4 (xhigh) 78.20 ± 1.85 Claude Opus 4.6 (Thinking) 78.20 ± 1.85 GPT-5.3-Codex 78.00 ± 1.85 Claude Sonnet 4.6 77.40 ± 1.87 Claude Opus 4.5 (Thinking) 76.40 ± 1.90 Gemini-3-Pro 76.40 ± 1.90

1.15 0.40 0.04 2.05 1.29 1.49 0.44 1.36 0.51 0.05 1.92 2.42 0.95 0.78 0.80 1.22 0.46 1.30 1.05 0.77

182.37 180.00 201.07 356.18 576.99 962.37 634.61 426.43 326.46 187.13 566.95 441.99 254.13 312.26 307.12 350.76 246.53 511.79 320.54 394.49

97.00 ± 0.76 96.20 ± 0.86 95.60 ± 0.92 96.00 ± 0.88 97.00 ± 0.76 85.80 ± 1.56 82.80 ± 1.69 85.60 ± 1.57 82.40 ± 1.70 80.20 ± 1.78 91.00 ± 1.28 86.00 ± 1.55 84.00 ± 1.64 83.40 ± 1.67 83.80 ± 1.65 84.40 ± 1.62 83.00 ± 1.68 82.80 ± 1.69 81.80 ± 1.73 82.20 ± 1.71

1.80 0.98 0.10 4.50 2.25 2.10 0.83 2.45 0.83 0.21 2.65 3.10 1.42 1.26 1.23 1.86 0.79 1.92 1.64 1.31

205.34 196.71 218.46 381.27 472.18 824.63 418.52 398.74 347.82 213.55 482.73 396.52 276.84 328.75 322.68 341.28 267.39 438.64 336.28 381.73

+0.80 +0.80 +2.60 +1.00 0.00 +6.20 +5.40 +3.00 +9.40 +19.40 +2.40 +4.00 +5.20 +4.60 +5.60 +6.20 +5.00 +5.40 +5.40 +5.80

Baseline harness: Claude Code Claude Opus 4.8 85.80 ± 1.56

0.67

135.40

91.00 ± 1.28

2.65

482.73

+5.20

Baseline harness: Codex GPT-5.5

76.40 ± 1.90

0.65

108.54

85.60 ± 1.57

2.45

398.74

+9.20

–

–

–

80.60 ± 1.77

0.13

229.81

–

No baseline harness Qwen3-8B

Setup. We evaluate HERMES on four benchmarks: SWE-bench Verified [16], 500 human-validated GitHub issues; SWE Refactor Bench [82], 20 whole-repository migrations scored by a composite; Terminal-Bench 4.0 [17], 66 terminal tasks run five times each; and DevOps-Gym [83], scored by the unweighted mean over its build and configuration, monitoring, issue-resolving, and test-generation stages. The baselines are the harnesses of published configurations, among them mini-SWE-agent [8], Claude Code [5], Codex [4], and OpenHands [9], and HERMES runs on the backbone and reasoning effort of each, so that the comparison isolates the harness. The backbones include GPT-5.6 Sol, Terra, and Luna, Claude Opus 5 and Sonnet 5, DeepSeek-V4, and Qwen3-8B [169], with medium reasoning effort unless stated otherwise and 𝐵 = 3 revision rounds. Main Results. With GPT-5.6 Sol at medium effort, HERMES improves on the matched baseline harnesses by 12.4 pp on average over the four benchmarks. On SWE-bench Verified (Table 12), it raises the resolved rate over mini-SWE-agent for 19 of the 20 backbones, ties on Claude Opus 5 at 97.00%, and improves on Claude Code and Codex by 5.20 and 9.20 pp. The gains are largest where the baseline is weakest, 19.40 pp for GPT-5-mini, from 60.80% to 80.20%, and smallest on frontier backbones: with GPT-5.6 Sol, HERMES resolves 97.00 ± 0.76% against 96.20 ± 0.86%, at $1.80 against $1.15 per task. Coordination need not cost time, since with Claude Sonnet 5 HERMES gains 6.20 pp while average latency falls from 962.37 s to 824.63 s. With Qwen3-8B in all roles, it resolves 80.60%. The largest gain with GPT-5.6 Sol at medium effort is on whole-repository migration: on SWE Refactor Bench (Table 31), HERMES raises the composite by 24.5 pp, from 6.5% under Codex to 31.0%, and at high effort from 25

The Last Mile to the Real World

Experiments

Table 13: HERMES against same-backbone harnesses on Terminal-Bench 4.0. Resolution (%, mean ± standard deviation over five runs of the 66 tasks) with tokens and cost totalled over each complete evaluation, per model and reasoning effort, and Δ the change of HERMES in pp. Qwen3-8B is run with HERMES only [49]. Green marks the best and amber the second-best value in each Resolution, Tokens, and Cost column. Lower is better for tokens and cost. Baseline Results Model

Effort

HERMES Results

Resolution (%) Tokens (B) Cost ($k) Resolution (%) Tokens (B) Cost ($k)

Δ Res.

Baseline harness: Codex medium GPT-5.6 Sol max

33.0 ± 3.4 37.3 ± 3.8

3.6 4.4

1.9 2.5

51.8 ± 2.2 55.5 ± 3.1

6.8 8.9

3.4 4.7

+18.8 +18.2

GPT-5.6 Terra

medium max

18.2 ± 3.0 21.5 ± 3.3

4.4 5.7

1.2 1.7

44.5 ± 3.3 48.2 ± 2.5

7.2 9.1

2.2 2.9

+26.3 +26.7

GPT-5.6 Luna

medium max

14.5 ± 2.6 17.3 ± 2.8

9.1 11.6

0.2 0.3

29.1 ± 2.9 33.3 ± 3.2

13.2 15.8

0.4 0.5

+14.6 +16.0

Baseline harness: Claude Code medium 47.0 ± 3.2 Claude Opus 5 max 51.8 ± 3.4

5.2 6.5

4.8 6.0

48.5 ± 3.6 53.0 ± 2.1

7.3 9.4

6.7 8.6

+1.5 +1.2

Claude Fable 5

max

44.5 ± 3.8

3.8

7.3

49.7 ± 3.3

6.2

10.1

+5.2

GLM-5.3

max

41.8 ± 3.2

8.7

2.7

46.4 ± 2.8

11.5

3.9

+4.6

Claude Opus 4.8

max

23.6 ± 3.6

6.4

6.5

34.2 ± 3.8

9.0

8.4

+10.6

Claude Sonnet 5

medium max

10.6 ± 2.7 12.4 ± 3.1

15.9 21.6

7.1 9.6

35.8 ± 2.3 40.0 ± 3.5

13.4 16.8

5.8 7.4

+25.2 +27.6

Baseline harness: mini-SWE-agent DeepSeek-V4 max 28.8 ± 3.4

7.8

0.7

39.4 ± 3.1

10.1

1.1

+10.6

Gemini-3.8-Flash high

19.1 ± 3.4

17.2

1.8

30.3 ± 3.7

20.4

2.6

+11.2

Gemini-3.7-Flash high

11.2 ± 2.4

11.1

1.3

24.2 ± 3.4

14.8

1.9

+13.0

Baseline harness: Grok Build Grok 4.6 high

20.3 ± 3.1

4.0

3.6

31.5 ± 2.5

6.2

5.2

+11.2

Grok 4.5

12.4 ± 2.6

3.4

2.1

25.5 ± 2.2

5.0

3.0

+13.1

–

–

–

27.6 ± 2.7

14.3

0.7

–

high

No baseline harness Qwen3-8B –

19.0% to 36.5%; three repetitions of the default configuration give 31.0 ± 1.3. On Terminal-Bench 4.0 (Table 13), HERMES with GPT-5.6 Sol resolves 51.8% against 33.0% under Codex at medium effort and 55.5% against 37.3% at maximum effort, and GPT-5.6 Terra gains 26.3 and 26.7 pp. The smallest gains, 1.5 and 1.2 pp, are for Claude Opus 5, whose Claude Code configurations are the two strongest baselines. Most configurations spend more tokens, but with Claude Sonnet 5 at maximum effort, HERMES raises resolution by 27.6 pp, from 12.4% to 40.0%, while total cost falls from $9.6k to $7.4k. On DevOps-Gym (Table 14), the stage mean rises with GPT-5.6 Sol from 49.89% under Codex to 55.41%, with gains in all four stages. Across all baseline harnesses, HERMES raises the mean by 4.78 to 31.70 pp, and with Claude Opus 5 it reaches the highest mean, 57.10%. Analysis. Removing one mechanism at a time shows where the gain arises (Table 15). Among the four removals, diagnosis feedback matters most: without it, Terminal-Bench 4.0 resolution falls from 51.8% to 41.2% and the SWE Refactor composite from 31.0 to 20.0, the largest drop on each benchmark. Once edits are artifact-local, a failure observed at the repository level must be routed back to the component that owns the responsible implementation, and diagnosis provides this routing. Execution feedback ranks second. Removing inter-primitive communication lowers the Refactor composite to 23.5 and Terminal-Bench resolution to 46.1%, larger losses than on SWE-bench Verified (97.0 to 94.2), in line with its role in coordinating changes that span files. Keeping all initially identified Dev-Primitives active lowers the scores by 1.2 to 3.0 points at 1.61 to 1.88 times the cost (Table 52). The primitives add value beyond the choice of components: one editor given the same plan and components resolves 38.8% of

26

The Last Mile to the Real World

Experiments

Table 14: HERMES against same-backbone harnesses on the four DevOps-Gym stages. Stage success rates (%), with Avg. the unweighted mean over stages and Δ the change of HERMES in pp. DeepSeek-V4 uses maximum reasoning effort. HERMES is run once per backbone, so a backbone compared with several harnesses repeats its HERMES results, and Qwen3-8B is run with HERMES only [49]. Green marks the best and amber the second-best value in each stage and Avg. column. Baseline Results Model

Build Monitor. Issue

Baseline harness: Codex GPT-5.6 Sol 77.78 GPT-5.6 Terra 74.07

HERMES Results

Test

Avg. Build Monitor. Issue

Test

Avg.

Δ Avg.

38.24 35.29

40.97 42.58 49.89 83.33 38.71 40.32 47.10 79.63

44.12 41.18

46.13 48.06 55.41 43.55 44.84 52.30

+5.52 +5.20

Baseline harness: Claude Code Claude Opus 5 79.63 41.18 Claude Sonnet 5 68.52 29.41 Claude Sonnet 4 51.85 20.56

42.90 44.52 52.06 85.19 34.84 36.45 42.31 75.93 23.87 13.87 27.54 62.96

47.06 38.24 26.47

47.10 49.03 57.10 40.65 41.94 49.19 29.35 25.48 36.07

+5.04 +6.88 +8.53

Baseline harness: OpenHands DeepSeek-V4 75.93 35.29 Claude Sonnet 4 42.59 14.70 o4-mini 24.07 8.82 Qwen3-Coder-30B 20.37 5.89 Gemini-2.5-Pro 16.66 11.76 DeepSeek-V3.1 11.11 0.00

39.35 41.29 47.97 81.48 23.87 11.61 23.19 62.96 10.32 8.70 12.98 35.19 13.22 6.13 11.40 31.48 10.96 2.90 10.57 29.63 14.20 3.22 7.13 27.78

41.18 26.47 14.71 11.76 17.65 8.82

44.19 29.35 18.06 20.00 17.42 21.61

Baseline harness: mini-SWE-agent GPT-5.6 Sol 75.93 35.29 GPT-5.6 Luna 66.67 29.41 Claude Sonnet 4 29.62 2.91

39.35 40.97 47.89 83.33 32.90 34.84 40.96 72.22 5.16 0.98 9.67 62.96

44.12 35.29 26.47

46.13 48.06 55.41 +7.52 38.71 40.32 46.64 +5.68 29.35 25.48 36.07 +26.40

Baseline harness: SageAgent GPT-5.3-Codex 81.82 35.29

32.14 37.99 46.81 85.19

41.18

37.42 42.58 51.59

Baseline harness: Aider Claude Sonnet 4 5.55

0.00

9.67

2.25

4.37

62.96

26.47

29.35 25.48 36.07 +31.70

No baseline harness Qwen3-8B

–

–

–

–

59.26

26.47

31.29 32.90 37.48

–

45.48 25.48 16.45 14.84 13.23 12.90

53.08 +5.11 36.07 +12.88 21.10 +8.12 19.52 +8.12 19.48 +8.91 17.78 +10.65

+4.78

–

Table 15: Removing HERMES’s messages, on-demand activation, execution feedback or diagnosis. GPT-5.6 Sol, each variant keeping the backbone, reasoning effort, revision budget and environment; resolved rate (%) on SWE-bench Verified and Terminal-Bench 4.0, composite score on SWE Refactor and mean stage success (%) on DevOps-Gym, with the change from full HERMES in pp in parentheses [49]. Green marks the best and amber the second-best value in each column. Setting

SWE-bench

Refactor

Terminal

DevOps

HERMES

97.0

31.0

51.8

55.41

94.2 (−2.8) 95.8 (−1.2) 93.8 (−3.2) 92.6 (−4.4)

23.5 (−7.5) 28.0 (−3.0) 21.5 (−9.5) 20.0 (−11.0)

46.1 (−5.7) 49.4 (−2.4) 43.6 (−8.2) 41.2 (−10.6)

50.34 (−5.07) 53.26 (−2.15) 48.17 (−7.24) 46.73 (−8.68)

w/o Inter-Primitive Comm. w/o On-Demand Activation w/o Execution Feedback w/o Diagnosis Feedback

Terminal-Bench tasks, below each single-mechanism removal, and 45.5% at matched compute, against 51.8% for HERMES (Table 32). The Dev-Primitives also run well on a small model. With GPT-5.6 Sol for activation and diagnosis, Qwen3-8B DevPrimitives stay within 4.5 points of the homogeneous Sol configuration on all four benchmarks, at 26.2% lower Terminal-Bench cost, and a stronger model pays off more in diagnosis than in activation (Tables 50 and 51). We provide the full framework of HERMES, its overview figure, and additional results on component selection, the revision budget, and related multi-agent frameworks in Appendix B.6. HERMES chooses its Dev-Primitives before each round; when what a query needs changes as the search proceeds, the choice of primitives must follow the needs that remain open.

27

The Last Mile to the Real World

Experiments

4.8. Navigation Primitives for Large Information Spaces In a large information space, the requirements of a query change as evidence arrives: new evidence can resolve or refine a requirement, reveal another, and redirect the search [187]. At the last mile, coordination must therefore follow what remains unresolved, whereas partition-driven multi-agent systems such as LongAgent [109] and Chain of Agents (CoA) [110] assign one member to each chunk of the input, so their coordination grows with the space. Navigation Primitives are workers that each search one territory of the space, and ANTMAN [50] coordinates them through a revisable Need Graph that tracks a query’s unresolved information needs and decides which primitives take part in the search (Section 3.4). Table 16: ANTMAN in structured information spaces. RepoProbe-Python and SWE-QA-Pro scores, linearly rescaled to 0–100 from their original 10- and 50-point scales, and GAIA-Text-103 accuracy (%) by difficulty level; ANTMAN uses GPT-4.1, ANTMAN-H† keeps the GPT-4.1 orchestrator and runs every other component on Qwen3-8B, methods run in ANTMAN’s evaluation harness share its question sets and model setting, and – marks a benchmark outside a method’s evaluation scope [50]. Green marks the best and amber the second-best value in each column. GAIA-Text-103 Method

RepoProbe SWE-QA-Pro

L1

L2

55.74 60.90 58.30 60.40

28.2 53.8 43.6 25.6

17.3 8.3 42.3 33.3 26.9 25.0 19.2 0.0

25.86 –

64.92 –

46.2 30.8 53.8 38.5

8.3 8.3

34.0 40.8

Repository-specific agents ReAct + RepoGraph RepoDistill

25.71 28.27

67.38 65.10

– –

– –

– –

– –

Benchmark-specific reference SWE-QA-Pro Agent

–

77.58

–

–

–

–

34.10 38.21

74.60 79.78

69.2 55.8 25.0 69.2 53.8 41.7

57.3 58.3

Retrieval and iterative retrieval baselines Direct 31.08 Iterative Sparse Retrieval 33.15 Dense Retrieval 31.51 S2G-RAG 22.87 General tool-using agents ReAct OWL

†

ANTMAN-H ANTMAN

L3

Overall 20.4 45.6 33.0 19.4

Setup. Three benchmarks require navigation in struc- Table 17: Activation as the information space grows 16× at tured spaces: RepoProbe-Python, 108 questions over approximately fixed demand. Growth from 32K to 512K in eight Python repositories [159]; a frozen 80-question ANTMAN’s study and the change in F1 on its fractional scale; all methods use GPT-4.1 [50]. Green marks the best and amber the set of SWE-QA-Pro [160]; and GAIA-Text-103, the second-best value in each growth column. Lower is better for coortext subset of GAIA for tool-using information seek- dination, calls, and cost growth. ing [18]. Two controlled interventions grow the space Coord. Calls Cost and the demand separately, and multi-document QA Method Growth Growth Growth Δ F1 over HotpotQA [93], 2WikiMultiHopQA [157], and Multi-agent baselines MuSiQue [158] checks answer quality on standard tasks. LongAgent 15.29× 12.81× 14.22× +0.023 CoA 15.33× 11.30× 16.11× −0.004 The baselines include direct, iterative sparse, and dense retrieval, S2G-RAG [108], the tool-using agents ReANTMAN 1.23× 1.21× 6.90× +0.065 Act [1] and OWL [113], the repository agents ReAct with RepoGraph [167] and RepoDistill [168], and the benchmark-specific SWE-QA-Pro Agent. ANTMAN runs all components on GPT-4.1, as do the other methods in its evaluation harness, and ANTMAN-H runs all components except the orchestrator, the Navigation Primitives included, on Qwen3-8B [169]. Main Results. ANTMAN obtains the best overall score on all three structured benchmarks (Table 16): 38.21 on RepoProbe against 33.15 for iterative sparse retrieval, 79.78 on SWE-QA-Pro against 77.58 for the SWE-QAPro Agent and 67.38 for ReAct with RepoGraph, and 58.3% on GAIA-Text-103 against 45.6% for iterative sparse 28

The Last Mile to the Real World

Experiments

retrieval. On GAIA, a web-and-tool substrate with one territory-backed worker, unresolved needs decide only where the search continues, so the gain is not tied to repository-specific navigation machinery. ANTMAN-H scores 34.10, 74.60, and 57.3%, above all baselines on RepoProbe and GAIA-Text-103, and its GAIA gap to ANTMAN lies at Level 3, 25.0% against 41.7%. Space Scaling. The space intervention tests Proposition 3.1, which bounds participation by realized demand whatever the number of territories, |ℒ𝑞 |. In LongAgent’s multi-needle Needle-in-a-Haystack PLUS setting, filler grows the distractor space around the preserved evidence of ten questions from 32K to 512K tokens, at approximately fixed demand. Under this 16× growth, the mean number of Navigation Primitives that participate per query, the active coordination, grows 1.23×, against 15.29× for LongAgent and 15.33× for CoA, whose members multiply with the partitions (Table 17). ANTMAN’s calls grow 1.21× and its cost 6.90×, against cost growth of 14.22× and 16.11× for the baselines, while its F1 rises by 0.065. Table 18: Activation as required evidence grows from 1 to 16 units at a fixed 512K space. ANTMAN with GPT-4.1 on matched contexts that keep distractors, evidence locations and worker organization fixed; active coordination is the mean number of participating workers, calls and cost are per query, and complete evidence records whether every required unit was recovered [50]. Required Active Calls Cost Complete Answer Evidence (units) Coordination / Query / Query ($) Evidence (%) Acc. (%) ANTMAN 1 4 16

5.0 4.3 8.0

Demand Scaling. The reverse intervention fixes the 512K space, with its distractors, evidence locations, and eight territories, and raises the required evidence from 1 to 4 to 16 units (Table 18). The answer combines all required values, so omitting one unit leaves it ambiguous. Calls rise from 35.5 to 86.2 per query and cost from $0.107 to $0.293, and ANTMAN recovers the complete evidence and answers correctly at all three levels. Active coordination dips from 5.0 workers at one unit to 4.3 at four units and reaches 8.0 at 16 units; at one unit, most worker activity is exploration that finds no required evidence. Coordination thus stays nearly flat when the space grows and rises when the query needs more.

35.5 43.6 86.2

0.107 0.111 0.293

100 100 100

100 100 100

Table 19: Removing ANTMAN’s need state, need revision, rerouting or recovery. Scores under matched budgets with the relative change from full ANTMAN in % in parentheses, on the 512K controlled condition (F1 on a 0–100 scale) and on 40-question SWE-QA-Pro and GAIA subsets, where full ANTMAN scores 81.43 and 60.00; on the evaluation sets of Table 16 it scores 79.78 and 58.3 [50]. Green marks the best and amber the second-best value in each column. Variant

512K

SWE-QA-Pro

GAIA

F1

Score

Accuracy

68.58 (−15.78%) 78.88 (−3.13%) 75.32 (−7.50%) 72.13 (−11.42%) 76.52 (−6.03%)

50.00 (−16.67%) 50.00 (−16.67%) 42.50 (−29.17%) 60.00 (0.00%) 52.50 (−12.50%)

81.43

60.00

Adaptive coordination ablations Static ANTMAN 40.32 (−52.02%) Graph-free Adaptive 64.01 (−23.82%) w/o Need Revision 74.44 (−11.41%) w/o Adaptive Rerouting 82.20 (−2.18%) w/o Recovery 60.12 (−28.45%) ANTMAN

84.03

Evolving Need State. The ablations in Table 19, all under a budget of 100 LLM calls per query, separate the explicit need state from generic adaptive control. Graph-free Adaptive keeps adaptive worker selection, rerouting, and recovery but replans from the accumulated evidence and interaction history, without any explicit Need Graph. Its 512K F1 falls from 84.03 to 64.01 while it activates a comparable number of distinct workers, 3.20 against 3.17, so the gap does not come from wider participation; on the 40-question GAIA subset, its accuracy falls from 60.00 to 50.00. Static ANTMAN freezes the initial Need Graph and routing plan, and its 512K F1 falls to 40.32. Single removals play complementary roles: removing recovery gives 60.12 at 512K, removing need revision gives 42.50 on GAIA, and removing adaptive rerouting gives 72.13 against 81.43 on SWE-QA-Pro and leaves single-worker GAIA unchanged. Further Results. On standard multi-document QA (Table 33), ANTMAN has the best macro average of nine methods, 0.711 EM and 0.814 F1, against 0.700 and 0.799 for S2G-RAG; it is best on HotpotQA and MuSiQue, and 29

The Last Mile to the Real World

Experiments

S2G-RAG leads on 2WikiMultiHopQA. Narrowing the context searched per need also makes it robust to evidence position: its Middle F1 minus the mean of its Early and Late F1 is +0.015, against −0.181 for direct full-context inference, which shows the lost-in-the-middle pattern [39]. We provide the full method of ANTMAN with its overview figure, implementation details, and these results in Appendix B.7. Across these experiments, unresolved needs decide which Navigation Primitives run, under a control program that is the same for all queries; Harness Primitives compose that program itself for each task.

4.9. Harness Primitives for Agent Harnesses For a fixed task-solving model, the actor, an agent’s success depends on its harness: the control program that constructs the actor’s requests, executes its actions through the available tools, returns observations, and decides when to terminate [51, 143]. The last mile then reaches the harness itself, which must fit the task. Hand-designed harnesses such as SWE-agent [8] and Codex [4], and the harnesses that optimizers such as MetaHarness find, apply one design to all tasks, yet a mechanism that improves one task can impose overhead or context distraction on another. Generating harness code for each task avoids this mismatch but adds generation and debugging cost, with execution risks that compound as more mechanisms are generated. Harness Primitives make task-specific harnesses possible without generating or repairing mechanism code at test time. They are control operations implemented and validated during development, each with an application scope and a composition contract, and STITCH [51] selects the primitives that suit each task and compiles them into its harness before the actor starts. Setup. We evaluate on 100 randomly sampled tasks of SWE-bench Verified [16], which asks for repairs of real repository issues, and on 45 randomly sampled tasks of Terminal-Bench 2.0 [17], which covers tasks in a terminal environment; results are broken down by repository (Django, SymPy, and the others) and by difficulty. Both samples are disjoint from the 100 SWE-bench and 44 Terminal-Bench development tasks on which the primitives were developed. Each task gets two attempts, and we report Pass@1, the fraction of successful attempts; Pass@2, the fraction of tasks solved in at least one attempt [183]; and Pass2 , the fraction solved in both. GPT-5.6 Luna at medium reasoning effort [170] serves as both the actor and the composer, and GPT-5.6 Sol is the development model. The baselines are the mini-SWE-agent 2.4.6 seed harness [8], SWE-agent, Codex, and the harness optimizers MemoHarness [144] and MetaHarness [143], which start from the same seed. Two controls keep the primitive implementations and vary the composition alone: a fixed harness that applies one focused primitive, with its required dependencies and shared supporting operations, to all tasks, and a random composer. Table 20: STITCH’s task-specific harnesses against fixed harnesses on SWE-bench Verified. GPT-5.6 Luna actor on 100 sampled tasks with two attempts each, where Pass@1 is the fraction of successful attempts and Pass@2 and Pass2 the fractions of tasks solved in at least one and in both attempts (%) [51]. Green marks the best and amber the second-best value in each column. Overall Method

Django

Pass@1 Pass@2 Pass

2

SymPy

Pass@1 Pass@2 Pass

2

Other repos 2

Pass@1 Pass@2 Pass

Pass@1 Pass@2 Pass2

78.8 71.2 80.8 76.9 78.8

71.2 36.5 76.9 63.5 69.2

75.0 42.9 64.3 67.9 64.3

78.6 50.0 71.4 71.4 71.4

71.4 35.7 57.1 64.3 57.1

69.1 39.7 85.3 72.1 76.5

76.5 55.9 91.2 79.4 82.4

61.8 23.5 79.4 64.7 70.6

One focused Harness Primitive with its dependencies, fixed across tasks Change-surface Tracer 73.5 77.0 70.0 76.9 80.8 Compatibility-envelope Gate 72.0 77.0 67.0 76.9 80.8 Contract-case Explorer 72.5 78.0 67.0 69.2 76.9 Discriminating-oracle Runner 71.5 76.0 67.0 75.0 78.8 Domain-trace Template 73.0 77.0 69.0 76.0 80.8 Environment-capability Runner 76.0 79.0 73.0 77.9 80.8

73.1 73.1 61.5 71.2 71.2 75.0

60.7 57.1 71.4 57.1 60.7 67.9

64.3 64.3 78.6 57.1 64.3 71.4

57.1 50.0 64.3 57.1 57.1 64.3

73.5 70.6 77.9 72.1 73.5 76.5

76.5 76.5 79.4 79.4 76.5 79.4

70.6 64.7 76.5 64.7 70.6 73.5

Random composer STITCH

71.2 75.0

64.3 75.0

69.6 78.6

58.9 71.4

73.9 83.8

79.0 88.2

68.8 79.4

mini-SWE-agent (seed) SWE-agent Codex MemoHarness MetaHarness

73.0 47.5 79.0 70.5 73.5

73.3 80.5

78.0 63.0 83.0 77.0 79.0

78.0 85.0

68.0 32.0 75.0 64.0 68.0

68.6 76.0

75.0 53.8 78.8 70.2 74.0

75.4 79.8

79.6 84.6

30

The Last Mile to the Real World

Experiments

Table 21: STITCH’s task-specific harnesses against fixed harnesses on Terminal-Bench 2.0. GPT-5.6 Luna actor on 45 sampled tasks grouped by difficulty, two attempts each; metrics as in Table 20 [51]. Green marks the best and amber the second-best value in each column. Overall Method

Easy

Medium

Hard

Pass@1 Pass@2 Pass2 Pass@1 Pass@2 Pass2 Pass@1 Pass@2 Pass2 Pass@1 Pass@2 Pass2 100.0 33.3 66.7 66.7 33.3

65.5 13.8 67.2 62.1 62.1

75.9 17.2 79.3 75.9 72.4

55.2 10.3 55.2 48.3 51.7

38.5 7.7 30.8 26.9 38.5

46.2 15.4 38.5 38.5 46.2

30.8 0.0 23.1 15.4 30.8

One focused Harness Primitive with its dependencies, fixed across tasks State Guard 66.7 75.6 57.8 83.3 100.0 66.7 Check Replay 65.6 75.6 55.6 100.0 100.0 100.0 100.0 100.0 100.0 First-Failure Localizer 62.2 68.9 55.6 Experiment Keeper 58.9 68.9 48.9 83.3 100.0 66.7 Execution Supervisor 65.6 75.6 55.6 100.0 100.0 100.0

77.6 74.1 72.4 65.5 70.7

86.2 86.2 82.8 75.9 82.8

69.0 62.1 62.1 55.2 58.6

38.5 38.5 30.8 38.5 46.2

46.2 46.2 30.8 46.2 53.8

30.8 30.8 30.8 30.8 38.5

71.3 81.0

81.0 89.7

61.5 72.4

39.1 46.2

44.9 46.2

33.3 46.2

mini-SWE-agent (seed) SWE-agent Codex MemoHarness MetaHarness

Random composer STITCH

60.0 13.3 56.7 53.3 55.6

63.5 72.2

68.9 17.8 66.7 66.7 66.7

71.9 77.8

51.1 8.9 46.7 40.0 44.4

55.2 66.7

100.0 33.3 66.7 83.3 66.7

94.4 100.0

100.0 33.3 66.7 100.0 100.0

100.0 100.0

88.9 100.0

Main Results. STITCH attains the highest overall Pass@1, Pass@2, and Pass2 on both benchmarks (Tables 20 and 21). On SWE-bench Verified it reaches 80.5% Pass@1, 7.5 pp above the seed harness, 1.5 pp above Codex, and 7.0 pp above MetaHarness, the stronger harness optimizer. On Terminal-Bench 2.0 it reaches 72.2%, 12.2, 15.5, and 16.6 pp above the same three baselines. The gains extend to repeated success: Pass2 rises over the seed from 68.0% to 76.0% on SWE-bench Verified and from 51.1% to 66.7% on Terminal-Bench 2.0. The controls attribute the gain to the choice made for each task. STITCH exceeds the best fixed primitive by 4.5 pp on SWE-bench Verified and by 5.5 pp on Terminal-Bench 2.0, whereas the random composer, at 73.3% and 63.5%, stays within the range of the fixed primitives. The breakdowns show why a fixed choice falls short: the same primitive can help on some tasks and hurt on others. Contract-case Explorer raises Pass@1 above the seed on the other repositories, 77.9% against 69.1%, but lowers it on Django, 69.2% against 75.0%, and on SymPy no fixed primitive reaches the seed’s 75.0%, which STITCH matches. On Terminal-Bench 2.0, State Guard leads the fixed arms on medium tasks but lowers easy-task Pass@1 from 100.0% to 83.3%, while Execution Supervisor leads them on hard tasks; STITCH reaches 81.0% on medium and 46.2% on hard tasks and keeps 100.0% on easy ones. This is the task-mechanism mismatch behind Proposition 3.2: a fixed harness either pays a primitive’s penalty outside its application scope or forgoes its gain inside it. Composition also scales with the library. Growing the test-time library along one order of addition, from the seed harness (size 0) to eight entries on SWE-bench Verified and six on Terminal-Bench 2.0, raises Pass@1 from 73.0% to 80.5% and from 60.0% to 72.2% (Figure 15). Transfer and Cost. The primitives and the composer carry over to actors not used in development. With implementations and contracts unchanged and the scopes adjusted on development tasks, STITCH averages 84.0% Pass@1 on SWE-bench Verified over GPT-5.6 Terra, DeepSeek-V4-Flash, and Claude-4.5-Haiku, against 75.2% for the seed, 78.5% for Codex, and 79.5% for the best fixed primitive (Table 34); with the GPT-5.6 Terra actor, Codex leads at 88.0% against 86.5%. Each fixed primitive also stays above the seed on this average, so the primitives encode control mechanisms that generalize across actors. Composition is cheap and reliable. It costs 4.3% of one actor attempt on SWE-bench Verified and 2.7% on Terminal-Bench 2.0; a lower bound for writing the complete harness at test time, which counts an oracle coder’s output tokens alone, is about 543× and 638× the cost of composition. Each selected primitive activated and executed without runtime errors (Figure 16): the selected primitives take effect during the actor’s execution, free of the reliability risk of generated harness code. We provide the full framework of STITCH, its overview figure, and the per-actor, library-size, and composition-cost results in Appendix B.8.

31

The Last Mile to the Real World

Analysis

5. Analysis We further analyze the gains of Section 4 by setting the controls of the eight companion studies side by side. The analysis asks four questions: where the benefit of a primitive comes from, what the channels between agents and primitives carry, what decides which primitives run and what is revised after a failure, and what a task costs and where a strong model pays off. Each question draws on more than one family, so its answer concerns the method that the eight families share.

5.1. Where the Benefit Comes From With the backbone and the reused resources fixed, the controls below vary how a primitive applies the resource it owns (Definition 3.1), what it carries, and how much of the library its assessment reaches. Application versus Access. At matched backbone, calls, and tokens, the gain comes from the primitive, which extra inference does not replace. HEART’s budget-matched control reallocates the calls of the Tool Primitives to value resolution, which self-refines its arguments over raw schemas; with Qwen3-8B, it reaches 34.2% ToolBench pass rate, against 75.1% with Tool Primitives (Table 26). Nor does access to the same components suffice. When LEGO shows the caller the retrieved components with their contracts and tests as read-only context, delivery reaches 0.3544, close to the 0.3469 of diagnosis without any library and below the 0.5134 of Code Primitives (Table 6). Primitive Fields. Most of the margin over generic editing lies in the fields of 𝑃𝑖 (Table 28). A generic editor over the same retrieved components scores 0.4093, and adding the carried tests 𝑉𝑖 , the interface contract and provenance 𝐼𝑖 , 𝑋𝑖 , and the dependency closure 𝐷𝑖 raises it to 0.5039, within 0.0095 of primitive adaptation (𝑝 = 0.49) [45]. Withholding 𝑉𝑖 from an otherwise complete primitive lowers delivery to 0.4796. Source-Model Content. A source model adds more than context. When each retrieval counterpart receives SourceLearn’s per-question context budget, filled from further down its own ranking, SourceLearn leads on three of the four sources tested with GPT-5.6 Luna (Table 29). On the QA benchmarks with Luna, its gain over Hybrid RAG is 12.1 pp on regions that no guidance task touched and 14.9 pp on touched regions (Table 45). Reach of Assessment. Assessment over the whole library reaches what reranking of a retrieved pool misses. By Proposition 3.3(ii), the hit@𝑘 of any reranker of a similarity pool is capped by whether the pool holds an annotated contract (Eq. (10)). Across three selector backbones, the best reranker of a top-10 pool plateaus between 0.842 and 0.858 Putnam hit@5, while self-advertisement over all 82 contracts (Eq. (3)) reaches up to 0.975 (Table 7). Tool Primitives are found by similarity retrieval and so face the same ceiling, which is high here: retrieval over the 25,519 schema descriptors of ToolFace places 95.8% of ToolBench and 97.2% of NESTFUL gold tools in its top ten (Table 35). Validation and Grounding Audits. HEART’s result validation agrees with human labels on 200 sampled ToolBench and NESTFUL decisions at 0.97 precision and 0.96 recall, with an 8% false-pass rate (Table 36), so the failures it reports can drive revision (Section 5.3) [44]. SourceLearn’s grounding audit finds 13.3% and 3.3% of the sampled units of the final StudentAid and pytest source models unsupported by the source elements they cite, while the fully supported share of the HuggingFace model rises from 65.0% to 84.2% (Table 43). The source model therefore complements the source, which stays authoritative and directly accessible [47]. Together, these controls place the benefit in what the primitive owns and in applying it where it lives. What a primitive returns must then reach the agents and primitives that use it.

5.2. Communication Within and Between Primitives What crosses between agents and primitives carries part of the result (Section 3.3). Two families each disrupt one channel while keeping backbones, primitives, and components fixed.

32

The Last Mile to the Real World

Analysis

KV Cache Communication. Agent Primitives run all agents on one LLM, so that one agent’s cache is readable by another. When a receiver reads a sender’s cache after its own system-level cache, the sender’s keys are reindexed to their new positions to preserve RoPE semantics [139]. Without this re-indexing, the composed MAS on R1/Llama-70B falls from 85.3% to 31.1% on HumanEval+, and on Qwen3-8B it loses 13.3 pp on AIME25 (Table 30). Each primitive used alone on R1/Llama-70B also loses 26.7 to 59.7 pp, so the loss already arises in the KV cache communication inside each primitive, before any composition [48]. Messages Between Dev-Primitives. A Dev-Primitive’s directed message 𝑚𝑖 carries the task-relevant information or requirements that it communicates to other components (Section 3.4). With GPT-5.6 Sol, removing these messages while keeping each Dev-Primitive’s local reasoning and edits lowers the SWE Refactor Bench composite from 31.0 to 23.5 and Terminal-Bench 4.0 resolution from 51.8% to 46.1%, larger drops than on SWE-bench Verified (97.0 to 94.2), consistent with messages coordinating changes that span files (Table 15). On Terminal-Bench 4.0, one editor given the same plan and components resolves 38.8%, and 45.5% at matched compute, against 51.8% for HERMES (Table 32), so component-local reasoning with direct communication leads at equal compute. In both families, a cache read at the wrong positions or a requirement that never reaches its component costs accuracy, although each agent and Dev-Primitive performs its own step as before.

5.3. Activation and Revision With the primitives and their channels fixed, COMPOSE decides which primitives run (Eq. (4)), and revision decides what changes after a failure (Eq. (6); Figure 2, middle and bottom). For ANTMAN and STITCH, the interventions below examine the behaviour that Propositions 3.1 and 3.2 predict. Space and Demand. ANTMAN satisfies the premises of Proposition 3.1: each step routes one unresolved need to at most one territory-backed worker (Eq. (48)), and each need is routed at most 𝜌 times. Participation accordingly follows demand and changes little with the size of the space: it grows 1.23× when distractors grow the space 16× around the same evidence, against 15.29× and 15.33× for the members of LongAgent [109] and CoA [110] (Table 17), and it reaches 8.0 workers when the required evidence grows to 16 units in a fixed space (Table 18). Excess Activation. Activating more than the task needs costs accuracy as well as calls. Keeping all Dev-Primitives identified in HERMES’s initial analysis active throughout raises primitive calls from 9.6 to 28.7 per SWE-bench Verified task and lowers the score on each of the four benchmarks by 1.2 to 3.0 points (Tables 15 and 52). Task-Conditioned Choice. Per-task composition beats the best fixed choice, and a random composition falls to the level of the fixed choices. STITCH’s fixed primitives raise Pass@1 on some repositories and difficulty levels and lower it on others (Section 4.9), the heterogeneity that Proposition 3.2 needs. With the GPT-5.6 Luna actor, per-task composition reaches 80.5% Pass@1 on SWE-bench Verified, against 76.0% for the best primitive applied to all tasks and 73.3% for a random composer (Table 20), and Terminal-Bench 2.0 keeps this order (Table 21). The Organizer of Agent Primitives composes per query and reaches 73.3% on Qwen3-8B AIME25 (Table 10). Random structures from its Knowledge Pool reach 66.7%, the level of the best single primitives, and the Organizer without its pool reaches 63.3%. With Claude-4 as the Organizer, composition also reaches 73.3% (Tables 46 and 47). For Agent Primitives, the benefit thus comes mainly from problem-aware structure selection and not from a particular Organizer model [48]. For Harness Primitives, it comes from choosing complementary primitives for each task [51]. Library Size. Growing STITCH’s test-time library of accepted primitives along one order of addition raises Pass@1 from 73.0% for the seed harness (size 0) to 80.5% at size 8 on SWE-bench Verified and from 60.0% to 72.2% at size 6 on Terminal-Bench 2.0 (Figure 15). Drivers of Revision. Removing the mechanism that turns evidence from the run into revision (Eq. (6)) lowers the result in each family that ablates it. Result validation is HEART’s sole judgment of whether a step’s result answers the query; without it, ToolBench pass rate falls from 75.1% to 47.6%, the same value as with no revision rounds (Tables 4 and 37). Without the diagnosis feedback that names the components that revision reactivates 33

The Last Mile to the Real World

Analysis

(Eq. (45)), HERMES’s Terminal-Bench 4.0 resolution falls from 51.8% to 41.2%, the largest drop among its four mechanism removals (Table 15). Without task-local recovery of stalled needs, ANTMAN’s 512K F1 falls from 84.03 to 60.12 (Table 19). Across these interventions, the task decides what runs and what is revised: its unresolved needs bound participation, its features select the composition, and the evidence of its failures and stalled needs decides what changes.

5.4. Cost and Model Allocation What runs and what is revised also sets what a task costs, which each study records in its own unit. Per-Task Cost. At matched cost, reuse delivers more: execution feedback with eight attempts reaches 0.3437 delivery at $2.07 per task, against 0.5134 at $2.08 for Code Primitives, which lead at each attempt limit (Tables 27 and 40). Composing Harness Primitives costs 4.3% of one actor attempt on SWE-bench Verified and 2.7% on Terminal-Bench 2.0, while the output-token lower bound for an oracle that writes the complete harness is about 543× and 638× the cost of composition (Figure 16). As the space grows 16×, ANTMAN’s calls grow 1.21× and its cost 6.90×, against cost growth of 14.22× for LongAgent and 16.11× for CoA (Table 17). Strong-Model Placement. For Dev-Primitives, whose LLM can differ from the activation and diagnosis models, a strong model pays off most in diagnosis (Table 50). With GPT-5.6 Sol in activation and diagnosis, Qwen3-8B DevPrimitives resolve 50.6% of Terminal-Bench 4.0 tasks, against 27.6% with Qwen3-8B in all three roles. A stronger diagnosis model alone raises resolution to between 44.5% and 47.9%, and a stronger activation model alone to at most 38.2%. Sol activation with Qwen3-8B Dev-Primitives and GPT-5.5 diagnosis reaches 49.7% at $2.13k, against 51.8% at $3.40k with Sol in all three roles, a 2.1-pp loss for 37.4% lower expenditure (Table 51). Smaller Models Inside Primitives. With GPT-OSS-20B as the resident LLM of all Code Primitives and as the diagnosis model, LEGO keeps 95.1% of its delivery (0.4880) at 24.0% lower recorded-call cost (Table 27). ANTMAN-H, which runs all components except the GPT-4.1 orchestrator on Qwen3-8B, reaches 57.3% against 58.3% on GAIA-Text-103 and 25.0% against 41.7% on its hardest level (Table 16). With GPT-5.4 in decomposition, value resolution, and result validation and Qwen3-8B in the Tool Primitives, HEART reaches 𝜏 2 -Bench Pass1 of 0.84, 0.75, and 0.67 on retail, airline, and telecom, close to the 0.86, 0.78, and 0.68 of GPT-5.4 in all roles (Appendix B.1) [44]. In these three families, as in Dev-Primitives, a smaller model takes the work inside the primitives while a stronger model coordinates. Summary. Across the four questions, the analyses locate the benefit in applying each resource inside its primitive, in what the primitive carries, in channels that keep their meaning, and in activation and revision controlled by the task. At matched budgets the primitives deliver more, and smaller models inside them keep most of the result at lower cost while a stronger model coordinates. The end-to-end outcome of a real-world task thus depends both on the competence that each primitive keeps with its resource and on the mechanisms that HEELO builds around the primitives (Section 3.3), which each family realizes as its task needs.

6. The heelo.dev Platform for Real-World Tasks On the research benchmarks of Section 4, as in the cross-family protocol of Appendix E, the environment is frozen and the grader is fixed in advance. Real-world tasks also run against live services, accounts, and people. To test HEELO under these conditions, we built heelo.dev (https://heelo.dev), a platform that exposes real-world actions as agent-native primitives (Figure 3), and we compared agents with and without HEELO on 46 such tasks.

6.1. Platform Overview heelo.dev is a tool execution backend for AI agents. It exposes third-party actions, such as flights, calendars, email, social posting, company research, and browser automation, as tools behind one invocation endpoint, and each tool 34

The Last Mile to the Real World

The heelo.dev platform

Figure 3: The heelo.dev platform. Agents such as Claude Code, Codex, or Cursor reach real-world tools through HEELO, which orchestrates agent-native primitives for each task: it plans the task, routes each need to a primitive, executes the primitives, and verifies the result, revising the plan when a check fails. Four loops test the platform end to end: distribution creates and publishes content across channels, acquisition finds potential customers and sends personalized outreach, the personal assistant takes over everyday tasks across apps and tools, and validation discovers real problems and validates market demand.

follows the agent-native design of Section 3.2. A tool accepts the fields of its input schema or a natural-language request and encapsulates its own execution strategy, from LLM calls to browser automation and third-party APIs, and it keeps its state in a session. When a request lacks a detail, the tool returns a clarification question with the session’s identifier, and the caller continues the session until the tool returns a result or an error. Callers reach the tools in four ways: direct invocation; a natural-language router that selects a tool from the descriptions and examples of all registered tools; a ReAct loop that chains tools step by step, each result informing the next action; and an MCP server through which external agents, such as Claude Code, connect to HEELO. Tool calls are recorded in a shared log.

6.2. Four Real-World Loops The real-world tasks of the platform are organized as four loops, each with one end-to-end task (Table 22). Each loop adds three tool families to the platform, such as mail.read, doc.extract, and doc.render for the assistant, and draws on tools that the platform already has, such as calendars, Notion, and the posting tools of single platforms. Distribution publishes a campaign’s content to several channels, each under its own platform rules, and hosts the pages that capture visitors. Acquisition finds people and companies with a matching need and sends them multi-step outreach from an authenticated domain. The assistant turns an invoice email into checked fields, approved writes, and a receipt, and validation mines the problems that people report and tests the demand for one of them. The loops also feed one another: distribution attributes each click from acquisition’s outreach to the contact who clicked and stops that contact’s sequence once the contact is captured, and validation reaches real people through acquisition. A loop’s task is therefore solved only when the tools compose within a loop and across loops, each tool’s result reaching the tool that consumes it. The four tasks need 26 capabilities: distribution, for

35

The Last Mile to the Real World

The heelo.dev platform

Table 22: The four real-world loops of heelo.dev. The end-to-end task of each loop and the tools that it composes, grouped by category. A name that ends in * stands for all tools of a family, such as notion.* for reading, creating, updating, and deleting Notion pages. Loop

Distribution

Task description Publish a campaign’s landing page, post one text adapted to each platform’s rules, render its share cards, and attribute each click to the contact who clicked

Category

Tools

Page publishing Click attribution Cross-platform posting Single-platform posting

inbound.page.publish, inbound.page.audit inbound.link.wrap, inbound.stats channel.expand twitter.post, linkedin.post, instagram.post, bluesky.post_browser media.render.card

Share cards

Acquisition

Find people or companies with a matching need, verify their contacts, and send them a multi-step sequence from an authenticated domain, checking consent and suppression before every send

Find leads Research companies Manage contacts Send outreach

Read emails

Assistant

Read an invoice email, extract and check its fields, and, after a person approves, add the calendar event and the Notion page and render a PDF receipt

Manage documents

Calendars and notes Travel and forms

Validation

Mine the problems that people report in a domain, size the market for one of them, and test its demand with an experiment that reaches real people through acquisition

Scraping Problem discovery Trend analysis Market and competitors Demand experiment

demand.radar icp.search, icp.export_results contact.resolve, outbound.mail.contact_*, outbound.mail.consent_record outbound.mail.sender_*, outbound.mail.sequence_*, outbound.mail.suppress_* mail.read_list, mail.read_get, mail.read_attachment, mail.read_watch doc.extract_text, doc.extract_fields, doc.review_decide, doc.render_pdf, doc.render_docx, doc.render_xlsx google.calendar_*, outlook_calendar.*, apple_calendar.*, notion.* kiwi.book_flight, booking_com.reserve_stay, web.fill_form scrape.structured.extract problem.mine trends.signals market.size.assess validation.run.*

instance, needs page publishing, click attribution, safety and input checks, multi-platform posting, budget control, share-card rendering, and the launch pipeline that chains them.

6.3. Evaluation Protocol The acceptance criterion 𝑅𝑞 of a loop’s task (Section 3.1) is met only when the whole chain runs live: the assistant loop’s live runner, for instance, checks the chain against real accounts link by link, each link reported as passed, failed, or not run, and a link that has not run never counts as passed. To measure what the primitives add, we pose the 26 capabilities as 46 tasks that name no heelo.dev tool. The pass criteria of each task are fixed before its first run as facts that a person can check, such as an invoice total, the pages of a PDF, or a refused channel, and the tasks include failure paths, such as a wrong contact identifier that must return a clean error or a page that robots.txt disallows. The tasks also serve as acceptance tests of the primitives: when a run exposes a defect in a primitive, the primitive is fixed and the task runs again, and we report the runs after the fix. We run two agents, Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex, both at medium reasoning effort, in two arms each: on their own, and connected to HEELO through the MCP server of heelo.dev, which gives the agent HEELO’s primitives as tools; in five validation tasks, Codex with HEELO is also prompted to use them. Each agent keeps the built-in tools that a task allows, from web search and web fetch to file editing and a sandboxed shell. On its own, an agent receives no credentials, and each run starts in an empty directory outside the heelo.dev repository, in a new session unless it follows up an earlier task; Claude Code cannot read this repository, and Codex, whose shell can read outside its directory, runs with its user configuration and connected apps turned off. A task 36

The Last Mile to the Real World

The heelo.dev platform

with side effects uses test accounts and allowlisted recipients only, and an agent on its own, which cannot act on any account, is scored by the pass criteria that its work meets, from a ready draft to a script that needs only a credential. A connected agent calls the staging deployment of heelo.dev, or a local build where a task needs what staging lacks, such as a sandbox channel, a second tenant, a CMS site, or a browser for Lighthouse, and the validation tasks run on a local build with the loop’s latest fixes. Most tasks run three times per arm under a fixed time limit. In distribution and the assistant loop, Claude Sonnet 5.5, which is not one of the tested agents, grades the answers blind, with the arm names and heelo.dev’s tool names removed, and a person reviews its grades against the run records; acquisition and validation runs are checked against their criteria from the answers or tool results, the artifacts they link, and the request logs. Each run records whether it passes, its wall-clock time and tokens, and its cost at API prices, which for each model call is its tokens multiplied by the model’s price per token, with cached input tokens at the cache price. A run scores 1 when it passes. A run that does not pass scores the share of its pass criteria that it meets, or of its target that it reaches, where a task measures one, as in distribution, lead discovery, problem mining, and the six-stage end-to-end validation, and 0 otherwise; a task’s score averages its runs. A capability’s score averages its tasks, a loop’s score the loop’s tasks, and the overall score all 46 tasks. A task is done when all its runs pass. An arm’s time and cost per completed task are its time and cost summed over 43 tasks, which leave out three tasks whose runs another task already counts, divided by the number of tasks it completes out of 46. In the six-stage end-to-end validation, only the time of the agent’s sessions counts, not that of the launch, which runs on without the agent.

6.4. Results Table 23: Results on the 46 real-world tasks of heelo.dev. Task score (%) over the tasks of each loop (14, 15, 7, and 10) and over all 46 tasks, tasks completed out of 46, and time and cost per completed task, for Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex, both at medium reasoning effort, each on its own and connected to HEELO through MCP (+ HEELO). Time and cost per completed task divide an arm’s totals over 43 tasks, which leave out three tasks whose runs another task already counts, by the number of tasks it completes out of 46. Green marks the best and amber the second-best value in each column; lower is better for time and cost. Task score (%) Arm

Per completed task

Distribution Acquisition Assistant Validation All

Completed

Time (min) Cost ($)

Claude Code + HEELO

23 98

41 84

0 100

35 81

28 90

9 39

6.3 3.4

1.24 0.80

Codex + HEELO

14 90

34 84

0 83

46 73

25 83

9 35

8.0 3.5

0.65 0.42

Overall Results. Connected to HEELO, Claude Code completes 39 of the 46 tasks instead of 9, and Codex 35 instead of 9, and their overall scores rise from 28% to 90% and from 25% to 83%; Figure 1(c) and Table 24 give the score of every capability, and Table 23 the scores by loop. Of the 26 capabilities, Claude Code completes every task in 2 on its own and in 21 with HEELO, and Codex in 3 and in 19. On their own, both agents score zero in 12 capabilities, most of which act on an account or a service: publishing a page, sending a sequence, finding a contact, reading a mailbox, or holding an invoice’s writes for a person’s approval. Distribution and Acquisition. In distribution, Claude Code scores 98% with HEELO and Codex 90%, against 23% and 14% on their own. Claude Code with HEELO completes every task except the card check over four templates and four sizes, in which one of three runs stops at the time limit, and Codex with HEELO every task except two of the three multi-platform posts. In acquisition, both agents score 84% with HEELO, against 41% and 34% on their own, and with HEELO they complete every task in all capabilities but lead discovery. On their own, the agents score mainly where the open web and their own code suffice, as in multi-platform posting (56% and 39%) and source coverage within robots.txt (100%). Lead discovery, which asks for people who state a matching need on public platforms, is the only capability in which no arm scores above 20%: both connected agents reach 20%, and Claude Code and Codex on their own 6% and 3%. 37

The Last Mile to the Real World

The heelo.dev platform

Table 24: Results of the 26 capabilities of heelo.dev. Task score (%), time per task, and cost per task of Claude Opus 5.5 with Claude Code (CC) and GPT-6.1 Sol with Codex, each on its own and connected to HEELO through MCP (+H), for each capability of Figure 1(c). Time and cost are means over the capability’s tasks, with costs at API prices: each model call’s tokens multiplied by the model’s price per token, with cached input tokens at the cache price. Shaded columns are the agents connected to HEELO. Task score (%) Loop

Capability

Time per task (s)

+H

Codex

+H

0 0 0 56 0 77 0

100 100 100 100 100 83 100

0 17 0 39 0 0 33

100 16.3 100 18.8 100 12.1 56 88.7 100 55.4 100 353.6 100 61.0

156.4 58.2 92.1 0.050 0.734 241.8 92.5 218.3 0.053 0.590 104.6 54.6 97.7 0.046 0.530 107.0 112.4 167.5 0.281 0.390 98.8 51.8 108.4 0.169 0.467 422.7 65.1 181.6 0.535 1.145 217.3 102.7 346.8 0.197 0.584

0.053 0.066 0.056 0.091 0.039 0.057 0.096

0.152 0.204 0.152 0.152 0.200 0.195 0.322

Outreach pipeline 0 Sequenced sending 0 Deliverability 75 Acquisition Consent and compliance 50 Contact finding 0 Source coverage and robots.txt 100 Lead discovery 6

100 100 100 100 100 100 20

0 0 50 50 0 100 3

100 95.0 100 124.0 100 110.0 100 67.5 100 201.0 100 97.0 20 88.5

293.0 57.8 56.3 53.1 456.1 134.5 199.1

192.8 116.3 90.8 110.5 137.5 59.4 211.7

754.2 85.5 112.4 115.5 596.5 227.7 244.9

0.287 0.414 0.363 0.193 1.125 0.299 0.424

1.733 0.380 0.407 0.375 3.045 0.825 1.094

0.148 0.053 0.075 0.107 0.381 0.139 0.450

1.776 0.109 0.158 0.139 2.843 0.592 0.831

Assistant

Mail and attachments Cost routing Invoice fields and checks Receipt rendering Review and approval

0 0 0 0 0

100 100 100 100 100

0 0 0 0 0

100 100 100 0 80

25.8 26.1 26.4 18.3 23.4

16.7 17.6 16.7 13.7 15.0

24.6 25.6 29.3 20.9 21.1

0.054 0.054 0.055 0.049 0.053

0.116 0.116 0.116 0.147 0.116

0.029 0.032 0.033 0.023 0.024

0.053 0.053 0.053 0.080 0.055

Validation

Scraping compliance Crawling and extraction Request audit log Problem mining Lead hand-off Funnel math End-to-end validation

50 33 0 0 0 100 42

100 100 100 57 0 100 75

33 67 0 100 100 0 29

100 14.6 17.7 25.3 14.5 0.049 0.072 100 43.8 44.0 41.4 31.0 0.123 0.178 100 10.1 12.9 10.6 8.5 0.212 0.301 100 340.9 287.0 431.6 603.3 1.218 0.560 67 97.1 1200.0 677.5 662.3 0.286 2.240 0 22.8 38.5 26.2 29.8 0.115 0.207 29 47.7 768.2 67.1 105.9 0.259 3.223

0.029 0.066 0.007 0.822 1.212 0.034 0.105

0.024 0.061 0.008 1.272 1.365 0.053 0.177

Page publishing Click attribution Safety and input checks Distribution Multi-platform posting Budget control Share-card rendering Launch pipeline

CC +H Codex +H

CC

10.7 10.3 10.8 7.7 9.4

+H

Codex

Cost per task ($) +H

CC

Assistant. On their own, both agents score 0% in the assistant loop, because an agent on its own cannot read the mailbox, the calendar, or Notion. With HEELO, Claude Code completes every assistant task and passes all 15 runs behind them, from an invoice email to its planned writes, through an invoice whose total does not add up and is held for correction, to the receipt of an approved invoice. Codex with HEELO completes five of the seven tasks and scores 83%: in the three runs that end with the receipt of an approved invoice, heelo.dev returns the receipt, but Codex’s answer leaves it out. Every link of the loop’s chain also passes live against real accounts: the invoice email is read, its eight fields are extracted by the rule-based tier, the arithmetic checks pass, nothing is written until a person approves, and the approval then produces the calendar event, the Notion page, and the PDF receipt. Validation. In validation, Claude Code scores 81% with HEELO and Codex 73%, against 35% and 46% on their own, and they complete 7 and 6 of the 10 tasks, against 2 and 3. Only the connected agents record the status code of every request in a scrape. On a self-contained version of the mining task that asks for problems ranked with their trends, Codex completes the task with and without HEELO; for Claude Code with HEELO, heelo.dev’s problem miner gives a trend for 8 of the 14 mined problems. On its own, Codex also completes the hand-off of 100 people with a draft of 100 candidate contacts; with HEELO, it passes two of its three runs, and Claude Code with HEELO stops at the time limit, because the agent resolves the contacts one at a time, with a call for each. Codex scores zero on the funnel math with and without HEELO: its answers compute the rate but leave out the caveat of a small sample. Of the two end-to-end tasks, Claude Code with HEELO passes all 12 runs of the shorter one and Codex with HEELO 7. In the six-stage end-to-end validation, Claude Code with HEELO passes stages two to four, up to a launch that finds 110 people at 77 companies, and Codex with HEELO stops before the assessment, because its fresh scans return no assessable problem. 38

The Last Mile to the Real World

Conclusion

Time and Cost. In most tasks, a connected run costs more than a run on its own, because it reads more tokens, from the tool menu to the structured results of the primitives: the median assistant run of Claude Code with HEELO reads 160,701 tokens and takes 23.4 seconds, against 19,456 tokens and 9.4 seconds on its own. Because the connected agents complete most of their tasks, they need less time and lower cost per completed task: Claude Code with HEELO needs 3.4 minutes and $0.80, against 6.3 minutes and $1.24 on its own, and Codex with HEELO 3.5 minutes and $0.42, against 8.0 minutes and $0.65, which is 1.9× and 2.3× less time and 1.55× lower cost. Per completed task, Codex with HEELO is the cheapest of the four arms. Repeated work also costs less with HEELO: when a task revisits a JavaScript page whose extraction recipe heelo.dev stored on the first visit, the second visit costs less than half the first in every run of both connected agents, against one of three runs of Codex on its own. Across the four loops, the gap between an agent on its own and the same agent connected to HEELO opens where a task must act on an account or a service: an agent on its own can find a page, draft a message, or write a script, but a published page, a sent sequence, or an invoice read from a mailbox takes a primitive that holds the account, applies its resource in the task’s context, and verifies the outcome before it reports success.

7. Conclusion How can an agent handle a real-world task end to end? Our answer is to engineer each task’s harness around agent-native primitives, which address the last mile to the real world by keeping the competence to apply a resource with the resource itself. An agent-native primitive equips a resource with a resident LLM that applies it to a natural-language request in context and sends other primitives the requirements that its result places on them; where the domain offers an exact procedure, a program takes the place of the resident LLM. HEELO builds each task’s harness from a library of such primitives through three mechanisms. Library selection and composition retrieves and assesses candidates and composes the few that the task needs into a task-specific graph. Evidence-based execution and improvement runs the graph, verifies its result against evidence from the task’s environment, and after a failure revises only the primitives that the evidence implicates, within a budget of revision rounds. Validation and publishing admits new or improved primitives into the library only after validation, on a path separate from any task. Each of the eight primitive families comes from the solution we built for one kind of real-world task, realizes the mechanisms that its task needs, and improves on the reference baseline of its companion study. On the tasks where frontier harnesses fall short, Dev-Primitives lead Codex by 18.2 points on Terminal-Bench 4.0 with GPT-5.6 Sol at maximum effort, and composing Harness Primitives per task raises Terminal-Bench 2.0 Pass@1 from 60.0% for the seed harness to 72.2%, above the 56.7% of Codex. The analyses show that binding inside a Tool Primitive beats budget-matched self-refinement over raw schemas and that the fields a Code Primitive carries account for most of its margin over generic editing. What crosses between agents and primitives must keep its meaning: exchanged KV caches need positional re-encoding, and requirements must reach the components they concern. The task, in turn, decides which primitives run, as active Navigation Primitives grow 1.23× while distractors grow the information space 16× around the same evidence. The families share one method, so a new kind of task can join HEELO as a new primitive family that realizes the same definition and mechanisms. Beyond the research benchmarks, heelo.dev exposes real-world actions to agents as agent-native primitives. On its 46 real-world tasks, Claude Code and Codex complete 9 tasks each on their own and 39 and 35 when connected to HEELO through MCP, with 1.9× and 2.3× less time and 1.55× lower cost per completed task. Overall, primitives let the competence developed for one task outlast that task: once validated and published, the primitive that carries this competence becomes a building block that later tasks compose.

39

The Last Mile to the Real World

References

References [1] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023. [2] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551, 2023. [3] Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In Forty-first International Conference on Machine Learning, 2024. [4] OpenAI. Introducing Codex. https://openai.com/index/introducing-codex/, May 2025. [5] Anthropic. Claude Code. https://www.anthropic.com/claude-code, 2025. [6] Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. ToolLLM: Facilitating large language models to master 16000+ real-world APIs. arXiv preprint arXiv:2307.16789, 2023. [7] Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. Gorilla: Large language model connected with massive APIs. arXiv preprint arXiv:2305.15334, 2023. [8] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWEagent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528–50652, 2024. [9] Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, et al. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations, volume 2025, pages 65882–65919, 2025. [10] Leonardo de Moura and Sebastian Ullrich. The Lean 4 theorem prover and programming language. In International Conference on Automated Deduction, pages 625–635, 2021. [11] The mathlib Community. The Lean mathematical library. In Proceedings of the 9th ACM SIGPLAN International Conference on Certified Programs and Proofs (CPP), pages 367–381. ACM, 2020. doi: 10.1145/3372885.3373824. [12] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Processing Systems, 33:9459–9474, 2020. [13] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G Patil, Ion Stoica, and Joseph E Gonzalez. MemGPT: Towards LLMs as operating systems. arXiv preprint arXiv:2310.08560, 2023. [14] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023. [15] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, et al. MetaGPT: Meta programming for a multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, 2024. [16] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWEbench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, volume 2024, pages 54107–54157, 2024. [17] Mike Merrill, Alexander Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Shin, Thomas Walshe, E Kelly Buchanan, et al. Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces. In International Conference on Learning Representations, volume 2026, pages 40903–40986, 2026. [18] Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for general AI assistants. In B. Kim, Y. Yue, S. Chaudhuri, K. Fragkiadaki, M. Khan, and Y. Sun, editors, International Conference on Learning Representations, volume 2024, pages 9025– 9049, 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/file/ 25ae35b5b1738d80f1f03a8713e405ec-Paper-Conference.pdf. [19] Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, et al. ToolACE: Winning the points of LLM function calling. arXiv preprint arXiv:2409.00920, 2024.

40

The Last Mile to the Real World

References

[20] Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. [21] Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. RAPTOR: Recursive abstractive processing for tree-organized retrieval. In International Conference on Learning Representations, 2024. [22] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2023. [23] Andrew Zhao, Daniel Huang, Quentin Xu, Matthieu Lin, Yong-Jin Liu, and Gao Huang. ExpeL: LLM agents are experiential learners. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19632–19642, 2024. [24] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023. [25] Zora Zhiruo Wang, Jiayuan Mao, Daniel Fried, and Graham Neubig. Agent workflow memory. arXiv preprint arXiv:2409.07429, 2024. [26] Zhiruo Wang, Daniel Fried, and Graham Neubig. TroVE: Inducing verifiable and efficient toolboxes for solving programmatic tasks. arXiv preprint arXiv:2401.12869, 2024. [27] Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. GPTSwarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, 2024. [28] Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. arXiv preprint arXiv:2408.08435, 2024. [29] Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating agentic workflow generation. arXiv preprint arXiv:2410.10762, 2024. [30] Guibin Zhang, Leo Lu, Fangzhou Xie, Kang Zhu, Junhao Wang, Zhifei Xie, Zhaochen Yu, Zihang Liu, Zhongxiang Sun, Qiankun Li, Yue Liao, Heng Chang, Xiaobin Hu, Qibing Ren, Wangchunshu Zhou, Chuanrui Hu, Yafeng Deng, and Shuicheng Yan. JIT-Agent: Scaling harness intelligence via just-in-time harness evolution. arXiv:2608.25593, 2026. URL https://arxiv.org/abs/2608.25593. [31] Victor Barres, Honghua Dong, Soham Ray, Xujie Si, and Karthik Narasimhan. 𝜏 2 -Bench: Evaluating conversational agents in a dual-control environment. arXiv preprint arXiv:2506.07982, 2025. [32] Kinjal Basu, Ibrahim Abdelaziz, Kiran Kate, Mayank Agarwal, Maxwell Crouse, Yara Rizk, Kelsey Bradford, Asim Munawar, Sadhana Kumaravel, Saurabh Goyal, et al. NESTFUL: A benchmark for evaluating LLMs on nested sequences of API calls. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 33538–33547, 2025. [33] Jiarui Lu, Thomas Holleis, Yizhe Zhang, Bernhard Aumayer, Feng Nan, Haoping Bai, Shuang Ma, Shen Ma, Mengyu Li, Guoli Yin, et al. ToolSandbox: A stateful, conversational, interactive evaluation benchmark for LLM tool use capabilities. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1160–1183, 2025. [34] Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 𝜏 -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045, 2024. [35] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. RepoCoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2471–2484, 2023. [36] Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, et al. CrossCodeEval: A diverse and multilingual benchmark for cross-file code completion. Advances in Neural Information Processing Systems, 36:46701–46723, 2023. [37] Alan Bundy. The use of explicit plans to guide inductive proofs. In International conference on automated deduction, pages 111–120. Springer, 1988. [38] Erica Melis and Jörg Siekmann. Knowledge-based proof planning. Artificial Intelligence, 115(1):65–105, 1999. [39] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics,

41

The Last Mile to the Real World

References

12:157–173, 2024. doi: 10.1162/tacl\_a\_00638. URL https://aclanthology.org/2024.tacl-1.9/. [40] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In International conference on learning representations, volume 2024, pages 9112– 9141, 2024. [41] Clemens Szyperski, Dominik Gruntz, and Stephan Murer. Component software: Beyond object-oriented programming. Pearson Education, 2002. [42] Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1–2):181–211, 1999. doi: 10.1016/S0004-3702(99) 00052-1. [43] Bertrand Meyer. Applying “design by contract”. Computer, 25(10):40–51, 1992. doi: 10.1109/2.161279. [44] Haibo Jin, Suijin Wang, Xucheng Yu, Haojing Luo, and Haohan Wang. Harness engineering in LLM tool use via agent-native reusable tool primitives. arXiv preprint arXiv:2609.01736, 2026. [45] Haibo Jin, Peng Kuang, Xucheng Yu, Jerry Wang, Dehao Wu, and Haohan Wang. Large-scale repository engineering via agent-native reusable code primitives. arXiv preprint arXiv:2610.09079, 2026. [46] Xiaopeng Yuan, Suijin Wang, Yanli Wang, Haibo Jin, Peng Kuang, Jerry Wang, Lijun Yu, and Haohan Wang. Let the library speak: Self-advertised method selection for formal proving. arXiv preprint arXiv:2610.09401, 2026. [47] Lucheng Fu, Kejing Xia, Yiyang Wang, Yiqiao Jin, Jinjin He, Xiyuan Yang, Haoxin Liu, Ye Yu, Haibo Jin, Yijia Xiao, Wenke Lee, B. Aditya Prakash, and Haohan Wang. From knowledge access to source learning: Developing sourcespecific competence. arXiv preprint arXiv:2610.02150, 2026. [48] Haibo Jin, Peng Kuang, Ye Yu, Xiaopeng Yuan, and Haohan Wang. Agent primitives: Reusable latent building blocks for multi-agent systems. arXiv preprint arXiv:2602.03695, 2026. [49] Haibo Jin, Xinjie Li, Peng Kuang, and Haohan Wang. Harness engineering for software engineering via modular executable dev-primitives. arXiv preprint arXiv:2610.07832, 2026. [50] Jerry Wang, Haibo Jin, Xiaopeng Yuan, Peng Kuang, and Haohan Wang. ANTMAN: Adaptive need tracking for multiagent navigation in large information spaces. arXiv preprint arXiv:2609.33326, 2026. [51] Peng Kuang, Haibo Jin, Dehao Wu, Feiyang Deng, Xiaopeng Yuan, Jerry Wang, and Haohan Wang. Composing taskspecific agent harnesses at test time with reusable primitives. arXiv preprint arXiv:2609.38912, 2026. [52] Zuxin Liu, Thai Hoang, Jianguo Zhang, Ming Zhu, Tian Lan, Shirley Kokane, Juntao Tan, Weiran Yao, Zhiwei Liu, Yihao Feng, et al. APIGen: Automated pipeline for generating verifiable and diverse function-calling datasets. Advances in Neural Information Processing Systems, 37:54463–54482, 2024. [53] Fan Yin, Zifeng Wang, I-Hung Hsu, Jun Yan, Ke Jiang, Yanfei Chen, Jindong Gu, Long T. Le, Kai-Wei Chang, Chen-Yu Lee, Hamid Palangi, and Tomas Pfister. Magnet: Multi-turn tool-use data synthesis and distillation via graph translation. arXiv:2503.07826, 2025. URL https://arxiv.org/abs/2503.07826. [54] Qiqiang Lin, Muning Wen, Qiuying Peng, Guanyu Nie, Junwei Liao, Jun Wang, Xiaoyun Mo, Jiamu Zhou, Cheng Cheng, Yin Zhao, et al. Hammer: Robust function-calling for on-device language models via function masking. arXiv preprint arXiv:2410.04587, 2024. [55] Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. API-Bank: A comprehensive benchmark for tool-augmented LLMs. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 3102–3116, 2023. [56] Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. StableToolBench: Towards stable large-scale benchmarking on tool learning of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 11143–11156, 2024. [57] Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Vernon Austel, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, et al. API-BLEND: A comprehensive corpora for training and benchmarking API LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12859–12870, 2024. [58] Chen Chen, Xinlong Hao, Weiwen Liu, Xu Huang, Xingshan Zeng, Shuai Yu, Dexun Li, Yuefeng Huang, Xiangcheng Liu, Wang Xinzhi, et al. ACEBench: A comprehensive evaluation of LLM tool usage. Findings of the Association for Computational Linguistics: EMNLP, 2025:12970–12998, 2025.

42

The Last Mile to the Real World

References

[59] Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. AgentDojo: A dynamic environment to evaluate prompt injection attacks and defenses for LLM agents. arXiv preprint arXiv:2406.13352, 2024. [60] OpenAPI Initiative. html, 2024.

OpenAPI specification, version 3.1.1.

https://spec.openapis.org/oas/v3.1.1.

[61] Model Context Protocol Contributors. Model Context Protocol specification. https://modelcontextprotocol. io/specification/latest, n.d. Living specification. [62] Agent2Agent Project Contributors. Agent2Agent protocol specification. latest/specification/, n.d. Living specification.

https://a2a-protocol.org/

[63] Agent Skills Contributors. Agent Skills specification. https://agentskills.io/specification, n.d. Living specification. [64] Charles W Krueger. Software reuse. ACM Computing Surveys (CSUR), 24(2):131–183, 1992. [65] Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke Hewitt, Luc Cary, Armando Solar-Lezama, and Joshua B. Tenenbaum. DreamCoder: Bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the ACM SIGPLAN Conference on Programming Language Design and Implementation, pages 835–850, 2021. [66] Gabriel Grand, Lionel Wong, Maddy Bowers, Theo X Olausson, Muxin Liu, Joshua B Tenenbaum, and Jacob Andreas. LILO: Learning interpretable libraries by compressing and documenting code. In International Conference on Learning Representations, volume 2024, pages 30399–30446, 2024. [67] Elias Stengel-Eskin, Archiki Prasad, and Mohit Bansal. ReGAL: Refactoring programs to discover generalizable abstractions. arXiv preprint arXiv:2401.16467, 2024. [68] Tianle Cai, Xuezhi Wang, Tengyu Ma, Xinyun Chen, and Denny Zhou. Large language models as tool makers. In International Conference on Learning Representations, volume 2024, pages 54067–54089, 2024. [69] Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi Fung, Hao Peng, and Heng Ji. CRAFT: Customizing LLMs by creating and retrieving from specialized toolsets. In International Conference on Learning Representations, volume 2024, pages 40097–40125, 2024. [70] Shuai Lu, Nan Duan, Hojae Han, Daya Guo, Seung-won Hwang, and Alexey Svyatkovskiy. ReACC: A retrievalaugmented code completion framework. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6227–6240, 2022. [71] Shuyan Zhou, Uri Alon, Frank F Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. DocPrompting: Generating code by retrieving the docs. arXiv preprint arXiv:2207.05987, 2022. [72] Jiahao Qiu, Xinzhe Juan, Yimin Wang, Ling Yang, Xuan Qi, Tongcheng Zhang, Jiacheng Guo, Yifu Lu, Zixin Yao, Hongru Wang, et al. AgentDistill: Training-free agent distillation with generalizable MCP boxes. arXiv preprint arXiv:2506.14728, 2025. [73] Yanzhou Li, Yiran Zhang, Xiaoyu Zhang, Xiaoxia Liu, and Yang Liu. CODESKILL: Learning self-evolving skills for coding agents. arXiv preprint arXiv:2605.25430, 2026. [74] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Agentless: Demystifying LLM-based software engineering agents. arXiv preprint arXiv:2407.01489, 2024. [75] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. AutoCodeRover: Autonomous program improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 1592–1604, 2024. [76] Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, et al. CodeR: Issue resolving with multi-agent and task graphs. arXiv preprint arXiv:2406.01304, 2024. [77] Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. MAGIS: LLM-based multiagent framework for GitHub issue resolution. Advances in Neural Information Processing Systems, 37:51963–51993, 2024. [78] Yifei Wang, Ruiyin Li, Peng Liang, Qiong Feng, Zengyang Li, Mojtaba Shahin, and Arif Ali Khan. CodeTeam: An LLM-powered multi-agent framework for repository-level code generation. arXiv preprint arXiv:2606.22082, 2026.

43

The Last Mile to the Real World

References

[79] Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, et al. DevEval: A manually-annotated code generation benchmark aligned with real-world code repositories. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3603–3614, 2024. [80] Tue Le, Minh VT Thai, Dung Nguyen Manh, Huy Phan Nhat, and Nghi DQ Bui. SWE-EVO: Benchmarking coding agents in long-horizon software evolution scenarios. arXiv preprint arXiv:2512.18470, 2025. [81] Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, et al. NL2Repo-Bench: Towards long-horizon repository generation evaluation of coding agents. arXiv preprint arXiv:2512.12730, 2025. [82] Deyao Hong, Yizhe Chi, Wenyi Li, Xiaoqiu Wang, Mingju Gao, Kaisen Yang, Bingxiang He, Youjie Zheng, Calvin Xiao, and Qinhuai Na. SWE Refactor Bench: Can coding agents complete a long-horizon, whole-repository stack migration? arXiv preprint arXiv:2608.23564, 2026. [83] Yuheng Tang, Kaijie Zhu, Bonan Ruan, Chuqi Zhang, Michael Yang, Hongwei Li, Suyue Guo, Tianneng Shi, Zekun Li, Christopher Kruegel, et al. DevOps-Gym: Benchmarking AI agents in software DevOps cycle. In International Conference on Learning Representations, volume 2026, pages 13021–13045, 2026. [84] Shukai Liu, Bo Jiang, Jian Yang, Yizhi Li, Jinyang Guo, Xianglong Liu, and Bryan Dai. Context as a tool: Context management for long-horizon SWE-agents. In Findings of the Association for Computational Linguistics: ACL 2026, pages 20604–20617, 2026. [85] Boshi Wang, Weijian Xu, Yunsheng Li, Mei Gao, Yujia Xie, Huan Sun, and Dongdong Chen. Improving code localization with repository memory. In International Conference on Learning Representations, volume 2026, pages 111266–111285, 2026. [86] Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR, 2020. [87] Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wentau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.emnlp-main.550. URL https://aclanthology.org/2020.emnlp-main.550/. [88] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving retrieval with chain-ofthought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 10014–10037, 2023. [89] Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 7969–7992, 2023. [90] Soyeong Jeong, Jinheon Baek, Sukmin Cho, Sung Ju Hwang, and Jong C Park. Adaptive-RAG: Learning to adapt retrieval-augmented large language models through question complexity. In Proceedings of the 2024 conference of the north american chapter of the association for computational linguistics: Human language technologies (volume 1: Long papers), pages 7036–7050, 2024. [91] Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884, 2024. [92] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. arXiv preprint arXiv:1903.00161, 2019. [93] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium, October-November 2018. Association for Computational Linguistics. doi: 10.18653/v1/D18-1259. URL https://aclanthology.org/D18-1259/. [94] Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph RAG approach to query-focused summarization. arXiv preprint arXiv:2404.16130, 2024. [95] Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. LightRAG: Simple and fast retrieval-augmented generation. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng, editors, Findings of 44

The Last Mile to the Real World

References

the Association for Computational Linguistics: EMNLP 2025, pages 10746–10761, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/2025.findings-emnlp.568. URL https://aclanthology.org/2025.findings-emnlp.568/. [96] Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, et al. GraphReader: Building graph-based agent to enhance long-context abilities of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12758–12786, 2024. [97] Bernal J Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. HippoRAG: Neurobiologically inspired long-term memory for large language models. Advances in neural information processing systems, 37:59532–59569, 2024. [98] Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From RAG to memory: Non-parametric continual learning for large language models. In ICML, volume 267, pages 21497–21515, 2025. [99] Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, and Tiejun Huang. MemoRAG: Boosting long context processing with global memory-enhanced retrieval augmentation. In Proceedings of the ACM on Web Conference 2025, pages 2366–2377, 2025. [100] Zhuoqun Li, Xuanang Chen, Haiyang Yu, Hongyu Lin, Yaojie Lu, Qiaoyu Tang, Fei Huang, Xianpei Han, Le Sun, and Yongbin Li. StructRAG: Boosting knowledge intensive reasoning of LLMs via inference-time hybrid information structurization. In International Conference on Learning Representations, volume 2025, pages 36107–36124, 2025. [101] Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th annual acm symposium on user interface software and technology, pages 1–22, 2023. [102] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. MemoryBank: Enhancing large language models with long-term memory. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pages 19724– 19731, 2024. [103] Petr Anokhin, Nikita Semenov, Artyom Sorokin, Dmitry Evseev, Andrey Kravchenko, Mikhail Burtsev, and Evgeny Burnaev. AriGraph: Learning knowledge graph world models with episodic memory for LLM agents. In IJCAI, 2025. [104] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: Agentic memory for LLM agents. In D. Belgrave, C. Zhang, H. Lin, R. Pascanu, P. Koniusz, M. Ghassemi, and N. Chen, editors, Advances in Neural Information Processing Systems, volume 38, Main Conference, pages 17577–17604. Curran Associates, Inc., 2025. doi: 10.52202/085713-0593. URL https://proceedings.neurips.cc/paper_files/ paper/2025/file/19909c36f51abc4856b4560aff3d36d6-Paper-Conference.pdf. [105] Zouying Cao, Jiaji Deng, Li Yu, Weikang Zhou, Zhaoyang Liu, Bolin Ding, and Hai Zhao. Remember me, refine me: A dynamic procedural memory framework for experience-driven agent evolution. In Findings of the Association for Computational Linguistics: ACL 2026, pages 16803–16822, 2026. [106] Rongzhi Zhu, Xiangyu Liu, Zequn Sun, Yiwei Wang, and Wei Hu. Mitigating lost-in-retrieval problems in retrieval augmented multi-hop question answering. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 22362–22375, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 9798-89176-251-0. doi: 10.18653/v1/2025.acl-long.1089. URL https://aclanthology.org/2025.acl-long. 1089/. [107] Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12991–13013, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653/v1/2024.acl-long.702. URL https://aclanthology.org/2024.acl-long.702/. [108] Minghan Li, Junjie Zou, Xinxuan Lv, Chao Zhang, and Guodong Zhou. S2G-RAG: Structured sufficiency and gap judging for iterative retrieval-augmented QA. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25846–25862, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.1185. URL https://aclanthology.org/2026. acl-long.1185/.

45

The Last Mile to the Real World

References

[109] Jun Zhao, Can Zu, Xu Hao, Yi Lu, Wei He, Yiwen Ding, Tao Gui, Qi Zhang, and Xuanjing Huang. LONGAGENT: Achieving question answering for 128k-token-long documents through multi-agent collaboration. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16310–16324, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-main.912. URL https://aclanthology.org/2024.emnlp-main. 912/. [110] Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ö. Arik. Chain of Agents: Large language models collaborating on long-context tasks. arXiv:2406.02818, 2024. URL https://arxiv.org/abs/ 2406.02818. [111] Guoxin Chen, Minpeng Liao, Peiying Yu, Dingmin Wang, Zile Qiao, Chao Yang, Xin Zhao, and Kai Fan. C-3PO: Compact plug-and-play proxy optimization to achieve human-like retrieval-augmented generation. In Aarti Singh, Maryam Fazel, Daniel Hsu, Simon Lacoste-Julien, Felix Berkenkamp, Tegan Maharaj, Kiri Wagstaff, and Jerry Zhu, editors, Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 8530–8551. PMLR, 13–19 Jul 2025. URL https://proceedings.mlr.press/v267/ chen25an.html. [112] Chia-Yuan Chang, Zhimeng Jiang, Vineeth Rakesh, Menghai Pan, Chin-Chia Michael Yeh, Guanchu Wang, Mingzhi Hu, Zhichao Xu, Yan Zheng, Mahashweta Das, and Na Zou. MAIN-RAG: Multi-agent filtering retrieval-augmented generation. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors, Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2607– 2622, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/ v1/2025.acl-long.131. URL https://aclanthology.org/2025.acl-long.131/. [113] Mengkang Hu, Yuhang Zhou, Wendong Fan, Yuzhou Nie, Ziyu Ye, Bowei Xia, Tao Sun, Zhaoxuan Jin, Yingru Li, Zeyu Zhang, Yifeng Wang, Qianshuo Ye, Bernard Ghanem, Ping Luo, and Guohao Li. OWL: Optimized workforce learning for general multi-agent assistance in real-world task automation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=MBJ46gd1CT. [114] Huajian Xin, Daya Guo, Zhihong Shao, Zhizhou Ren, Qihao Zhu, Bo Liu, Chong Ruan, Wenda Li, and Xiaodan Liang. DeepSeek-Prover: Advancing theorem proving in LLMs through large-scale synthetic data. arXiv preprint arXiv:2405.14333, 2024. [115] Emily First, Markus N Rabe, Talia Ringer, and Yuriy Brun. Baldur: Whole-proof generation and repair with large language models. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, pages 1229–1241, 2023. [116] Haoxiong Liu, Jiacheng Sun, Zhenguo Li, and Andrew C Yao. ProofAug: Efficient neural theorem proving via finegrained proof structure analysis. arXiv preprint arXiv:2501.18310, 2025. [117] Azim Ospanov, Farzan Farnia, and Roozbeh Yousefzadeh. APOLLO: Automated LLM and Lean collaboration for advanced formal reasoning. Advances in Neural Information Processing Systems, 38:41599–41633, 2025. [118] Borja Requena, Austin Letson, Krystian Nowakowski, Izan Beltran-Ferreiro, and Leopoldo Sarra. A minimal agent for automated theorem proving. arXiv:2602.24273, 2026. URL https://arxiv.org/abs/2602.24273. [119] Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J. Prenger, and Anima Anandkumar. LeanDojo: Theorem proving with retrieval-augmented language models. Advances in Neural Information Processing Systems, 36:21573–21612, 2023. [120] Thomas Zhu, Joshua Clune, Jeremy Avigad, Albert Qiaochu Jiang, and Sean Welleck. Premise selection for a Lean hammer. In International Conference on Learning Representations, volume 2026, pages 127594–127607, 2026. [121] Guoxiong Gao, Zeming Sun, Jiedong Jiang, Yutong Wang, Jingda Xu, Peihao Wu, Bryan Dai, and Bin Dong. LeanSearch v2: Global premise retrieval for Lean 4 theorem proving. arXiv preprint arXiv:2605.13137, 2026. [122] Haiming Wang, Huajian Xin, Chuanyang Zheng, Zhengying Liu, Qingxing Cao, Yinya Huang, Jing Xiong, Han Shi, Enze Xie, Jian Yin, et al. LEGO-Prover: Neural theorem proving with growing libraries. In International Conference on Learning Representations, volume 2024, pages 31566–31597, 2024. [123] Weiwei Sun, Lingyong Yan, Xinyu Ma, Shuaiqiang Wang, Pengjie Ren, Zhumin Chen, Dawei Yin, and Zhaochun Ren. Is ChatGPT good at search? Investigating large language models as re-ranking agents. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 14918–14937, 2023.

46

The Last Mile to the Real World

References

[124] Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, et al. Large language models are effective text rankers with pairwise ranking prompting. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1504–1518, 2024. [125] Orion Weller, Kathryn Ricci, Eugene Yang, Andrew Yates, Dawn Lawrie, and Benjamin Van Durme. Rank1: Test-time compute for reranking in information retrieval. arXiv preprint arXiv:2502.18418, 2025. [126] Shengyao Zhuang, Xueguang Ma, Bevan Koopman, Jimmy Lin, and Guido Zuccon. Rank-R1: Enhancing reasoning in LLM-based document rerankers via reinforcement learning. arXiv preprint arXiv:2503.06034, 2025. [127] David B Leake, Andrew Kinley, and David Wilson. Case-based similarity assessment: Estimating adaptability from experience. In AAAI/IAAI, pages 674–679, 1997. [128] Barry Smyth and Mark T Keane. Adaptation-guided retrieval: Questioning the similarity assumption in reasoning. Artificial intelligence, 102(2):249–293, 1998. [129] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, et al. ChatDev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15174–15186, 2024. [130] Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chi-Min Chan, Heyang Yu, Yaxi Lu, Yi-Hsin Hung, Chen Qian, et al. AgentVerse: Facilitating multi-agent collaboration and exploring emergent behaviors. In The Twelfth International Conference on Learning Representations, 2024. [131] Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first International Conference on Machine Learning, 2024. [132] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 17889–17904, 2024. [133] Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic LLM-powered agent network for taskoriented agent collaboration. In First Conference on Language Modeling, 2024. URL https://openreview. net/forum?id=XII0Wp1XA9. [134] Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Siheng Chen, and Jing Shao. MAS-GPT: Training LLMs to build LLM-based multi-agent systems. arXiv preprint arXiv:2503.03686, 2025. [135] Yujia Zheng, Zhuokai Zhao, Zijian Li, Yaqi Xie, Mingze Gao, Lizhu Zhang, and Kun Zhang. Thought communication in multiagent collaboration. arXiv preprint arXiv:2510.20733, 2025. [136] Tianyu Fu, Zihan Min, Hanling Zhang, Jichao Yan, Guohao Dai, Wanli Ouyang, and Yu Wang. Cache-to-Cache: Direct semantic communication between large language models. arXiv preprint arXiv:2510.03215, 2025. [137] Jiaru Zou, Xiyuan Yang, Ruizhong Qiu, Gaotang Li, Katherine Tieu, Pan Lu, Ke Shen, Hanghang Tong, Yejin Choi, Jingrui He, et al. Latent collaboration in multi-agent systems. arXiv preprint arXiv:2511.20639, 2025. [138] Jacob Fein-Ashley, Dhruv Parikh, Rajgopal Kannan, and Viktor Prasanna. Mixture of Thoughts: Learning to aggregate what experts think, not just what they say. arXiv:2509.21164, 2025. URL https://arxiv.org/abs/2509. 21164. [139] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. RoFormer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024. [140] Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-Refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594, 2023. [141] Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv preprint arXiv:2310.03714, 2023. [142] Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. SelfHarness: Harnesses that improve themselves. arXiv:2606.09498, 2026. URL https://arxiv.org/abs/2606. 09498. [143] Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-Harness: End-toend optimization of model harnesses. arXiv:2603.28052, 2026. URL https://arxiv.org/abs/2603.28052.

47

The Last Mile to the Real World

References

[144] Yue Huang, Wenjie Wang, Han Bao, Yuchen Ma, Xiaonan Luo, Yi Nian, Haomin Zhuang, Zheyuan Liu, Yue Zhao, and Xiangliang Zhang. MemoHarness: Agent harnesses that learn from experience. arXiv:2607.14159, 2026. URL https://arxiv.org/abs/2607.14159. [145] Jun Nie, Yonggang Zhang, Jun Song, Qianshu Cai, Dahai Yu, Yike Guo, Xinmei Tian, and Bo Han. TTHE: Test-time harness evolution. arXiv:2607.08124, 2026. URL https://arxiv.org/abs/2607.08124. [146] Song Feng, Siva Sankalp Patel, Hui Wan, and Sachindra Joshi. MultiDoc2Dial: Modeling dialogues grounded in multiple documents. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6162–6176, 2021. [147] Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317–328, 2018. [148] Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen, and Xiaodong Gu. SWE-QA: Can language models answer repository-level code questions? In Findings of the Association for Computational Linguistics: ACL 2026, pages 8230–8245, 2026. [149] Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian. AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16022–16076, 2024. [150] math-ai. AIME 2025 dataset. 2025. URL https://huggingface.co/datasets/math-ai/aime25. [151] Maxwell-Jia. AIME 2024 dataset. 2024. URL https://huggingface.co/datasets/Maxwell-Jia/ AIME_2024. [152] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. arXiv preprint arXiv:2103.03874, 2021. [153] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [154] Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems, 36:21558–21572, 2023. [155] Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. What disease does this patient have? A large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. [156] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. GPQA: A graduate-level Google-proof Q&A benchmark. In First Conference on Language Modeling, 2024. [157] Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong, editors, Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online), December 2020. International Committee on Computational Linguistics. doi: 10.18653/v1/2020.coling-main.580. URL https://aclanthology.org/2020.coling-main.580/. [158] Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. doi: 10.1162/tacl_a_00475. URL https://aclanthology.org/2022.tacl-1.31/. [159] Yuexi Yang, Alyssa Wu, Ji Luo, Richeng Xuan, Zhichao Hu, Yuhong Liu, and Zhen Qin. RepoProbe: Benchmarking architecture-aware repository comprehension with checklists. In Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE ’26), New York, NY, USA, 2026. ACM. [160] Songcheng Cai, Zhiheng Lyu, Yuansheng Ni, Xiangchao Chen, Baichuan Zhou, Shenzhe Zhu, Yi Lu, Haozhe Wang, Chi Ruan, Benjamin Schneider, Weixu Zhang, Xiang Li, Andy Zheng, Yuyu Zhang, Ping Nie, and Wenhu Chen. SWE-QA-Pro: A representative benchmark and scalable training recipe for repository-level code understanding. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Findings of the Association for Computational Linguistics: ACL 2026, pages 16958–16999, San Diego, California, United States, July 2026. Associ48

The Last Mile to the Real World

References

ation for Computational Linguistics. ISBN 979-8-89176-395-1. doi: 10.18653/v1/2026.findings-acl.837. https://aclanthology.org/2026.findings-acl.837/.

URL

[161] Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333–389, 2009. doi: 10.1561/1500000019. [162] Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513–523, 1988. [163] Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005, 2022. [164] Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, et al. Qwen3 Embedding: Advancing text embedding and reranking through foundation models. arXiv preprint arXiv:2506.05176, 2025. [165] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chainof-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. [166] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022. [167] Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. RepoGraph: Enhancing AI software engineering with repository-level code graph. In Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors, International Conference on Learning Representations, volume 2025, pages 30098–30121, 2025. URL https://proceedings.iclr.cc/paper_files/paper/2025/file/ 4a4a3c197deac042461c677219efd36c-Paper-Conference.pdf. [168] Xin Yin, Zixiang Ding, Yiang Zhang, Qiang Wang, Rui Wang, Chao Ni, and Zhe Cui. RepoDistill: Distilling repository knowledge through compression-aware budget allocation and policy optimization. In Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens, editors, Findings of the Association for Computational Linguistics: ACL 2026, pages 4425–4443, San Diego, California, United States, July 2026. Association for Computational Linguistics. ISBN 979-8-89176-395-1. doi: 10.18653/v1/2026.findings-acl.217. URL https://aclanthology.org/2026. findings-acl.217/. [169] Qwen Team. Qwen3 technical report. arXiv:2505.09388, 2025. URL https://arxiv.org/abs/2505.09388. [170] OpenAI. GPT-5.6 Luna. OpenAI API documentation, 2026. URL https://developers.openai.com/api/ docs/models/gpt-5.6-luna. Accessed: 2026-09-20. [171] Anyi Xu, B Li, Bangcai Lin, Bing Xue, BingCheng Xian, Bingzheng Xu, Bochao Wu, Bowei Zhang, Boyi Deng, CC Yu, et al. DeepSeek-V4.1-Flash: Pushing the limits of KV cache compression. arXiv preprint arXiv:2609.19969, 2026. [172] Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925, 2025. [173] DeepSeek-AI. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv:2501.12948, 2025. URL https://arxiv.org/abs/2501.12948. [174] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [175] Anthropic. Introducing Claude 4. https://www.anthropic.com/news/claude-4, 2025. [176] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in neural information processing systems, 36:46595–46623, 2023. [177] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An instruction-following LLaMA model. https://github.com/tatsu-lab/ stanford_alpaca, 2023. [178] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774,

49

The Last Mile to the Real World

References

2023. [179] Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, GP Shrivatsa Bhargav, Maxwell Crouse, Chulaka Gunasekara, et al. Granite-function calling model: Introducing function calling abilities via multi-task learning of granular tasks. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 1131–1139, 2024. [180] Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401.04088, 2024. [181] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. DeepSeek-V3 technical report. arXiv preprint arXiv:2412.19437, 2024. [182] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. GPT-4o system card. arXiv preprint arXiv:2410.21276, 2024. [183] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv:2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. [184] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. [185] Cong Lu, Shengran Hu, and Jeff Clune. Intelligent Go-Explore: Standing on the shoulders of giant foundation models. arXiv preprint arXiv:2405.15143, 2024. [186] Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: A task-solving agent through multi-persona self-collaboration. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 257–279, 2024. [187] Marcia J. Bates. The design of browsing and berrypicking techniques for the online search interface. Online Review, 13(5):407–424, 05 1989. ISSN 0309-314X. doi: 10.1108/eb024320. URL https://doi.org/10.1108/ eb024320. [188] C. A. R. Hoare. An axiomatic basis for computer programming. Communications of the ACM, 12(10):576–580, 1969. doi: 10.1145/363235.363259. [189] George L. Nemhauser, Laurence A. Wolsey, and Marshall L. Fisher. An analysis of approximations for maximizing submodular set functions—I. Mathematical Programming, 14(1):265–294, 1978. doi: 10.1007/BF01588971. [190] Jiawen Shi, Zenghui Yuan, Guiyao Tie, Pan Zhou, Neil Zhenqiang Gong, and Lichao Sun. Prompt injection attack to tool selection in LLM agents. arXiv preprint arXiv:2504.19793, 2025. [191] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. [192] Jun Wang, Jiamu Zhou, Xihuai Wang, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Cheng Jin, Muning Wen, Weinan Zhang, Qiuying Peng, and Jun Wang. HammerBench: Fine-grained function-calling evaluation in real mobile assistant scenarios. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3350–3376, 2025.

50

The Last Mile to the Real World

Proofs

A. Proofs for Section 3.5 Section 3.5 states three results that the companion studies prove about selection: demand-bounded activation in ANTMAN, the comparison of fixed and task-conditioned composition in STITCH, and the coverage and retrieval ceiling in self-advertised method selection [46, 50, 51]. This appendix proves them in that order. Each proof follows the proof of its study and uses the study’s notation, which holds only within the subsection of that result, and a closing paragraph records what each premise contributes. Section 3.5 states the same results in the notation of the model, with the following correspondence. For demand-bounded activation, ANTMAN’s territory-backed workers are the primitives, its information needs are the needs that COMPOSE routes, its realized need count 𝐻(𝑞) is 𝐷(𝑞), and its number of territories 𝑀 is |ℒ𝑞 |. For task-conditioned composition, STITCH’s task 𝑥, task distribution 𝒟, primitive 𝑝𝑖 , and library size 𝑀 are 𝑞, 𝒬, 𝑃𝑖 , and 𝑁 ; its utility 𝑢𝑖 and penalty 𝑐𝑖 are 𝑔𝑖 and 𝜅𝑖 ; its activation vector a is z; and its validity indicator 𝑉 (𝐻) is 1[𝐻 executes validly]. For the coverage results, the study’s Lean goal 𝑔 with its local context, its contract descriptions, and its contract library 𝒞 are the task 𝑞 with its context, the interfaces, and ℒ; its set 𝐴(𝑔) of annotated reference contracts is 𝒰𝑞 ; its information variable 𝐹 is 𝑍; and its top-𝑚 similarity pool 𝑃𝑚 (𝑔) is 𝒮𝑚 (𝑞).

A.1. Demand-Bounded Activation The proof follows ANTMAN’s proof of its Proposition 1 [50], in ANTMAN’s notation (Appendix B.7). At coordination step 𝑡, ANTMAN selects one unresolved need 𝑛𝑡 of the Need⋃︀Graph 𝐺𝑡 = (𝑁𝑡 , ∆𝑡 ) and routes it to the 𝑇𝑞 territory-backed worker 𝑖𝑡 (Eq. (48)). The realized need set ℋ(𝑞) = 𝑡=0 𝑁𝑡 of Eq. (49), with 𝐻(𝑞) = |ℋ(𝑞)|, collects every need that the graph holds during execution, and a reframed or newly introduced need counts as a new member; it is distinct from the attempt history ℋ𝑡 (𝑛) that each need carries in Eq. (47). For a realized need 𝑛, 𝑎(𝑛) counts its routing attempts, initial routing and every rerouting caused by recovery, with 𝑎(𝑛) ≤ 𝜌 under the ⋃︀its 𝑇𝑞 −1 execution budget, and 𝒜(𝑞) = 𝑡=0 {𝑖𝑡 } is the set of distinct workers activated (Eq. (50)). Proof of Proposition 3.1. Every worker activation in Algorithm 2 occurs through a routing decision at Line 7, and each such decision is associated with the one need 𝑛𝑡 selected at that step, which belongs to 𝑁𝑡 ⊆ ℋ(𝑞). Summing the routing events of all needs encountered during execution therefore counts every worker-dispatch event once: ∑︁ 𝐾(𝑞) = 𝑎(𝑛). 𝑛∈ℋ(𝑞)

A single routing event activates at most one territory-backed worker. Mapping each activated worker to the first routing event that activated it therefore sends distinct workers to distinct events, so the number of distinct workers activated cannot exceed the number of routing events, |𝒜(𝑞)| ≤ 𝐾(𝑞). Each need is routed at most 𝜌 times, so ∑︁ ∑︁ 𝐾(𝑞) = 𝑎(𝑛) ≤ 𝜌 = 𝜌 |ℋ(𝑞)| = 𝜌 𝐻(𝑞). 𝑛∈ℋ(𝑞)

𝑛∈ℋ(𝑞)

Combining the two inequalities gives |𝒜(𝑞)| ≤ 𝐾(𝑞) ≤ 𝜌𝐻(𝑞). ANTMAN contains only 𝑀 territory-backed workers, so |𝒜(𝑞)| ≤ 𝑀 , and together |𝒜(𝑞)| ≤ min{𝑀, 𝜌𝐻(𝑞)}, which is Eq. (8). Role of the Premises. 𝑀 enters only as the size of the available candidate pool and not as a factor of the demanddependent bound 𝜌𝐻(𝑞), so enlarging the information space and increasing 𝑀 does not by itself force additional ¯ and 𝜌 does not depend on 𝑀 , then workers to become active. If realized demand stays bounded, 𝐻(𝑞) ≤ 𝐻, ¯ = 𝑂(1) with respect to 𝑀 , which is ANTMAN’s Corollary 1. A harder query can still produce |𝒜(𝑞)| ≤ 𝜌𝐻 more needs, revisions, or recovery attempts and so raise 𝐻(𝑞) or 𝐾(𝑞); the proposition attributes this growth to task demand. The argument uses three properties of ANTMAN’s coordination: every activation goes through a routing decision on one selected need, a routing event activates at most one worker, and each need is routed at most 𝜌 times, 51

The Last Mile to the Real World

Proofs

with a reframed need counted as a new need. These are the properties that Section 3.5 requires of a COMPOSE that activates a primitive only by routing an unresolved need to it. The bound concerns active coordination, that is, whether worker participation is structurally coupled to the number of available territories; the retrieval stage that ranks WorkerCards for each routing decision is a separate cost, which can grow with the information space.

A.2. Fixed versus Task-Conditioned Composition The proof follows STITCH’s proofs of its Proposition 1 and of part 1 of its Proposition 2 [51], in STITCH’s notation (Section 3.5 and Appendix B.8). Expectations include task sampling and any randomness in primitive selection. Because the additive expression lies in [0, 1] for every task and activation vector, ∆𝑖 (𝑥) = 𝑅(𝑥, e𝑖 ) − 𝑅(𝑥, 0), where e𝑖 activates 𝑝𝑖 alone and 0 activates no primitive, lies in [−1, 1], and every expectation below exists. Proof of Proposition 3.2. Part (i). Under the pointwise sign assumptions, ∆𝑖 (𝑥) = 𝑢𝑖 (𝑥) > 0 for 𝑥 ∈ Ω𝑖 and ∆𝑖 (𝑥) = −𝑐𝑖 (𝑥) < 0 otherwise. Since 𝑅(𝑥, a) is additive and every combination is admissible, one oracle activation that maximizes 𝑅(𝑥, a) on each task is 𝑎*𝑖 (𝑥) = 1(𝑥 ∈ Ω𝑖 ). The expected performance of the oracle task-adaptive harness is 𝑀 𝑀 ∑︁ ∑︁ [︀ ]︀ [︀ ]︀ E𝑥∼𝒟 𝑅(𝑥, a* (𝑥)) = E[𝑅0 (𝑥)] + E𝑥 1(𝑥 ∈ Ω𝑖 ) 𝑢𝑖 (𝑥) = E[𝑅0 (𝑥)] + 𝜇𝑖 𝑢 ¯𝑖 . 𝑖=1

𝑖=1

For a fixed harness a ∈ {0, 1}𝑀 , E𝑥∼𝒟 [𝑅(𝑥, a)] = E[𝑅0 (𝑥)] + gives, for each 𝑖,

∑︀𝑀

𝑖=1 𝑎𝑖 E𝑥 [∆𝑖 (𝑥)], and the law of total expectation

E𝑥 [∆𝑖 (𝑥)] = Pr(𝑥 ∈ Ω𝑖 ) E[𝑢𝑖 (𝑥) | 𝑥 ∈ Ω𝑖 ] − Pr(𝑥 ∈ / Ω𝑖 ) E[𝑐𝑖 (𝑥) | 𝑥 ∈ / Ω𝑖 ] = 𝜇 𝑖 𝑢 ¯𝑖 − (1 − 𝜇𝑖 )¯ 𝑐𝑖 . The sum separates over 𝑖, so the optimal fixed harness sets 𝑎*𝑖,fixed = 1 if 𝜇𝑖 𝑢 ¯𝑖 > (1 − 𝜇𝑖 )¯ 𝑐𝑖 and 𝑎*𝑖,fixed = 0 otherwise, and 𝑀 ∑︁ [︀ ]︀ (︀ )︀ * E𝑥∼𝒟 𝑅(𝑥, afixed ) = E[𝑅0 (𝑥)] + max 0, 𝜇𝑖 𝑢 ¯𝑖 − (1 − 𝜇𝑖 )¯ 𝑐𝑖 . 𝑖=1

)︀ ∑︀𝑀 (︀ Subtracting this from the expected performance of the oracle gives Γfixed = 𝑖=1 𝜇𝑖 𝑢 ¯𝑖 −max(0, 𝜇𝑖 𝑢 ¯𝑖 −(1−𝜇𝑖 )¯ 𝑐𝑖 ) , and the identity 𝑦 − max(0, 𝑦 − 𝑧) = min(𝑦, 𝑧) with 𝑦 = 𝜇𝑖 𝑢 ¯𝑖 and 𝑧 = (1 − 𝜇𝑖 )¯ 𝑐𝑖 gives the first line of Eq. (9). Because 𝜇𝑖 ∈ (0, 1), 𝑢 ¯𝑖 > 0, and 𝑐¯𝑖 > 0, each term of the sum is strictly positive, so Γfixed > 0. Part (ii). Write 𝐴 = E𝑥 [𝑅(𝑥, a* (𝑥))] and 𝐵 = E[𝐽(𝑥, 𝐻comp )]. The final activation 𝑎𝑖 of 𝑝𝑖 in 𝐻comp depends on the task and on the composer’s randomness. Conditioning on the application scope and using the definitions of 𝛼𝑖 and 𝛽𝑖 , [︀ ]︀ [︀ ]︀ [︀ ]︀ E 𝑎𝑖 ∆𝑖 (𝑥) = 𝜇𝑖 E 𝑎𝑖 𝑢𝑖 (𝑥) | 𝑥 ∈ Ω𝑖 − (1 − 𝜇𝑖 ) E 𝑎𝑖 𝑐𝑖 (𝑥) | 𝑥 ∈ / Ω𝑖 = 𝜇𝑖 (1 − 𝛽𝑖 ) 𝑢 ¯𝑖 − (1 − 𝜇𝑖 ) 𝛼𝑖 𝑐¯𝑖 , because E[𝑎𝑖 𝑢𝑖 (𝑥) | 𝑥 ∈ Ω𝑖 ] = 𝑢 ¯𝑖 − E[(1 − 𝑎𝑖 )𝑢𝑖 (𝑥) | 𝑥 ∈ Ω𝑖 ] = (1 − 𝛽𝑖 ) 𝑢 ¯𝑖 . The compositions execute validly, 𝑉 (𝐻comp ) = 1, so 𝐽(𝑥, 𝐻comp ) = 𝑅(𝑥, a) at the final activations, and summing over 𝑖 under the additive model gives 𝐵 =𝐴−

𝑀 ∑︁ [︀

]︀ 𝜇𝑖 𝛽𝑖 𝑢 ¯𝑖 + (1 − 𝜇𝑖 )𝛼𝑖 𝑐¯𝑖 .

𝑖=1 * As in STITCH, the best static harness 𝐻fixed , which activates a*fixed on every task, executes validly, so part (i) gives * E[𝐽(𝑥, 𝐻fixed )] = 𝐴 − Γfixed . Comparing the two expressions, 𝐵 > 𝐴 − Γfixed if and only if the utility-weighted selection loss is smaller than Γfixed , which is the second line of Eq. (9).

52

The Last Mile to the Real World

Proofs

Role of the Premises. Additivity is the modeling assumption: it abstracts away interactions between primitives, so it excludes primitives that help only together or that interfere. Admissibility of every combination lets the maximum over fixed harnesses separate over primitives, and the range [0, 1] keeps every expectation finite. Part (i) compares two oracle values. Part (ii) adds valid execution, 𝑉 (𝐻comp ) = 1, which STITCH assumes for compositions of validated primitives built by its deterministic compiler; every selected primitive activated and executed without runtime errors, an activation rate of 100% that STITCH reads as support for part 2 of its Proposition 2 (Figure 16). The composer need not be perfect, since its selection loss need only stay below Γfixed . Each 𝑎𝑖 is the final activation of mechanism 𝑝𝑖 , after selection, bounded correction, and fallback to the seed harness, and the dependencies that the compiler supplies belong to the mechanism that requires them, as in STITCH’s fixed arms. Part 2 of STITCH’s Proposition 2, the comparison with raw code generation under independent implementation failures, is Eq. (53), proved in Appendix B.8. Each fixed arm of Section 4.9 applies one focused primitive with its required dependencies and shared supporting operations to every task. It is one of the fixed activations over which the maximum in Eq. (9) ranges, so the best fixed arm bounds the value of the best fixed harness from below.

A.3. Coverage and Retrieval Ceiling The three parts of Proposition 3.3 restate the information bound (Proposition 2), the candidate-pool ceiling (Corollary 1), and the multi-method coverage result (Proposition 4) of the study of self-advertised method selection [46]. The proofs follow the study’s appendix, in its notation (Section 3.5 and Appendix B.3). A Method Contract makes its own guarantee conditional on explicit premises, as program contracts do [43, 188]: once Pre𝑐 (𝜃) and Obl𝑐 (𝜃) are proved for a binding 𝜃, its action yields 𝐺𝑐 (𝜃) (Proposition B.1). The three parts proved here concern selection instead, that is, which contracts a selector can place in its shortlist. Throughout, 𝑋 = (𝑔, Γ, contract descriptions) is the selector’s input, the nonempty annotated set 𝐴 = 𝐴(𝑔) is random given 𝑋, and 𝑇 = 𝜑(𝑋) holds the features available to a similarity-only selector. Proof of Proposition 3.3. Part (i). For each value 𝑡 of 𝑇 , choose a set 𝑆(𝑡) with |𝑆(𝑡)| = 𝑘 that attains the maximum in the definition of 𝑉𝑘 (𝑇 ); a maximizer exists because 𝒞 is finite. Then [︀ ]︀ (︀ )︀ [︀ ]︀ 𝑉𝑘 (𝑇 ) = E Pr(𝑆(𝑇 ) ∩ 𝐴 ̸= ∅ | 𝑇 ) = Pr 𝑆(𝑇 ) ∩ 𝐴 ̸= ∅ = E Pr(𝑆(𝜑(𝑋)) ∩ 𝐴 ̸= ∅ | 𝑋) , where the middle equality is the tower property and the last one uses that 𝑆(𝜑(𝑋)) is a function of 𝑋. For every 𝑥, the set 𝑆(𝜑(𝑥)) is one of the sets over which the inner maximum in 𝑉𝑘 (𝑋) is taken, so Pr(𝑆(𝜑(𝑥)) ∩ 𝐴 ̸= ∅ | 𝑋 = 𝑥) ≤ max|𝑆|=𝑘 Pr(𝑆 ∩ 𝐴 ̸= ∅ | 𝑋 = 𝑥). Taking expectations over 𝑋 gives 𝑉𝑘 (𝑇 ) ≤ 𝑉𝑘 (𝑋). For strictness, fix 𝑛 with 𝑘 < 𝑛 ≤ |𝒞|, and let 𝑋 be uniform over 𝑛 distinct values 𝑥1 , . . . , 𝑥𝑛 with 𝜑(𝑥𝑖 ) = 𝑡 for every 𝑖. Let 𝑐1 , . . . , 𝑐𝑛 be distinct contracts and set 𝐴 = {𝑐𝑖 } whenever 𝑋 = 𝑥𝑖 . Given 𝑋 = 𝑥𝑖 , any shortlist containing 𝑐𝑖 hits with probability one, so 𝑉𝑘 (𝑋) = 1. Given 𝑇 = 𝑡, which carries no information about 𝑖, a shortlist 𝑆 hits with probability 𝑛1 |𝑆 ∩ {𝑐1 , . . . , 𝑐𝑛 }| ≤ 𝑘/𝑛, so 𝑉𝑘 (𝑇 ) = 𝑘/𝑛 < 1. Part (ii). By construction, the first 𝑚 positions of the final list are a permutation of 𝑃𝑚 (𝑔), and the remaining positions hold 𝒞 ∖ 𝑃𝑚 (𝑔). For 𝑘 ≤ 𝑚, therefore, 𝜋𝑘 (𝑔) ⊆ 𝑃𝑚 (𝑔), and hence 𝜋𝑘 (𝑔) ∩ 𝐴(𝑔) ⊆ 𝑃𝑚 (𝑔) ∩ 𝐴(𝑔). Taking the indicator of nonemptiness of both sides, and their cardinalities divided by |𝐴(𝑔)|, gives the two inequalities of Eq. (10). Their right-hand sides are the first-stage selector’s own hit@𝑚 and recall@𝑚 on 𝑔, since 𝑃𝑚 (𝑔) is its top𝑚 list, and averaging over problems preserves both inequalities. The same containment bounds the reciprocal rank: if 𝑃𝑚 (𝑔) ∩ 𝐴(𝑔) = ∅, the first annotated contract appears at position 𝑚 + 1 or later, so 1/ rank(𝑔) ≤ 1/(𝑚 + 1). Part (iii). Fix a realization of 𝐴 and let ℎ𝐴 (𝑆) = 1[𝑆 ∩𝐴 ̸= ∅]. If 𝑆 ⊆ 𝑆 ′ , then 𝑆 ∩𝐴 ⊆ 𝑆 ′ ∩𝐴, so ℎ𝐴 (𝑆) ≤ ℎ𝐴 (𝑆 ′ ), and ℎ𝐴 is monotone. For 𝑐 ∈ / 𝑆 ′ , the marginal gain of 𝑐 at 𝑆 is ℎ𝐴 (𝑆 ∪ {𝑐}) − ℎ𝐴 (𝑆) = 1[𝑐 ∈ 𝐴] · 1[𝑆 ∩ 𝐴 = ∅], which is nonincreasing in 𝑆 because 1[𝑆 ′ ∩ 𝐴 = ∅] ≤ 1[𝑆 ∩ 𝐴 = ∅] when 𝑆 ⊆ 𝑆 ′ , so ℎ𝐴 is submodular. Since 𝐻𝑥 (𝑆) = E[ℎ𝐴 (𝑆) | 𝑋 = 𝑥] is an average of the functions ℎ𝐴 with nonnegative weights, and both 53

The Last Mile to the Real World

Proofs

properties are preserved under such averages, 𝐻𝑥 is monotone and submodular, with 𝐻𝑥 (∅) = 0. The greedy guarantee is the classical bound for monotone submodular maximization under a cardinality constraint [189]. Let 𝑆 opt maximize 𝐻𝑥 over sets of size 𝑘, and let 𝑆𝑖 be the greedy set after 𝑖 steps, with 𝑆0 = ∅. For each 𝑖 < 𝑘, monotonicity and submodularity give (︀ )︀ 𝐻𝑥 (𝑆 opt ) ≤ 𝐻𝑥 𝑆𝑖 ∪ 𝑆 opt ∑︁ [︀ ]︀ ≤ 𝐻𝑥 (𝑆𝑖 ) + 𝐻𝑥 (𝑆𝑖 ∪ {𝑐}) − 𝐻𝑥 (𝑆𝑖 ) 𝑐∈𝑆 opt ∖𝑆𝑖

[︀ ]︀ ≤ 𝐻𝑥 (𝑆𝑖 ) + 𝑘 𝐻𝑥 (𝑆𝑖+1 ) − 𝐻𝑥 (𝑆𝑖 ) , where the last step uses |𝑆 opt ∖ 𝑆𝑖 | ≤ 𝑘 and the fact that the greedy step picks the largest marginal gain. Writing 𝛿𝑖 = 𝐻𝑥 (𝑆 opt ) − (︀𝐻𝑥 (𝑆𝑖 ), this rearranges to 𝛿𝑖+1 ≤ (1 − 1/𝑘) 𝛿𝑖 , so 𝛿𝑘 ≤ (1 − 1/𝑘)𝑘 𝛿0 = (1 − 1/𝑘)𝑘 𝐻𝑥 (𝑆 opt ). )︀ 𝑘 opt Hence 𝐻𝑥 (𝑆𝑘 ) ≥ 1 − (1 − 1/𝑘) 𝐻𝑥 (𝑆 ) ≥ (1 − 1/𝑒) 𝐻𝑥 (𝑆 opt ), using (1 − 1/𝑘)𝑘 ≤ 𝑒−1 . Role of the Premises. Part (i) is a statement about information and leaves open whether a particular model extracts it. An LLM reranker reads full descriptions, but only of the contracts that a similarity stage passes to it, so similarity still determines what the reranker can consider. Part (ii) requires only that the second stage reorder the pool and place the remaining contracts after it, so it holds for every reranking backbone. The study’s rerankers follow this design with 𝑚 = 10, and every reported 𝑘 satisfies 𝑘 ≤ 𝑚, so an annotated contract that similarity ranks low cannot be recovered by any of them (Figure 17). Self-advertisement judges all 82 contracts from their proposals, and its coverage is limited instead by the information in 𝑋 and by how accurately the model judges expected ⃒ hit@𝑘.]︀ Expected recall@𝑘 is additive over contracts, [︀ ∑︀ applicability. Part (iii) concerns 𝑅𝑥 (𝑆) = 𝑐∈𝑆 𝑤𝑐 (𝑥) with 𝑤𝑐 (𝑥) = E 1[𝑐 ∈ 𝐴]/|𝐴| ⃒ 𝑋 = 𝑥 , so sorting by 𝑤𝑐 (𝑥) maximizes it, and the two objectives coincide when a goal has a single reference contract. When several methods apply, a near-duplicate of a contract that is likely to apply adds little to expected hit@𝑘, so sorting per-contract scores need not maximize it. The study sorts per-contract scores, which is optimal for expected recall@𝑘, and part (iii) identifies the objective that a diversity-aware selector would target.

B. Framework Details of the Eight Primitive Families This appendix gives the full framework of each primitive family, one subsection per family and in the order of Section 4: Tool Primitives and HEART [44], Code Primitives and LEGO [45], Math Primitives and self-advertised method selection in Lean [46], Knowledge Primitives and SourceLearn [47], Agent Primitives [48], Dev-Primitives and HERMES [49], Navigation Primitives and ANTMAN [50], and Harness Primitives and STITCH [51]. Each subsection gives one family’s framework in the notation of its own paper, with the paper’s equations, its algorithm where the paper has one, and its implementation specifics, such as models, budgets, and retrieval sizes. The symbols of a subsection hold only within that subsection, and each subsection ends with a short mapping of its framework to the method of Section 3.

B.1. Tool Primitives: Natural-Language Invocation and Harness Engineering Tool Primitives give each tool a natural-language interface [44], and HEART, a Harness Engineering framework via Agent-native, Reusable Tool Primitives, makes these primitives practical at repository scale. With ToolFace storing the tools, the primitives representing them, and the harness supplying what this abstraction leaves open, HEART decouples tool storage, representation, and invocation, which prior methods couple by exposing raw schemas in the model’s context. In Figure 4, an illustrative request to freeze the cards in a stolen wallet passes through these parts, from the clarification that request decomposition asks for to the validation of each step.

54

The Last Mile to the Real World

1

Framework details

Request Decomposition PLAN

No

2

Invocation Plan

Value Grounding

Yes

Resolve each step’s placeholders from the query and prior results

Sufficient Information?

[From query] "the debit card ending 1234" [From r_1] lock_id = LX-990 [Config] retry = 5 [priority] = high

User User Intent Analysis

My wallet recently got stolen... can you help me freeze my credit cards?

Understand user intent and identify required tasks

Not Passed: Feedback

Result Validation

Evaluation Passed

1

Decomposition Intent Analysis & Decomposition

Evaluation Metrics

Resolution Map Arguments & Settings

Mapping

ToolFace Organize & Store Tools

Function 1

Schema 2

Function 2

Schema N

Function N

Tool Primitives LLM interface that handles schema resolution and execution

4

Execution Results {"output": "Card [**1234] is FROZEN", "data": { "lock_id": "LX-990", "timestamp": "02:33:05"} }

Pass / Fail decision Error signals (if any) Constraint checks Revision feedback

3

Function 1:1

Schema 1

Validation Summary

Task Completion? Argument Consistency? Execution Validity? Constraint Satisfaction?

2

Schema

"Freeze the user's debit card ending 1234.“ Potential Tool:…

Break down into subtasks and identify needed tools

5

ToolFace

Invocation Request

Request Decomposition

Query

3

Value Resolution

4

Execution Run & Return

5

Validation Check Results & Feedback

Figure 4: Overview of HEART. (1) Request decomposition analyses the user’s intent and checks whether the context supports a plan, soliciting missing details if not. (2) Value resolution grounds each step’s placeholders from the query and from prior results into a naturallanguage invocation request. (3) ToolFace stores tools as schema–function pairs, each wrapped as a Tool Primitive that resolves the schema internally. (4) Execution returns the structured result, which (5) result validation checks against task completion, argument consistency, execution validity, and constraint satisfaction; on failure it returns feedback that triggers a targeted revision [44].

ToolFace. ToolFace is a centralized registry 𝒯 = {(𝑠𝑖 , 𝑓𝑖 )}𝑁 𝑖=1 of schema–function pairs. Each manually authored schema 𝑠𝑖 gives the parameter types, constraints, and return specifications of tool 𝑖, and 𝑓𝑖 is its executable implementation. The 25,519 tools comprise 16,464 live APIs from ToolBench [6], 40 mathematical reasoning and 4,348 coding tools from NESTFUL [32], 68 tools from 𝜏 2 -Bench [31], 4,538 bilingual APIs from ACEBench [58], 2 web-search and 5 memory tools from BFCLv4 [20], and 54 manually crafted interaction tools such as form filling. Semantic search over schema descriptors retrieves tools on demand, so the model need not know the catalogue upfront and context pressure follows the number of candidates retrieved per step. A Tool Primitive is defined by the tool it wraps, so the set of primitives is induced by ToolFace and grows with it, and invocations chain along the data dependencies between calls. Tool Primitives. A Tool Primitive is an LLM-based wrapper around one ToolFace tool, and all Tool Primitives share one base LLM ℳ, conditioned on different schemas: (︀ )︀ 𝒫𝑖 (𝑥; 𝑐) = ℳ [ 𝑠𝑖 ; 𝑐; 𝑥 ] , (12) where 𝑥 is the natural-language request of the upstream caller, 𝑐 is optional context such as a prior primitive’s result, empty when there is none, and [·] denotes prompt concatenation. 𝒫𝑖 maps each value that the request describes to the argument space of 𝑠𝑖 , with type coercion, enum matching, unit conversion, required-field, range, and format checks, schema defaults, and binding of fields referenced in the upstream context; it then invokes 𝑓𝑖 and returns the execution result 𝑟𝑖 . On success, 𝑟𝑖 holds the tool name, the bound arguments, the function’s result, and a one-sentence summary that lets it serve as the context 𝑐 of a downstream primitive. A failed schema resolution or a runtime error is not retried silently but returned as a structured failure of type SCHEMA_RESOLUTION, CONSTRAINT_VIOLATION, or RUNTIME_ERROR, with the parameter or constraint that failed. The resulting properties are natural-language

55

The Last Mile to the Real World

Framework details

invocation, inter-tool communication, and execution isolation; under the last, a primitive never calls another tool, modifies the plan, or recovers from a failure on its own. Harness Engineering. HEART supplies what the abstraction leaves undetermined with three mechanisms. Request decomposition is needed because a primitive answers the request it is handed and cannot ask what the user wants. Intent analysis identifies the target task and its objective, the relevant tool categories, and the user’s explicit constraints or preferences. Information sufficiency checking then decides whether the context 𝒞𝑡 of query, acquired information, and execution history supports a fully specified plan: 𝛿𝑡 = D ECOMPOSE(𝑞, 𝒞𝑡 ) ∈ {sufficient, insufficient}.

(13)

While 𝛿𝑡 = insufficient, decomposition returns to the user a targeted clarification request 𝑐𝑡 for the missing information, and the reply enters the context. It then writes the invocation plan Π = (𝜋1 , . . . , 𝜋𝐾 ), whose steps give a tool category, a tool hint, the step’s objective, and the steps it depends on. Value resolution is needed because a step’s values come from the user query and from earlier results, which the primitive does not see. It resolves the values of step 𝜋𝑘 from 𝒞𝑡 into a natural-language invocation request 𝑥𝑘 that names the target Tool Primitive and embeds the values inline, together with execution hyperparameters: (︀ )︀ 𝛽𝑘 = R ESOLVE(𝜋𝑘 , 𝒞𝑡 ) = 𝑥𝑘 , H-Params𝑘 . (14) Values may come from the query, earlier clarifications, or the result of any upstream step, referenced as $step_j. <field>. Value resolution never invents an ungrounded value, never overrides the primitive chosen by decomposition, and resolves a value shared by several steps once. H-Params𝑘 sets retry in [1, 5], priority to low, normal, or high, and timeout_s, 30 seconds by default. The step’s primitive then runs with the previous result as its context, 𝑟𝑘 ← 𝒫𝑘 (𝛽𝑘 | 𝑟𝑘−1 ). Result validation is needed because a primitive reports whether its own call executed, not whether the result answers the query. It decides 𝑣𝑘 = VALIDATE(𝑟𝑘 , 𝜋𝑘 , 𝑠𝑘 ) ∈ {pass, fail} (15) on four criteria, each assessed independently: task completion of the step’s objective, argument consistency with the user’s intent and the constraints of 𝑠𝑘 , execution validity, without runtime errors and with output of the expected schema, and constraint satisfaction of task-level or domain-specific constraints such as rate limits, access permissions, or business rules. A step passes if all four hold and fails otherwise; execution then proceeds to 𝜋𝑘+1 or, after 𝜋𝐾 , returns the final result. On a failure, 𝜑𝑘 ← VALIDATE.F EEDBACK(𝑟𝑘 , 𝜋𝑘 , 𝑠𝑘 ) names each failed criterion and why, with the root cause first. Targeted revision returns this feedback to decomposition, which re-plans from the enriched context: 𝒞𝑡+1 = 𝒞𝑡 ∪ {𝜑𝑘 },

Π′ = D ECOMPOSE(𝑞, 𝒞𝑡+1 ).

(16)

The revised plan addresses the diagnosed failure, so a failed call is re-planned instead of terminating the trajectory, and recovery stays confined to the step that failed validation. Revision repeats for at most 𝐵 rounds, after which the current result is returned (Algorithm 1). Implementation. HEART is training-free, unlike tool-calling models fine-tuned on curated trajectories [19, 52, 54]. Qwen3-8B [169] runs decomposition, value resolution, result validation, and the Tool Primitives, which share its weights and differ in their system prompts. All experiments use 𝐵 = 3 and retrieval over the full repository, with results averaged over three runs, and over eight on 𝜏 2 -Bench. A backbone study on 𝜏 2 -Bench separates the orchestration layer of the three mechanisms from the execution layer of the Tool Primitives: with GPT-5.4 in the first and Qwen3-8B in the second, HEART reaches Pass1 of 0.84, 0.75, and 0.67 on retail, airline, and telecom, close to the 0.86, 0.78, and 0.68 of GPT-5.4 in all roles, so the two layers can be scaled independently. The default Qwen3-8B configuration reaches 0.82, 0.73, and 0.63 at more than 12× lower cost than GPT-5.4 in all roles. 56

The Last Mile to the Real World

Framework details

Algorithm 1 End-to-end execution flow of HEART [44] Require: user query 𝑞, ToolFace 𝒯 , maximum revision budget 𝐵 Ensure: final execution result or structured failure report 1: 𝒞 ← {𝑞}, 𝑏 ← 0 2: while D ECOMPOSE(𝑞, 𝒞) = insufficient do 3: 𝑐 ← D ECOMPOSE.C LARIFY(𝑞, 𝒞); 𝒞 ← 𝒞 ∪ {U SER(𝑐)} 4: end while 5: Π ← D ECOMPOSE.P LAN(𝑞, 𝒞) 6: while 𝑏 ≤ 𝐵 do 7: for each step 𝜋𝑘 ∈ Π do 8: 𝛽𝑘 ← R ESOLVE(𝜋𝑘 , 𝒞); 𝑟𝑘 ← 𝒫𝑘 (𝛽𝑘 | 𝑟𝑘−1 ) 9: if VALIDATE(𝑟𝑘 , 𝜋𝑘 , 𝑠𝑘 ) = fail then 10: 𝜑𝑘 ← VALIDATE.F EEDBACK(𝑟𝑘 , 𝜋𝑘 , 𝑠𝑘 ); 𝒞 ← 𝒞 ∪ {𝜑𝑘 } 11: Π ← D ECOMPOSE.P LAN(𝑞, 𝒞); 𝑏 ← 𝑏 + 1; break 12: end if 13: end for 14: if all 𝐾 steps passed validation then return 𝑟𝐾 15: end while 16: return FAILURE R EPORT(𝒞) Matched Backbone and Budget. HEART pairs its harness with a small backbone, so its gains could come from the scaffold, the backbone, or the tool catalogue. Holding the backbone at Qwen3-8B and the catalogue at ToolFace, we compare HEART with ReAct, retrieval-only tool selection, a planner–executor pipeline, and DFSDT (Table 25). HEART reaches 75.1% ToolBench pass rate against 55.8% for DFSDT, the strongest of the four there, and 0.44 NESTFUL full accuracy against 0.19 for the planner–executor, the strongest on that metric. Table 25: HEART against scaffolds with the same backbone and tool catalogue. Every row uses Qwen3-8B and ToolFace; ToolBench reports average pass and win rates (%), NESTFUL full-sequence accuracy and win rate (fractions) [44]. Green marks the best and amber the second-best value in each column. ToolBench Method

NESTFUL

Pass Win Full Acc. Win

ReAct Retrieval-only selection Planner–executor DFSDT

33.5 41.0 52.6 55.8

46.2 49.5 55.8 57.4

0.08 0.10 0.19 0.13

0.10 0.12 0.28 0.16

HEART

75.1 75.7

0.44

0.75

Removing ToolFace and the Tool Primitives also removes one model call per invocation step. The budget-matched control of Table 26 gives these calls to value resolution, which self-refines its schema-compliant arguments with backbone, total call count, and token budget matched to full HEART. It raises the 16.1% pass rate and 0.06 full accuracy of the plain ablation to 34.2% and 0.15, against 75.1% and 0.44 with Tool Primitives. Further Benchmarks. HEART reaches the highest ACEBench Overall score, 86.9, and the highest BFCLv4 Web Search and Memory Overall scores, 86.0 and 69.38; outside HEART, the same Qwen3-8B reaches 12.0 on BFCLv4 Web Search. The 50 real-world tasks, 10 in each of travel planning, healthcare, e-commerce, finance, and local services, are grounded in user personas, range from a single-tool lookup to a multi-tool workflow of search, comparison, and conditional booking, and are scored by human annotators on whether the system completes the task. Through their product interfaces and built-in tools, GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro complete 20%

57

The Last Mile to the Real World

Framework details

Table 26: Removing Tool Primitives at a matched call budget. The equal-budget row reassigns the removed primitive calls to valueresolution self-refinement over raw schemas, with backbone, total call count and token budget matched to full HEART; the second row is the ablation of Table 4 without reallocation [44]. Green marks the best and amber the second-best value in each column. Variant

ToolBench Avg. Pass (%) NESTFUL Full Acc.

Equal-budget value-resolution self-refinement (no Tool Primitives) w/o ToolFace & Tool Primitives, no budget reallocation

34.2 16.1

0.15 0.06

HEART

75.1

0.44

to 24% of these tasks; given the ToolFace tools through native function calling, they complete 22% to 32%, a gain of at most 8 pp, against 84% for HEART. Validation and Revision. Because the recovery loop is conditioned on result validation, HEART’s study compares 200 sampled validation decisions from ToolBench and NESTFUL trajectories with human pass/fail labels (Table 36): precision is 0.97, recall 0.96, and the false-pass and false-fail rates are 8% and 4%. Most false passes fall on the taskcompletion criterion for partially satisfied objectives, and fewer than 3% of trajectories end on a false pass; false fails trigger at least one redundant revision round in 10–15% of trajectories without changing outcomes. ToolBench pass rate grows from 47.6% without revision to 51.4%, 65.9%, and 75.1% with one to three rounds and saturates at 75.3% with five (Table 37), a plateau indicating that the loop terminates because validation converges. Retrieval and Robustness. Over the full repository, Recall@5 is 92.4 on ToolBench and 94.7 on NESTFUL, and the top ten hold 95.8% and 97.2% of the gold tools (Table 35). Retrieval misses account for 15–20% of HEART’s residual end-task failures, and validation-triggered revision recovers part of the first-pass misses by reformulating the tool query. On subsampled repositories, ToolBench Recall@5 is 94.6 at 5k tools and 93.7 at 10k, against 92.4 at 25,519. Because tool schemas stay in each primitive’s own prompt and never enter the caller’s, HEART’s attack success rate on ToolBench is 0.0% when a malicious tool document is injected into the retrieved candidates following ToolHijacker [190], under both gradient-free and gradient-based attacks, against 66.0–82.2% for GPT-5.4, Claude-4.6-Sonnet, and Gemini-3.1-Pro. Mapping to the Method. Schema resolution with its structured failure is the carried validation of a Tool Primitive (Section 3.2), and running 𝒫𝑘 (𝛽𝑘 | 𝑟𝑘−1 ) is the act step of Eq. (5), in which the summary in 𝑟𝑘−1 carries the earlier result along a dependency of the plan.

B.2. Code Primitives: Adaptation with Carried Tests Code Primitives are reusable software components that adapt their implementations, interfaces, and dependencies to target repositories [45]. Tool Primitives and Agent Primitives expose tools and reusable behaviours through interfaces that support invocation; a Code Primitive instead provides an interface for adaptation, because it becomes part of the delivered repository. LEGO, Large-scale repository Engineering via aGent-native reusable cOde primitives, builds repositories with Code Primitives drawn from CodeFace, a library mined once from real repositories. Adaptation is performed by the component that owns the implementation, while constraints that cross components are resolved at the repository level. In Figure 5, CodeFace is mined offline, and a single illustrative OAuth request then passes through adaptation, execution, and diagnosis. Code Primitives. A Code Primitive is a reusable executable software component equipped with a resident LLM, 𝑃𝑖 = (𝐶𝑖 , 𝐼𝑖 , 𝐷𝑖 , 𝑉𝑖 , 𝑋𝑖 , 𝑀𝑖 ), and the resident LLM is its natural-language interface. Its boundary follows functional coherence and the dependencies required for independent execution, so its granularity is induced by mining: entries have a median of 214 implementation lines (p25 96, p75 468) across 2 files (p75 4), and they export 4 symbols and carry 9 tests at the median. Given a requirement 𝑟 and construction context 𝒞, 𝑀𝑖 runs in two modes: (𝑃̃︀𝑖 , 𝜇𝑖 ) = 𝑀𝑖adapt (𝑃𝑖 , 𝑟, 𝒞),

(𝑠𝑖 , 𝑢𝑖 ) = 𝑀𝑖assess (𝑃𝑖 , 𝑟, 𝒞),

(17)

where 𝑠𝑖 is the relevance assessment, 𝑢𝑖 the usage proposal, and 𝜇𝑖 = {𝜇𝑖→𝑗 }𝑗 is empty for locally contained changes. ASSESS never modifies the primitive, and it answers UNSUITABLE when the capability does not support 58

The Last Mile to the Real World

1

Framework details

my-project

Public

frontend/ package.json tsconfig.json backend/ auth.py TS api.ts models.py tests/ README.md

data-service

ml-platform

Repository Mining Pipeline

1

Parse Repository

3 Retrieval & Activation

“Add OAuth login, update schema, and generate tests.”

Library of executable code primitives

Create edit plan

Primitive Adaptation & Collaboration

4

Natural Language Communication

models_prim

auth_prim models_prim

4

def login(...) class User(...) async def get(...)

Public

•••

2

Primitive Mining

3

Retrieval Activation

build_prim

test_prim

Execution Feedback

7

test_prim

6

native + synthesized tests $ run tests collected 12 items ....

Component-level primitives with metadata and dependencies.

4

5

test_prim

Repository Engineering Actions

@@ def login(...): - pass + def oauth_login(...): + ... ...

backend/models.py… @@ class User(Base): - name = Column(...) + oauth_id = Column(...) + ... ...

…

TS backend/api.ts

@@ router ... - // old route + router.get("/oauth") + ... ...

… tests/test_auth.py @@ def test_login(): - pass + def test_oauth_login(): + ... ...

Revise

5 Tests / Configs / Docs

6 Synthesize Code Primitives

API api_prim

</>

build_prim

Infer Interfaces

api_prim

models_prim build_prim

backend/auth.py …

API

3 Segment into Components

auth_prim API

User Request

api_prim

Public

Selected Primitives

Understand task Retrieve primitives

2 Build Dependency Graph

src/ train.py data.py utils.py tests/ docs/ README.md

Real-world Repositories

CodeFace

Task-Driven Repository Engineering

auth_prim

src/ worker.py scheduler.py configs/ build.yml docker/ tests/ README.md

1

2

Repository Mining & Primitive Construction

Real-world Repositories

12 passed

Primitive Collaboration

5

Repository Editing

6

Verification & Diagnosis Localize failing components Task completion Build success Test pass rate Interface consistency Code quality

Execution Feedback

8

7 Verification & Diagnosis

Updated Repository main

4 files changed +120

−18

Open Pull Request

8

Repository Construction

Figure 5: Overview of LEGO. Left: real-world repositories are mined into component-level Code Primitives, each carrying its interface, dependencies, tests, configuration, and documentation, and organized in CodeFace. Right: a user request activates the required primitives, which adapt themselves and exchange the requirements their changes impose, before the adapted implementations are edited into the repository and revised under execution feedback and diagnosis [45].

the request or when meeting the request would require replacing the capability itself. ADAPT may change the implementation, exported signatures, data representations, dependencies, and configuration while preserving the selected capability. The primitive then updates its interface and dependency records and reruns 𝑉𝑖 , and this carried validation distinguishes primitive adaptation from generic editing. Prompt Contract. A primitive receives a fixed four-part input: its own state (𝐶𝑖 , 𝐼𝑖 , 𝐷𝑖 , 𝑉𝑖 , 𝑋𝑖 ) verbatim; the requirement 𝑟 as the backbone wrote it; a context slice naming the target module path, the symbols the module must export, and the signatures of the sibling components it binds to; and the diagnostics 𝑒 of a previous attempt, if any, with the incoming requirements 𝜇→𝑖 in a construction round. It sees no native test, no other module’s source, and neither the repository-level plan nor its diagnosis. ADAPT returns a complete replacement implementation, the updated exports, behavioural contract, and dependencies, the outgoing requirements 𝜇𝑖→𝑗 , a mapping that marks each carried test kept, rewritten, or dropped, and the execution log of 𝑉𝑖 on the rewritten component. A test may be dropped when it checks donor-specific behaviour, never because the adapted implementation fails it. A return that cannot be parsed, misses a field, or leaves a retained test failing is re-requested with the failure attached up to 3 times and then returns to repository-level handling as an unsuccessful adaptation. CodeFace and Mining. CodeFace organizes the validated primitives as a library ℱ = {𝑃𝑖 }𝑁 𝑖=1 . The evaluated snapshot holds 1,424 entries: 871 mined, 343 harvested from construction runs whose repository-level suite verified the component in context, and 210 web-sourced for capabilities that the mined corpus lacked, all admitted under the same requirements. Mining parses a source repository ℛ into program entities and relations and proposes component boundaries by weighted community discovery, 𝐺ℛ = (𝑉, 𝐸call ∪ 𝐸data ∪ 𝐸struct ),

(0)

(0)

(0) 𝒮ℛ = COMMUNITY(𝐺ℛ ) = {𝑆1 , . . . , 𝑆𝑚 },

59

(18)

The Last Mile to the Real World

Framework details

whose edges capture calls, data and type dependencies, and structural relations such as imports, configuration references, and implementation–test associations. Semantic refinement then splits separable functions or merges entities that serve one capability into a component 𝑆𝑖 . Across its dependency boundary 𝜕𝑆𝑖 , local helpers and configuration state are internalized into 𝐶𝑖 , external requirements are recorded in 𝐷𝑖 , and the exposed interactions, associated tests, and documentation with provenance supply 𝐼𝑖 , 𝑉𝑖 , and 𝑋𝑖 . Each candidate is validated in isolation, and its build, runtime, and test diagnostics 𝑒𝑡𝑖 refine it, 𝜕𝑆𝑖 = {𝑣 ∈ / 𝑆𝑖 | ∃𝑢 ∈ 𝑆𝑖 , (𝑢, 𝑣) ∈ 𝐸},

(𝐶𝑖𝑡+1 , 𝐷𝑖𝑡+1 , 𝐼𝑖𝑡+1 ) = REFINE(𝐶𝑖𝑡 , 𝐷𝑖𝑡 , 𝐼𝑖𝑡 , 𝑒𝑡𝑖 ),

(19)

until it is admitted, with executable closure, a passing component suite, and a recorded source repository, commit, and license, or excluded. A security hold set is applied when the library is loaded and saved, and because each entry records its source repository, same-repository donors can be excluded per target (Table 39). Retrieval and Activation. The construction context 𝒞𝑡 holds the repository state, the constructed components with their recorded interfaces and dependencies, the decisions of earlier rounds, and the accumulated diagnostics. LEGO decomposes the request and activates primitives: (︀ )︀ ⋃︀ Π𝑡 = (𝑟1 , . . . , 𝑟𝐾 ) = DECOMPOSE(𝑞, 𝒞𝑡 ), 𝒜𝑡 = MATCH Π𝑡 , {ℱ𝑘 }𝐾 𝑘=1 , {(𝑠𝑖 , 𝑢𝑖 )} ⊆ 𝑘 {𝑖 : 𝑃𝑖 ∈ ℱ𝑘 }. (20) Each requirement 𝑟𝑘 names a capability with the interface it must expose and the dependencies it may assume. Retrieval runs outside the LLM, by embedding and reranking a query built from these constraints, and returns |ℱ𝑘 | = 𝑚 candidates, with 𝑚 = 2. MATCH weighs coverage of Π𝑡 and compatibility with the target interfaces, the other selected primitives, and the existing repository. It activates the smallest mutually compatible set for the round, and a requirement that no suitable primitive covers goes to task-specific code 𝑔𝑡 . Primitive Adaptation and Collaboration. Round 𝑡 instantiates ADAPT as (︀ )︀ (𝑃̃︀𝑖 , 𝜇𝑖 ) = 𝑀𝑖adapt 𝑃𝑖 , 𝑟𝑘(𝑖) , 𝒞𝑡 ∪ 𝜇→𝑖 , 𝑖 ∈ 𝒜𝑡 ,

(21)

with 𝑘(𝑖) the capability that 𝑃𝑖 was selected to cover and 𝜇→𝑖 = {𝜇𝑗→𝑖 : 𝑗 ∈ 𝒜𝑡 } the requirements addressed to it. A requirement 𝜇𝑖→𝑗 , such as a changed signature, representation, dependency, or shared behavioural assumption, reaches its consumer in 𝒜𝑡 within the same round, along the dependency edges that Π𝑡 records, and the consumer adapts its own implementation; no primitive edits another’s component. Editing, Execution, and Diagnosis. Repository Editing combines the adapted primitives with integration code into 𝒮𝑡 = INTEGRATE({𝑃̃︀𝑖 }𝑖∈𝒜𝑡 , 𝑔𝑡 ) and resolves the composition constraints across primitive boundaries, Γ({𝑃̃︀𝑖 }𝑖∈𝒜𝑡 ) = {Γapi , Γtype , Γdep , Γconfig },

(22)

namely API bindings, shared data representations and types, package dependencies, and configuration; the harness, not diagnosis, evaluates Γ. Execution Feedback runs the target’s build checks and tests together with unit and integration tests synthesized from Π𝑡 and records 𝑜𝑡 = EXECUTE(𝒮𝑡 ): build status, test outcomes and failing cases, runtime errors, and traces. Each task runs in a separate environment; suites share a 900-second timeout, and each test has a 180-second limit, tightened to 45 seconds inside repair after a timed-out round. Verification and Diagnosis computes (𝑣𝑡 , 𝜑𝑡 ) = DIAGNOSE(𝑞, Π𝑡 , 𝒮𝑡 , 𝑜𝑡 ),

𝑣𝑡 ∈ {PASS, FAIL},

𝜑𝑡 = (𝑒𝑡 , 𝑐𝑡 , 𝑢𝑡 ),

(23)

on task completion, build success, test pass rate, interface consistency, and code quality. Build success and test pass rate come from execution, so 𝑣𝑡 = FAIL whenever the build or a required test fails. Diagnosis ties each observed failure in 𝑒𝑡 to execution evidence, finds the smallest set of components 𝑐𝑡 ⊆ 𝒜𝑡 or interactions that explains it, classifies it among seven types, from an incorrect implementation or a missing capability to an integration failure, 60

The Last Mile to the Real World

Framework details

and gives each implicated component a revision objective in 𝑢𝑡 . It modifies no file and uses no held-out grading outcomes, original implementations, or floor and ceiling counts. Revision. On failure, 𝒞𝑡+1 = 𝒞𝑡 ∪ {𝜑𝑡 }, and the primitives in 𝑐𝑡 are reactivated for another round of Eq. (21), while components that the new evidence does not contradict are preserved. The context accumulates diagnostics and is never truncated within a task. Π𝑡 is recomputed each round, so a requirement unmatched in round 𝑡 can be matched once neighbouring interfaces settle, and additional primitives are reserved for capabilities or dependencies that the evidence shows to be missing. A run stops when 𝑣𝑡 = PASS, after 𝑏 = 5 rounds per module, after two consecutive rounds without an additional passing test, or after two timeout rounds, and it returns the updated repository and its patch. Controls. Configurations are named by their active stages: retrieval (R), primitive adaptation (A), execution feedback (V), and diagnosis (D). Each stage ablation is one substitution. Without D, revision runs on the raw execution evidence 𝑜𝑡 in place of 𝜑𝑡 ; without A, the retrieved 𝑃𝑖 is integrated in place of 𝑃̃︀𝑖 ; and without R, 𝒜𝑡 is empty. Scoped generic editing (E) replaces primitive adaptation over the same retrieved components and differs from a primitive in the prompt contract above: the editor receives none of the primitive’s carried tests, metadata, or execution state. Vendor-and-invoke (I) copies the selected primitive unchanged into a vendored package and calls it, and read-only context (C) places its implementation, contract, and carried tests in the repository-level prompt without delivering its files. The independently developed agents run their own released harnesses with default scaffolding, tools, and stopping rules. A task adapter presents each LEGO-REPO task in the input format that the agent expects, and no agent receives the frozen test identifiers, the floor and ceiling counts, or any CodeFace entry. Implementation. GPT-5.6 Terra is the default backbone, adaptation model, and diagnosis model, and one adaptation model instantiates all 𝑀𝑖 ; the study sweeps 13 backbones. The harness runs 10 concurrent model calls with at most 9,000 interface characters per module, top-2 retrieval, and 𝑏 = 5. A task uses on average 856k input and 48k output tokens for repository-level construction, 11k input and 2.3k output tokens per adaptation over 7.4 successful adaptations, and 18k input and 2.6k output tokens per diagnosis round, and embedding and reranking cost a fixed $0.011. Priced at the hosted rates of the evaluated deployments, recorded calls cost $1.55 per task for execution feedback and $2.08 for LEGO with GPT-5.6 Terra. Model Allocation. Table 27 varies the models in the backbone, adaptation, and diagnosis positions. Under a GPT5.6 Terra backbone, GPT-OSS-20B in adaptation and diagnosis yields 0.4880 at $1.58 per task, above feedback with diagnosis at 0.3469 and $1.79, whereas a DeepSeek-V3.2 backbone with GPT-5.6 Terra in the other two positions reaches 0.2471 at $0.79. Table 27: Model allocation across LEGO’s backbone, adaptation and diagnosis. Configurations name the backbone / adaptation / diagnosis models (Terra is GPT-5.6 Terra, OSS-20B is GPT-OSS-20B, Sonnet 5 is Claude Sonnet 5), of which only adaptation runs inside the Code Primitives; cost is recorded-call USD per task, counting backbone and diagnosis calls, successful adaptation calls only, and a fixed retrieval charge, with open-weight models priced as hosted deployments; % LEGO is relative to the full LEGO score, and the last two rows are library-free references [45]. Green marks the best and amber the second-best value in the All and $/task columns. Lower is better for $/task. Configuration (c/a/d)

All

$/task % LEGO

Terra / Terra / Terra Terra / OSS-20B / OSS-20B Sonnet 5 / OSS-20B / OSS-20B DeepSeek-V3.2 / Terra / Terra

0.5134 0.4880 0.4661 0.2471

2.08 1.58 3.31 0.79

100.0 95.1 90.8 48.1

Feedback + Diagnosis, 𝑏 = 5 Feedback, 𝑏 = 8

0.3469 0.3437

1.79 2.07

67.6 66.9

Worked Adaptation. CodeFace entry cf-0412/schema-coercion, mined from marshmallow, spans 261 lines across 2 files, exports a field-coercion dispatcher and its error type, and carries 14 tests. Reconstructing pandera, LEGO required it to coerce untyped input to declared field types and to collect per-field errors, each with a location path, into one report. The adaptation replaced the donor’s exception-per-field control flow with an 61

The Last Mile to the Real World

Framework details

accumulating error collector, renamed the exported error type to the target’s contract, and moved the location path from a string to the target’s tuple representation. It dropped 3 tests that asserted on marshmallow’s message formatting, rewrote 5, and kept 6, and the 11 retained tests passed before repository-level composition bound the component to 4 sibling modules. A generic editor can produce the same rewrite but performs no carried-test check. (a) the construction ladder 0.2172

execution feedback +0.1008 unadapted retrieval +0.0172

0.2314 0.3180 0.3352

relative gain over Feedback (%)

Single-shot Retrieval, one attempt Feedback Retrieval + Feedback Feedback + Diagnosis Retrieval + Diagnosis RAG + Edit LEGO, same-repo donors excl. Retrieval + Adapt LEGO (ours)

(b) relative gain by repository scale

0.3469 0.3980 0.4093 0.4743 0.4795

retrieval + primitive adaptation +0.1665

0.5134

0

.1 .2 .3 .4 .5 delivery score (all 522 tasks)

150

LEGO LEGO, same-repo donors excluded

+162.0%

+98.5%

100 +76.3% +52.6%

50

+30.7%

+68.2%

+63.6%

+71.3%

+50.3%

+30.1%

Feedback (reference)

0

D1 D2 D3 D4 D5 difficulty band (mean 6 / 19 / 32 / 71 / 119 modules)

Figure 6: Where Code Primitives add delivery, and how the gain scales with the repository. (a) Delivery of the construction paradigms of Table 5 on all 522 tasks, with the increments of execution feedback, unadapted retrieval, and retrieval with primitive adaptation. (b) Relative gain over execution feedback by difficulty band, with and without same-repository donors. The absolute gain is nearly flat across bands (Table 38) [45].

Construction Ladder. Figure 6(a) plots the ten configurations of Table 5 as a ladder of delivery: execution feedback adds 0.1008 to single-shot generation, unadapted retrieval 0.0172 to feedback, and retrieval with primitive adaptation 0.1665 to feedback with diagnosis. Panel (b) gives the relative gain over execution feedback by band, and the gap between the curves with and without same-repository donors at D4 and D5 is the part of the gain that those donors account for. Primitive Fields. Table 28 locates LEGO’s 0.1041 margin over generic editing in the fields of 𝑃𝑖 . A scoped generic editor over the retrieved components scores 0.4093, and adding the carried tests 𝑉𝑖 , then the interface contract and provenance 𝐼𝑖 , 𝑋𝑖 , then the dependency closure 𝐷𝑖 raises it to 0.4431, 0.4818, and 0.5039. Added in this order, the fields recover 0.0946 of the margin, and the residual 0.0095 has a 95% interval of [−0.0071, +0.0148] with Wilcoxon 𝑝 = 0.49. Withholding the carried tests from an otherwise complete primitive lowers delivery to 0.4796, a loss of 0.0338 that equals their forward increment [45]. Table 28: The fields of 𝑃𝑖 that make a Code Primitive useful. Rows add carried tests 𝑉𝑖 , interface contract and provenance 𝐼𝑖 , 𝑋𝑖 , and dependency closure 𝐷𝑖 cumulatively to RAG with generic editing (RE DV), with retrieval, diagnosis and feedback fixed, and the fifth row withholds 𝑉𝑖 from a complete primitive; delivery on the [0, 1] scale, with Δ to RE DV and to LEGO [45]. Green marks the best and amber the second-best value in the All column. Configuration

All

RAG + Edit (RE DV) + carried tests 𝑉𝑖 + contract and provenance 𝐼𝑖 , 𝑋𝑖 + dependency closure 𝐷𝑖 primitive without 𝑉𝑖 LEGO (RADV)

Δ RE DV Δ LEGO

0.4093 ref. 0.4431 +0.0338 0.4818 +0.0725 0.5039 +0.0946 0.4796 +0.0703 0.5134 +0.1041

−0.1041 −0.0703 −0.0316 −0.0095 −0.0338 ref.

Mapping to the Method. Within the method of Section 3, ADAPT returns 𝑃̃︀𝑖 as the result of Eq. (2) and 𝜇𝑖 as its messages, Repository Editing and Execution Feedback together form EXECUTE, DIAGNOSE is VERIFY, and admission into CodeFace is LEGO’s validation and publishing step, Eq. (7). 62

The Last Mile to the Real World

Framework details

B.3. Math Primitives: Self-Advertised Method Selection 1 Problem + method library

2 Selection: proposals, gate, ranking

Target theorem: Cauchy–Schwarz

Goal, context 82 contract descriptions

Lean goal g + context Γ n : Nat a : Fin n → Real

82 method contracts One reusable unit; 82 in the library

Method Contract

× 82

Selection side

Execution side

read by the selector

Lean-checked, sent to the prover

Selector model One batched call Shared backbone

3 Execution: prove + verify

Proposals from all 82 contracts Target + action

Facts + support

Offer

Reals given; compiled scaffold

Selected package

Cauchy– Schwarz

Square of sum; pair a with all ones

Main

AM–GM

Compare means

Nonnegativity not given; unsupported

Demote main

• Compiled scaffold • Worked instance • Mathlib anchors • Proof template • Instantiation pattern

Pigeonhole

Terms repeat; no concrete target/action

No cited support

Demote main

Every proposal carries

built from Putnam 2000–2014 solutions

Offer e.g.

Cauchy–Schwarz

AM–GM

Pigeonhole

Role

Score

Target

Action

Putnam 2000–2014

Author review

1 Offers itself: main or support 2 Specifies a concrete target

Compiled contracts

3 Specifies a concrete action

4 Cites compiled material

Similarity-only retrieval (baseline) Ranks by resemblance, not by applicability 1

AM–GM

✗ a_i ≥ 0 not given

2

Pigeonhole

✗ no concrete target

⋮ 27 Cauchy–Schwarz

✓ applicable, missed

bar = similarity score. Looking related is not the same as offering a usable step; panel 2 asks each method how it would be used.

Conditions Citation

Commitment gate Screening, not proof verification

Library construction — offline

Execution-side package: Cauchy–Schwarz

Contract

Tiered ranking (score within tiers) 1

Gated main nominations

2

Other offering contracts

3

Remaining contracts

PASS: keep main role

Does not prove math premises. FAIL: main → support Same score

Optional runoff

Goal

Conditions

Real entries: given. No extra sign condition. Algebra remains

AxProverBase + contract Selected package in initial prompt. Generate full proof; check obligations.

Complete proof attempt Lean 4 code

Goal + context

Proof steps 1 Choose vectors 2 Apply inequality

3 Simplify 4 Conclude the goal

Lean 4 kernel PASS At most 10 compilations

Lean errors

Accepted complete proof

Whole-proof repair

e.g., Cauchy–Schwarz Demoted supports: AM–GM, Pigeonhole

Re-rank top 10 pairwise, same backbone Boosts hit@1

AxProverBase revises all steps

Budget used up: unresolved

Contracts that do not offer themselves

Figure 7: Math Primitives: self-advertised method selection on an illustrative Cauchy–Schwarz goal. (1) The task is a Lean goal with its local context. The library holds 82 Method Contracts, built offline from Putnam 2000–2014 solutions with every extracted method reviewed by the authors, and a selector reads only their selection sides: the method, its typical target, when it applies and when not, and a manifest of its verified material. Similarity retrieval alone proposes AM–GM, whose condition 𝑎𝑖 ≥ 0 the goal does not give. (2) In one batched call of a shared backbone, every contract proposes a target, an action, and the facts its use requires, cites verified material, and states its offer, role, and score. The commitment gate screens these proposals and demotes a vague (Pigeonhole) or unsupported (AM–GM) main nomination to a supporting role; the tiered ranking and an optional runoff over the top ten then order the contracts. (3) The execution side of the top-ranked contract enters the initial prompt of AxProverBase, Lean compiles each complete proof attempt, and its errors drive the prover’s whole-proof repair within ten compilations [46].

A Math Primitive stores a proof method as a Method Contract, with the method’s required conditions, intended action, and residual obligations, and self-advertised method selection lets each contract propose its own use on a goal [46]. We give the full method and its experimental settings in that study’s notation. Figure 7 follows an illustrative Cauchy–Schwarz goal through the three stages of the method: the library whose selection sides the selector reads, self-advertisement with its commitment gate and tiered ranking, and proving under Lean compilation. Problem and Contracts. Let 𝑔 be a Lean goal with local context Γ, and let 𝒞 be a finite library of Method Contracts. A contract 𝑐 has parameters 𝑥𝑐 , a selection-facing description, required conditions Pre𝑐 (𝑥𝑐 ), a target statement 𝐺𝑐 (𝑥𝑐 ) with an intended action, an execution scaffold 𝑆𝑐 , and a list of residual obligations Obl𝑐 (𝑥𝑐 ). The action reduces the target to the required conditions and the obligations: once Pre𝑐 (𝜃) and Obl𝑐 (𝜃) are proved for a binding 𝜃 of 𝑥𝑐 , the action yields 𝐺𝑐 (𝜃). A selector receives 𝑔, Γ, and the contract descriptions and returns an ordered list 𝜋(𝑔) = (𝑐1 , . . . , 𝑐|𝒞| ) of the whole library. A contract is applicable to 𝑔 if some correct proof of 𝑔 contains a step that instantiates it. The experiments measure against a nonempty set 𝐴(𝑔) of annotated reference contracts. Correct annotations are applicable, and other proofs may use other methods, so agreement with 𝐴(𝑔) understates how often a shortlist contains a usable method. Selection and Execution Sides. The selection side, which is what a selector reads, states the method, its typical target, when it applies and when not, and a manifest naming the verified material of the execution side without its 63

The Last Mile to the Real World

Framework details

Lean code. The execution side, which is supplied to the prover, holds a Lean scaffold over 𝑥𝑐 , a compiled worked instance, the Mathlib declarations it relies on, Pre𝑐 as Lean hypotheses, Obl𝑐 , 𝐺𝑐 , and a verification record. Self-Advertisement. One batched model call reads 𝑔, Γ, and the selection sides of all contracts and returns a proposal for each, so similarity plays no part in ranking or filtering the candidates. The proposal of 𝑐 makes a ˆ in the goal’s own objects commitment that can be checked against the goal: a target 𝑡𝑐 , which instantiates 𝐺𝑐 (𝜃) ˆ ˆ It also cites verified ̂︂ at a proposed binding 𝜃; an action 𝑎𝑐 ; and the required facts Pre𝑐 , which instantiate Pre𝑐 (𝜃). material 𝜎𝑐 from the contract and records whether the contract offers itself (𝑜𝑐 ), its role 𝜌𝑐 ∈ {main, support}, and an integer applicability score 𝑠ˆ𝑐 . A main nomination must pass the commitment gate Gate(𝑐) = 𝑜𝑐 ∧ Spec(𝑡𝑐 ) ∧ Spec(𝑎𝑐 ) ∧ (𝜎𝑐 ̸= ∅),

(24)

where Spec holds when a target or action names a concrete object of the goal; a main nomination that fails the gate is demoted to a supporting role with its score unchanged. The tiered ranking orders contracts lexicographically by (𝜏𝑐 , −ˆ 𝑠𝑐 ), where 𝜏𝑐 = 1 for main nominations that pass the gate, 𝜏𝑐 = 2 for other contracts that offer themselves, and 𝜏𝑐 = 3 otherwise. Ties are broken by contract identifier, so the ranking is deterministic given the model outputs. Runoff. A prover that receives one contract depends on hit@1, and integer scores often leave the leading proposals close or tied. The optional runoff re-ranks the top 𝑞 contracts in one top-down pass of 𝑞 − 1 calls: each call sees a contract, the contract above it, and that contract’s earlier proposal, writes a proposal for the lower contract, and swaps the two when that proposal fits the proof better. Contracts outside the window keep their order. The runoff uses the selector’s own backbone, so any gain it brings comes from direct comparison, and 𝑞 = 10 matches the rerankers’ candidate count. Conditional Scaffold Soundness. The study’s first result states what a checked scaffold establishes. Proposition B.1 (Conditional scaffold soundness [46]). Suppose a contract provides a Lean-checked term with no sorry or admitted proof holes, of type 𝑆𝑐 : ∀𝑥𝑐 , Pre𝑐 (𝑥𝑐 ) → Obl𝑐,1 (𝑥𝑐 ) → · · · → Obl𝑐,𝑟 (𝑥𝑐 ) → 𝐺𝑐 (𝑥𝑐 ).

(25)

For any binding 𝜃 whose required conditions and obligations have checked proofs, applying 𝑆𝑐 yields a checked proof of the target 𝐺𝑐 (𝜃). Proof sketch. Instantiate the checked term at 𝜃 and apply it to the checked proofs of its premises. Lean checks that the resulting term proves 𝐺𝑐 (𝜃). The proposition applies to scaffolds checked in this parameterized form; a compiled example certifies that one example, not arbitrary new bindings. It concerns reuse, not discovery: a proposal identifies the remaining work for ˆ and Obl𝑐 (𝜃), ˆ without discharging it. the new goal, Pre𝑐 (𝜃) Similarity and Coverage. Let 𝑋 = (𝑔, Γ, contract descriptions) be the selector’s input, with 𝐴 = 𝐴(𝑔) random given 𝑋, and let 𝑇 = 𝜑(𝑋) be the features With information 𝐹 , the best [︀ available to a similarity-only selector. ]︀ achievable expected hit@𝑘 is 𝑉𝑘 (𝐹 ) = E max𝑆⊆𝒞, |𝑆|=𝑘 Pr(𝑆 ∩ 𝐴 ̸= ∅ | 𝐹 ) . Proposition 3.3 restates the study’s information bound, 𝑉𝑘 (𝑋) ≥ 𝑉𝑘 (𝑇 ); its candidate-pool ceiling for a selector that reorders the similarity top-𝑚 pool 𝑃𝑚 (𝑔) and places the other contracts after it; and its multi-method coverage result, under which 𝐻𝑥 (𝑆) = Pr(𝑆 ∩ 𝐴 ̸= ∅ | 𝑋 = 𝑥) is monotone and submodular, so greedy selection with the true marginal gains attains at least 1 − 1/𝑒 of the best size-𝑘 coverage [189]. The proofs of all three are in Appendix A. ⃒ [︀ ]︀ Ranking Regret. With 𝑤𝑐 (𝑥)∑︀= E 1[𝑐 ∈ 𝐴]/|𝐴| ⃒ 𝑋 = 𝑥 for a fixed input 𝑋 = 𝑥, a shortlist 𝑆 has expected recall@𝑘 equal to 𝑅𝑥 (𝑆) = 𝑐∈𝑆 𝑤𝑐 (𝑥), and self-advertisement can be read as estimating, from each contract’s proposal, the ordering by 𝑤𝑐 (𝑥) that maximizes it.

64

The Last Mile to the Real World

Framework details

Proposition B.2 (Top-𝑘 ranking regret [46]). Let 𝑆𝑘* contain the 𝑘 contracts with the largest 𝑤𝑐 (𝑥), let 𝑆̂︀𝑘 contain the 𝑘 contracts with the largest scores 𝑠ˆ𝑐 , and suppose there is a strictly increasing map 𝑓 such that |𝑓 (ˆ 𝑠𝑐 ) − 𝑤𝑐 (𝑥)| ≤ 𝜖 for every 𝑐 ∈ 𝑆𝑘* ∪ 𝑆̂︀𝑘 . Then 𝑅𝑥 (𝑆𝑘* ) − 𝑅𝑥 (𝑆̂︀𝑘 ) ≤ 2𝑘𝜖. Estimation errors affect the shortlist when they fall on contracts that are, or appear to be, among the top 𝑘. The gate moves contracts that appear to be among the top 𝑘 but cannot state a checkable use below all supported nominations, and the runoff compares the leading proposals directly, since absolute scores from different contracts need not lie on the common scale that the premise requires. Selection and Proof Success. For the event 𝐸𝑘 that the first 𝑘 positions contain an annotated contract and the event 𝑍𝐵 that the prover completes a Lean proof within 𝐵 compile calls, any prover satisfies Pr(𝑍𝐵 ) = Pr(𝑍𝐵 | ¬𝐸𝑘 ) + Pr(𝐸𝑘 ) ∆𝑘 with ∆𝑘 = Pr(𝑍𝐵 | 𝐸𝑘 ) − Pr(𝑍𝐵 | ¬𝐸𝑘 ). Selection controls Pr(𝐸𝑘 ), while ∆𝑘 depends on the contract’s execution guidance and residual obligations; the contract arm of Table 8 is the case 𝑘 = 1, 𝐵 = 10. Contract Library. The 82 contracts are built once, offline, in three steps from solutions to Putnam 2000–2014 problems, none of them an evaluation problem. GPT-5.6 Sol extracts the methods used in the reference solutions and proposes merges between methods that play the same role in different problems, and the authors review each extracted method and each proposed merge. The model then drafts both sides of each contract from its source solutions, and each execution-side Lean artifact is compiled with Lean v4.27.0 and Mathlib at commit a3a10db. All 82 contracts pass, each with a scaffold stated for arbitrary parameters and a compiled worked instance. Construction and proving share this environment, so a scaffold that compiled at construction stays valid for the prover, as Proposition B.1 requires, and the runner exits at startup if the library’s recorded Mathlib revision differs from the environment’s Lake manifest. Experimental Configuration. The selection baselines are random ordering, BM25 [161], TF-IDF cosine [162], OpenAI’s text-embedding-3-small and text-embedding-3-large [163], Qwen3-Embedding-0.6B [164], and a listwise LLM reranker over the top 10 candidates of each embedding model [123]. All LLM-based selectors run with DeepSeek-V4.1-Flash [171], GPT-5.6 Luna [170], and GPT-5.6 Terra, one backbone per block of Table 7. With 𝜋𝑘 (𝑔) = {𝑐1 , . . . , 𝑐𝑘 } and rank(𝑔) = min{𝑖 : 𝑐𝑖 ∈ 𝐴(𝑔)}, the metrics over the 𝑁 problems of a benchmark are hit@𝑘 =

]︀ 1 ∑︁ [︀ 1 𝜋𝑘 (𝑔) ∩ 𝐴(𝑔) ̸= ∅ , 𝑁 𝑔 MRR =

recall@𝑘 =

1 ∑︁ |𝜋𝑘 (𝑔) ∩ 𝐴(𝑔)| , 𝑁 𝑔 |𝐴(𝑔)|

1 ∑︁ 1 . 𝑁 𝑔 rank(𝑔)

For proving, GPT-5.6 Luna runs a whole-proof repair loop with a budget of 𝐵 = 10 Lean compilations per problem. Base is one attempt without retrieval; pass@20 draws 20 independent samples and so has twice the budget; APOLLO alternates generation with Lean-guided repair [117]; AxProverBase, Ax-Prover run without contracts under the same budget, is the direct control [118]; and the contract arm adds the top-ranked self-advertised contract to AxProverBase’s initial prompt. Mapping to the Method. Within the method of Section 3, the selection side of a contract is its interface 𝑑𝑖 , and a proposal is the relevance assessment of Eq. (3): the score 𝑠ˆ𝑐 plays the role of 𝑠𝑖 , and the target, action, and required facts form the proposed use 𝑢𝑖 .

B.4. Knowledge Primitives: Source Learning Each Knowledge Primitive owns a source model, an explicit and persistent state that holds the understanding of one authoritative source while remaining grounded in it. SourceLearn [47] is the framework that develops this model 65

The Last Mile to the Real World

Framework details

Figure 8: Overview of SourceLearn. Initial Construction builds a provisional source model 𝑀0 from the authoritative source 𝐷. SelfDirected Source Learning deepens and connects source understanding through adaptive study, producing 𝑀𝑆 . Task-Guided Source Learning uses guidance-task failures and recurring representation needs to further refine the model into 𝑀𝑇 . At inference time, Source-Model Activation combines relevant source-model context with directly retrieved evidence for downstream tasks [47].

through repeated study and use of the source, and we describe it in its own notation: 𝑀 is a source model, 𝐹 the task solver, 𝐵 a token budget, 𝑅(𝐷, 𝑞) the source context of a task 𝑞 and 𝑅 alone a source region, and Π0 and Π1 are instructions on how to represent a source. Initial construction builds a provisional source model 𝑀0 from the source 𝐷, Self-Directed and Task-Guided Source Learning refine it into 𝑀𝑆 and then 𝑀𝑇 through grounded source reconstruction, and at test time the solver receives the activated 𝑀𝑇 beside the retrieved evidence (Figure 8). Problem. A persistent authoritative source 𝐷 grounds recurring tasks (𝑞, 𝑦) ∼ 𝒫𝐷 , where 𝑦 is the desired outcome of 𝑞. Knowledge access, source learning, and the use of the learned understanding are (︀ )︀ (︀ )︀ 𝑦ˆ = 𝐹 𝑞, 𝑅(𝐷, 𝑞) , 𝑀𝑡 = Learn(𝐷, 𝐻𝑡 ), 𝑦ˆ = 𝐹 𝑞, 𝑅(𝐷, 𝑞), 𝑀𝑡 , (26) where 𝑅 may be retrieval, long-context access, or a structured source representation, and 𝐻𝑡 is the accumulated study and task experience. Repeated access alone can leave reusable knowledge fragmented or in need of reconstruction for each task. Source learning therefore faces two questions: what source knowledge to learn before future tasks are known, and how task feedback can improve source-specific competence beyond the task that produced it, since retaining an observed answer or correction risks learning the task rather than the source. Self-Directed Source Learning addresses the first question and Task-Guided Source Learning the second: 𝑀0 = Construct(𝐷),

𝑀𝑆 = SelfLearn(𝐷, 𝑀0 ),

𝑀𝑇 = TaskLearn(𝐷, 𝑀𝑆 ; 𝒢),

(27)

where 𝒢 = {(𝑞𝑡 , 𝑦𝑡 )}𝑇𝑡=1 holds guidance tasks grounded in the same source, with reference outcomes 𝑦𝑡 . Source Model. 𝑀 is organized around source entities 𝑒, each with a reusable representation 𝑚𝑒 of its properties, mechanisms, governing conditions, procedures, exceptions, distinctions, and relations to other entities, and higherlevel structure organizes dependencies across entities. Each unit makes one reusable semantic commitment and 66

The Last Mile to the Real World

Framework details

cites the source elements it rests on; facts that a reader can look up stay in 𝐷. Construct reads the source entity by entity within coherent regions, such as code modules or document sections, under a generic modeling instruction Π0 that favors major entities, relations, procedures, and governing conditions over redundant low-level content. The resulting 𝑀0 is deliberately provisional. Grounded Source Reconstruction. All later updates follow one principle: learning signals determine what should be reconsidered, while persistent knowledge is reconstructed from the authoritative source. For an affected source region 𝑅 and a learning signal 𝜉, (︀ )︀ 𝑂𝑅 = Inspect(𝐷𝑅 , 𝑀𝑅 ; 𝜉), 𝑀 ′ = GroundApply 𝑀, Reconstruct(𝑀𝑅 , 𝑂𝑅 ); 𝐷𝑅 , (28) where the temporary observations 𝑂𝑅 , gathered by rereading the region’s source content against the current model, are never copied into 𝑀 . Each proposed unit passes through the grounding gate, which judges a statement supported when its cited excerpts entail it and a negative claim supported when an excerpt states the denial; any other unit counts as unsupported. The region rewrite is committed if the preservation check finds each previously supported meaning still stated, entailed, or covered by a broader unit, and withheld otherwise. Source-Model Activation. At task time, the solver receives at most 𝐵 = 24k tokens of 𝑀 in addition to the evidence retrieved for the query: {︃ (︀ )︀ 𝑀, tok(𝑀 ) ≤ 𝐵, 𝐴𝐵 (𝑀, 𝑞) = 𝑦ˆ = 𝐹 𝑞, 𝑅(𝐷, 𝑞), 𝐴𝐵 (𝑀, 𝑞) , (29) ActivateRegions(𝑀, 𝑞; 𝐵), tok(𝑀 ) > 𝐵, where activation selects task-relevant regions among those established during construction and preserves the internal structure of each region it selects. Self-Directed Source Learning. Before future task requirements are known, one Inspect–Study–Consolidate cycle decides what deserves deeper study: 𝑂 = Inspect(𝐷, 𝑀 ),

𝑂+ = Study(𝐷, 𝑀, 𝑂),

𝑀 ′ = Consolidate(𝐷, 𝑀, 𝑂+ ).

(30)

Model-conditioned source inspection rereads the source content 𝑆𝑒 of each entity against its representation, 𝑂𝑒 = Observe(𝑆𝑒 , 𝑚𝑒 ), and records what the representation still fails to explain well; observations do not modify 𝑀 . Adaptive source study then selects, at planning round 𝑏, study actions from the current model, the accumulated observations 𝑂𝑏 , and the study history 𝐻<𝑏 : {︀ }︀ Γ𝑏 = PlanStudy(𝑀, 𝑂𝑏 , 𝐻<𝑏 ), Γ𝑏 ⊆ D EEPEN(𝑒, 𝑞 study ), C ONNECT(𝑒𝑖 , 𝑒𝑗 , 𝑞 study ) . (31) D EEPEN investigates an unresolved aspect of one entity, and C ONNECT jointly studies two entities whose dependency remains insufficiently understood. The temporary study question 𝑞 study of each action guides retrieval and renewed reading and is not stored. Evidence consolidation then applies Eq. (28) once, a read-many, write-once pattern, and turns 𝑀0 into 𝑀𝑆 . Task-Guided Source Learning. In one pass over 𝒢, SourceLearn first returns to the source for each guidance task: ̃︀𝑡 ), (𝐸𝑡* , 𝐶𝑡 ) = Diagnose(𝑞𝑡 , 𝑦𝑡 , 𝐸

̃︀𝑡 = RetrieveGuide(𝐷, 𝑞𝑡 , 𝑦𝑡 ), 𝐸

(32)

where 𝐸𝑡* is the source evidence that supports the reference outcome and 𝐶𝑡 the source requirements of the task. No other step of source learning directly observes 𝑦𝑡 , and neither 𝑦𝑡 nor 𝐸𝑡* is stored as persistent knowledge. The task is then answered through Eq. (29); when the answer 𝑦ˆ𝑡 is wrong, failure-guided local refinement rereads from 𝐷 the region behind the requirements that the model does not represent and revises it through Eq. (28): 𝐶𝑡− = {𝑐 ∈ 𝐶𝑡 : 𝑐 is not represented in 𝑀𝑡 }, 67

𝑀𝑡+1 = Refine(𝐷, 𝑀𝑡 ; 𝐶𝑡− ).

(33)

The Last Mile to the Real World

Framework details

Requirements that 𝑀𝑡 represents trigger no writing, which separates a representational deficiency from an error made although the required understanding is available. The pass yields the locally refined model 𝑀local . Cross-task representation learning draws from each guidance task at most one lesson about how, not what, to represent, and aggregates the recurring lessons with Π0 into a representation policy Π1 , which introduces no source knowledge: (︀ )︀ ℓ𝑡 = Lesson(𝑞𝑡 , 𝐶𝑡 , 𝑀𝑡 , 𝑦ˆ𝑡 ), Π1 = Aggregate Π0 , {ℓ𝑡 }𝑇𝑡=1 . (34) Recalibration then revisits the source under Π1 and incorporates structure that is missing or represented at an inappropriate granularity through Eq. (28): 𝑀𝑇 = Recalibrate(𝐷, 𝑀local ; Π1 ).

(35)

𝑀𝑇 stays fixed for the test tasks.

Figure 9: Both learning stages and both task-guided pathways add accuracy. Accuracy (%) with GPT-5.6 Luna on two sources per QA benchmark. Left: the full source model 𝑀𝑇 against removing self-directed or task-guided learning and against the initial model 𝑀0 . Right: removing failure-guided local refinement, cross-task representation learning, or both, which recovers 𝑀𝑆 , under shared task-time access [47].

Sources and Counterparts. MultiDoc2Dial contributes four government domains, DMV, SSA, StudentAid, and VA, each a source of 92–149 documents with 60 questions. NarrativeQA contributes 12 stories of its official test split, six movie scripts and six books with 27–30 questions each, read in full. SWE-QA contributes six of its fifteen repositories, streamlink, sphinx, xarray, pytest, flask, and conan, each a source at its pinned commit with 48 questions. APIBench has one source per model hub, TorchHub with 186 instructions and HuggingFace and TensorFlow Hub with 100 each, whose accuracies are averaged; a prompt that exceeds a backend’s context window counts as a failure. AppWorld’s API documentation, 457 API specifications across nine simulated apps, is a single source. RAPTOR [21] runs its official implementation with default settings (100-token chunks, recursive clustering, and collapsed-tree retrieval of the top 10 nodes within 3,500 tokens). HippoRAG 2 [98] runs its official package in its reproduction setting and passes its top 8 passages, matching the evidence budget of Hybrid RAG. AWM [25] induces workflows from the successful guidance-task experiences of each source and split and places them in the prompt of each test task of that split, beside the same Hybrid RAG evidence that SourceLearn receives. Implementation. GPT-5.6 Luna [170] and GPT-OSS-120B [172] run at medium reasoning effort, and DeepSeekV4.1-Flash [171] at high effort on the QA benchmarks and medium effort on APIBench and AppWorld; all methods embed with text-embedding-3-large. 𝑀0 is constructed once per source and backend. Self-Directed Source Learning runs one cycle with 𝐾 = 20 study actions, and its 𝑀𝑆 is shared by the three splits. Hybrid RAG, which 68

The Last Mile to the Real World

Framework details

supplies 𝑅(𝐷, 𝑞), splits documents into sections and code into files and symbol-level units, fuses the top 30 elements by BM25 [161] and by dense retrieval [87] by reciprocal rank fusion, and passes the top 8 to the solver. On AppWorld, the ReAct-style code agent [1] runs for at most 50 steps, its documentation calls answered by a channel that returns each method’s own source context, with the built-in documentation app removed. The three 30%/70% splits of the QA benchmarks and APIBench keep related tasks on the same side, and AppWorld’s 90 training and 168 test-normal tasks serve as guidance and test tasks. The QA judge, GPT-5.6 Luna at medium reasoning effort, accepts an answer that makes the key claims of the reference without contradicting it, and accuracy is macro-averaged over sources and then averaged over splits; APIBench uses its AST-based evaluator, and AppWorld reports task goal completion. The ablations in Figure 9 use two sources per QA benchmark, SSA and VA, pytest and conan, and the scripts of All About Eve and The Breakfast Club, with the guidance and test splits of the main results. Removing Self-Directed Source Learning starts task-guided learning from 𝑀0 , removing Task-Guided Source Learning leaves 𝑀𝑆 , removing cross-task representation learning leaves 𝑀local , and removing failure-guided local refinement recalibrates 𝑀𝑆 under Π1 . Additional Results. The activated source model gives Table 29: SourceLearn against retrieval given the same perthe solver more task-time context than the native re- question context budget. Accuracy (%) with GPT-5.6 Luna on trieval baselines receive. We therefore give each retrieval four sources, each retrieval counterpart reading further down its own ranking up to SourceLearn’s evidence tokens (last row) [47]. baseline the per-question context budget of SourceLearn, Green marks the best and amber the second-best value in each colwhich it fills by reading further down its own ranking umn. without changing its ranking or index (Table 29). SourceMultiDoc2Dial SWE-QA APIBench Learn remains the best on three of the four sources and Method DMV VA Sphinx HuggingFace second on Sphinx, so task-time context alone does not Hybrid RAG 76.2 70.2 62.7 48.6 explain its performance. The remaining analyses also RAPTOR 70.2 73.8 61.8 52.9 use GPT-5.6 Luna with the sources, splits, and judge of HippoRAG 2 71.4 71.5 73.5 51.4 the main results. On the QA benchmarks, the gain over SourceLearn 77.0 77.8 72.5 56.7 Hybrid RAG is 14.9 pp on source regions that guidance Evidence tokens 16.5k 18.2k 23.4k 17.1k tasks touched and 12.1 pp on regions that none touched (Table 45), so the learned understanding carries over to parts of the source that no guidance task exposed. On MultiDoc2Dial, the source model alone, with no retrieved evidence, performs comparably to Hybrid RAG with all eight retrieved elements, and removing the gold document costs Hybrid RAG about 25 pp but SourceLearn about 8 pp (Figure 18). The audits of Tables 43 and 44 cover StudentAid, pytest, and HuggingFace: at most 1% of the sampled units of 𝑀0 , 𝑀𝑆 , and 𝑀𝑇 are contradicted by their cited source elements, and 93.3% to 100% of the claims that a learning stage removed remain represented. Some Task-Guided units on the document and code sources are partially supported or unsupported; on StudentAid, the fully supported share falls from 79.2% in 𝑀0 to 63.3% in 𝑀𝑇 , with 13.3% unsupported. The learned model therefore complements the source with reusable understanding, and the source stays authoritative and directly accessible through 𝑅(𝐷, 𝑞). Mapping to the Method. As in Section 3.4, each source model is the resource of one Knowledge Primitive, sourcemodel activation specializes COMPOSE, and grounded source reconstruction realizes VALIDATE on the publishing path of Eq. (7), along which the two learning stages improve each source model apart from the test tasks.

B.5. Agent Primitives: Reusable Latent Building Blocks Agent Primitives [48] address two limitations of existing multi-agent systems (MAS): they lack a reusable, taskagnostic abstraction for organizing multi-agent computation, so their complexity scales with task complexity, and they communicate in natural language, which degrades in long contexts and under accumulated noise. Drawing on neural network design, where complex models are built from reusable blocks such as residual blocks and attention heads [191], the study decomposes existing MAS architectures into a few recurring, minimal computation patterns, such as iterative review, multi-agent consensus, and plan–execute decomposition, and instantiates three representative primitives that communicate via key–value (KV) cache. Latent communication methods mainly improve how 69

The Last Mile to the Real World Knowledge Pool

Framework details

(a) Primitive-based Multi-Agent System

Access

Centralized

Equiv.

Distributed

Hierarchical

(e) Knowledge Pool MAS Output

User Query Organizer

System + User Query x1 x2 xn

…

MAS Structure

Primitive-based MAS Primitives

Primitives

Input Embedding

Primitives

e1

e2

…

en

Key Cache Value Cache Agent A

Agent A y2 y1

… ym

Key Cache Value Cache Agent B

Output Embedding en+1

en+2 … en+m x1

Compute K, Q, V Latent Reasoning Trace

Executor 1

Solver 1

Critic

Solver 2

…

Selector

Planner

…

Refined Latent Feedback

Executor N

Solver N

(b) Review

(c) Voting and Selection

(d) Planning and Execution

xN

…

e1

e2 …

en

Compute K, Q, V

Directly Use

Executor 2

System x2 … xn

Input Embedding

…

… Agent A Key-Value Cache

Aggregation

Solver

…

Query

Query

Query

…

Agent A Key-Value Cache Key-Value Cache Communication

… Concatenation

Agent B Key-Value Cache

Agent B

(f) Agent-Agent Communication via Key-Value Cache

Figure 10: Overview of Agent Primitives. (a) Agent Primitives enable system-level functionality to be constructed by composing reusable latent operators, which replace manually designed agent roles and natural-language interaction protocols. (b)–(d) Three representative primitives: Review, Voting and Selection, and Planning and Execution, each encapsulating a recurring multi-agent computation pattern as a reusable latent operator. (e) A Knowledge Pool stores previously observed queries and effective system structures, providing structural guidance for system construction. (f) Internal coordination within and across primitives is carried out via latent KV cache communication [48].

information passes between agents [135–137]; Agent Primitives use KV cache communication to implement the computation structure of each primitive. For each query, an LLM Organizer selects and composes the primitives into a primitive-based MAS, guided by a lightweight Knowledge Pool (Figure 10). Latent Communication. For a Transformer decoder of 𝐿 layers [191], the KV cache after 𝑛 positions is 𝒵𝑛 = ℓ ℓ ℓ ℓ ℓ ℓ {(𝐾𝑛ℓ , 𝑉𝑛ℓ )}𝐿 ℓ=1 , and decoding step 𝑡 appends its key and value, 𝐾𝑡 = [𝐾𝑡−1 ; k𝑡 ] and 𝑉𝑡 = [𝑉𝑡−1 ; v𝑡 ]. Latent communication is stated for two agents 𝐴 and 𝐵 for simplicity; the Selector of Voting and Selection reads the caches of all its Solvers. The input prompt 𝑥𝐴 1:𝑛𝐴 of 𝐴 holds its system prompt and the user query. After processing 𝐴 𝐴 𝐴 its full sequence 𝑠 = [𝑥1:𝑛𝐴 ; 𝑦𝑛𝐴 +1:𝑛𝐴 +𝑚𝐴 ] of length 𝑇𝐴 = 𝑛𝐴 + 𝑚𝐴 , 𝐴 holds the final KV cache 𝒵𝑇𝐴𝐴 , a latent 𝐵 representation of 𝑠𝐴 , and the fixed system prompt 𝑥𝐵 1:𝑛𝐵 of 𝐵, processed alone, yields the system-level cache 𝒵𝑛𝐵 : 𝒵𝑇𝐴𝐴 = {(𝐾𝑇𝐴,ℓ , 𝑉𝑇𝐴,ℓ )}𝐿 ℓ=1 , 𝐴 𝐴

𝒵𝑛𝐵𝐵 = {(𝐾𝑛𝐵,ℓ , 𝑉𝑛𝐵,ℓ )}𝐿 ℓ=1 . 𝐵 𝐵

(36)

A combined KV cache concatenates the latent states of 𝐴 and the system prompt of 𝐵 at each layer: ˜ 𝐵,ℓ = [𝐾𝑛𝐵,ℓ ; 𝐾 𝐴,ℓ ], 𝐾 𝑇𝐴 𝐵

; 𝑉𝑇𝐴,ℓ 𝑉˜ 𝐵,ℓ = [𝑉𝑛𝐵,ℓ ]. 𝐵 𝐴

(37)

˜ 𝐵,ℓ , 𝑉˜ 𝐵,ℓ )}𝐿 replaces the KV cache of 𝐵, which then decodes autoregressively The resulting cache 𝒵˜𝐵 = {(𝐾 ℓ=1 without natural-language communication from 𝐴. Input-Output Alignment Assumption. KV cache communication assumes that the decoder conditions future tokens on its accumulated key–value states alone, so that conditioning on 𝒵 𝐴 is distributionally equivalent to conditioning on the token sequence 𝑠𝐴 : 𝐴 𝐵 𝑝(𝑦 | 𝒵 𝐴 , 𝑥𝐵 (38) 1:𝑛𝐵 ) ≈ 𝑝(𝑦 | 𝑠 , 𝑥1:𝑛𝐵 ). The approximation holds when both agents share the same model parameters and positional encoding scheme, so all agents use the same LLM, and all evaluations stay within one model architecture. KV Cache Positional Re-encoding. Rotary positional encoding (RoPE) [139] rotates the key at position 𝑡 as RoPE(𝐾𝑡ℓ ) = 𝑅(𝑡)𝐾𝑡ℓ , where 𝑅(𝑡) is a deterministic block-diagonal rotation operator whose 2D rotation angles are linear in 𝑡. The concatenated cache is treated as one continuous sequence, in which the system prompt of 𝐵

70

The Last Mile to the Real World

Framework details

occupies positions 1 to 𝑛𝐵 and the cache of 𝐴 positions 1 to 𝑇𝐴 . To preserve RoPE semantics, the KV states of 𝐴 are re-indexed by the offset 𝑛𝐵 , for each layer ℓ and 𝑡 ∈ {1, . . . , 𝑇𝐴 }, RoPE(𝐾𝑡𝐴,ℓ ) → 𝑅(𝑡 + 𝑛𝐵 )𝐾𝑡𝐴,ℓ ,

(39)

so that the concatenated cache can be used directly in standard decoding. Communication Bottleneck. Two stress tests with Qwen3-8B [169] motivate KV cache as the medium. In longcontext task injection, an auxiliary instruction is injected at the beginning, midpoint, or end of the long contexts of 351 En.QA examples from InfiniteBench. Without injection, natural-language communication reaches 51.6% accuracy and KV cache communication 61.3%. At the midpoint, injection compliance falls to 15.6% for natural language, against 73.3% for KV cache, while accuracy stays stable across positions, so the degradation stems from communication failure. In communication noise, 100 correctly solved GSM8K [153] traces, truncated to 6,000 tokens, are continued by another Qwen3-8B within 5,000 tokens while 0 to 25 sentences of unrelated reasoning are injected. With one and three noise sentences, natural language solves 91% and 73% of the problems and KV cache all of them; with 10 and 25, 47% and 40% against 93% and 77%. Primitive Design. Internal coordination runs exclusively through latent KV cache exchange, so a primitive’s behavior is independent of text-level message passing, and each primitive can serve as a drop-in replacement for a single agent. The role system prompts fix role boundaries independently of task-specific content and encode no explicit problem-solving strategy. Review, an iterative self-critique operator, connects a Solver 𝐴 and a Critic 𝐵 through a latent feedback channel: 𝐵 consumes the KV cache of 𝐴 to generate corrective feedback, and the modified latent representation returns to 𝐴 for refinement until a stopping condition derived from the intermediate latent states holds, which gives adaptive depth of refinement. The Critic provides evaluative signals and neither revises the solution nor answers. In Voting and Selection, a consensus operator, parallel Solvers produce latent candidates independently, and a Selector votes and selects over these candidates directly in latent space, computing an aggregated latent state that produces the output; it judges correctness, consistency, and overall solution quality without introducing reasoning beyond the candidates. In Planning and Execution, a latent decomposition operator, a Planner 𝑃 produces a latent plan that decomposes the task into intermediate steps or subgoals without solving it, and an Executor 𝐸 follows the plan without modifying it to generate the output. Primitive-Based MAS. A primitive-based MAS is a composition graph of primitives, which an LLM Organizer, GPT-5.2 by default, builds per query. The Organizer’s prompt supplies the query, the three primitives, and the Knowledge Pool, and instructs it to analyze the task requirements and complexity, to select primitives, and to determine their execution order and composition structure. The Organizer reasons at the level of primitives, designs no task-specific roles or natural-language protocols, and does not solve the task. Its primitive composition plan specifies (i) the primitives to instantiate and (ii) their execution structure in code, at 200–400 output tokens per query. The lightweight Knowledge Pool holds 45 previously observed queries paired with effective MAS structures from frameworks including Multi-Agent Debate [131], DyLAN [133], Self-Refine [140], AFlow [29], and MAS-GPT [134]. The Organizer retrieves entries by the input query as structural guidance and abstracts them into primitive compositions, replacing task-specific agents with Agent Primitives. The system then executes the plan, coordinating within and across primitives through KV cache, and externally behaves like a standard MAS. Implementation. The study runs five worker backbones, Qwen3-4B, Qwen3-8B, Qwen3-14B, DeepSeek-R1Distill-Qwen-32B, and DeepSeek-R1-Distill-Llama-70B, of which Table 10 reports three. With Qwen3-4B and Qwen3-14B, the primitive-based MAS reaches 69.6% and 77.6% average accuracy, against 55.7% and 65.5% for a single agent. LatentMAS [137] runs in its sequential configuration with 𝑚 = 40, its best setting, and four agents, so TextMAS and the single-primitive baselines use four as well: two rounds for Review, three Solvers and one Selector for Voting and Selection, and one Planner with three Executors for Planning and Execution. In the primitive-based MAS, the Organizer sets the number of agents per query. The metrics are accuracy, token usage, and average inference time per query.

71

The Last Mile to the Real World

Framework details

RoPE Re-indexing. Table 30 disables the re-indexing of Eq. (39) while the backbones and primitives stay unchanged. With Qwen3-8B, the composed MAS loses 13.3 pp on AIME25 and at most 2.7 pp on HumanEval+ and MedQA. With R1/Llama-70B, each primitive and the composed MAS lose between 26.7 and 59.7 pp, so positional re-encoding is critical for stable KV cache communication, most of all on the Llama-based backbone. Table 30: RoPE re-indexing in the KV cache communication of Agent Primitives. Accuracy (%) with and without positional re-encoding for each primitive used alone and for the composed MAS, all communicating agents using the same LLM (R1/Llama-70B is DeepSeek-R1Distill-Llama-70B), with the change from the re-indexed run in pp in parentheses [48]. Model

RoPE Method

AIME25

HumanEval+

MedQA

w/

Review Voting Planning Primitives-based MAS

60.0 66.7 66.7 73.3

78.6 81.0 78.6 82.3

64.2 70.3 67.0 76.7

w/o

Review Voting Planning Primitives-based MAS

56.7 (−3.3) 60.0 (−6.7) 56.7 (−10.0) 60.0 (−13.3)

73.8 (−4.8) 79.9 (−1.1) 78.0 (−0.6) 81.1 (−1.2)

61.5 (−2.7) 66.8 (−3.5) 65.2 (−1.8) 74.0 (−2.7)

w/

Review Voting Planning Primitives-based MAS

50.0 56.7 50.0 56.7

82.3 82.9 82.3 85.3

72.9 77.8 74.0 81.9

w/o

Review Voting Planning Primitives-based MAS

16.7 (−33.3) 26.7 (−30.0) 23.3 (−26.7) 26.7 (−30.0)

22.6 (−59.7) 29.9 (−53.0) 24.4 (−57.9) 31.1 (−54.2)

31.4 (−41.5) 34.7 (−43.1) 30.5 (−43.5) 36.6 (−45.3)

Qwen3-8B

R1/Llama-70B

Further Results. Table 48 prices the methods of Table 11 per query at market output-token prices of $1.75 per million tokens for Llama-3-70B-Instruct and $14 per million for GPT-5.2, and divides accuracy by cost. With the same Llama-3-70B-Instruct workers, Claude-4 as Organizer reaches 72.8%, 93.2%, 81.1%, and 53.6% on MATH, GSM8K, HumanEval+, and GPQA, Qwen3-32B 71.3%, 89.8%, 78.6%, and 49.1%, and Llama-3-70B-Instruct itself, the iso-model setting, 72.0%, 91.8%, 81.0%, and 52.2%, so the framework does not rely solely on a proprietary Organizer. Over 100 examples per task category, Review accounts for 15%, 22%, and 48% of the selected primitives on MATH, HumanEval, and MedQA, Voting and Selection for 52%, 18%, and 35%, and Planning and Execution for 33%, 60%, and 17%, with 3.4, 2.8, and 2.5 primitives per query, so the Organizer adapts its composition to the task. On ARC-Challenge, which has no entry in the Knowledge Pool, the Organizer without the pool outperforms a single agent by 2.5 and 2.3 pp with Qwen3-4B and Qwen3-8B and random structure selection by 1.9 and 1.1 pp, and the pool adds a further 1.2 pp on average. Mapping to the Method. The Organizer realizes COMPOSE (Eq. (4)), with the Knowledge Pool as the structural guidance it reads, and its primitive composition plan is the plan of that equation. Along the plan’s dependencies, agents that share one LLM pass the combined cache of Eq. (37). The Critic and the Selector judge results inside their primitives, the Review loop is the family’s revision, and the primitives and the pool stay fixed across queries.

B.6. Dev-Primitives: Activation, Collaboration, Execution, and Diagnosis Dev-Primitives are modular executable interfaces that turn repository components into active participants in software engineering [49]. HERMES, the harness engineering framework that instantiates them, consists of an activation module, a dynamically activated set of Dev-Primitives, an executable environment, and a diagnosis module. Dynamic activation and bug diagnosis, both implemented by prompting an LLM, are the two mechanisms that it supplies beyond the Dev-Primitive abstraction. Figure 11 gives the general workflow, in which bug diagnosis maps the observed failure back to the components that must be revised and the next round reactivates them. In one

72

The Last Mile to the Real World

Framework details

1

4  Failing test cases

Dynamic Primitive Activation

SWE Task

Bug Localization

Bug Report

Dependency Expansion

Primitive Selection

• •

Objective Assignment

3/5 test cases passed failed: test_run_bar

Repository of Dev-Primitive

Localize & Select

Failure Attribution

2

Results

3

Runtime

Python environment Install dependencies Capture logs & traces

YAML

file_a.py

utils.py

file_b.py

config

test.py

Test Run unit tests Collect test results

Natural Language Inter-file Communication

Example: Dev-Primitive for file_a.py

Identity: file_a.py Type: Python Module Function: API Logic Brief: file_a.py

Core API functions for calculation

C:\

Self (Modification)

Dev-Primitive 1 1

- return x * 2 + return x * 3

• old line

new line

• …

Shell/Terminal Run commands Execute code

Execution Environment

modify / edit code

5

Activation

 Suspected root cause • console log: missing key "threshold" • error traces in utils.py:42

…

2

1

Refined Patch Self-correcting Patch

Collaboration Inter-file Communication

3 Execution Run & Test

4 Diagnosis Locate & Reactivate

5 Refined Patch Generate Patch

Figure 11: Overview of HERMES. Dynamic activation selects the repository components relevant to the task and instantiates DevPrimitives, which modify their own artifacts and exchange requirements in natural language. The modified repository is executed, and bug diagnosis maps the observed failure back to the components that must be revised, reactivating only those for the next round [49].

SWE-bench Verified trajectory with Qwen3-8B, six activated Dev-Primitives exchange 15 directed messages, and the issue is resolved in the first round. Dev-Primitives. A repository consists of 𝑁 software components ℛ = {𝑎𝑖 }𝑁 𝑖=1 , and each component 𝑎𝑖 is associated with a Dev-Primitive 𝑃𝑖 instantiated with an LLM ℳ: (︀ )︀ (𝑎′𝑖 , 𝑚𝑖 ) = 𝑃𝑖 (𝑎𝑖 , 𝑥𝑖 , 𝒞𝑖 ) = ℳ [ 𝑎𝑖 ; 𝑥𝑖 ; 𝒞𝑖 ] , (40) where 𝑥𝑖 is the local objective assigned to 𝑃𝑖 , 𝒞𝑖 the communication context of messages received from other DevPrimitives, 𝑎′𝑖 the optionally modified artifact (𝑎′𝑖 = 𝑎𝑖 when no modification is required), and 𝑚𝑖 a natural-language message with task-relevant information or requirements for other components. In local modification, the primitive reasons over and edits its own artifact. In inter-primitive communication, it sends requirements such as interface changes, dependency updates, or testing requirements to the components that its local reasoning implicates, in practice its callers, callees, configurations, and tests. Each message is placed verbatim into its target’s communication context, and no central agent interprets, filters, or rewrites it. Participation does not imply modification: a primitive may contribute information while leaving its artifact unchanged. State and Granularity. A Dev-Primitive persists across the revision rounds of a task with its current artifact and the communication accumulated during the trajectory; model hidden states are not preserved across calls. A new persistent file that an active Dev-Primitive creates is registered as a Dev-Primitive, while temporary outputs, caches, and logs are environment observations. Files are the default granularity, and on Terminal-Bench the components are the persistent filesystem artifacts, such as shell scripts, configuration files, and service definitions. Dynamic Primitive Activation. The activation module converts the issue 𝑞 into a plan over repository components, Π = ACTIVATE(𝑞, ℛ) = {(𝑖, 𝑥𝑖 )}𝑖∈𝒜 ,

73

𝒫𝒜 = {𝑃𝑖 | 𝑖 ∈ 𝒜},

(41)

The Last Mile to the Real World

Framework details

where 𝒜 ⊆ {1, . . . , 𝑁 } indexes the selected components; HERMES instantiates 𝒫𝒜 , and inactive components receive no model calls. Activation is a single LLM call chain of issue analysis, component localization, dependency analysis, task decomposition, and edit planning. Here ℛ denotes access to the repository, not its concatenation into one context: the module retrieves candidate files and expands along dependencies by reading them, without a precomputed call graph. For each activated component, the plan gives an objective, dependencies, communication targets, and an expected action (inspect, modify, or validate), followed by an execution plan. Collaboration and Execution. The activated primitives operate as one round, and HERMES applies their edits: (︀ )︀ (︀ )︀ {𝑎′𝑖 }𝑖∈𝒜 ← C OLLABORATE {𝑃𝑖 }𝑖∈𝒜 , {𝑥𝑖 }𝑖∈𝒜 , ℛ′ ← U PDATE R EPOSITORY ℛ, {𝑎′𝑖 }𝑖∈𝒜 . (42) Communication is interleaved with local modification and follows the repository dependencies encountered during execution, so a recipient may revise its modification after receiving a requirement within the round. The modified repository is executed in the benchmark-provided environment, (︀ )︀ 𝑜 = EXECUTE(ℛ′ ) = 𝑜shell , 𝑜test , 𝑜runtime , 𝑜trace , (43) which records shell commands, outputs, exit codes, and build status; test results; runtime behaviour and exceptions; and logs and stack traces. During solving, all modules use task-visible checks, such as repository tests, build commands, linters, and reproduction scripts written while solving. Held-out evaluation tests, such as the FAIL_TO_PASS and PASS_TO_PASS tests of SWE-bench Verified, are withheld until HERMES terminates. Bug Diagnosis and Revision. Given the issue, the plan, the modified repository, and the execution observation, one LLM call returns the verdict 𝑣 = DIAGNOSE(𝑞, Π, ℛ′ , 𝑜) ∈ {PASS, FAIL}, (44) and, on FAIL, structured feedback 𝜑 = (𝑒, 𝑐, 𝑢): the observed failure 𝑒, the suspected root cause 𝑐 together with the components implicated by the evidence, including components that activation may have missed, and revision guidance 𝑢 for each component. Diagnosis does not modify artifacts and cannot override deterministic execution failures: a trajectory with failing task-visible tests or unsuccessful compilation cannot be marked PASS on model judgment alone. Activation then revises the plan, Π′ = ACTIVATE(𝑞, ℛ′ , Π, 𝜑),

(45)

retaining the modifications that new execution evidence does not contradict, updating objectives, removing components that are no longer relevant, and activating newly implicated Dev-Primitives. HERMES returns ℛ′ on PASS and a structured failure report FAILURE R EPORT(𝑞, ℛ′ , 𝑜) once the budget 𝐵 is spent. 𝐵 counts the revision rounds after the initial plan–collaborate–execute–diagnose round, so 𝐵 = 0 performs that round without revision. Implementation. Each Dev-Primitive is instantiated with its own LLM, which may differ from the activation and diagnosis models (Table 50). The primary backbones are GPT-5.6 Sol, Terra, and Luna, Claude Sonnet 5, Claude Opus 5, DeepSeek-V4, and Qwen3-8B [169]. Reasoning effort is medium where configurable unless a table states otherwise, and 𝐵 = 3. Qwen3-8B is served locally with Ollama on NVIDIA A40 GPUs, with its default sampling parameters (temperature 0.6, top-𝑝 0.95, top-𝑘 20) and Ollama’s 32K context window; all other models are accessed through their official APIs. SWE-bench Verified uses the official snapshots, SWE Refactor Bench its composite score, Terminal-Bench 4.0 five runs per task, and DevOps-Gym the unweighted average of its build and configuration, monitoring, issue-resolving, and test-generation categories. Token counts and costs include all activation, Dev-Primitive, and diagnosis calls. Baseline results come from the benchmark leaderboard or published evaluation where the model, harness, reasoning effort, and benchmark version are reported, and are otherwise run under the same snapshot and protocol as HERMES. On SWE Refactor Bench, each configuration runs on all 20 migration tasks, and the Claude and GPT-5.6 models are evaluated at medium and high effort (Table 31). Component Selection. Selection is measured against a target component set 𝒢, with 𝒜 the Dev-Primitives activated across all rounds: recall is |𝒜 ∩ 𝒢|/|𝒢| and precision |𝒜 ∩ 𝒢|/|𝒜|, micro-averaged within each benchmark. 74

The Last Mile to the Real World

Framework details

Table 31: HERMES against same-backbone harnesses on SWE Refactor Bench. Composite score (%) and cost per task (USD) on the 20 whole-repository migration tasks, per model and reasoning effort, with Δ the composite change of HERMES in pp. Qwen3-8B is run with HERMES only [49]. Green marks the best and amber the second-best value in each Comp. and Cost column. Lower is better for cost. Baseline Results Model

Effort

HERMES Results

Comp. (%) Cost ($) Comp. (%) Cost ($)

Baseline harness: Claude Code medium 28.5 Claude Opus 5 high 34.5

Δ Comp.

38.4 55.7

36.5 41.5

58.6 82.4

+8.0 +7.0

Claude Sonnet 5

medium high

15.0 6.0

11.9 24.6

22.0 25.5

22.7 34.8

+7.0 +19.5

Kimi-K3

max

19.5

28.9

27.0

43.8

+7.5

Qwen3.8-Max

max

10.0

14.5

18.5

22.6

+8.5

DeepSeek-V4

max

7.0

4.3

15.0

11.6

+8.0

GLM-5.2

max

6.5

17.5

14.5

27.2

+8.0

Baseline harness: Codex medium GPT-5.6 Sol high

6.5 19.0

6.0 7.7

31.0 36.5

24.8 39.6

+24.5 +17.5

GPT-5.6 Terra

medium high

4.5 11.5

3.4 4.8

26.0 30.0

14.6 23.9

+21.5 +18.5

GPT-5.6 Luna

medium high

0.0 4.0

1.7 1.8

16.0 20.0

4.9 7.2

+16.0 +16.0

–

–

10.5

8.3

–

No baseline harness Qwen3-8B –

Where a reference patch exists, 𝒢 holds the files that it modifies, and otherwise the components that benchmarkspecific supervision implicates; DevOps-Gym’s monitoring tasks are excluded. With GPT-5.6 Sol, recall is 95.4% on SWE-bench Verified, 83.3% on SWE Refactor Bench, 84.5% on Terminal-Bench 4.0, and 85.7% on DevOpsGym, precision ranges from 64.9% to 72.6%, and 4.7 to 12.6 Dev-Primitives are activated per task. Solved tasks exceed failed tasks in recall by 16.5 to 24.7 points, where a SWE Refactor Bench task counts as solved when its composite score is non-zero. Revision Budget. Raising 𝐵 from 0 to 3 adds 12.5 points on SWE Refactor Bench, 12.1 on Terminal-Bench, 9.50 on DevOps-Gym, and 4.8 on SWE-bench Verified, while raising it from 3 to 5 adds at most 0.9, which makes 𝐵 = 3 the default. Controls and Ablations. Table 32 separates the Dev- Table 32: A centralized editor over the same components. Primitives from the choice of components on Terminal- Terminal-Bench 4.0 with GPT-5.6 Sol at medium effort; all setBench 4.0. The single editor keeps GPT-5.6 Sol, the tings share the selected components, environment, diagnosis module and revision budget, and tokens (B) and cost ($k) are evaluation activation-selected components, the environment, the di- totals [49]. Green marks the best and amber the second-best value agnosis module, and the revision budget, and replaces in the Resolution column. the Dev-Primitives and their messages by one editor that Resolution Tokens Cost receives the whole plan with the contents of all selected Setting (%) (B) ($k) components and modifies them jointly. Across the four GPT-5.6 Sol benchmarks, it uses 10–14% fewer tokens and 11–16% Single editor 38.8 5.9 2.90 + compute matching 45.5 6.7 3.35 lower cost than HERMES. On Terminal-Bench 4.0, it reHERMES 51.8 6.8 3.40 solves 38.8% of the tasks against 51.8% for HERMES. Its compute-matched variant generates several candidate revisions per round, evaluated with the same environment and diagnosis, until tokens and cost approximately match, 6.7B against 6.8B tokens, and reaches 45.5%. Over the same components, the editor also loses 3.6 pp on SWE-bench Verified, 14.2 pp on SWE Refactor Bench, and 11.20 pp on DevOps-Gym, so on all benchmarks except SWE-bench Verified the replacement costs more than removing any single mechanism. Among the single-mechanism removals 75

The Last Mile to the Real World

Framework details

of Table 15, removing inter-primitive communication keeps each Dev-Primitive’s local reasoning and editing but exchanges no messages, and lowers SWE-bench Verified from 97.0% to 94.2%. Without on-demand activation, all Dev-Primitives identified in the initial task analysis stay active through all rounds, and primitive calls per SWEbench Verified task rise from 9.6 to 28.7 (Table 52). Without execution feedback, diagnosis judges the repository by static inspection alone, and Terminal-Bench resolution falls to 43.6%. Without diagnosis feedback, the raw execution outputs remain available and later rounds proceed, but no structured diagnosis or guidance is produced. Three repetitions of the default configuration on SWE Refactor Bench give composite scores of 29.5%, 32.0%, and 31.5%, that is, 31.0 ± 1.3. Related Frameworks. A Dev-Primitive is defined by the artifact it owns, so the set of primitives is induced by the repository, whereas the libraries of Agent and Tool Primitives are built ahead of their tasks [44, 48]. MAGIS and CodeTeam instead assign work to designer-specified roles or centrally coordinated file-level developers [77, 78], skill-based methods encapsulate reusable, task-agnostic behaviours [73], and context-management methods reorganize information within one centralized trajectory [84, 85]. In their original evaluation settings, MAGIS reports 13.9% resolution on SWE-bench against 36.6% for HERMES with GPT-5.6 Luna, and CodeTeam reports average test pass rates of 34.6% and 42.3% on NL2Repo-Bench [81] for its prompting and supervised fine-tuning variants, against 44.9% for HERMES. Relation to the Method. Dynamic Primitive Activation (Eq. (41)) realizes COMPOSE (Eq. (4)), and Dev-Primitive Collaboration (Eq. (42)) the act step of Eq. (5). Execution and Bug Diagnosis (Eqs. (43) and (44)) realize EXECUTE and VERIFY, with 𝜑 = (𝑒, 𝑐, 𝑢) as the feedback 𝜑𝑡 , and the re-planned activation of Eq. (45) is the revision of Eq. (6). The repository induces the library, which grows by the persistent files that active Dev-Primitives create.

B.7. Navigation Primitives: Need-Conditioned Coordination Navigation Primitives are scope-specialized workers, each searching one territory of an information space, and ANTMAN [50], an adaptive coordination framework, decides which of them search for a query. ANTMAN answers a query 𝑞 from an information space ℐ whose available information can grow large while the information that 𝑞 requires stays comparatively small. Newly encountered information can change the searcher’s understanding of the problem and redirect the search [187], so ANTMAN makes evolving unresolved information needs the runtime control state for coordination, which lets worker activation and search decisions adapt as those needs change. The method has three parts, shown in Figure 12: a static substrate map, a revisable Need Graph, and need-conditioned coordination and recovery. Static Substrate Map. The map describes where ANTMAN can search. A substrate-specific partitioning procedure 𝑃 deterministically splits ℐ into 𝑀 territories, bounded searchable regions, and each territory receives a WorkerCard, lightweight metadata that summarizes its scope and content: 𝒯 = 𝑃 (ℐ) = {𝜏1 , . . . , 𝜏𝑀 },

𝑐𝑖 = Card(𝜏𝑖 ),

𝒞 = {𝑐1 , . . . , 𝑐𝑀 }.

(46)

The worker 𝑤𝑖 of territory 𝜏𝑖 is scope-specialized: ANTMAN requires no distinct model architectures, tool sets, or personas, and within a substrate all workers share one interface for searching, inspecting, and returning evidence. Eq. (46) defines the workers that are available, not those that must participate in a query, and the map stays fixed during execution. Revisable Need Graph. The Need Graph describes what remains unresolved. The coordinator initializes 𝐺0 = InitializeNeedGraph(𝑞) and maintains the runtime state (︀ )︀ 𝐺𝑡 = (𝑁𝑡 , ∆𝑡 ), 𝑧𝑡 (𝑛) = 𝑠𝑡 (𝑛), ℰ𝑡 (𝑛), ℋ𝑡 (𝑛), 𝑝𝑡 (𝑛) , (47) where 𝑁𝑡 is the set of information needs at step 𝑡, ∆𝑡 holds their dependencies, and each need 𝑛 carries its resolution status 𝑠𝑡 (𝑛), accumulated evidence ℰ𝑡 (𝑛), attempt history ℋ𝑡 (𝑛), and progress state 𝑝𝑡 (𝑛). The unresolved needs 76

The Last Mile to the Real World

Framework details

substrate map 1 Static Fixed territories define where search is possible.

Need Graph 2 Revisable Explicit state for what remains unresolved.

Information substrate

User Query clients/ client.py defaults.py

services/ api.py handlers.py

Code

Repos

tests/ test_timeout.py test_api.py

docs/ Readme.md test_result.md

Docs

frontend/ main.html app.css

scripts/ runtime.py check.py

Which client.send callers need a timeout update?

Unresolved information needs

n1

n3

n2

n4 Resolved Status

1

Select one open need

2

Discover + route

Attempts

4

Deterministic partition Territory 1 clients/ services/ Territory 2 docs/ frontend/

Territory 3 tests/ scripts/

summarize

summarize

summarize

Wrappers and defaults

WorkerCard 2 Stats and Website

Retrieve

Add

Delete

n5

Timeout checks

Follow references

n1

n2′

Resolved

Reframed

n4 n3

New need Emerged

Dispatch

Call sites Defaults Checks

services/ 1 active

Bounded local search ·services/ Search and read within the selected territory Find references → read snippet 42 client.send(payload, **kwargs) Territory 2 only

Structured report

Evidence Progress Uncertainty

Evidence, progress, and uncertainty

Composer assessment

Update

Updated information needs

WorkerCard 3

Shared worker toolbox Read local region

LLM Graph Revision

WorkerCard 1

RRF WorkerCards 1 services/ 2 clients/ 3 tests/

services/api.py

Progress

Update from worker report

Hybrid retrieval and RRF ranking of fixed WorkerCards

Sparse / exact / dense

3

Which callers set timeout?

Status ·dependencies ·progress

Candidates

Waiting

Open

Evidence

coordination 3 Need-conditioned Route, search, revise, and recover as needs evolve.

api.py:42 + snippet Caller found Wrapper timeout?

Repeated no progress?

Resolved

Partial

Unresolved

Close need

Refine need

Keep open

Reframe Reroute Fallback

5

Update Need Graph

6

Resolved evidence → synthesis → answer

Update needs; repeat 1–5 while needs remain

Use accumulated evidence from resolved needs only

n1 Resolved

n2′ Open

n3 Waiting

api.py:42; handlers.py:87

Default still unknown

Not included

Termination gate

Partial answer at budget limit

Required needs resolved or budget spent

Call sites: api.py:42; handlers.py:87 Timeout evidence: unresolved

Figure 12: Overview of ANTMAN. (1) Static substrate map: a deterministic partition splits the information substrate, such as code, repositories, and documents, into fixed territories, each summarized by a WorkerCard, and every worker uses one shared toolbox. (2) Revisable Need Graph: the user query becomes unresolved information needs whose status, evidence, attempts, and progress are updated from each worker report; an LLM graph revision then adds, deletes, or updates needs, so that a need can be resolved or reframed and a new need can emerge. (3) Need-conditioned coordination: one open need is selected and routed through WorkerCard retrieval to one territory worker, which searches only its territory and returns evidence, progress, and uncertainty; the assessment closes, refines, or keeps the need open, repeated lack of progress triggers reframing, rerouting, or fallback, and the answer is synthesized from resolved evidence once the required needs are resolved or the budget is spent. The example follows the query “Which client.send callers need a timeout update?” over a small repository [50].

form 𝒰𝑡 = {𝑛 ∈ 𝑁𝑡 | 𝑠𝑡 (𝑛) ̸= resolved}; resolved needs stay in 𝑁𝑡 , and final synthesis reads their evidence. After each report 𝑟𝑡 , the task-local update 𝐺𝑡+1 = 𝑈 (𝐺𝑡 , 𝑟𝑡 ) may resolve a need, keep it unresolved, reframe it into 𝑛′ , or add requirements and dependencies that new evidence reveals, so needs may branch during execution. Within 𝑈 , evidence selection keeps the evidence most relevant to the query and discards redundant or unsupported material, and need-resolution assessment judges the need resolved, partial, or unresolved: a resolved need is closed, a partial need is replaced or refined into a more specific unresolved requirement, and an unresolved need stays active. Need Graph consolidation adds each proposed need as a new requirement, attaches it to an existing dependency, merges it with an overlapping need, subsumes it under an existing one, or discards it if redundant. Need-Conditioned Coordination and Recovery. At step 𝑡, ANTMAN selects an unresolved need, routes it with the Need Graph and the WorkerCards, and invokes the corresponding territory-backed worker: 𝑛𝑡 = Select(𝐺𝑡 ),

𝑖𝑡 = Route(𝑛𝑡 , 𝐺𝑡 , 𝒞),

𝑟𝑡 = Exec(𝑤𝑖𝑡 , 𝑛𝑡 , 𝜏𝑖𝑡 ),

(48)

where 𝑖𝑡 ∈ {1, . . . , 𝑀 } identifies the selected worker. Route has two stages: a non-LLM retrieval stage ranks the WorkerCards by sparse lexical, exact-term, and dense matching, combined by reciprocal-rank fusion, into a bounded candidate set, and the orchestrator then decides from 𝐺𝑡 whether and where to dispatch the need, respecting the dependencies between needs. Retrieval bounds the WorkerCard information within one routing decision, while the Need Graph governs the sequence of decisions and when search ends. The dispatched worker plans a bounded sequence of actions with its substrate’s tools, searches its own territory, and returns a report 𝑟𝑡 of evidence, progress, 77

The Last Mile to the Real World

Framework details

and remaining uncertainty; global state remains in 𝐺𝑡 . If repeated attempts make no progress, Stuck(𝑛𝑡 , 𝐺𝑡+1 ) holds and Recover returns Reframe, Reroute to another territory, or Fallback to an alternative resolution path, leaving the workers and the map unchanged. When the required needs are resolved or the execution budget 𝐵 is spent, ANTMAN synthesizes the answer from the evidence of resolved needs, which completes the execution procedure of Algorithm 2. Algorithm 2 ANTMAN execution procedure [50]. The substrate map remains fixed, while the Need Graph is revised during execution to select, route, and recover unresolved information needs. Require: query 𝑞, information space ℐ, execution budget 𝐵 Ensure: final answer 𝑦 1: 𝒯 ← 𝑃 (ℐ) |𝒯 | 2: 𝒞 ← {Card(𝜏𝑖 )}𝑖=1 3: 𝐺0 ← InitializeNeedGraph(𝑞) 4: 𝑡 ← 0 5: while not Terminate(𝐺𝑡 , 𝐵) do 6: 𝑛𝑡 ← Select(𝐺𝑡 ) 7: 𝑖𝑡 ← Route(𝑛𝑡 , 𝐺𝑡 , 𝒞) 8: 𝑟𝑡 ← Exec(𝑤𝑖𝑡 , 𝑛𝑡 , 𝜏𝑖𝑡 ) 9: 𝐺𝑡+1 ← 𝑈 (𝐺𝑡 , 𝑟𝑡 ) 10: if Stuck(𝑛𝑡 , 𝐺𝑡+1 ) then 11: 𝑎𝑡 ← Recover(𝑛𝑡 , 𝐺𝑡+1 , 𝒞) 12: if 𝑎𝑡 = Reframe then 13: 𝐺𝑡+1 ← ReframeNeed(𝑛𝑡 , 𝐺𝑡+1 ) 14: else if 𝑎𝑡 = Reroute then 15: MarkForRerouting(𝑛𝑡 , 𝐺𝑡+1 ) 16: else if 𝑎𝑡 = Fallback then 17: 𝐺𝑡+1 ← Fallback(𝑛𝑡 , 𝐺𝑡+1 ) 18: end if 19: end if 20: 𝑡←𝑡+1 21: end while 22: ℰ * ← SelectResolvedEvidence(𝐺𝑡 ) 23: 𝑦 ← Synthesize(𝑞, ℰ * ) 24: return 𝑦 Runtime Coordination Demand. number the initial set 𝑁0 :

◁ fixed territories ◁ static WorkerCards

◁ choose an unresolved need ◁ select territory-backed worker ◁ bounded local information seeking ◁ update evidence, status, and dependencies

Because the graph is revised, the needs realized over a query can out-

ℋ(𝑞) =

𝑇𝑞 ⋃︁

𝐻(𝑞) = |ℋ(𝑞)|,

𝑁𝑡 ,

(49)

𝑡=0

where 𝑇𝑞 is the final coordination step and a reframed or newly introduced need counts as new. A need 𝑛 receives 𝑎(𝑛) ≤ 𝜌 routing attempts, its initial routing and any reroutes, and the worker-dispatch events and distinct activated workers are 𝑇𝑞 −1

𝐾(𝑞) =

∑︁

𝒜(𝑞) =

𝑎(𝑛),

⋃︁

{𝑖𝑡 }.

(50)

𝑡=0

𝑛∈ℋ(𝑞)

Each routing decision at Line 7 of Algorithm 2 concerns one need and activates at most one worker, so |𝒜(𝑞)| ≤ 𝐾(𝑞) ≤ 𝜌𝐻(𝑞), and with |𝒜(𝑞)| ≤ 𝑀 this gives ANTMAN’s Proposition 1, |𝒜(𝑞)| ≤ min{𝑀, 𝜌𝐻(𝑞)}, which 78

The Last Mile to the Real World

Framework details

Proposition 3.1 restates in the notation of Section 3 and Appendix A proves together with its corollary. A partitiondriven method that activates a constant fraction of its 𝑀 partitions instead has |𝒜partition (𝑞)| = Θ(𝑀 ). A larger space enlarges the candidate pool, which stays passive until the Need Graph creates demand for it. Implementation. ANTMAN runs all components on GPT-4.1, and ANTMAN-H keeps the GPT-4.1 orchestrator and runs all non-orchestrator components on Qwen3-8B [169]. The information-access layer adapts to each substrate, while the Need Graph, the worker organization, and the coordination procedure stay the same. In the controlled long-context setting, workers use lexical and semantic retrieval with bounded reading of selected document regions; repository navigation adds structural navigation over symbols, imports, references, inheritance, and call relations. GAIA-Text-103 uses a single territory-backed worker with web search, backed by Tavily, webpage text extraction, and isolated Python execution, an interface that all methods on GAIA share. Agentic methods on RepoProbe-Python and all ablation variants run under at most 100 LLM calls per query, and the iterative sparse retrieval baseline allows up to three lexical retrieval rounds, in which the LLM judges whether the accumulated evidence suffices and reformulates the query. Evaluation Settings. Multi-document QA scores 30 questions from each of HotpotQA [93], 2WikiMultiHopQA [157], and MuSiQue [158] for nine methods, 810 predictions in all, with one frozen answer-extraction and evaluation pipeline, and its baselines also run on GPT-4.1. In the space intervention, each per-length entry averages 30 evaluations, ten questions with evidence at early, middle, and late positions. The demand study uses ten matched 512K base contexts, each with 16 planted evidence records over a fixed pool of eight territories; documents, distractors, evidence locations, territory assignment, and WorkerCards stay fixed across demand levels, while the query and its gold answer change with the level. Each an independently generated value (︀∑︀record holds )︀ 𝑒𝑖 ∈ {0, . . . , 7}, and at level 𝑘 ∈ {1, 4, 16} the answer is 𝑦𝑘 = 𝑒𝑖 ∈𝐷𝑘 𝑒𝑖 mod 8 over nested required-evidence sets 𝐷1 ⊂ 𝐷4 ⊂ 𝐷16 , so omitting any required unit leaves the answer ambiguous over all eight classes. Runs are capped at ten coordination rounds, answers are scored by exact match, and Complete Evidence counts a unit as recovered when its content is extracted, rather than when its document is accessed. The ablations use the 512K condition, a 40-question SWE-QA-Pro subset sampled with seed 42, and a fixed 40-question GAIA subset. Their variants keep the territories, WorkerCards, models, tools, execution budget, and synthesis procedure, and the Need-Graph variants also share the initial decomposition and Need Graph. Graph-free Adaptive receives the same query, WorkerCards, accumulated evidence, and interaction history but no explicit Need Graph; w/o Need Revision prevents changes to the graph’s structure or formulation, w/o Adaptive Rerouting fixes each need’s worker after its initial routing, and w/o Recovery disables stuck detection, reframing, and fallback. Additional Results. On multi-document QA (Table 33), ANTMAN’s macro average of 0.711 EM and 0.814 F1 leads the specialized S2G-RAG [108], at 0.700 and 0.799, so need-driven coordination preserves answer quality on standard tasks, and ANTMAN-H, at 0.622 and 0.763, is above all baselines except S2G-RAG. Across the lengths of the space intervention, ANTMAN keeps its active coordination nearly constant and its F1 stable, while the active coordination of LongAgent and CoA grows with the space (Figure 13 and Table 53). Because it narrows the context searched for each unresolved need, ANTMAN is also robust to evidence position: averaged over the 32K to 128K contexts, its Middle F1 minus the mean of its Early and Late F1 is +0.015, against −0.181 for direct full-context inference, which shows the lost-in-the-middle pattern [39]. Under growing demand, the trajectory-level split of worker activity traces the dip in active coordination to exploration at low demand: at one unit, 4.1 of 5.0 worker activities are unrelated to the required evidence (Table 54). Mapping to the Method. Within the method of Section 3, a territory 𝜏𝑖 is the resource 𝑎𝑖 of a Navigation Primitive, and its WorkerCard 𝑐𝑖 is the interface 𝑑𝑖 . Select and the two-stage Route realize COMPOSE one need at a time, the need-resolution assessment inside 𝑈 plays the role of VERIFY, and recovery is the revision step of Eq. (6).

79

The Last Mile to the Real World

Framework details

Table 33: ANTMAN on standard multi-document QA. EM and token-level F1 on their native [0, 1] scales over 30 questions per benchmark, 810 predictions in all, scored with one frozen answer-extraction and evaluation pipeline, with macro averages over the three benchmarks; all methods use GPT-4.1, and ANTMAN-H† keeps the GPT-4.1 orchestrator and runs all other components on Qwen3-8B [50]. Green marks the best and amber the second-best value in each column. HotpotQA

2WikiMultiHopQA

MuSiQue

EM

F1

EM

F1

EM

F1

EM

F1

0.533 0.533 0.567 0.667

0.684 0.696 0.711 0.810

0.367 0.267 0.733 0.800

0.515 0.379 0.791 0.858

0.433 0.200 0.500 0.633

0.582 0.292 0.654 0.730

0.444 0.333 0.600 0.700

0.594 0.456 0.719 0.799

Agentic and multi-agent baselines ReAct 0.533 0.734 0.567 LongAgent 0.433 0.655 0.467 CoA 0.567 0.769 0.600

0.781 0.697 0.778

0.467 0.626 0.522 0.714 0.400 0.494 0.433 0.616 0.533 0.682 0.567 0.743

ANTMAN-H† ANTMAN

0.796 0.823

0.467 0.651 0.622 0.763 0.667 0.777 0.711 0.814

Method Retrieval and RAG baselines Iterative Sparse Retrieval (BM25) Dense Retrieval (DPR) ChainRAG S2G-RAG

0.700 0.842 0.700 0.700 0.842 0.767

(a) Active coordination

(b) Cost per query

320

(c) Answer quality

1.5

0.85

LongAgent $1.397

LongAgent 290.5

CoA 78.2

80

0

32K

64K

128K

Information-space size

512K

0.6

0.0

LongAgent .754

0.75

0.9

ANTMAN $0.533

0.3

ANTMAN 3.17

ANTMAN .840

0.80

1.2

F1

160

Cost per query (USD)

Active coordination

CoA $1.303 240

Avg.

CoA .746

0.70 0.65 0.60 0.55

32K

64K

128K

512K

Information-space size

32K

64K

128K

512K

Information-space size

Figure 13: ANTMAN’s active coordination stays nearly flat as the information space grows. Active coordination (a), cost per query in USD (b), and F1 (c) of ANTMAN, LongAgent, and CoA as distractor filler grows the context from 32K to 512K tokens around preserved evidence at approximately fixed demand. The study uses ten questions with evidence at early, middle, and late positions and GPT-4.1 for all components. Per-length values are in Table 53 [50].

B.8. Harness Primitives: Composing Task-Specific Harnesses Harness Primitives are reusable harness mechanisms with a clear application scope and composition contract, mined from failed task trajectories [51]. Based on them, STITCH Selects suitable primitives given Task Information and compiles them into Task-speCific Harnesses at test time, keeping the development of Harness Primitives separate from their composition into each task’s harness. In Figure 14, development turns a recurring failure, checks that fail without triggering repair, into a validated repair primitive, and test-time composition selects and wires different primitives from the same library for each of three tasks. We state the framework below in STITCH’s own notation. Problem. Let 𝑥 be a task instruction and 𝑐𝑥 an execution specification that describes the task-solving language model, the actor, its interaction interface, and the task environment setup. A harness 𝐻 is the control program that mediates the actor’s interaction with the environment, and for the resulting trajectory 𝜏 , 𝑅𝑥 (𝜏 ) ∈ {0, 1} indicates task success. With the actor model and the environment fixed, a construction procedure 𝐹 maps the task information to a harness 𝐻𝑥 = 𝐹 (𝑥, 𝑐𝑥 ), and the goal is [︀ ]︀ 𝐻𝑥⋆ ∈ arg max E𝜏 ∼𝑝(·|𝑥,𝑐𝑥 ,𝐻) 𝑅𝑥 (𝜏 ) , (51) 𝐻∈ℋ(𝑐𝑥 )

where ℋ(𝑐𝑥 ) is the set of compatible harnesses and 𝑝(· | 𝑥, 𝑐𝑥 , 𝐻) the induced trajectory distribution. Primitives and Harness Graphs. The library 𝒫 = {𝑝1 , . . . , 𝑝𝑀 } holds implemented, reusable control operations. A primitive’s application scope describes when it is expected to facilitate the task and when its overhead or behavior 80

The Last Mile to the Real World

Framework details

1 Develop reusable harness primitives Development failures

Mine deficiencies

Separate development and evaluation tasks

Implement candidate

State Guard

Read the notes

Group failures

Originals edited

Details lost

Save copies Detect changes

Frozen primitive library

Behavior tests

Task improvement

State Guard

Save copies Detect edits Restore files

With / without Task success must improve

Check Replay Execution Supervisor

Restore / report

Unprotected originals

Validate before adding to the library

Both pass: add to the library

Scope + composition contract

Validation fails: revise

2 Compose before the first actor request Task input Original files (read)

Notes

Composer Task requirements Step budget Default workflow Primitive interfaces

Compile the workflow

3 Run the fixed harness

Hooks react, the graph stays fixed

Validate the graph Supported primitives

Before action Save copies of the originals

Compatible specification

Agenda

Summarize the notes in a separate file

State Guard scope

Intent: State Guard Protect: notes + agenda Read: original files Write: separate summary

Compare to copies Restore if changed

Reachable termination

Next request

Bounded loops

In: file locations Out: edit report + status Setup: saved copies

Actor request

On completion

Read the notes

Saved copies After each action: check the originals

Notes + agenda Before the action

NO

Originals changed?

Originals intact No net changes

YES Preserved originals

PASS Ready to run

Inputs and outputs

Separate output Exclude: tasks that must edit originals

Valid wiring

Report changes to the actor

Keep originals Edits may occur

After action

Before action

Completion No selection or retry limit reached: use default harness

Confirm originals are unchanged

Fixed implementations

Invalid graph: revise the selection within the retry limit

Next actor request Read report Use a copy

Continue the task

Restore Original content Report edits

Separate summary

Original files restored

Task evaluation Checks summary

Shared state: history, saved copies, observations

Figure 14: Overview of STITCH, shown with State Guard. (1) On development tasks kept apart from evaluation, recurring failures, such as an actor editing the original files whose details it later needs, are grouped into deficiencies; a candidate primitive is implemented and enters the frozen library only after it passes behavior tests and improves task success over the same harness without it, and each primitive carries an application scope and a composition contract. (2) Before the first actor request, the harness composer reads the task requirements, step budget, default workflow, and primitive interfaces and states a composition intent, which the deterministic compiler wires into a workflow; the graph is validated for supported primitives, compatible specifications, valid wiring, reachable termination, and bounded loops, an invalid graph is revised within a retry limit, and the default harness is used otherwise. (3) The compiled harness then runs fixed: its hooks save copies before each action, check the originals after it, and restore and report any change to the actor [51].

may be undesirable, according to its success on the evaluated tasks; its composition contract specifies its inputs, outputs, required environment setup, and dependencies. Scopes guide the harness composer, and contracts guide the implementation of the harness compiler. A harness is a directed graph 𝐺 = (𝑉, 𝐸) whose nodes are control operations, each invoking a primitive, and whose edges specify execution order and conditional branches; the operations communicate through shared execution state, which holds the interaction history and intermediate results. The seed harness 𝐻seed is abstracted as a graph 𝐺seed , and the task-specific graph 𝐺𝑥 represents 𝐻𝑥 . State Guard. State Guard, one of the five Terminal-Bench primitives, addresses a recurring failure in which actor actions destroy inputs needed later. It blocks recognized direct writes, restores files that an action changes, and reports the affected paths in a failed-action observation. Its scope selects it when local inputs are irreplaceable, actions plausibly risk modifying them, and outputs are separate, and it excludes files that the task requires editing in place. Its contract has the source paths to protect and the declared output paths as inputs, mutation reports and integrity-check results as outputs, file access and storage for recoverable copies as setup, and as dependencies a snapshot before the first actor action, a check after each action, violation feedback to the next actor request, and the final integrity check for completion logic. Primitive Development. A development agent, distinct in role from the actor, runs the actor under 𝐻seed on a development set 𝒟dev and analyzes the failed attempts (𝑅𝑥 (𝜏 ) = 0) for recurring deficiencies. For each deficiency, it proposes a mechanism, specifies when it should activate and which subsequent actor decision it should influence, and implements it. Mechanism tests check that the operation activates and that its output changes the relevant request, action, or control-flow decision, and matched development runs with and without the candidate 81

The Last Mile to the Real World

Framework details

measure its benefit to 𝑅𝑥 (𝜏 ). This evidence drives iterative revision and library acceptance: only primitives with a positive effect are accepted, development ends once the number of valid primitives passes a threshold 𝜃, and each accepted primitive then receives its scope and contract. The seed harness is mini-SWE-agent 2.4.6 [8], the development model GPT-5.6 Sol, and the actor during development GPT-5.6 Luna; the 100 SWE-bench and 44 Terminal-Bench development tasks are sampled at random and disjoint from the evaluation samples. For an actor model or a task domain not used in development, implementations and contracts stay unchanged, and the scopes are adjusted on development tasks. Test-Time Composition. Before the first actor request, the composer 𝜋, which shares the actor’s backbone (GPT5.6 Luna at medium reasoning effort), reads the task, 𝑐𝑥 , the budget, a compact description of 𝐺seed , and the whole primitive catalog in one initial_single_pass prompt. It applies each primitive’s choose_only_if criteria literally and returns should_mutate, a rationale, and an intent. With should_mutate=false and intent=null, it keeps the seed graph when no other primitive fully qualifies; otherwise the composition intent 𝐼𝑥 names primitive instances with their parameters at a design level, without generating their implementation code or handling their integration. The deterministic compiler 𝐾 then builds the candidate graph (︀ )︀ (︀ )︀ ̃︀ 𝑥 = 𝐾 𝐺seed , 𝐼𝑥 ; 𝒫, 𝑐𝑥 , 𝐺 (52) 𝐼𝑥 = 𝜋 𝑥, 𝑐𝑥 ; 𝐺seed , 𝒫 , connecting the selected primitives to the seed graph and supplying the dependencies that their contracts require; selecting a repository map, for instance, also inserts the map into the actor’s request. The candidate is checked for supported primitives, compatibility with 𝑐𝑥 , valid connections, reachable termination, and bounded loops. The diagnostics of an invalid candidate return to the composer for bounded correction, the seed harness is retained if correction fails, and 𝐾 builds the valid graph into 𝐻𝑥 = 𝐹 (𝑥, 𝑐𝑥 ), as summarized in Algorithm 3. During the run, the structure of 𝐺𝑥 stays fixed while its conditional branches respond to observations, and selected verification and repair primitives return a failed check to the actor for repair. We record whether each selected primitive is activated and executed without runtime errors, and the benchmark scores 𝑅𝑥 (𝜏 ) after the run. Algorithm 3 Test-time harness composition in STITCH. The procedure and the composer prompt follow the companion study [51]. Require: task 𝑥, execution specification 𝑐𝑥 , budget, seed graph 𝐺seed , fixed library 𝒫 with application scopes and composition contracts 1: (should_mutate, rationale, 𝐼𝑥 ) ← 𝜋(𝑥, 𝑐𝑥 ; 𝐺seed , 𝒫) ◁ before the first actor request 2: if should_mutate is false then 3: return 𝐺seed ◁ intent=null retains the seed graph 4: end if ̃︀ 𝑥 ← 𝐾(𝐺seed , 𝐼𝑥 ; 𝒫, 𝑐𝑥 ) 5: 𝐺 ◁ connect primitives, supply contract dependencies 6: while the candidate fails the check for supported primitives, compatibility with 𝑐𝑥 , valid connections, reachable termination, or bounded loops do 7: if the correction bound is reached then 8: return 𝐺seed ◁ correction failed; the seed harness is retained 9: end if 10: send the diagnostics to 𝜋, which corrects 𝐼𝑥 ̃︀ 𝑥 ← 𝐾(𝐺seed , 𝐼𝑥 ; 𝒫, 𝑐𝑥 ) 11: 𝐺 12: end while ̃︀ 𝑥 13: return the harness 𝐻𝑥 = 𝐹 (𝑥, 𝑐𝑥 ) that 𝐾 builds from the valid graph 𝐺𝑥 = 𝐺 Comparison with Code Generation. STITCH’s analysis assigns zero utility to invalid harnesses: with 𝑉 (𝐻) ∈ {0, 1} indicating the absence of the modeled implementation failures and 𝑅(𝑥, 𝐻) the success probability given valid execution, a harness earns 𝐽(𝑥, 𝐻) = 𝑉 (𝐻)𝑅(𝑥, 𝐻). Proposition 3.2 restates STITCH’s Proposition 1 and part 1 of its Proposition 2, and part 2 compares composition with raw code generation. With Ω𝑖 the application 82

The Last Mile to the Real World

Framework details

∑︀𝑀 scope of 𝑝𝑖 and 𝜇𝑖 its task prevalence (Section 3.5), let 𝑘(𝑥) = 𝑖=1 1(𝑥 ∈ Ω𝑖 ) be the number of oracle-selected mechanisms, generated in one attempt without repair. If, conditional on 𝑥, each mechanism fails independently with probability 𝜖 ∈ (0, 1) and the harness is valid if and only if none of these failures occurs, then P(𝑉 (𝐻gen ) = 1 | 𝑥) = (1 − 𝜖)𝑘(𝑥) , while compositions of validated ∑︀𝑀 primitives by the deterministic compiler are assumed to execute validly, 𝑉 (𝐻comp ) = 1. Let 𝑘¯ = E𝑥 [𝑘(𝑥)] = 𝑖=1 𝜇𝑖 , assume 𝑘(𝑥) = 𝑘¯ for all tasks, although the selected mechanisms may differ across tasks, and suppose that valid generated harnesses attain the performance 𝑅(𝑥, a* (𝑥)) of the oracle activation 𝑎*𝑖 (𝑥) = 1(𝑥 ∈ Ω𝑖 ). If E[𝐽(𝑥, 𝐻comp )] > 0, composition outperforms this oracle generator if and only if (︀ )︀ ln E[𝑅(𝑥, a* (𝑥))] / E[𝐽(𝑥, 𝐻comp )] 1 E[𝑅(𝑥, a* (𝑥))] * ¯ 𝑘>𝑘 = ≈ ln , (53) − ln(1 − 𝜖) 𝜖 E[𝐽(𝑥, 𝐻comp )] where the approximation holds for small 𝜖. With 𝐴 = E𝑥 [𝑅(𝑥, a* (𝑥))] and 𝐵 = E[𝐽(𝑥, 𝐻comp )], the failure ¯ model gives E[𝐽(𝑥, 𝐻gen )] = E𝑥 [(1 − 𝜖)𝑘(𝑥) 𝑅(𝑥, a* (𝑥))] = (1 − 𝜖)𝑘 𝐴. Since 0 < 𝐵 ≤ 𝐴, the inequal¯ 𝑘 ity 𝐵 > (1 − 𝜖) 𝐴 holds if and only if 𝑘¯ > ln(𝐴/𝐵)/(− ln(1 − 𝜖)), and the approximation follows from − ln(1 − 𝜖) = 𝜖 + 𝑂(𝜖2 ). Part 2 thus characterizes a reliability trade-off under a constant mechanism count and independent implementation failures. Baselines and Settings. Each fixed-primitive baseline evaluates a fixed harness configuration that contains one designated focused control mechanism, its required dependencies, and shared supporting operations, which together form a minimal compatible set of operations handled by the compiler. The harness optimizers MemoHarness [144] and MetaHarness [143] start from the same mini-SWE-agent seed and use GPT-5.6 Sol as their development model. The actors not used in development are GPT-5.6 Terra with medium reasoning, DeepSeekV4-Flash without reasoning, and Claude-4.5-Haiku. Appendix C reports the transfer of primitives across task domains and the composer’s selection patterns: the composer keeps the seed harness for about half of the tasks and otherwise mostly selects a single primitive. Additional Results. On SWE-bench Verified with the three actors not used in development (Table 34), STITCH has the best average on all three metrics and the best scores with DeepSeek-V4-Flash and Claude-4.5-Haiku, while Codex leads with GPT-5.6 Terra, at 88.0% against 86.5% Pass@1; each fixed primitive exceeds the seed on the average. Table 34: Harness Primitives with actor models not used to develop the primitives, on SWE-bench Verified. Implementations and composition contracts are unchanged, while each application scope is adjusted according to the new actor’s execution on development tasks; Overall averages the three actors, and metrics are as in Table 20 [51]. Green marks the best and amber the second-best value in each column. Overall Method mini-SWE-agent (seed) SWE-agent Codex

GPT-5.6 Terra

Pass@1 Pass@2 Pass 75.2 73.3 78.5

Pass@1 Pass@2 Pass

Pass@1 Pass@2 Pass

Pass@1 Pass@2 Pass2

82.5 89.5 82.0

89.0 93.0 90.0

76.0 86.0 74.0

61.0 56.0 65.5

68.0 63.0 72.0

54.0 49.0 59.0

One focused Harness Primitive with its dependencies, fixed across tasks Change-surface Tracer 77.7 83.7 71.7 79.5 84.0 Compatibility-envelope Gate 77.3 83.3 71.3 79.5 85.0 Contract-case Explorer 78.7 83.7 73.7 80.0 83.0 Discriminating-oracle Runner 76.3 81.0 71.7 77.5 82.0 Domain-trace Template 78.7 85.0 72.3 81.5 86.0 Environment-capability Runner 79.5 84.3 74.7 83.5 86.0

75.0 74.0 77.0 73.0 77.0 81.0

87.5 85.5 88.5 86.5 84.5 89.0

92.0 90.0 93.0 90.0 91.0 93.0

83.0 81.0 84.0 83.0 78.0 85.0

66.0 67.0 67.5 65.0 70.0 66.0

75.0 75.0 75.0 71.0 78.0 74.0

57.0 59.0 60.0 59.0 62.0 58.0

Random composer STITCH

76.9 83.0

85.9 91.5

90.9 95.0

80.9 88.0

65.6 74.0

73.3 80.0

58.0 68.0

71.9 79.7

82.0 74.5 88.0

Claude-4.5-Haiku 2

80.0 66.0 84.0

82.8 88.3

70.0 67.0 72.3

DeepSeek-V4-Flash 2

84.0 83.0 92.0

77.4 84.0

80.3 79.7 84.7

2

80.6 86.5

84.4 90.0

As the test-time library grows along a single order of addition (Figure 15), Pass@1 rises from 73.0% for the seed harness (size 0) to 80.5% at size 8 on SWE-bench Verified and from 60.0% to 72.2% at size 6 on Terminal-Bench 2.0, ending above Codex on both benchmarks.

83

The Last Mile to the Real World

Framework details

Figure 15: Pass@1 of STITCH grows with the primitive library. SWE-bench Verified (left) and Terminal-Bench 2.0 (right) as the testtime library grows from the seed harness (size 0) to 8 and 6 entries, along STITCH’s single order of addition. The curve labelled Ours is STITCH. The dashed line marks Codex, and the dotted line marks the fixed-primitive arms at their mean Pass@1, whose best members reach 76.0% and 66.7% (Tables 20 and 21) [51].

The activation rate is 100% for each selected primitive (Figure 16), which supports part 2, the comparison with raw code generation: free of the reliability risk of generated harness code, the selected primitives take effect during the actor’s execution. Composition costs 4.3% of one actor attempt on SWE-bench Verified and 2.7% on TerminalBench 2.0, whereas the output-token lower bound for an oracle that writes the complete harness, with no input, reasoning, or debugging cost, is 23.35 and 17.23 actor attempts, about 543× and 638× the cost of composition.

Figure 16: Each selected Harness Primitive activates, and composition costs a few percent of one actor attempt. Left: selection and activation rates (%) per selected primitive, where activation means that the selected primitive was activated and executed without runtime errors. Right: token cost of composition (Ours) relative to one actor attempt on SWE-bench Verified and Terminal-Bench 2.0, against the output-token lower bound for an oracle that writes the complete harness with no input, reasoning, or debugging cost, on a logarithmic axis [51].

Relation to the Method. The compiled graph 𝐺𝑥 is the plan of Eq. (4): the composition intent 𝐼𝑥 names its primitive instances and their parameters, and the compiler adds the dependencies along which results flow; the shared execution state is the context of Eq. (5), and primitive development is STITCH’s form of the validation and publishing of Section 3.3 (Eq. (7)).

84

The Last Mile to the Real World

Additional results

C. Additional Experimental Results This appendix completes the experimental record of the eight primitive families, in the order of Section 4. Each subsection extends the family’s subsection there with complete benchmark breakdowns, retrieval and verification diagnostics, model allocations, provenance and budget controls, grounding audits, scaling trajectories, or selection patterns, and it names the mechanism of the family that its material bears on. Appendix B states each family’s full method in its own notation, and all values are copied from the eight companion studies, whose settings and evidence Appendix D records. The tables of this appendix follow the table convention of Section 4.1.

C.1. Tool Primitives: Breakdowns, Retrieval, and Validation Section 4.2 compares HEART with the reference systems of ToolBench, NESTFUL, and 𝜏 2 -Bench and removes one component at a time, a harness mechanism or ToolFace with its Tool Primitives (Tables 1, 2, 3, and 4); the controls at a matched backbone and tool catalogue and at a matched call budget accompany the framework in Appendix B.1 (Tables 25 and 26). This subsection adds the diagnostics of retrieval, validation, and revision. HEART is evaluated on five benchmarks [44]. ToolBench covers large-scale API invocation [6], NESTFUL nested call sequences, and 𝜏 2 -Bench interactive dual-control execution [31, 32]. ACEBench targets fine-grained multi-tool workflows, and BFCLv4 agentic web search and memory function calling [20, 58]. The ACEBench and BFCLv4 results, which Section 4.2 summarizes, appear in full in the companion study’s own tables. Three further suites are related benchmarks: StableToolBench, which replaces ToolBench’s live APIs with deterministic virtual servers [56], HammerBench [192], and ToolSandbox [33]. Each benchmark table follows its benchmark’s own protocol. The ToolBench reference rows include Vicuna [176], Alpaca [177], and GPT-4 [178]. The NESTFUL comparison also lists Granite’s function-calling model [179], Llama 3.1 [174], Mixtral [180], DeepSeek-V3 [181], and GPT4o [182]. Their configurations and scores follow the companion study. NESTFUL’s own metrics are F1 for function and parameter names, partial and full sequence accuracy, and win rate. The last three tables concern how HEART’s harness finds Tool Primitives and judges their results. Table 35 measures how often semantic retrieval over the 25,519 schema descriptors returns the gold tools. A plan step can invoke only a tool that retrieval returns, the containment behind the candidate-pool ceiling of Proposition 3.3(ii), and validationtriggered revision recovers part of the first-pass misses by reformulating the tool query. Table 36 compares result validation, which judges every step’s result on four criteria, with human labels. HEART audits validation because its recovery loop is conditioned on it: most false passes fall on the task-completion criterion for partially satisfied objectives, fewer than 3% of trajectories end on a false pass, false fails cost redundant revision rounds without changing outcomes, and the plateau at 𝐵 = 3 indicates that the loop terminates because validation converges (Appendix B.1). Table 37 varies HEART’s maximum revision budget 𝐵; 𝐵 = 0 disables revision. Table 35: Retrieval recall over the 25,519 tools of ToolFace. Recall@𝑘 (%) of semantic retrieval over schema descriptors, with each task’s gold tools as ground truth, measures candidate access separately from invocation success [44].

Table 36: Decision quality of HEART’s result validation. Result validation shares the Qwen3-8B backbone of the other roles; HEART’s study compares 200 sampled validation decisions from ToolBench and NESTFUL trajectories with human pass/fail labels [44].

Benchmark Recall@1 Recall@3 Recall@5 Recall@10 ToolBench NESTFUL

76.5 81.3

88.2 91.6

92.4 94.7

95.8 97.2

Metric Precision Recall Value

85

0.97

0.96

False-pass rate False-fail rate (%) (%) 8

4

The Last Mile to the Real World

Additional results

Table 37: HEART under different revision budgets. The maximum revision budget 𝐵 bounds the revision rounds after a failed validation; 𝐵 = 0 disables revision, and the shaded 𝐵 = 3 is HEART’s default; metrics as in Table 25 [44]. Green marks the best and amber the secondbest value in each column. ToolBench Revision budget 𝐵=0 𝐵=1 𝐵=2 𝐵=3 𝐵=5

NESTFUL

Pass Win Full Acc. Win 47.6 51.4 65.9 75.1 75.3

49.6 57.9 66.8 75.7 76.2

0.15 0.27 0.42 0.44 0.44

0.33 0.49 0.68 0.75 0.75

C.2. Code Primitives: Breakdowns, Controls, and Uncertainty Section 4.3 compares LEGO with construction paradigms, consumption and field controls, and model allocations (Tables 5, 6, 28, and 27). The tables below break delivery down by band and domain, restrict donor provenance, fix the attempt budget, give the uncertainty over decoding seeds and paired tasks, and report the independently developed agents by band with their cost. Table 38 breaks LEGO-REPO delivery down by difficulty band and domain. Across the five bands, the absolute gain over feedback stays between +0.161 and +0.209, so the larger relative gains on harder bands come mostly from a falling feedback score. Its coverage column measures offline reachability in the library. Table 39 restricts library eligibility step by step, from excluding same-repository donors to a library re-mined from a disjoint corpus. These are measured library variants, and the gain over feedback stays positive under every restriction. The restrictions are possible because every CodeFace entry is admitted with a recorded source repository, commit, and license (Section 3.3). Table 40 fixes the same integer attempt limit for both configurations; at every reported limit from 1 to 8, the library configuration stays above feedback alone [45]. Table 38: Feedback and LEGO by difficulty band and software domain. The two configurations of Table 5 partitioned two ways, with Δ their difference in mean delivery over each partition’s tasks; coverage is the offline module-level reachability of the library [45]. Δ

Coverage (%)

0.686 0.583 0.483 0.409 0.338

+0.161 +0.201 +0.209 +0.203 +0.209

75.3 71.2 65.8 61.2 56.7

0.2498 0.3390 0.3640 0.3195 0.3179 0.3711 0.3880

0.4368 0.4946 0.6104 0.5055 0.5213 0.5877 0.6139

+0.1870 +0.1556 +0.2464 +0.1860 +0.2034 +0.2166 +0.2259

56.3 66.5 68.5 60.4 58.3 62.4 52.1

0.3180

0.5134 +0.1954

61.4

Partition

Tasks Modules Feedback LEGO

D1 (mean 6 modules) D2 (mean 19 modules) D3 (mean 32 modules) D4 (mean 71 modules) D5 (mean 119 modules)

117 111 116 96 82

721 2,064 3,654 6,823 9,723

0.525 0.382 0.274 0.206 0.129

machine-learning databases-storage compilers-languages graphics-vision scientific-computing quant-finance crypto-security

158 149 131 32 26 14 12

9,649 5,537 4,166 1,420 1,322 465 426

all tasks

522

22,985

Table 41 evaluates four configurations over three decoding seeds and gives paired contrasts on seed 1, with bootstrap intervals over the 522 tasks. LEGO averages 0.5116 ± 0.0024 over the seeds, and its paired gain over feedback with diagnosis is +0.1665, with 95% interval [+0.1503, +0.1847] [45]. Table 42 gives the four independently developed repository agents of Section 4.3 by difficulty band, with their recorded cost per task. OpenHands and Claude Code cost $2.41 and $2.16 per task, against $2.08 for LEGO, and SWE-agent and Agentless cost $1.86 and $0.94. Every agent loses most of its delivery from D1 to D5, as execution feedback does, and LEGO leads all of them in every band [45].

86

The Last Mile to the Real World

Additional results

Table 39: Reuse under progressively stricter donor-provenance controls. Every row is a measured library variant, from donor filters applied per target to a library re-mined from a corpus disjoint from the benchmark repositories; coverage as in Table 38 [45]. Green marks the best and amber the second-best value in the All column.

Library eligibility

Entries

Coverage (%)

All

Δ vs Feedback

all entries (default) − same-repository donors − same-org donors − any fork or near-duplicate re-mined from disjoint corpus

1,424 1,424 1,424 1,365 518

61.4 41.2 38.6 33.1 44.7

0.5134 0.4743 0.4597 0.4455 0.4528

+0.1954 +0.1563 +0.1417 +0.1275 +0.1348

Table 40: Executed attempt budgets with and without the library. Both configurations use the same integer attempt limit 𝑏 per module on the same 522 tasks; 𝑏 = 5 is the default, 𝑏 = 1 rows are first-round prefixes, and $/task is recorded-call cost as in Table 27 [45]. Green marks the best and amber the second-best value in each All column. Feedback (---V) LEGO (RADV) Budget 𝑏 1 2 3 4 5 (default) 8

All

$/task

All

$/task

0.2172 0.2420 0.2785 0.3066 0.3180 0.3437

0.86 1.05 1.21 1.36 1.55 2.07

0.3954 0.4412 0.4663 0.5071 0.5134 0.5493

1.13 1.38 1.61 1.82 2.08 2.79

Δ

+0.1782 +0.1992 +0.1878 +0.2005 +0.1954 +0.2056

Table 41: Repeated-seed and paired uncertainty for repository construction. All configurations use the same 522 tasks; panel (a) gives means and standard deviations over three decoding seeds, and panel (b) gives paired contrasts on seed 1 with 95% bootstrap confidence intervals over tasks, Wilcoxon signed-rank 𝑝-values and Holm correction across the four contrasts [45]. Green marks the best and amber the second-best value in each column of panel (a). (a) Configuration means over decoding seeds Configuration

Seed 1

Seed 2

Seed 3

Mean ± s.d.

Feedback (---V) Feedback + Diagnosis (--DV) RAG + Edit (RE DV) LEGO (RADV)

0.3180 0.3469 0.4093 0.5134

0.3202 0.3434 0.4036 0.5125

0.3150 0.3493 0.4121 0.5089

0.3177 ± 0.0026 0.3465 ± 0.0030 0.4083 ± 0.0043 0.5116 ± 0.0024

𝑝

𝑝Holm

(b) Paired contrasts on seed 1 Δ

Contrast LEGO − Feedback LEGO − (Feedback + Diagnosis) LEGO − (RAG + Edit) LEGO − LEGO, re-mined disjoint corpus

95% CI

−40

[+0.1762, +0.2134] < 10 [+0.1503, +0.1847] < 10−30 [+0.0868, +0.1190] < 10−16 [+0.0472, +0.0768] 2.1 × 10−9

+0.1954 +0.1665 +0.1041 +0.0606

< 10−39 < 10−29 < 10−15 2.1 × 10−9

Table 42: Independently developed repository agents on all 522 tasks. Delivery score by difficulty band and over all tasks, the shares of tasks that reach the identity ceiling (Ceiling) and that score zero (Zero), and recorded-call cost per task (USD). Each agent runs under its own harness with the backbone of column two, with only the task set, environment, and grading held fixed, and SWE-agent and Agentless run through an issue-to-patch adapter [45]. Green marks the best and amber the second-best value in each column. Lower is better for Zero and $/task. System

Backbone

D1

D2

D3

D4

D5

All

Ceiling (%) Zero (%) $/task

OpenHands Claude Code SWE-agent Agentless

GPT-5.6 Terra Claude Sonnet 5 GPT-5.6 Terra GPT-5.6 Terra

0.556 0.543 0.482 0.421

0.401 0.394 0.345 0.276

0.269 0.271 0.238 0.181

0.198 0.190 0.152 0.098

0.129 0.104 0.081 0.043

0.3263 0.3170 0.2750 0.2181

9.4 9.6 6.7 5.4

41.6 39.8 46.2 55.4

2.41 2.16 1.86 0.94

Feedback LEGO

GPT-5.6 Terra GPT-5.6 Terra

0.525 0.382 0.274 0.206 0.129 0.3180 0.686 0.583 0.483 0.409 0.338 0.5134

8.8 22.6

41.0 18.8

1.55 2.08

C.3. Math Primitives: Per-Problem Method Ranks Section 4.4 compares self-advertisement with similarity retrieval and reranking (Table 7) and reports downstream proof completion (Table 8). Figure 17 shows where the coverage gains of Table 7 come from [46]. Each dot is one problem, placed by the rank of its highest-ranked annotated method under self-advertisement and by that method’s rank under one similarity retriever. Dots in the shaded region are methods that similarity ranks outside its top five but self-advertisement places inside. Many of them lie outside the top ten of their panel’s retriever. In the three

87

The Last Mile to the Real World

Additional results

embedding panels, this places them outside the pool of the reranker built on that retriever; BM25 has no reranker. This is the candidate-pool ceiling of Proposition 3.3(ii) at work: no reranking backbone can recover a method that its pool omits, whereas self-advertisement judges every contract from its proposal.

Figure 17: Per-problem rank of the highest-ranked annotated method. Each dot is one Putnam or IMO ProofBench problem, placed vertically by the rank of its highest-ranked annotated method under self-advertisement with GPT-5.6 Luna, shared across panels, and horizontally by that method’s rank under the panel’s similarity retriever (BM25, OpenAI-small, OpenAI-large, or Qwen3-Emb0.6B); dashed lines mark the top five, and the shaded region holds annotated methods that similarity ranks outside its top five but self-advertisement places inside [46].

C.4. Knowledge Primitives: Grounding, Preservation, and Transfer Table 43: Grounding audit of the stored source-model units. Sampled units (%) of the initial 𝑀0 , self-directed 𝑀𝑆 and task-guided 𝑀𝑇 source models of one document, one code and one API source, judged against their cited source elements: supported if all claims are entailed, partial if only the main claim is, unsupported if the evidence does not establish the claim, and contradicted if it states otherwise [47].

Table 44: Meaning preservation under source-model reconstruction. Units removed by a learning stage are compared with the new model on the same entity, a claim counting as preserved if it remains represented after rewording, merging or condensation (%); 𝑀0 → 𝑀𝑆 is self-directed and 𝑀𝑆 → 𝑀𝑇 task-guided learning [47].

Model Supported Partial Unsupported Contradicted

Dropped by Preserved Lost Contradicted

MultiDoc2Dial (StudentAid) 𝑀0 79.2 19.2 𝑀𝑆 75.0 20.8 𝑀𝑇 63.3 22.5

1.7 4.2 13.3

0.0 0.0 0.8

MultiDoc2Dial (StudentAid) 𝑀0 → 𝑀𝑆 98.3 1.7 𝑀𝑆 → 𝑀𝑇 98.3 0.0

0.0 1.7

SWE-QA (pytest) 𝑀0 89.2 𝑀𝑆 86.7 𝑀𝑇 65.0

10.8 13.3 31.7

0.0 0.0 3.3

0.0 0.0 0.0

SWE-QA (pytest) 𝑀0 → 𝑀𝑆 93.3 𝑀𝑆 → 𝑀𝑇 100.0

5.0 0.0

1.7 0.0

APIBench (HuggingFace) 𝑀0 65.0 32.5 𝑀𝑆 76.7 18.3 𝑀𝑇 84.2 15.8

APIBench (HuggingFace) 𝑀0 → 𝑀𝑆 96.7 3.3 𝑀𝑆 → 𝑀𝑇 100.0 0.0

0.0 0.0

1.7 5.0 0.0

0.8 0.0 0.0

Section 4.5 compares SourceLearn with retrieval and memory baselines on five benchmarks and at a matched context budget (Tables 9 and 29). The material below audits the learned source models, splits the gains by region, and reduces the evidence. Tables 43 and 44 are SourceLearn’s post-hoc audits of grounding and source authority, the principle that learning signals determine what is reconsidered while persistent updates are reconstructed from the authoritative source (Eq. (28)) [47]. The first judges sampled units of 𝑀0 , 𝑀𝑆 , and 𝑀𝑇 against their cited source elements, and the second checks whether units removed by learning remain represented in the revised model. 88

The Last Mile to the Real World

Additional results

Table 45: Source-model gains inside and outside the regions touched by guidance tasks. Accuracy (%) with GPT-5.6 Luna, where seen regions overlap at least one guidance task and unseen regions none, with the change from Hybrid RAG in the same stratum in pp in parentheses [47]. Green marks the best and amber the second-best value in each row within each stratum. 𝑀0

Hybrid RAG Benchmark

Seen Unseen

Seen

𝑀𝑆 Unseen

Seen

𝑀𝑇 (SourceLearn) Unseen

Seen

Unseen

MultiDoc2Dial 64.2 SWE-QA 57.3 NarrativeQA 65.8

65.3 56.4 66.2

69.0 (+4.8) 72.3 (+7.0) 77.4 (+13.2) 72.0 (+6.7) 78.9 (+14.7) 76.7 (+11.4) 70.3 (+13.0) 61.1 (+4.7) 66.2 (+8.9) 64.6 (+8.2) 72.9 (+15.6) 66.3 (+9.9) 74.2 (+8.4) 74.0 (+7.8) 77.2 (+11.4) 78.4 (+12.2) 80.1 (+14.3) 79.9 (+13.7)

All QA

62.9

71.6 (+9.2) 69.6 (+6.7) 73.3 (+10.9) 72.2 (+9.3) 77.3 (+14.9) 75.0 (+12.1)

62.4

Figure 18: Missing-evidence robustness. Accuracy (%) of Hybrid RAG and SourceLearn on MultiDoc2Dial with GPT-5.6 Luna as fewer source elements of the Hybrid RAG evidence 𝑅(𝐷, 𝑞) are retrieved (a) and with the gold document removed (b) [47].

Table 45 splits test questions by whether their source regions overlapped the guidance tasks, for GPT-5.6 Luna. The gain over Hybrid RAG on unseen regions shows that the benefit of the learned source model extends beyond the regions that the guidance tasks touched. Figure 18 reduces the retrieved evidence or removes the gold document on MultiDoc2Dial. SourceLearn degrades more slowly than Hybrid RAG, so the source model supplies source understanding when task-local retrieval is incomplete. Figure 19 reports what the source model holds at each learning stage and the share of units with an explicit condition. It also reports how much of the test questions’ required content is represented, and the association between represented content and accuracy (Appendix D).

89

The Last Mile to the Real World

Additional results

Figure 19: Representation changes through source learning. Share of units by role group (a) and with an explicit condition (b) in 𝑀0 , 𝑀𝑆 , and 𝑀𝑇 , coverage of the claims that test questions require at each stage (c), and accuracy by the coverage of a question’s required claims (d) [47].

C.5. Agent Primitives: Structure Selection and Cost Section 4.6 compares the primitive-based MAS with a single agent, text-based and latent MAS, and each single primitive applied to every query, and it removes positional re-encoding (Tables 10 and 30). The eight benchmarks of Table 10 combine mathematical problem solving in AIME, MATH, and GSM8K [150–153]; executable code evaluation through HumanEval+ and MBPP+, the stricter test suites built on HumanEval and MBPP [154, 183,

90

The Last Mile to the Real World

Additional results

184]; and medical and graduate-level question answering [155, 156]. The benchmark mean therefore aggregates heterogeneous evaluation procedures, and the individual columns report each of them. Table 46: The LLM Organizer against random structure selection. Accuracy (%) of the composed MAS when the GPT-5.2 Organizer is replaced by Claude-4 or by random selection of MAS structures from the Knowledge Pool (R1/Llama-70B is DeepSeek-R1-Distill-Llama70B), with the change from GPT-5.2 in pp in parentheses [48]. Model

Organizer AIME25 HumanEval+

MedQA

Qwen3-8B

GPT-5.2 Claude-4 Random

73.3 73.3 (0.0) 66.7 (−6.6)

82.3 81.1 (−1.2) 77.4 (−4.9)

76.7 76.2 (−0.5) 70.6 (−6.1)

GPT-5.2 R1/Llama-70B Claude-4 Random

56.7 56.7 (0.0) 50.0 (−6.7)

85.3 83.5 (−1.8) 78.6 (−6.7)

81.9 81.9 (0.0) 69.5 (−12.4)

Table 47: Removing the Organizer’s Knowledge Pool. Accuracy (%) of the composed MAS with (w/) and without (w/o) the Knowledge Pool, whose 45 entries pair previously observed queries with effective MAS structures and guide the Organizer’s composition, models as in Table 46, with the change from w/ in pp in parentheses [48]. Model

Pool

AIME25

HumanEval+

MedQA

Qwen3-8B

w/ w/o

73.3 63.3 (−10.0)

82.3 77.4 (−4.9)

76.7 71.6 (−5.1)

R1/Llama-70B

w/ w/o

56.7 50.0 (−6.7)

85.3 73.4 (−11.9)

81.9 75.9 (−6.0)

Table 48: Accuracy and estimated inference cost of the methods of Table 11. Cost is the Agent Primitives study’s estimated USD per query from market output-token prices, accuracy (Acc.) is in %, efficiency (Eff.) is accuracy per $0.01, and the two composed-MAS rows differ in the Organizer model [48]. MATH Method Single CoT Self-Consistency LLM-Debate Self-Refine Quality-Diversity SPP AgentVerse GPTSwarm DyLAN MAS-GPT

GSM8K Eff.

Cost

Acc.

HumanEval+ Eff.

Cost

Acc.

GPQA

Cost

Acc.

Eff.

Cost

Acc. Eff.

0.0024 0.0033 0.0204 0.0081 0.0055 0.0132 0.0052 0.0080 0.0074 0.0085 0.0082

50.6 210.8 0.0016 92.4 573.9 0.0037 75.8 204.9 0.0117 36.7 31.4 53.2 161.2 0.0025 92.8 369.4 0.0041 77.0 186.6 0.0130 35.3 27.2 61.6 30.2 0.0153 95.0 62.1 0.0352 75.8 21.5 0.0729 37.2 5.1 61.4 75.8 0.0101 91.6 90.9 0.0151 74.5 49.3 0.0250 34.4 13.8 58.5 106.1 0.0038 90.8 240.5 0.0086 62.7 73.1 0.0268 38.3 14.3 60.5 45.9 0.0090 93.0 103.5 0.0205 70.2 34.2 0.0352 33.6 9.5 51.7 99.8 0.0034 92.8 273.7 0.0067 73.3 109.3 0.0198 35.1 17.7 55.6 69.6 0.0061 93.4 152.3 0.0161 73.9 45.9 0.0248 40.2 16.2 55.4 74.9 0.0061 93.2 153.6 0.0152 73.9 48.6 0.0232 36.5 15.7 59.6 70.1 0.0062 91.2 147.1 0.0165 75.8 45.9 0.0283 36.0 12.7 68.7 83.5 0.0065 93.4 143.7 0.0174 78.9 45.4 0.0257 37.6 14.6

Primitives-based MAS, GPT-5.2 Organizer 0.0056 72.4 129.3 0.0048 93.8 195.4 0.0118 82.3 69.7 0.0171 53.2 31.1 Primitives-based MAS, Llama-3-70B Organizer 0.0028 72.4 258.6 0.0020 93.8 469.0 0.0063 82.3 130.6 0.0122 53.2 43.6

Table 49: Output tokens per query of the methods of Table 11. Counts with Llama-3-70B-Instruct workers cover the whole pipeline, including the Organizer’s 200–400 output tokens per query; Mean averages the four benchmarks, and the two composed-MAS rows differ in the Organizer model [48]. Green marks the best and amber the second-best value in each column. Lower is better for every column. Method

MATH GSM8K HumanEval+ GPQA Mean

Single CoT Self-Consistency LLM-Debate Self-Refine Quality-Diversity SPP AgentVerse GPTSwarm DyLAN MAS-GPT

1,375 1,874 11,673 4,612 3,162 7,526 2,957 4,572 4,216 4,873 4,714

934 1,447 8,742 5,747 2,149 5,137 1,934 3,504 3,473 3,541 3,706

2,129 2,371 20,114 8,635 4,891 11,710 3,832 9,216 8,698 9,427 9,916

6,674 2,778 7,408 3,275 41,675 20,551 14,278 8,318 15,350 6,388 20,127 11,125 11,345 5,017 14,165 7,864 13,267 7,414 16,188 8,507 14,672 8,252

Primitives-based MAS, GPT-5.2 Organizer Primitives-based MAS, Llama-3-70B Organizer

1,524 1,609

1,017 1,134

3,779 3,618

6,882 6,974

91

3,301 3,334

The Last Mile to the Real World

Additional results

Tables 46 and 47 hold the Organizer ablations [48]. They replace the GPT-5.2 Organizer by Claude-4 [175] or by random selection of structures from the Knowledge Pool, and they remove the Knowledge Pool. They thus vary the model and the information behind the Organizer’s choice of primitives, down to a random choice, and the study concludes that the main benefit comes from problem-aware structure selection rather than from a specific Organizer model. Table 11 compares the primitive-based MAS with eight independently designed MAS methods and two single-model prompting baselines on Llama-3-70B-Instruct [174], separately from the backbones of Table 10. The prompting baselines are Chain-of-Thought [165] and Self-Consistency [166]; the MAS methods include SelfRefine [140], a Quality-Diversity comparator [185], and SPP [186]. Table 48 adds the cost accounting for the same methods. Its row for a Llama-3-70B Organizer repeats the GPT-5.2 accuracies, an inconsistency of the Agent Primitives study that Appendix D records. Table 49 gives the output tokens behind these costs. With either Organizer, the primitive-based MAS uses fewer tokens than every MAS method on each benchmark: 3,301 per query on average with the GPT-5.2 Organizer, against 8,252 for MAS-GPT, 2.5× fewer.

C.6. Dev-Primitives: Model Allocation and On-Demand Activation Table 50: Assigning models separately to activation, the Dev-Primitives and diagnosis. The lower blocks keep the Dev-Primitives on Qwen3-8B and scale activation, diagnosis or both, with the change from the homogeneous Qwen3-8B configuration in pp in parentheses [49]. SWE-bench Activation

Dev-Primitives Diagnosis

SWE Refactor Terminal-Bench DevOps-Gym

Resolved (%) Composite (%)

Resolved (%)

Avg. (%)

Homogeneous backbones Qwen3-8B Qwen3-8B GPT-5.6 Luna GPT-5.6 Luna DeepSeek-V4 DeepSeek-V4 GPT-5.6 Sol GPT-5.6 Sol

Qwen3-8B GPT-5.6 Luna DeepSeek-V4 GPT-5.6 Sol

80.6 95.6 82.8 97.0

10.5 16.0 15.0 31.0

27.6 29.1 39.4 51.8

37.48 46.64 53.08 55.41

Diagnosis scaling Qwen3-8B Qwen3-8B Qwen3-8B Qwen3-8B Qwen3-8B Qwen3-8B Qwen3-8B Qwen3-8B

DeepSeek-V4 GPT-5.5 Claude Sonnet 5 GPT-5.6 Sol

84.8 (+4.2) 86.4 (+5.8) 86.0 (+5.4) 87.2 (+6.6)

14.5 (+4.0) 16.5 (+6.0) 16.0 (+5.5) 17.5 (+7.0)

44.5 (+16.9) 46.1 (+18.5) 45.5 (+17.9) 47.9 (+20.3)

41.62 (+4.14) 43.25 (+5.77) 43.01 (+5.53) 44.36 (+6.88)

Activation scaling GPT-5.6 Luna Qwen3-8B DeepSeek-V4 Qwen3-8B GPT-5.5 Qwen3-8B Claude Sonnet 5 Qwen3-8B GPT-5.6 Sol Qwen3-8B

Qwen3-8B Qwen3-8B Qwen3-8B Qwen3-8B Qwen3-8B

82.4 (+1.8) 83.6 (+3.0) 84.4 (+3.8) 84.0 (+3.4) 85.2 (+4.6)

12.0 (+1.5) 13.0 (+2.5) 13.5 (+3.0) 13.0 (+2.5) 14.0 (+3.5)

32.7 (+5.1) 34.8 (+7.2) 36.7 (+9.1) 36.1 (+8.5) 38.2 (+10.6)

39.82 (+2.34) 40.77 (+3.29) 41.58 (+4.10) 41.20 (+3.72) 42.31 (+4.83)

Activation + diagnosis scaling GPT-5.6 Luna Qwen3-8B GPT-5.5 Qwen3-8B Claude Sonnet 5 Qwen3-8B GPT-5.6 Sol Qwen3-8B GPT-5.6 Sol Qwen3-8B

GPT-5.5 GPT-5.5 Claude Sonnet 5 GPT-5.5 GPT-5.6 Sol

89.4 (+8.8) 91.2 (+10.6) 92.4 (+11.8) 93.6 (+13.0) 94.2 (+13.6)

20.0 (+9.5) 22.5 (+12.0) 23.5 (+13.0) 25.0 (+14.5) 26.5 (+16.0)

47.6 (+20.0) 49.1 (+21.5) 48.8 (+21.2) 49.7 (+22.1) 50.6 (+23.0)

46.02 (+8.54) 48.73 (+11.25) 49.66 (+12.18) 51.44 (+13.96) 52.30 (+14.82)

Section 4.7 compares HERMES with same-backbone harnesses, with a single editor over the same components, and with variants that remove one mechanism at a time (Tables 12, 13, 32, 15, and 14). Table 50 assigns models independently to activation, the Dev-Primitives, and diagnosis; its values in parentheses are improvements, in pp, over the homogeneous Qwen3-8B configuration [49]. With Qwen3-8B Dev-Primitives, a stronger diagnosis model alone raises Terminal-Bench resolution from 27.6% to between 44.5% and 47.9%. A stronger activation model alone reaches at most 38.2%. Table 51 adds the Terminal-Bench 4.0 inference cost of five of these configurations. Table 52 counts primitive calls with and without on-demand activation on all four benchmarks, which shows how much instantiating only the activated Dev-Primitives 𝒫𝒜 saves within one task (Section 3.3). Its ablation keeps every component found by the initial analysis active for the whole trajectory. Repeated calls of one primitive after messages or revisions make the call counts exceed the number of distinct active components. 92

The Last Mile to the Real World

Additional results

Table 51: Cost and resolution of mixed model assignments on Terminal-Bench 4.0. Configurations name the activation / DevPrimitive / diagnosis models, cost is the total inference expenditure of the evaluation, and the last row is HERMES with GPT-5.6 Sol in every role [49]. Configuration

Table 52: Primitive calls with and without on-demand activation. Average calls per task, including repeated calls for communication and revision; without on-demand activation all Dev-Primitives identified in the initial analysis stay active throughout, and cost is relative to HERMES [49].

Resolved (%) Cost ($k)

Qwen3-8B / Qwen3-8B / Qwen3-8B Qwen3-8B / Qwen3-8B / GPT-5.5 GPT-5.6 Sol / Qwen3-8B / GPT-5.5 GPT-5.6 Sol / Qwen3-8B / GPT-5.6 Sol GPT-5.6 Sol / GPT-5.6 Sol / GPT-5.6 Sol

27.6 46.1 49.7 50.6 51.8

Avg. Primitive Calls Benchmark

0.70 1.56 2.13 2.51 3.40

Relative Cost

HERMES w/o Activation w/o Activation

SWE-bench Verified SWE Refactor Bench Terminal-Bench 4.0 DevOps-Gym

9.6 25.1 14.2 11.7

28.7 64.3 39.6 33.2

1.61× 1.88× 1.73× 1.67×

C.7. Navigation Primitives: Scaling and Activity Diagnostics Table 53: Per-length results of the controlled information-space scaling. Active coordination is the mean number of participating workers or members per query, and calls and cost are per query [50]. Green marks the best and amber the second-best value in each column within each length. Lower is better for active coordination, calls, and cost. Method

Active Calls Cost Length Coord. / Query / Query ($)

EM

F1

0.0983 0.1797 0.3549 1.3974

0.533 0.533 0.500 0.567

0.731 0.737 0.670 0.754

7.10 12.00 22.00 80.23

0.0809 0.1613 0.3216 1.3027

0.567 0.567 0.400 0.500

0.750 0.734 0.564 0.746

27.47 29.53 28.30 33.10

0.0773 0.1069 0.1276 0.5332

0.600 0.667 0.600 0.667

0.775 0.784 0.768 0.840

Multi-agent baselines 32K 19.00 64K 37.00 LongAgent 128K 73.10 512K 290.50

22.83 39.50 75.60 292.50

32K 64K 128K 512K

5.10 10.00 20.00 78.20

32K 64K ANTMAN 128K 512K

2.57 2.87 3.07 3.17

CoA

Table 54: Relevant and irrelevant worker activity as required evidence grows. ANTMAN’s trajectory-level worker activity at a fixed 512K space, split by whether it contributes to acquiring required evidence; the split attributes the dip in active coordination from 1 to 4 units (Table 18) to exploration overhead, and at 16 units ANTMAN’s study reports 8.2 worker activities against 8.0 actively coordinating workers [50]. Required Worker Relevant Irrelevant Evidence (units) Activity Activity Activity ANTMAN 1 4 16

5.0 4.3 8.2

0.9 3.2 8.0

4.1 1.1 0.2

Section 4.8 reports ANTMAN in structured information spaces, under a growing space and a growing demand, and with its need tracking, rerouting, or recovery removed (Tables 16, 17, 18, and 19). The structured-navigation comparison in Table 16 includes RepoDistill [168]. Table 53 gives every space size and baseline row of the controlled intervention from 32K to 512K, which examines the behaviour that Proposition 3.1 predicts at approximately fixed demand [50]. It reports active coordination, calls and cost per query, and answer quality (EM and F1) for each space size. Table 54 decomposes trajectory-level worker activity under growing demand, measured by required evidence in place of 𝐻(𝑞), into activity that contributes required evidence and activity that does not. This decomposition explains why active coordination dips from 5.0 to 4.3 workers between one and four required units: at one unit, 93

The Last Mile to the Real World

Additional results

most activity is exploration that finds no required evidence. Worker activity is a trajectory-level count and differs from the number of distinct workers in Table 18.

C.8. Harness Primitives: Catalogs, Transfer, and Selection Section 4.9 compares STITCH’s task-conditioned composition with the seed harness and with fixed and random compositions, and it repeats the comparison with actors not used in development (Tables 20, 21, and 34). MemoHarness [144] is a harness-optimization comparator in Tables 20 and 21, evaluated with the same mini-SWE-agent seed as STITCH [51]. The test-time libraries hold eight and six entries (Appendix D). Figure 20 evaluates each benchmark with only the primitives developed on the other one. Implementations and contracts are kept, and scopes are adjusted on development tasks. With these out-of-domain primitives, STITCH’s Pass@1 stays below its in-domain result but above the seed on both benchmarks. Its Pass@2 and Pass2 also stay above the seed, and on Terminal-Bench 2.0 its Pass@2 exceeds the in-domain run. Figure 21 shows how many primitives the composer selects per task and which combinations it selects. The composer keeps the seed graph for about half of the tasks and otherwise mostly selects one primitive. This sparse selection complements the selection and activation rates of Figure 16.

Performance (%)

80

78.0 73.0 68.0

80.0 74.5 69.0

Pass@2 84.0 77.5 71.0

Pass² 85.0 80.5 76.0

60 40 20 0

Pass@1

100

Performance (%)

Pass@1

100

80

Pass@2

75.6 68.9

60

62.2

60.0 51.1

Pass²

80.0 71.1 62.2

77.8 72.2 66.7

48.9

40 20

Seed harness (Mini-SWE-agent)

Best fixed primitive

0

STITCH STITCH (OOD primitives) (ID primitives)

Seed harness (Mini-SWE-agent)

Best fixed primitive

STITCH STITCH (OOD primitives) (ID primitives)

Figure 20: Harness Primitive transfer across task domains. Left, SWE-bench Verified, whose out-of-domain primitives were developed on Terminal-Bench 2.0; right, Terminal-Bench 2.0, whose out-of-domain primitives were developed on SWE-bench Verified. Implementations and composition contracts are retained while scopes are adjusted using development tasks; in-domain and out-of-domain libraries are compared with the seed and with the best fixed primitive that STITCH reports for this out-of-domain evaluation [51].

94

The Last Mile to the Real World SWE-bench

Study settings Change-surface Domain-trace Compatibility- Contract-case DiscriminatingTracer Explorer Template envelope Gate oracle Runner

Terminal-Bench

Tasks

60

3

Tasks (%)

51 23

22

2

39

40

1 1

20

1

7

0

2

1

0

1

2

3

1

4

1

Number of Selected Primitives

Figure 21: Task-specific Harness Primitive selection patterns. Share of tasks by the number of primitives that STITCH’s composer selects (left; bar labels give task counts) and the multi-primitive combinations it selects, with their task counts (right), complementing the activation frequencies in Figure 16 [51].

D. Study Settings and Evidence Ledger The quantitative record of this paper is the eight companion manuscripts. For each study, this appendix records its primitive in the study’s own terms, its evaluated settings, the evidence that Sections 4 and 5 use, and the scope of that evidence. Each subsection carries the identifier (S01–S08) of the companion manuscript’s source snapshot from which its values are copied, the same identifier that the bibliography uses; because the identifiers name manuscripts, they do not follow the family order. The subsections follow the family order of Section 4.

D.1. S02: Tool Primitives Primitive. A Tool Primitive is an LLM-based wrapper that encapsulates one ToolFace schema–function pair (𝑠𝑖 , 𝑓𝑖 ), 𝒫𝑖 (𝑥; 𝑐) = ℳ([𝑠𝑖 ; 𝑐; 𝑥]), and all Tool Primitives share one Qwen3-8B backbone [44]. HEART’s harness has three mechanisms: request decomposition plans the task, value resolution grounds each step’s values, and result validation judges every step’s result. On a failed validation, the structured feedback enriches the context, and targeted revision re-plans for at most 𝐵 = 3 rounds. ToolFace contains 25,519 tools, built by collecting tools and manually authoring their schemas, and the set of Tool Primitives is induced by ToolFace and grows with it. Controls. The matched-backbone table fixes Qwen3-8B and ToolFace (Table 25). ToolBench pass rates for ReAct, retrieval-only, planner–executor, DFSDT, and HEART are 33.5, 41.0, 52.6, 55.8, and 75.1%, and their NESTFUL full accuracies are 0.08, 0.10, 0.19, 0.13, and 0.44. The budget-matched self-refinement control without Tool Primitives reaches 34.2% and 0.15 (Table 26). Removing result validation gives 47.6% and 0.15 in the component ablation (Table 4). Scope. The matched-backbone comparison holds Qwen3-8B and ToolFace fixed to isolate the harness (Section 4.2), so it separates HEART from its backbone and catalogue. In the 𝜏 2 -Bench airline domain, where the reference systems run on their own backbones, HEART succeeds in all four trials on 0.63 of the tasks against 0.50 for Claude-4.6Sonnet (Table 3). The primitive-specific controls are the budget-matched control and the removal of ToolFace and the Tool Primitives, which leaves 16.1%. The audit of result validation samples 200 decisions from ToolBench and NESTFUL trajectories (Table 36). HEART’s robustness study injects a malicious tool document into the retrieved candidates to manipulate tool selection, following ToolHijacker [190], and reports a 0.0% attack-success rate on ToolBench, because tool schemas stay in each primitive’s own prompt and never enter the caller’s [44].

D.2. S04: Code Primitives Primitive. A Code Primitive is a reusable executable software component equipped with a resident LLM, 𝑃𝑖 = (𝐶𝑖 , 𝐼𝑖 , 𝐷𝑖 , 𝑉𝑖 , 𝑋𝑖 , 𝑀𝑖 ): the implementation, the exposed interface and behavioural contract, the dependency clo95

The Last Mile to the Real World

Study settings

sure, the carried validation tests, supporting context and provenance, and the resident LLM [45]. One adaptation model instantiates every 𝑀𝑖 . The resident LLM assesses the component’s relevance to a requirement and, once the primitive is selected, adapts the component and reruns 𝑉𝑖 on it. CodeFace contains 1,424 entries, 871 mined, 343 harvested from verified construction runs, and 210 web-sourced, each admitted with executable closure, a passing component suite, and a recorded source repository, commit, and license. Benchmark. LEGO-REPO contains 522 frozen reconstruction tasks from installable Python repositories, spanning seven domains, 22 capability tracks, and five difficulty bands. The benchmark removes an implementation while retaining its task interfaces, tests, and environment. Native passing-test counts are normalized between a floor and the original implementation’s ceiling and clipped to [0, 1], so a delivery score is not binary task success. A score counts only for an admissible run, in which execution completes, provenance confirms that the tests exercised the generated code, and the span between floor and ceiling is at least five tests; any other run scores zero (Eq. (11)). Controls and Costs. Feedback, feedback plus diagnosis, context-only reuse, unchanged import-and-call, structuredschema adaptation, and LEGO score 0.3180, 0.3469, 0.3544, 0.3430, 0.4974, and 0.5134 (Tables 5 and 6). With GPT-5.6 Terra in every model position, LEGO costs $2.08 per task; with GPT-OSS-20B in the adaptation and diagnosis positions, it costs $1.58 and scores 0.4880 (Table 27). The DeepSeek-V3.2 row of Table 27 is one configuration of the study’s sweep over thirteen backbones, in which adaptation and diagnosis stay on GPT-5.6 Terra. Scope. Frozen native tests supply the acceptance specification. The field controls of Table 28 add fields cumulatively in one order, so their gains are increments in that order. In a model allocation, only the adaptation position belongs to the primitive. The study reports source-repository exclusion and corpus-disjoint controls.

D.3. S05: Math Primitives Primitive. The library holds 82 Method Contracts, one per reusable proof method extracted from Putnam 2000– 2014 solutions, and each pairs a selection side, which the selector reads, with an execution side, which goes only to the prover [46]. The execution side holds a Lean scaffold stated for arbitrary parameters, a compiled worked instance, and Mathlib anchors. It also states the contract’s required conditions Pre𝑐 as Lean hypotheses and the residual obligations Obl𝑐 left after the method’s step. Lean provides the proof language and kernel [10], and Mathlib provides the reusable mathematical library [11]. GPT-5.6 Sol extracts the methods and proposes merges, all of which the authors review, and then drafts both sides of each contract. Every execution-side Lean artifact is compiled in Lean 4.27.0 with Mathlib commit a3a10db, the third of the three construction steps, and all 82 contracts pass. One batched call of a shared backbone writes, for each contract, a target, an action, required facts, cited verified material, an offer flag, a role, and a score. Residual obligations follow from the contract and its binding and are not proposal fields. Evaluation. Method selection uses 120 Putnam 2015–2025 problems and 60 IMO ProofBench problems. Reference methods are annotated from official solutions with model assistance and human checking, and fixed before selector evaluation. Each reranker reorders the top 10 contracts of an embedding ranking with the same backbone. For GPT5.6 Luna, self-advertisement reaches Putnam hit@5 of 0.950, against 0.842 for Qwen3 with reranking, whose hit@1 is 0.683 (Table 7). OpenAI-large with reranking has the best hit@1, 0.692, at hit@5 0.825, and self-advertisement’s hit@1 is 0.658. IMO ProofBench hit@5 is 0.883 for these rerankers and 0.917 for self-advertisement. Adding the top-ranked contract to the prompt of AxProverBase under ten Lean compilations changes its proof rates from 5.8 to 12.5% on Putnam and from 10.0 to 15.0% on IMO ProofBench (Table 8); pass@20 receives twice that budget. Scope. Hits are scored against the annotated reference methods. A checked scaffold leaves its residual obligations to the prover, which completes 12.5% of Putnam and 15.0% of IMO ProofBench proofs with the contract within ten compilations (Table 8). The downstream comparison adds contract content as well as selection. The coverage gain belongs to self-advertisement as a whole, commitment gate included. The study traces much of it to annotated

96

The Last Mile to the Real World

Study settings

methods that similarity ranks outside its top five (Figure 17). Ranking improvements depend on the criterion, and an additional runoff raises some metrics and lowers others.

D.4. S06: Knowledge Primitives Primitive. A Knowledge Primitive is SourceLearn’s persistent, revisable source model 𝑀 of one authoritative source 𝐷, organized around source entities, whose representations capture properties, mechanisms, governing conditions, procedures, exceptions, distinctions, and relations to other entities, and whose units cite the source elements they rest on [47]. At task time, source-model activation 𝐴𝐵 (𝑀, 𝑞) supplies the whole model, or its task-relevant regions within 𝐵 = 24k tokens, to the task solver beside the Hybrid RAG evidence 𝑅(𝐷, 𝑞). The model changes only in the offline learning stages before test tasks, and every update after construction follows grounded source reconstruction: every proposed unit passes the grounding gate, which requires its cited source excerpts to entail it, and a region rewrite is committed only if the preservation check finds no previously supported meaning lost, unless a stronger statement subsumes it. Evaluation. Five benchmarks are MultiDoc2Dial, NarrativeQA, SWE-QA, APIBench, and AppWorld, and three backends yield fifteen settings. All methods share the backend, the embedding model, and the answer or agent prompt, including the AppWorld scaffold. SourceLearn and AWM receive the same Hybrid RAG evidence, whereas RAPTOR and HippoRAG 2 use their own retrieval. Guidance and evaluation use fixed disjoint splits, with AppWorld using its official train/test-normal partition. SourceLearn improves over Hybrid RAG in fourteen settings and leads all evaluated methods in thirteen (Table 9). The one setting without improvement is GPT-OSS-120B on AppWorld, from 30.4 to 27.4. On GPT-OSS-120B SWE-QA, SourceLearn improves on Hybrid RAG, but HippoRAG 2 leads with 56.7 against 53.4. Scope. All QA results use one LLM judge, GPT-5.6 Luna, which is also an evaluated backend. Grounding measures support by the cited source. Learned units enter only through grounded source reconstruction, whose grounding gate is itself a model judgment. In the grounding audit, contradicted units stay at or below 1%, but some Task-Guided units on the document and code sources are only partially supported or unsupported: full support of StudentAid units falls from 79.2% in 𝑀0 to 75.0% in 𝑀𝑆 and 63.3% in 𝑀𝑇 (Table 43). The learned model is therefore not a replacement for the source, which stays authoritative and directly accessible through 𝑅(𝐷, 𝑞). SourceLearn also receives more source context than Hybrid RAG; at matched budgets it leads on three of the four sources (Table 29). Figure 19(d) reports accuracy by the coverage of a question’s required claims, an association across test questions.

D.5. S01: Agent Primitives Primitive. Review, Voting and Selection, and Planning and Execution each encapsulate a recurring multi-agent computation pattern as a reusable latent operator behind the external interface of a standard LLM agent; their internal agents follow task-agnostic role system prompts and coordinate exclusively through KV cache exchange [48]. The study’s Input-Output Alignment Assumption states that conditioning on a sender’s KV cache approximates conditioning on its token sequence when the agents share model parameters and positional encoding, and the study re-indexes transferred keys under RoPE. An Organizer, GPT-5.2 by default, composes the three primitives per query, guided by the entries it retrieves from a Knowledge Pool of 45 MAS structures, and the system executes the selected primitives according to the composition plan. Evaluation and Extracted Evidence. The main table reports average accuracy over AIME25, AIME24, MATH, GSM8K, HumanEval+, MBPP+, MedQA, and GPQA-Diamond (Table 10). From the single agent to the composed MAS, average accuracy rises from 58.8% to 75.3% with Qwen3-8B, from 71.0% to 77.6% with DeepSeek-R1Distill-Qwen-32B, and from 70.1% to 76.4% with DeepSeek-R1-Distill-Llama-70B. Scope. The Organizer’s backbone and its variable structure and agent count are part of the evaluated pipeline. Internal communication runs between agents that share one backbone. The study also compares text with cache

97

The Last Mile to the Real World

Study settings

communication in two stress tests, an auxiliary instruction injected into a long context and unrelated reasoning sentences injected as noise, which Section 4.6 summarizes. On DeepSeek-R1-Distill-Llama-70B, the LatentMAS baseline reaches 60.0% average accuracy, below the single agent’s 70.1%, which the study attributes to its aggressive chunking of latent reasoning steps, in contrast to the conservative latent communication of Agent Primitives. Table 48 reproduces an inconsistency of the study. Its Llama-3-70B Organizer row repeats the GPT-5.2 accuracies, 72.4, 93.8, 82.3, and 53.2%, and derives its efficiencies from them and its own costs, whereas the study’s Organizersensitivity table reports 72.0, 91.8, 81.0, and 52.2% for that Organizer. The cost table prices the methods of the MAS comparison (Table 11).

D.6. S03: Dev-Primitives Primitive. Each component 𝑎𝑖 of the repository ℛ = {𝑎𝑖 }𝑁 𝑖=1 , a file by default, is associated with a Dev-Primitive 𝑃𝑖 that wraps it with a resident LLM; given its local objective 𝑥𝑖 and the communication context 𝒞𝑖 of directed messages, it returns the optionally modified artifact and its messages [49]. The set of primitives is induced by the repository, a persistent file that an active Dev-Primitive creates is registered as a new Dev-Primitive, and each persists across the revision rounds of a task. Bug diagnosis is one LLM call over the executed repository that returns pass or fail and, on fail, the structured feedback 𝜑 = (𝑒, 𝑐, 𝑢). The revision Π′ = ACTIVATE(𝑞, ℛ′ , Π, 𝜑) reactivates only the implicated primitives, within 𝐵 = 3 revision rounds after the initial round by default. Evaluation and Extracted Evidence. SWE-bench Verified uses 500 instances. With GPT-5.6 Sol, the mini-SWEagent comparator reaches 96.20 ± 0.86% and HERMES 97.00 ± 0.76%, reported as mean and standard error (Table 12). Recorded cost is $1.15 versus $1.80, and latency 182.37 versus 205.34 seconds. The DevOps-Gym comparison to Codex uses the same backbone and reports a four-stage average of 49.89 versus 55.41 (Table 14). On Terminal-Bench 4.0, over five runs of the 66 tasks with GPT-5.6 Sol, HERMES resolves 51.8% against 33.0% under Codex at medium effort and 55.5% against 37.3% at maximum effort (Table 13). On the 20 SWE Refactor Bench migrations, the composite rises with GPT-5.6 Sol at medium effort from 6.5% under Codex to 31.0%. Scope. The four-stage average is the unweighted mean of the stage success rates (Table 14), and task settings and repeated-run procedures vary across the benchmarks. In the backbone-scaling study (Table 50), only the configuration that keeps GPT-5.6 Sol for activation and diagnosis changes the Dev-Primitive model alone; it reaches 50.6% against 51.8% on Terminal-Bench. The configuration at 49.7% also moves diagnosis to GPT-5.5. HERMES’s Terminal-Bench version and task set differ from those of S08.

D.7. S07: Navigation Primitives Primitive. A Navigation Primitive is a territory-backed worker of ANTMAN’s static substrate map, and ANTMAN coordinates these scope-specialized workers by the information needs that remain unresolved [50]. A deterministic, substrate-specific partitioning procedure splits the query’s information space into 𝑀 territories, each with a WorkerCard that summarizes its scope and content, and within a substrate the workers share one search interface and, in each configuration, one backbone. The revisable Need Graph records each need’s status, evidence, attempt history, and progress, and it controls worker selection, routing, and recovery. The substrate map stays fixed during execution, and what carries across queries is the partitioning procedure, the worker interface, and the coordination policy. Evaluation. ANTMAN and all baselines use GPT-4.1; ANTMAN-H keeps the GPT-4.1 orchestrator with Qwen38B elsewhere. The structured-navigation comparison covers 108 RepoProbe questions across eight repositories, a frozen 80-question SWE-QA-Pro set, and GAIA-Text-103 (Table 16). ANTMAN’s scores are 38.21, 79.78, and 58.3; GAIA iterative sparse retrieval scores 45.6 under the shared tool interface. GAIA uses a single territorybacked worker, so the coordination mechanisms it tests are need tracking and recovery. The controlled distractorspace intervention keeps the underlying needs approximately fixed while increasing space from 32K to 512K. Under it, active coordination grows 1.23×, against 15.29× for LongAgent and 15.33× for CoA (Table 17). A separate 98

The Last Mile to the Real World

Study settings

demand intervention fixes 512K and increases required evidence from one to four to sixteen units. Multi-document QA scores 30 questions from each of HotpotQA, 2WikiMultiHopQA, and MuSiQue for nine methods with one frozen answer-extraction and evaluation pipeline. Scope. The demand intervention’s calls are 35.5, 43.6, and 86.2 and costs $0.107, $0.111, and $0.293 per query. The space intervention uses ten questions with evidence at early, middle, and late positions, and the demand intervention ten matched 512K contexts at each of its three demand levels [50]. Full ANTMAN scores 60.00% on the 40-question GAIA subset and 81.43 on the 40-question SWE-QA-Pro subset, sampled with seed 42, of Table 19, against 58.3% on GAIA-Text-103 and 79.78 on the 80-question SWE-QA-Pro set (Table 16). Methods run in ANTMAN’s evaluation harness share its question sets and model setting, and all GAIA methods share the tool interface.

D.8. S08: Harness Primitives Primitive. A Harness Primitive is an implemented, reusable control operation with an application scope and a composition contract [51]. Before the actor’s first request, a harness composer on the actor’s backbone selects primitives in a composition intent. A deterministic compiler connects them to the seed graph and supplies the dependencies that their contracts require, and the candidate is checked for supported primitives, compatibility with the execution specification 𝑐𝑥 , valid connections, reachable termination, and bounded loops. An invalid candidate returns diagnostics to the composer for bounded correction, and if correction fails, the seed harness is retained. A primitive is accepted into the library after mechanism tests and a positive matched-run effect on development tasks. Evaluation. SWE-bench Verified uses a 100-task evaluation subset and Terminal-Bench 2.0 a 45-task subset, each with two attempts per task and development tasks disjoint from evaluation. The actor and selector is GPT-5.6 Luna, and the development model is GPT-5.6 Sol. On SWE-bench Verified, seed and STITCH Pass@1 are 73.0 and 80.5%; Pass@2 is 78.0 and 85.0%; and success on both attempts is 68.0 and 76.0% (Table 20). On Terminal-Bench 2.0, seed and STITCH Pass@1 are 60.0 and 72.2%; Pass@2 68.9 and 77.8%; and success on both attempts 51.1 and 66.7% (Table 21). Codex reaches 79.0% Pass@1 on SWE-bench Verified and 56.7% on Terminal-Bench 2.0. Scope. These are sampled subsets and differ from S03’s settings. The catalogs and fixed baselines cover six SWEbench Verified and five Terminal-Bench 2.0 primitives, and the library-size curves of Figure 15 extend to test-time libraries of eight and six entries. The cross-actor evaluation (Table 34, SWE-bench Verified only) uses actors held out from primitive development and adjusts the application scopes on development tasks. The 543- and 638-fold cost comparisons on SWE-bench Verified and Terminal-Bench 2.0 use an optimistic lower bound that counts only the output tokens of the generated implementation and excludes input, reasoning, and debugging costs.

E. Evaluation Protocol for Cross-Family Composition This appendix specifies an evaluation of HEELO as one system that composes primitives of several families. Its experiments measure what the three results of Section 3.5 predict, namely activation against realized demand (Proposition 3.1), the gap between task-conditioned and fixed composition (Proposition 3.2), and the reach of a retrieval stage (Proposition 3.3), together with the reliability of the verification that each family brings. The companion studies measure these quantities within a single family: ANTMAN grows space and demand separately, STITCH compares fixed, random, and task-conditioned harnesses, and the study of self-advertised method selection compares self-advertisement with reranking of a similarity pool. The protocol specifies their measurement for compositions that span families.

E.1. Primary Hypothesis and Task Construction The primary hypothesis is that composing primitives through HEELO improves task success or total cost at a fixed evaluation standard, relative to comparable access to the same underlying assets. A benchmark should therefore 99

The Last Mile to the Real World

Evaluation protocol

contain requests that need at least two families and a task-level evaluation that cannot be replaced by adding the families’ own scores. Suitable task templates include documented API migrations with repository regression tests, source-grounded workflows with mocked external actions, and formalized transformations that need both method selection and a Lean-checked proof of the selected method’s required conditions and residual obligations. Tasks should use frozen artifacts and environments, with separate development, library-validation, and held-out request sets. Library construction must not use held-out outcomes. Donor provenance, any overlap between donors and task repositories, and possible public benchmark exposure should be documented. Reference outcomes of guidance tasks should not enter persistent state directly: as in SourceLearn, they serve only to locate source evidence and derive source requirements; updates are reconstructed from the source. A task’s success criteria and its grading process must be fixed before systems are compared. During a run, verification may use only the tests, builds, and evidence available in the task’s environment, as in LEGO and HERMES, while held-out tests and grading outcomes are reserved for final scoring, and revision (Eq. (6)) changes the plan but not the criteria.

E.2. Comparators and Controlled Changes 1. Monolithic Execution: one task-level agent uses the same underlying assets and authorized tools. 2. Retrieved Context: the same components, source material, and method descriptions are available as context, without primitive-local execution or adaptation. 3. Executable Reuse: components are invocable, but a task-level script manages selection, dependencies, evidence, and revision. 4. Fixed Composition: the best fixed subset of primitives, chosen on development tasks and applied to all tasks, * which plays the role of the best static harness 𝐻fixed of Proposition 3.2. 5. Full HEELO: library selection and composition and evidence-based execution and improvement as described in Section 3.3, with each family’s own mechanisms for selection, verification, and revision (Section 4) and each primitive’s permissions enforced where it runs. 6. Mechanism Ablations: remove one mechanism at a time, such as relevance assessment, the passing of requirements between primitives, verification, or revision, while keeping the underlying assets fixed and grading each run by the preregistered task-level evaluation. The evaluation should hold the backbone, tool access, underlying assets, and total inference budget fixed, and add quality–cost curves for allocation strategies that one matched budget cannot represent fairly. An oracle selection condition, which activates on each task the primitives whose single-primitive runs improve it, approximates the oracle task-conditioned activation z* (𝑞) of Proposition 3.2; it estimates how much error is attributable to selection and is reported separately from deployable performance. Besides the shared assets, all comparators receive the same tests and source evidence.

E.3. Measurements The primary quality measure is end-to-end task success under the preregistered task-level evaluation. The verdicts of each family’s own verification should be reported beside it, together with the runs that end because the revision budget 𝐵 is spent. Most of these verdicts are pass or fail, such as those of HEART’s result validation, of LEGO’s and HERMES’s diagnosis, of Lean’s compilation, and of STITCH’s validity check of each candidate graph. ANTMAN’s need-resolution assessment is three-valued (resolved, partial, or unresolved), and its runs end in a synthesized answer that the benchmark grades (Section 4.8). Hidden checks stronger than the task-level evaluation, run on the composed system’s own task distribution, should measure how often each verification passes a wrong result. HEART’s audit of result validation on 200 sampled decisions measures this reliability within one family (Table 36), 100

The Last Mile to the Real World

Evaluation protocol

and SourceLearn’s grounding and preservation audits measure how well the stored units of its source models are supported by their cited source and preserve earlier meaning (Tables 43 and 44). A composed system can route harder requests to a check than the check’s own study did, so its rates must be measured under the composed distribution. False passes matter most, because they separate a more reliable system from one that reports success more often by weakening a check. For Proposition 3.2, single-primitive arms, which apply one focused primitive with its required dependencies and shared supporting operations to all tasks, as STITCH’s fixed arms do, estimate each primitive’s effect ∆𝑖 (𝑞) against the arm without primitives. These estimates give the prevalence 𝜇𝑖 of the primitive’s application scope, its mean in-scope gain 𝑔¯𝑖 , and its mean out-of-scope penalty 𝜅 ¯ 𝑖 , and the final activations under Full HEELO give the rates 𝛼𝑖 and 𝛽𝑖 . Together they predict Γfixed and the utility-weighted selection loss of Full HEELO (Eq. (9)). Comparing the predictions with the measured gaps between the oracle selection condition, Fixed Composition, and Full HEELO tests the additive model on compositions that span families, where primitives of different families can help only together or interfere. Library-size experiments should vary the number of irrelevant primitives in the library independently of task demand and relevant coverage, as ANTMAN’s space intervention enlarges the distractor space at approximately fixed demand. They should measure candidate recall, assessment cost, the number of distinct active primitives against realized demand, and task quality. Where COMPOSE activates a primitive only by routing one unresolved need to it, as ANTMAN does, the routing count 𝐾(𝑞) and the realized demand 𝐷(𝑞) are observable, so each run can be checked against Eq. (8); elsewhere, required evidence can stand in for demand, as in ANTMAN’s demand intervention. This separates selective execution, which Proposition 3.1 bounds, from discovery cost, which can grow with the library. For Proposition 3.3, each retrieval stage should report its own hit@𝑚 and recall@𝑚 beside the final hit@𝑘 and recall@𝑘, which Eq. (10) bounds for every 𝑘 ≤ 𝑚. This separates misses of retrieval from misses of assessment, as Figure 17 does for Math Primitives and Table 35 does for the retrieval stage of Tool Primitives. Costs should include library construction and validation, indexing, relevance assessment, adaptation, execution, retries, final checks, hosting, and revalidation after updates. Model calls and tokens should be reported as well as any monetary values, with the recorded pricing convention and model labels. Latency should include queueing and p50/p95 values, together with cancellation and timeout rates. Cross-request reuse should report the one-time cost of building and validating the library separately from the per-task costs, together with the number of tasks that share it, as STITCH separates the offline development of its primitives from a composition cost of a few percent of one actor attempt. Diagnostic measurements should distinguish observed failures from causal attribution. Injected or independently verified defects can measure whether diagnosis implicates the primitives that own them, the set ℐ𝑡 of Eq. (5), and whether revision then reactivates only those primitives, as LEGO and HERMES do.

E.4. Stress Conditions and Statistical Analysis Concurrency experiments should run concurrent tasks over one shared library, which no task changes (Section 3.3), with conflicting and disjoint edits. In LEGO and HERMES, a primitive edits only the component it owns and passes cross-component requirements, within the same round, to the primitives that must satisfy them; conflicting edits test this ownership when primitives of different families touch the same artifact. Drift experiments should revise API schemas, source documentation, runtime dependencies, and model compatibility while measuring invalidation and revalidation. Fault experiments should include unavailable dependencies, stale dependency records such as a Code Primitive’s dependency closure 𝐷𝑖 or a Harness Primitive’s composition contract, wrong relevance assessments, truncated evidence, malicious tool output, and uncertain external completion in a mock environment. These interventions test each family’s own checks without irreversible real-world effects. Task runs should be paired across conditions, with repeated seeds or attempts where the system is stochastic. Confidence intervals should come from task-level outcomes with an appropriate paired resampling or test, not from 101

The Last Mile to the Real World

Evaluation protocol

unpaired, rounded table means. Software task families and source groups may require clustered resampling. Sample size and stopping rules should rest on a prespecified, practically meaningful effect and a pilot variance estimate. An evaluation supports the integrated system only if its task-level outcome and complete cost accounting justify composing primitives through HEELO. A failure of interoperability or economics is reported as a specific finding about the system, separately from the overall average.

102

Related documents

• Residuals Are Not Enough: Limits of Physics-Informed Pre-Training for Scientific Foundation Models#1144711
• nvidia/ToolScale#786801
• ScienceIDE: Turning World's Scientific Codebase into Agent Learnable Environments#1013337
• AutoSciBench: Autonomous Benchmark Generation for Evaluating Scientific Agents#1201153
• EarthVerse: Benchmarking Scientific Agents Across Dynamic Earth Systems and Natural Hazards#482163