Post-Training on Office Work Improves Software Engineering: A Behavioral Account of Cross-Domain Transfer

Post-Training on Office Work Improves Software Engineering: A Behavioral Account of Cross-Domain Transfer Logan Ritchie∗, Sushant Mehta, Liudas Panavas, and Edwin Chen Surge AI

arXiv:2608.01604v1 [cs.AI] 3 Aug 2026

Abstract Long-horizon tasks require agents to maintain coherent state and goals across nested and branching work. We call this capability goal-directed execution (GDE): the repeated application of four behaviors, namely selecting goals, constructing task-relevant state, maintaining fidelity to higher-level objectives, and verifying completion against the environment. We hypothesize that long-horizon posttraining strengthens these behaviors across domains. We test this by post-training Qwen3.5-122B-A10B on 363 Long-Horizon Multi-Tool Agent (LHMTA) tasks drawn from office workflows. The collection contained no software-engineering tasks, yet the model’s pass@1 improved by 5.8 points on SWE-Bench Pro. Matched trajectory analysis shows gains in all four GDE behaviors in both office workflows and software repositories. Aggregate SWE-Bench Pro statistics showed related changes in information gathering, implementation, and verification. Together, the results support a behavioral interpretation in which long-horizon post-training changed how the model organized and applied knowledge across tasks, with effects extending beyond the training domain.

1

Introduction

Post-training is often described in terms of the content and domain of its examples. However, long-horizon tasks may also exercise broader ways of organizing behavior. Across a complex task, an agent must decide what to achieve next, maintain an accurate view of the environment, preserve higher-level requirements during local work, and determine whether its actions succeeded. These demands can recur even when the tools and subject matter change. We use goal-directed execution (GDE) to describe the agent’s ability to stay on track: maintaining coherent goals and task-relevant state across nested and branching work, and using them to select, coordinate, and evaluate behavior over time. We operationalize GDE through four behavioral capabilities: 1. Goal formation: deriving the correct immediate target from the parent goal and current working state. 2. State construction: gathering, interpreting, integrating, and preserving the environment information needed to guide local and higher-level decisions. 3. Goal stability: maintaining higher-level requirements and intended outcomes while pursuing lowerlevel work. 4. Verification: determining what evidence would establish that the intended state has been reached and obtaining that evidence at intermediate and final boundaries. We represent task execution as a goal loop. At any level, the agent forms a goal, acts on or queries the environment, updates its working state from the result, and checks that state against the goal. An action that is too abstract to execute may decompose into lower-level goals, so complex tasks contain nested and branching goal loops. GDE is the capability required to execute these loops and coordinate their goals and state updates across levels. ∗ Correspondence to: research@surgehq.ai

1

We hypothesize that GDE is a domain-general requirement for complex agentic tasks. Training that strengthens this capability in one domain may therefore improve performance in others. We examine this using a domain-mismatched source and target. We post-trained Qwen3.5-122B-A10B on 363 LHMTA tasks: realistic workflows over documents, spreadsheets, web research, planning, file manipulation, and office services, exposed as RL environments through Model Context Protocol tools [Anthropic, 2024]. The collection contained no software-engineering tasks. The trained model nevertheless improved by 5.8 percentage points on SWE-Bench Pro [Deng et al., 2025]. The training tasks supplied no repository conventions, parser semantics, or code-level solutions. To examine what changed, we compared matched base and trained trajectories. The trained model more reliably formed local goals that served the broader task, built and maintained relevant working state, preserved parent requirements during lower-level work, and verified substantive completion conditions. The same differences appeared in office workflows and software repositories. Aggregate SWE-Bench Pro statistics provided broader but indirect evidence. With a similar volume of retrieval, the trained model repeated less of what it read; its patches overlapped more with the reference implementation while adding substantially fewer lines; and the share of trajectories running a formal test nearly doubled. Goal stability had no clean aggregate proxy, so evidence for it came from the matched trajectory comparisons. This paper makes three contributions. First, it documents cross-domain transfer from non-software long-horizon post-training to SWE-Bench Pro. Second, it operationalizes GDE in learned agents through a recursive goal-loop representation and four behavioral capabilities. Third, it provides paired trajectory evidence and aggregate behavioral measures consistent with these capabilities improving after post-training. GDE provides an explanatory account of the observed transfer, but the proposed causal link to long-horizon task structure remains a hypothesis.

2

Related Work

Agent architectures. Research on language agents has concentrated on how to build them: ReAct interleaves reasoning with actions [Yao et al., 2023b], Reflexion adds verbal self-reflection [Shinn et al., 2023], search wraps inference-time exploration around the model [Yao et al., 2023a, Zhou et al., 2024], and decomposition-based prompting reduces problems to simpler subproblems [Zhou et al., 2023b, Prasad et al., 2024]. Recent long-horizon scaffolds connect decomposition to context management more directly. HiAgent organizes working memory into subgoal-based chunks, replacing completed branches with summaries while allowing their details to be retrieved [Hu et al., 2025]. ReCAP recursively decomposes plans and reintroduces the parent plan after subtask execution so that new observations can update higher-level work [Zhang et al., 2025]. Both prescribe structures for improving inference-time behavior; we use related concepts to describe behavior learned through post-training. The pattern predates language models: hierarchical RL builds subtask hierarchies into the agent as learnable structure [Sutton et al., 1999, Dietterich, 2000], and HTN planning decomposes tasks through hand-authored methods [Ghallab et al., 2004]. Cognitive architectures provide the closest precedents for the full model: classically, Soar spawns a subgoal automatically whenever an impasse blocks progress, over a working memory representing the current situation [Laird et al., 1987]; more recently, CoALA carries the same concepts to language agents, organizing them around a working memory hub with distinct internal and external actions [Sumers et al., 2024]. All of these contributions answer an engineering question: what mechanism or scaffolding makes an agent act better? They coexist with a broader trend toward general methods that scale with computation rather than task-specific structure [Sutton, 2019]. We argue that structured accounts of task completion still hold value for interpretation and evaluation: a learned agent must still form goals, build state, and verify outcomes, whether or not anything inside it resembles a goal stack, so the classical vocabulary remains useful for describing agent behavior. Cognitive foundations. The goal-loop representation is functionally aligned with the test-operate-testexit (TOTE) framework of Miller et al. [1960]. Both describe behavior as recursive, hierarchical cycles that compare an intended condition with represented current state, operate to reduce the discrepancy, and test again. The working state resembles their Image: accumulated, organized knowledge about the organism and its environment. Their broader account of Plans and Images also includes plan formation, working memory, 2

coordination among plans, and knowledge updated through action. We operationalize this structure for learned tool-using agents through four behaviorally observable capabilities: goal formation, state construction, goal stability, and verification. Goals, actions, working state, and verification are inferred from reasoning, tool calls, environment feedback, edits, and tests rather than treated as a literal internal architecture. This makes the structure useful for auditing where real agent trajectories diverge and for comparing behavior before and after post-training. The decomposition structure also echoes problem spaces [Newell and Simon, 1972], GOMS [Card et al., 1983], and hierarchical task analysis [Annett and Duncan, 1967]. BDI architectures likewise made belief state and the stability of intentions explicit [Bratman, 1987, Rao and Georgeff, 1995]. Cognitive control research emphasizes the active maintenance of goals against interference [Miller and Cohen, 2001, Duncan et al., 2012]. Behavioral analysis of agent failures. Our method characterizes agents entirely by the structure of their behavior, following the behavioral study of machines proposed by Rahwan et al. [2019], the treatment of GPT-3 as a participant in cognitive-psychology experiments [Binz and Schulz, 2023], and machine psychology, which defines itself by analyzing input-output relationships rather than inner workings [Hagendorff et al., 2023]. We extend this paradigm from single-turn probes to long-horizon trajectories. A growing literature catalogs agent failures: repetitive loops [Yao et al., 2023b], degradation and non-recovery in multi-turn settings [Laban et al., 2026], planning shortfalls [Valmeekam et al., 2023, Kambhampati et al., 2024], and goal misgeneralization, in which an agent retains its capabilities yet pursues the wrong goal [Langosco et al., 2022, Shah et al., 2022]. Goal drift has also been measured behaviorally by exposing agents to competing objectives or requiring them to return from a temporary goal to the original one [Arike et al., 2025]. That work studies the stability of a prompt-specified top-level objective; our account also considers higher-level requirements that must remain stable across nested work. Each failure mode is typically named and measured within a single task family, which makes it difficult to say whether two differently named failures in different domains are the same breakdown. Prior work from our group derived a hierarchy of agentic capability levels from failure analysis of frontier models on a realistic RL environment [Ritchie et al., 2026]; here we describe general agent failures as breakdowns at specific locations within a nested goal-loop representation of tasks. What makes training data valuable. Data-selection research has shown that value does not scale straightforwardly with volume: small curated sets can match or beat full corpora [Zhou et al., 2023a, Xia et al., 2024, Liu et al., 2024], generalization scales with the number and variety of distinct tasks rather than with examples per task [Wei et al., 2022, Chung et al., 2024, Zhang et al., 2024], and the RL analogue holds for procedurally diverse environments [Cobbe et al., 2020]. In agentic RL, training domains with richer observations and longer policy trajectories have also been associated with stronger cross-domain retention than domains with greater realism or surface similarity [Liu et al., 2026]. That analysis covered four environments and intervened only on observation richness, so the role of planning complexity remained correlational. Two results from our own group point the same way: training a small model on roughly a thousand expertwritten constraint rubrics improved instruction following on benchmarks it never saw [Mehta et al., 2026a], and training a different base model on CoreCraft, a high-fidelity enterprise environment, improved out-ofdistribution tool-use benchmarks [Mehta et al., 2026b]. Our analysis extends this question to cross-domain transfer: Section 4 identifies structural demands that may help explain why behavior learned in one domain remains useful in another.

3

Goal-Directed Execution in Long-Horizon Agents

We use the goal loop to represent task execution and GDE to describe the capability required to run and coordinate these loops reliably. This is a behavioral model: it describes what the agent appears to pursue, what it does, what evidence becomes available, and how later behavior responds. It does not assume that the model contains a literal symbolic goal stack or state register. A goal loop contains three basic elements: • Goal: the desired state of the environment, including the agent’s knowledge of that environment. • Action: an interaction intended to move the environment toward that state. 3

• Working state: the agent’s current, behaviorally inferred representation of the environment as it relates to its active and parent goals. Environment state is what is actually true; working state is the agent’s current representation of it. The environment is broad enough to include what the agent knows, so an investigation can have a goal such as learning which files are relevant. At the top level, the goal is formed from the task prompt. Within the task, a local goal is the immediate desired state formed from a parent goal and the current working state. The agent compares its working state with the goal, chooses an action, observes the result, and updates its working state (Figure 1). If the updated state satisfies the goal, the loop exits or returns control to its parent; otherwise, the updated state informs the next action. Because the agent rarely begins with complete knowledge, actions can both change the environment and reveal information about it. Opening a file, querying a service, running a test, and inspecting a spreadsheet are all actions that update the agent’s picture of what is true. An action can also be an abstract sequence that must be decomposed into lower-level goals. The goal loop The agent acts toward a goal, updates its picture of the environment, and checks that picture against the goal.

VERIFY

GOAL

Desired environment state What needs to be true for the task to be complete?

ACTION

CHOOSE

WORKING STATE

Change or inspect the environment

UPDATE

Make changes, receive feedback.

Interpret what happened What changed? What is now known?

Figure 1: The goal loop. The agent forms a goal, acts on or queries the environment, updates its working state from the result, and compares the updated state with the goal; verification connects the working state back to the goal. Complex tasks contain a hierarchy of these loops. A task such as reviewing a company’s software subscriptions and recommending which vendor contracts to renew, renegotiate, or cancel is too abstract to execute directly. The agent must first decompose it: 1. Locate the contracts and the finance workbook that tracks spend. 2. Determine the decision criteria: the budget target, usage thresholds, and notice periods for cancellation. 3. Extract the active subscriptions and their renewal dates. 4. Pull usage data for each tool from its admin dashboard. 5. Normalize the figures so tools are comparable, such as monthly versus annual billing and per-seat versus flat pricing. 6. Match spend against usage and flag tools that are underused or redundant. 7. Investigate edge cases, such as a rarely used tool that supports a critical integration. 8. Produce the renew, renegotiate, or cancel recommendation. Each step is a subgoal that runs its own instance of the loop, and most decompose further before reaching concrete tool calls: pulling usage data splits into finding each tool’s dashboard, exporting activity reports, handling tools that offer no export, and merging the results into one comparable sheet. A modest office task expands into a tree several levels deep and dozens of loops wide (Figure 2). Some subgoals are sequential, while others form parallel branches whose results must later be synthesized. 4

Figure 2: A task becomes a tree of goals. The subscription-review task decomposes into branching subgoals, and different branches decompose to different depths before the agent reaches concrete work. Decomposition accounts for only part of the difficulty. The agent must fold low-level observations into one coherent working state and understand what each observation means both for the immediate subgoal and for the objectives above it. This bookkeeping can fail in characteristic ways: • A locally sensible goal can be globally wrong. Converting every contract to a monthly cost simplifies the comparison, but hides the fact that one annual contract auto-renews next week, which is the deadline the task is about. • A relevant fact can be lost between levels. While pulling usage data, the agent notes that most activity runs through service accounts, then carries only the human logins into the comparison, and a heavily used tool looks dormant. • A subtask can succeed while undermining a parent requirement. Cancelling a cheap, barely used tool closes the budget gap while silently breaking an integration that a retained tool depends on. • An intermediate result can be coherent yet incomplete. A spend list built from the finance workbook is internally consistent, but nothing in the list reveals the subscriptions expensed on individual cards that never reached the workbook. These examples illustrate breakdowns in the four capabilities through which we operationalize GDE (Figure 3). Table 1 states the general form of each breakdown.

5

Four capabilities within the goal loop Each capability can break down at any level of nested task execution.

PARENT GOAL

WORKING STATE

The higher-level requirement

What is known so far

FORM NEXT GOAL

3

Goal stability

1

Goal formation

2

GOAL

ACTION

The subgoal for this loop

Change or inspect the environment

CHOOSE

State construction

WORKING STATE

UPDATE

Interpret what happened

VERIFY

4

Verification

Figure 3: Four capabilities within the goal loop: goal formation, state construction, goal stability, and verification. Each can break down at any level of nested task execution.

Capability

Table 1: The four capabilities and examples of how each can break down. How it breaks

Goal formation

The model turns the parent objective and working state into the wrong local target: a simpler proxy, the wrong scope, or a locally plausible goal that is wrong in the broader context.

State construction

The model misses, discards, or fails to combine information needed to understand the environment in relation to the goal.

Goal stability

A local obstacle or complicated subgoal displaces a higher-level goal the model had originally represented correctly.

Verification

The model skips verification or relies on a partial result or proxy, at the level of the overall task or an intermediate subgoal.

The goal loop is functionally aligned with TOTE [Miller et al., 1960]: both describe recursive, hierarchical cycles that compare an intended condition with represented current state, operate to reduce the discrepancy, and test again. Our use of the structure differs in purpose. We infer its elements from the behavior of learned tool-using agents and use the four capabilities to compare trajectories across training conditions and domains.

4

Task Structure and the Transfer Hypothesis

Our transfer hypothesis is that tasks from different domains can place similar demands on GDE. LHMTA tasks were designed to reproduce the complexity of realistic work, and four task demands were targeted: • Deep decomposition: high-level objectives require breaking into many parallel and sequential subtasks. • Parallel investigation and synthesis: evidence needed for a decision is distributed across files, services, and tools. 6

• Entangled constraints: high-level requirements interact, making local decisions challenging. • Long dependent chains: later work depends on earlier extraction, interpretation, and modification steps being correct. These demands may exercise several capabilities at once (Figure 4). For example, deep decomposition repeatedly requires goal formation; parallel investigation requires state construction across distributed evidence; entangled constraints place pressure on goal stability; and long dependent chains make intermediate verification consequential because an unchecked error can contaminate every later step. Related crossdomain RL results associate observation richness and policy trajectory length with generalization [Liu et al., 2026], but do not isolate the recursive structures considered here. These relationships motivate the transfer hypothesis, but the present experiment did not isolate their causal effects. Task demands on goal-directed execution Each demand may exercise several capabilities; one common pressure is shown below.

1

Deep decomposition

2 Parent

High-level objectives branch into many sequential and parallel subtasks.

B C

B

CAN PRESSURE

CAN PRESSURE

Goal formation

State construction

Entangled constraints

A local goal must balance several parent goals

A

files, sources, and tools. A

3

Parallel investigation and synthesis

Evidence for one decision is spread across

4 P1

whose requirements constrain one another.

P2

1

2

3

conclusions being correct.

Local goal P4

Long dependent chains

Later work depends on earlier steps and

P3

CAN PRESSURE

CAN PRESSURE

Goal stability

Verification

Figure 4: Four task demands in LHMTA: deep decomposition, parallel investigation and synthesis, entangled constraints, and long dependent chains. Each may exercise several aspects of GDE; the figure highlights one common pressure for each. The breadth of tools and topics exposed the model to varied instances of these demands. A shortcut tied to one spreadsheet library, tool schema, or output template helps on only a small part of the collection, whereas improvements in relating actions to goals and environment state can remain useful across tasks. To illustrate the scale of one realistic task, we reconstructed a successful LHMTA trajectory as a recursive task graph. Each node records a goal, the action taken, and the resulting state; child nodes represent decomposed subtasks, parallel branches represent work that can proceed independently, and follow-on nodes combine their results. An evidence catalog links the modeled nodes to the corresponding trajectory events. The reconstructed hierarchy contains: Structural measure

Value

Task-level goal-loop nodes Maximum nesting Leaf subtasks Parallel fan-out points Widest parallel fan-out Sequential follow-on or synthesis nodes Tool calls in the source trajectory Normalized trajectory events

53 10 task levels 29 3 5 subtasks 21 42 120

The complete reconstruction, together with its evidence catalog, is available as an interactive task decomposition. It illustrates the structural complexity of a single task but was not used in the aggregate analyses of the dataset or training effects. 7

5

Experimental Setup and Results

The experiment compared a source domain of long-horizon office and general tool-use workflows with a target domain of software engineering. The training collection contained no software-engineering tasks, graders, or benchmark feedback. The two domains differed in subject matter and tools but shared the need to manage long, stateful tasks.

5.1

Training environment

LHMTA consists of realistic professional workflows exposed through Model Context Protocol servers. Its 27 task categories cover document, spreadsheet, and slide manipulation; search and retrieval; file management; scheduling and calendars; browser automation; planning; and tool use across office services. Individual tasks frequently require the agent to coordinate several services within one trajectory. The collection contains no software-engineering tasks. Each task has a deterministic Python grader that evaluates the final environment state against several criteria. A task receives a strict pass only when all criteria are satisfied, while the criterion-level structure provides a partial-credit training signal. Successful trajectories typically contain 30 to 40 tool-calling turns and 80,000 to 100,000 tokens. Task authors targeted realistic work that strong models could not reliably complete and the collection as a whole was monitored for diversity of tools, task types, and failure modes. The dataset snapshot used for this experiment contained 403 tasks: 363 for training and 40 reserved for in-distribution evaluation.

5.2

Training

The base model was Qwen3.5-122B-A10B, an open-weight mixture-of-experts model with 122 billion total and approximately 10 billion active parameters. We adapted it using LoRA on the attention and MLP projections [Hu et al., 2022]. Training proceeded in two stages. First, the model received a supervised warm-up on 3,000 trajectories generated by Kimi K2.6 on LHMTA tasks and rejection-sampled to retain trajectories scoring above 0.9. The warm-up made reinforcement learning feasible: the base policy rarely completed enough of a long-horizon task to receive a sparse pass/fail reward. Second, we trained on all 363 tasks using the GSPO sequence-level estimator [Zheng et al., 2025]. We sampled eight rollouts per prompt, and each full trajectory received a dense reward equal to the fraction of grader criteria it satisfied. Reward was assigned at the trajectory level rather than to individual tool-call prefixes. Evaluation used each benchmark’s native scoring rather than the dense training reward. The SFT teacher generated trajectories only on LHMTA tasks, and the external benchmarks provided no training tasks, graders, checkpoint-selection signal, hyperparameter-tuning signal, or reward-design signal.

5.3

Evaluation

We compared the base and trained checkpoints on the in-distribution holdout and on external benchmarks. Toolathlon measures long-horizon orchestration across heterogeneous tools [Li et al., 2026], BFCL-V4 evaluates function calling [Patil et al., 2025], and SWE-Bench Pro requires agents to resolve issues in realistic repositories [Deng et al., 2025]. All results are pass@1 under greedy decoding. Table 2 reports the in-distribution holdout and the external tool-use benchmarks. The holdout result establishes improvement in the target environment, and Toolathlon and BFCL-V4 show that some gains extend beyond LHMTA’s particular tasks, graders, and tool conventions. Table 2: In-distribution and external tool-use results for the base and trained checkpoints. Benchmark Domain relationship Base pass@1 Trained pass@1 Change LHMTA holdout Toolathlon BFCL-V4

In-distribution External tool use External function calling 8

10.0% 22.2% 55.7%

27.5% 31.8% 59.2%

+17.5pp +9.6pp +3.5pp

Table 3 reports the cross-domain result examined in this paper: SWE-Bench Pro improved even though the training collection contained no software-engineering tasks. Benchmark transfer alone does not identify what changed in the model’s behavior. Section 6 examines whether the transfer is consistent with stronger GDE. Table 3: Cross-domain SWE-Bench Pro performance for the same checkpoints. Benchmark Base pass@1 Trained pass@1 Change SWE-Bench Pro

20.5%

26.3%

+5.8pp

6

Behavioral Analysis of Cross-Domain Transfer

6.1

Method

We analyzed all 103 tasks on which the base model failed and the trained model passed: 12 LHMTA holdout cases, 77 SWE-Bench Pro cases, and 14 Toolathlon cases. Each case compared a base and a trained trajectory for the same task under the same prompt, environment, scaffold, and evaluation protocol, giving 206 trajectories in total. We selected this outcome-conditioned subset to study how newly successful behavior manifested. It does not estimate the prevalence of each capability across the full evaluation distribution. We used Claude Opus 4.8 as a root-cause-analysis agent. For each scored output failure, the agent traced backward through the failed trajectory to identify the earliest evidence-supported point at which it diverged from a successful approach. It examined task requirements, evaluator evidence, tool calls and results, produced artifacts, edits, tests, and model reasoning. Each report cited the specific trajectory events and artifacts supporting its causal account, allowing us to audit each report against the source material. The successful trajectory supplied contrast evidence for a task-correct path but was not treated as automatic ground truth. The reports accelerated the analysis by locating and documenting candidate divergence points; they did not independently validate the framework. We manually reviewed the reports against the corresponding raw trajectories, checking claims and quotations against the original reasoning messages, tool calls, environment responses, edits, and tests before using any case as an example. Through iterative comparison across cases, we organized recurring behavioral differences into four capabilities: goal formation, state construction, goal stability, and verification. These capabilities form an interpretive framework rather than mutually exclusive labels. We supplemented the case studies with deterministic measures over matched SWE-Bench Pro trajectories. These measures cover retrieval repetition and breadth, contact with files changed by the reference patch, patch size, and the presence and timing of formal tests. They provide broader but indirect signals of behavioral change.

6.2

Four capabilities in paired trajectories

Table 4 summarizes how each capability appears in the paired examples. The surface failures differ by domain, but each pair reflects the same broken relationship among the current goal, higher-level requirements, and environment evidence.

9

Table 4: The four capabilities across paired general tool-use and software-engineering cases. Capability General tool-use manifestation Software-engineering manifestation Goal formation

Chose a locally plausible calculation that conflicted with the meaning of the task

Reimplemented behavior despite existing helpers that already encoded the required semantics

State construction

Discarded formula-backed needed by the parent task

values

Failed to connect the new requirement to the repository’s existing validation structure

Goal stability

Reintroduced a course that violated a previously identified course-level constraint

Changed correct parser behavior to satisfy a stale local test

Verification

Treated a bounded extraction as a complete catalog

Checked new exports but left downstream consumers calling removed APIs

Goal formation. Every loop requires converting a prompt or parent action into a concrete target grounded in both the higher-level objective and the environment. A common failure mode is choosing a locally plausible target that is wrong in the broader context. SWE-Bench Pro: NodeBB selected fields. The task required NodeBB database callers to request selected fields from stored objects. Both models found existing helpers and tests that already implemented the required selected-field semantics, including preserving one result per key and representing requested-butmissing fields as null. The correct local goal was to route the public methods through these helpers. The base model instead reimplemented field filtering inside backend paths and changed missing-object behavior. The trained model reused the established helpers, including in the Postgres path. Toolathlon: A/B conversion rates. The task required calculating how often homepage clicks led to store views. The base model defined conversion as clicks / store_views before inspecting any values. When this produced rates above 100 percent, it noted that the result was strange but continued. The trained model interpreted the columns in relation to the parent objective and used store_views / clicks. In both domains, the difference lay in how environment information shaped the next local goal (Figure 5).

SOFTWARE ENGINEERING · SWE-BENCH PRO

GENERAL TOOL USE · TOOLATHLON

NodeBB selected-field APIs

Homepage A/B conversion

PARENT GOAL

WORKING STATE

PARENT GOAL

WORKING STATE

Add selected-field support without changing established

Existing helpers already encode missing-field nulls and stable

Measure how often homepage clicks lead visitors to view

Clicks exceed store views; store views are the downstream

API behavior.

one-result-per-key behavior.

the store.

event being measured.

FORM NEXT GOAL

BASE MODEL · NEXT GOAL

×

TRAINED MODEL · NEXT GOAL

FORM NEXT GOAL

✓

BASE MODEL · NEXT GOAL

×

TRAINED MODEL · NEXT GOAL

Reimplement selected-field handling across the backends

Route the new option through the existing helpers

Calculate clicks ÷ store views

Calculate store views ÷ clicks

Ignores existing knowledge of what the codebase already implements.

Uses the known helper path to add the feature without changing existing behavior.

Ignores the known event order and measures clicks per view instead of views per click.

Uses the known event order to measure downstream store views per homepage click.

✓

Figure 5: Goal formation across domains. From the same parent goal and working state, the base model forms a locally plausible but incorrect local goal, while the trained model forms the correct one.

10

State construction. The agent must decide what to inspect and integrate its observations into a working state that supports both local and higher-level decisions. Relevant evidence can be missed, misinterpreted, or understood only within a low-level subtask. SWE-Bench Pro: Ansible collection-name validation. The task required collection-name validation to reject reserved Python keywords. The base model inspected the relevant imports and validation structure, then introduced a helper that called keyword.iskeyword even though the module imported iskeyword directly. The resulting NameError caused 19 tests to fail. When a self-test exposed the problem, the model changed the test setup and left its implementation untouched. The trained model connected the requirement to the repository’s central AnsibleCollectionRef.is_valid_collection_name validator and reused that path. Toolathlon: formula-backed workbook values. A market-research task required year-over-year growth computed from formula-backed cells. The base model encountered formula strings during extraction and discarded the entries as non-numeric, even though their calculated values were required by the report. The trained model reopened the workbook using cached formula values and retained the calculated data. Both models encountered the same tool friction, but only the trained model incorporated the observation into the state needed by the parent task (Figure 6).

SOFTWARE ENGINEERING · SWE-BENCH PRO

GENERAL TOOL USE · TOOLATHLON

Collection-name validation

Market-research workbook

CURRENT GOAL

OBSERVED INFORMATION

CURRENT GOAL

OBSERVED INFORMATION

Reject reserved keywords without breaking the existing collectionvalidation path.

The repository has a central validator; the local module does not import the new dependency.

Calculate year-over-year growth using the market values stored in the workbook.

The required cells contain formulas rather than literal values.

BUILD WORKING STATE

BASE · WORKING STATE

×

TRAINED · WORKING STATE

BUILD WORKING STATE

✓

BASE · WORKING STATE

×

TRAINED · WORKING STATE

✓

A new local helper can call the validation module directly

The new rule belongs in the existing central validator

The market values are missing and can be discarded

The calculated values exist in the workbook cache

Drops the observed import boundary from its working picture, leaving the new module undefined.

Connects the requested behavior to the implementation path already present in the repository.

Treats the formula representation as missing data, losing values needed for the report.

Recognizes the formulas as another representation of relevant data and reopens the cached values.

Figure 6: State construction across domains. The base model drops or misrepresents a relevant observation; the trained model incorporates it into the working state used by the parent task. Goal stability. A difficult local subtask can displace a requirement the model previously represented correctly. SWE-Bench Pro: PowerShell CLIXML parsing. The parser had to decode escaped characters while preserving control characters such as carriage returns and line feeds. The base model initially implemented the requirement correctly. It then encountered an existing test expecting a trailing CRLF to be stripped and changed the implementation to satisfy the stale behavior. The trained model interpreted the test relative to the requested behavior and retained the CRLF-preserving implementation. LHMTA holdout: course scheduling. Every selected course had to be at the second-year 2xxx level. The base model recorded the constraint and correctly identified STAT1010 as invalid. Later, while repairing an overloaded semester, it selected STAT1010 as an elective and claimed that all courses satisfied the rule. The trained model preserved the constraint, selected ENG2010, and moved another course to resolve the overload. In both pairs, local repair pressure displaced a known parent criterion in the base trajectory but not in the trained one (Figure 7).

11

SOFTWARE ENGINEERING · SWE-BENCH PRO

LHMTA HOLDOUT

PowerShell CLIXML parsing

Course scheduling

PARENT GOAL

LOCAL REPAIR PRESSURE

PARENT GOAL

LOCAL REPAIR PRESSURE

Decode UTF-16 escapes while preserving decoded control characters, including CRLF.

A visible test encodes the old behavior and expects trailing CRLF to be removed.

Build a valid schedule using only second-year courses with codes beginning with 2.

The draft schedule exceeds a semester credit limit and needs a local repair.

FORM REPAIR GOAL

×

BASE MODEL · REPAIR GOAL

TRAINED MODEL · REPAIR GOAL

FORM REPAIR GOAL

✓

×

BASE MODEL · REPAIR GOAL

TRAINED MODEL · REPAIR GOAL

Strip the final CRLF so the visible test passes

Preserve decoded CRLF despite the stale visible test

Use STAT1010 as the elective

Use ENG2010 and move another course

Lets the stale local test replace the explicit requirement it had already understood.

Treats the test as old behavior and keeps the parent requirement in control of the repair.

Solves the local scheduling problem while dropping the known 2xxx course-level requirement.

Resolves the overload while keeping the parent course-level requirement active.

✓

Figure 7: Goal stability across domains. A parent-goal constraint enters a local repair loop; the base model displaces it, while the trained model preserves it. Verification. Verification closes goal loops at both task and subtask boundaries. The model must determine what evidence would establish that the intended environment state has been reached. SWE-Bench Pro: Element Web Pill refactor. The task required converting a Pill class component into a function and exporting helpers that had been static methods. The base model checked that the new helpers were exported and imported. Its validation stopped before searching for consumers of the removed static APIs, leaving downstream code that still called Pill.roomNotifPos and Pill.roomNotifLen. The trained model searched for stale usages, migrated the call sites, and ran a targeted TypeScript check. LHMTA holdout: catalog reconciliation. A library task required reconciling History books against records distributed across several files. The base model created a 44-book intermediate file from a bounded spreadsheet read and inspected that file, but never checked whether the read covered the full catalog. The trained model compared two workbook views and confirmed that each contained the same 347 History authors before using the extraction. The base model’s checks established that an artifact existed and was internally coherent; the trained model’s checks tested the relevant claim against the environment (Figure 8).

SOFTWARE ENGINEERING · SWE-BENCH PRO

LHMTA HOLDOUT

Element Web component refactor

Library-catalog reconciliation

COMPLETION CONDITION

AVAILABLE EVIDENCE

COMPLETION CONDITION

AVAILABLE EVIDENCE

Every downstream consumer uses the new helpers instead of the

Exports and imports show the new API exists, but not whether old

The extracted file represents the complete History catalog

A bounded read produced 44 books; two workbook views were available

removed class methods.

call sites remain.

before updates are decided.

for an independent comparison.

CHOOSE COMPLETION CHECK

BASE MODEL · CHECK

×

TRAINED MODEL · CHECK

CHOOSE COMPLETION CHECK

✓

BASE MODEL · CHECK

×

TRAINED MODEL · CHECK

✓

Confirm the new helpers are exported and imported

Search for stale calls and type-check the changed files

Inspect and grep the same 44-book intermediate file

Compare both workbook views: 347 authors, zero mismatch

Checks that the new path exists, not that every old consumer has actually been migrated.

Checks the actual completion condition: every consumer uses the replacement helpers.

Checks the partial artifact against itself, not whether it represents the full source catalog.

Uses independent source views to establish that the intermediate representation is complete.

Figure 8: Verification across domains. The base model relies on proxy or self-referential evidence; the trained model checks the relevant claim against the environment. 12

6.3

Aggregate behavioral evidence

The case studies show how individual tasks diverge; the aggregate SWE-Bench Pro measures examine whether related behavioral signals appear more broadly (Table 5). Table 5: Behavioral metrics from SWE-Bench Pro trajectories, computed over 731 paired tasks. Behavioral signal Base Trained Interpretation Mean retrieval calls per run Mean distinct retrieved information Repeated retrieved information Mean reference-patch files touched Mean added lines Runs containing a formal test Mean first-test position (tested runs) Runs with passing test followed by edit

25.6 7,985 22.5% 2.69 415.5 37.5% 70.8% 5.9%

23.9 9,019 14.3% 3.08 111.7 73.3% 62.0% 12.9%

Similar retrieval effort More unique retrieved content Less repeated investigation Greater reference-file overlap Smaller patches More frequent verification Earlier verification More intermediate verification

The metrics are defined as follows: • Mean retrieval calls per run. Read-only inspection commands, such as ls, grep, cat, and read-only git subcommands, that neither modify the environment nor redirect output. • Mean distinct retrieved information. Mean number of unique 12-token spans in retrieval-call outputs per run. • Repeated retrieved information. The share of spans already seen earlier in the same run. • Mean reference-patch files touched. Mean number of files per run changed by both the agent’s final patch and the task’s reference patch. • Mean added lines. Added lines in the agent’s final patch. • Runs containing a formal test. Runs with at least one command invoking a standard test runner such as pytest or go test. • Mean first-test position. How far through a run’s commands the first test appears, from zero to one, among tested runs. • Runs with a passing test followed by editing. Runs in which a passing test is followed by a further file edit. Appendix A gives the exact classification patterns and parsing rules behind each metric. The trained model made slightly fewer retrieval calls while covering more distinct information and repeating less content, a pattern consistent with a better-maintained picture of what has already been learned and what remains unknown. It also touched more of the files changed by the reference patch while adding approximately one quarter as many lines, consistent with more targeted implementation. Testing provides the clearest aggregate change. The trained model ran formal tests in nearly twice as many trajectories and began testing earlier. Passing tests were also more likely to be followed by further editing, suggesting that verification frequently informed intermediate work rather than serving only as a final check. These measures provide convergent behavioral signals rather than direct measurements of the four capabilities. Reference patches do not uniquely specify a valid implementation strategy, and smaller patches are not inherently better. The available telemetry also lacks a clean aggregate proxy for goal stability, so evidence for that capability comes from the paired cases.

13

7

What Long-Horizon Post-Training Teaches

The results support a behavioral interpretation in which post-training changed how the model organized work over long tasks. The training collection supplied no software-engineering tasks, repository conventions, or code-level solutions. Yet the same differences in goal formation, state construction, goal stability, and verification appeared in both office workflows and software repositories. GDE provides a common description of these changes without requiring the source and target tasks to share subject matter. One plausible account is that the training improved how reliably the model deployed knowledge acquired during pretraining. Prior work has argued that pretrained models already contain much of the knowledge required for downstream tasks and that instruction tuning improves their ability to apply it in response to user intent [Zhou et al., 2023a, Ouyang et al., 2022]. LHMTA may have provided repeated practice applying available knowledge while pursuing complex goals in unfamiliar, stateful environments. The software result is informative because the domain mismatch makes direct acquisition of repository-specific knowledge from the training set implausible. The causal explanation remains a hypothesis. The experiment did not isolate deep decomposition, parallel synthesis, entangled constraints, or dependent chains, and it did not establish which of these demands contributed to transfer. Cross-domain evaluation can reveal when training effects extend beyond surface workflows, while the four capabilities provide a vocabulary for more controlled questions: which task properties strengthen each capability, which capabilities transfer together, and whether the same patterns replicate across models, data mixtures, and training methods. Long-horizon agentic data is costly to author and train on; successful LHMTA trajectories often span 80,000 to 100,000 tokens. This raises the value of understanding what kinds of behavior a training experience exercises. The present results motivate that question but do not yet provide a recipe for engineering particular task demands.

8

Limitations

Single model and training run. All results come from one base model and one post-training run. We have not established reproducibility across seeds, scales, model families, training recipes, or data mixtures. Exploratory behavioral interpretation. The four capabilities were refined iteratively through qualitative analysis of trajectories. They describe observable relationships among goals, actions, environment evidence, and outcomes; they do not identify literal internal goals, state registers, or control loops and do not constitute an exhaustive taxonomy. Outcome-conditioned cases and analyst judgment. The analysis focused on tasks where the base model failed and the trained model passed, and the cases shown were selected for explanatory clarity rather than by random sampling. They illustrate how improvement manifested but do not estimate the prevalence of each capability. Reconstructing goals and causal chains from long trajectories also involves judgment, and individual failures can have multiple causes. No causal or domain-matched counterfactual. We did not ablate the proposed task demands or compare LHMTA with an equal budget of software-engineering training data. The experiment therefore does not establish which properties caused transfer or whether non-software training was the most efficient way to improve SWE-Bench Pro. Broader changes in exploration, persistence, instruction following, or tool use remain possible explanations for part of the gain. Proxy measures. Retrieval overlap, reference-file coverage, patch size, and testing commands are indirect behavioral signals. Reference patches are not unique ground truth.

14

9

Conclusion

Post-training Qwen3.5-122B-A10B on 363 long-horizon multi-tool tasks produced behavior that remained useful outside the training domain: SWE-Bench Pro improved even though the training collection contained no software-engineering tasks. Paired trajectories showed recurring improvements in goal formation, state construction, goal stability, and verification, while aggregate measures provided broader evidence of changes in investigation, implementation, and testing. GDE organizes these differences as a behavioral account of how the trained model pursued long tasks. The findings motivate studying post-training data as experience that exercises behavioral capabilities, not only as examples containing particular topics, tools, or answers. Establishing the causal relationship will require controlled comparisons across task demands, domains, models, and training methods. A science of post-training should explain not only whether a dataset improves a benchmark, but what kinds of behavioral capabilities the training experience develops and where those capabilities transfer.

References John Annett and Keith D. Duncan. Task analysis and training design. Occupational Psychology, 41:211–221, 1967. Anthropic. Introducing the Model Context Protocol. Anthropic, November 2024. URL https://www. anthropic.com/news/model-context-protocol. Rauno Arike, Elizabeth Donoway, Henning Bartsch, and Marius Hobbhahn. Technical report: Evaluating goal drift in language model agents, 2025. URL https://arxiv.org/abs/2505.02709. arXiv:2505.02709. Marcel Binz and Eric Schulz. Using cognitive psychology to understand GPT-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120, 2023. doi: 10.1073/pnas.2218523120. Michael E. Bratman. Intention, Plans, and Practical Reason. Harvard University Press, Cambridge, MA, 1987. Stuart K. Card, Thomas P. Moran, and Allen Newell. The Psychology of Human-Computer Interaction. Lawrence Erlbaum Associates, Hillsdale, NJ, 1983. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. URL https://jmlr.org/papers/v25/23-0870.html. Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 2048–2056. PMLR, 2020. URL https://proceedings.mlr.press/v119/cobbe20a.html. Xiang Deng, Jeff Da, Edwin Pan, Yannis Yiming He, Charles Ide, Kanak Garg, Niklas Lauffer, Andrew Park, Nitin Pasari, Chetan Rane, Karmini Sampath, Maya Krishnan, Srivatsa Kundurthy, Sean Hendryx, Zifan Wang, Vijay Bharadwaj, Jeff Holm, Raja Aluri, Chen Bo Calvin Zhang, Noah Jacobson, Bing Liu, and Brad Kenstler. SWE-Bench Pro: Can AI agents solve long-horizon software engineering tasks?, 2025. URL https://arxiv.org/abs/2509.16941. arXiv:2509.16941. Thomas G. Dietterich. Hierarchical reinforcement learning with the MAXQ value function decomposition. Journal of Artificial Intelligence Research, 13:227–303, 2000. doi: 10.1613/jair.639. John Duncan, Moritz Schramm, Russell Thompson, and Iroise Dumontheil. Task rules, working memory, and fluid intelligence. Psychonomic Bulletin & Review, 19(5):864–870, 2012. doi: 10.3758/s13423-012-0225-y. 15

Malik Ghallab, Dana Nau, and Paolo Traverso. Automated Planning: Theory and Practice. Morgan Kaufmann, San Francisco, CA, 2004. Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C. Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata, and Eric Schulz. Machine psychology, 2023. URL https://arxiv.org/abs/2303.13988. arXiv:2303.13988. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. HiAgent: Hierarchical working memory management for solving long-horizon agent tasks with large language model. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32779–32798. Association for Computational Linguistics, 2025. doi: 10.18653/v1/2025.acl-long.1575. URL https://aclanthology.org/2025.acl-long.1575/. Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Paul Saldyt, and Anil B. Murthy. Position: LLMs can’t plan, but can help planning in LLMmodulo frameworks. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 22895–22907. PMLR, 2024. URL https:// proceedings.mlr.press/v235/kambhampati24a.html. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. LLMs get lost in multi-turn conversation. In International Conference on Learning Representations, 2026. URL https://openreview.net/ forum?id=VKGTGGcwl6. John E. Laird, Allen Newell, and Paul S. Rosenbloom. SOAR: An architecture for general intelligence. Artificial Intelligence, 33(1):1–64, 1987. doi: 10.1016/0004-3702(87)90050-6. Lauro Langosco Di Langosco, Jack Koch, Lee D. Sharkey, Jacob Pfau, and David Krueger. Goal misgeneralization in deep reinforcement learning. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 12004–12019. PMLR, 2022. URL https://proceedings.mlr.press/v162/langosco22a.html. Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu, Xiaochen Wang, Rui Ge, Yuxuan Cao, Yuzhen Huang, Wei Liu, Junteng Liu, Zhaochen Su, Yiyang Guo, Fan Zhou, Lueyang Zhang, Juan Michelini, Xingyao Wang, Xiang Yue, Shuyan Zhou, Graham Neubig, and Junxian He. The tool decathlon: Benchmarking language agents for diverse, realistic, and long-horizon task execution. In International Conference on Learning Representations, 2026. URL https://openreview.net/forum?id=z53s5p0qhf. Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? A comprehensive study of automatic data selection in instruction tuning. In International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=lhk0QnyRVp. Zhihan Liu, Lin Guan, Yixin Nie, Kai Zhang, Zhuoqun Hao, Lin Chen, Asli Celikyilmaz, Zhaoran Wang, and Na Zhang. Paying less generalization tax: A cross-domain generalization study of RL training for LLM agents, 2026. URL https://arxiv.org/abs/2601.18217. arXiv:2601.18217. Sushant Mehta, Liudas Panavas, Eleanor Fleming, Paul Mains, and Edwin Chen. Complex-IF and beyond: Expert rubrics for RLVR. In Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM), pages 668–677, San Diego, California, USA, July 2026a. Association for Computational Linguistics. doi: 10.18653/v1/2026.gem-main.61. URL https://aclanthology.org/2026.gem-main.61/. Sushant Mehta, Logan Ritchie, Suhaas Garre, Ian Niebres, Nick Heiner, and Edwin Chen. EnterpriseBench Corecraft: Training generalizable agents on high-fidelity RL environments, 2026b. URL https://arxiv. org/abs/2602.16179. arXiv:2602.16179.

16

Earl K. Miller and Jonathan D. Cohen. An integrative theory of prefrontal cortex function. Annual Review of Neuroscience, 24(1):167–202, 2001. doi: 10.1146/annurev.neuro.24.1.167. George A. Miller, Eugene Galanter, and Karl H. Pribram. Plans and the Structure of Behavior. Holt, Rinehart and Winston, New York, 1960. Allen Newell and Herbert A. Simon. Human Problem Solving. Prentice-Hall, Englewood Cliffs, NJ, 1972. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022. doi: 10.52202/068431-2011. URL https://proceedings.neurips.cc/paper_files/paper/2022/hash/ b1efde53be364a73914f58805a001731-Abstract-Conference.html. Shishir G. Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E. Gonzalez. The Berkeley Function Calling Leaderboard (BFCL): From tool use to agentic evaluation of large language models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 48371–48392. PMLR, 2025. URL https: //proceedings.mlr.press/v267/patil25a.html. Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. ADaPT: As-needed decomposition and planning with language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4226–4252, Mexico City, Mexico, 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-naacl.264. URL https: //aclanthology.org/2024.findings-naacl.264/. Iyad Rahwan, Manuel Cebrian, Nick Obradovich, Josh Bongard, Jean-François Bonnefon, Cynthia Breazeal, Jacob W. Crandall, Nicholas A. Christakis, Iain D. Couzin, Matthew O. Jackson, Nicholas R. Jennings, Ece Kamar, Isabel M. Kloumann, Hugo Larochelle, David Lazer, Richard McElreath, Alan Mislove, David C. Parkes, Alex Sandy Pentland, Margaret E. Roberts, Azim Shariff, Joshua B. Tenenbaum, and Michael Wellman. Machine behaviour. Nature, 568(7753):477–486, 2019. doi: 10.1038/s41586-019-1138-y. Anand S. Rao and Michael P. Georgeff. BDI agents: From theory to practice. In Victor R. Lesser and Les Gasser, editors, Proceedings of the First International Conference on Multi-Agent Systems, pages 312–319, San Francisco, CA, 1995. MIT Press. ISBN 0-262-62102-9. URL https://www.aaai.org/Papers/ICMAS/ 1995/ICMAS95-042.pdf. Logan Ritchie, Sushant Mehta, Nick Heiner, Mason Yu, and Edwin Chen. The hierarchy of agentic capabilities: Evaluating frontier models on realistic RL environments, 2026. URL https://arxiv.org/abs/ 2601.09032. arXiv:2601.09032. Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal misgeneralization: Why correct specifications aren’t enough for correct goals, 2022. URL https://arxiv.org/abs/2210.01790. arXiv:2210.01790. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Advances in Neural Information Processing Systems, volume 36, pages 8634–8652. Curran Associates, Inc., 2023. doi: 10.52202/075280-0377. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 1b44b878bb782e6954cd888628510e90-Abstract-Conference.html. Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. Cognitive architectures for language agents. Transactions on Machine Learning Research, 2024. URL https://openreview.net/ forum?id=1i6ZCvflQJ. Richard S. Sutton. The bitter lesson. Incomplete Ideas, 2019. URL http://www.incompleteideas.net/ IncIdeas/BitterLesson.html. 17

Richard S. Sutton, Doina Precup, and Satinder Singh. Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1–2):181–211, 1999. doi: 10.1016/S0004-3702(99)00052-1. Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. PlanBench: An extensible benchmark for evaluating large language models on planning and reasoning about change. In Advances in Neural Information Processing Systems, volume 36, pages 38975–38987. Curran Associates, Inc., 2023. doi: 10.52202/075280-1693. URL https://proceedings.neurips. cc/paper_files/paper/2023/hash/7a92bcdede88c7afd108072faf5485c8-Abstract-Datasets_and_ Benchmarks.html. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=gEZrGCozdqR. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. LESS: Selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 54104–54132. PMLR, 2024. URL https://proceedings.mlr.press/v235/xia24c.html. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems, volume 36, pages 11809–11822. Curran Associates, Inc., 2023a. doi: 10.52202/075280-0517. URL https://proceedings.neurips.cc/paper_files/paper/2023/hash/ 271db9922b8d1f4dd7aaef84ed5ac703-Abstract-Conference.html. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In International Conference on Learning Representations, 2023b. URL https://openreview.net/forum?id=WE_vluYUL-X. Dylan Zhang, Justin Wang, and François Charton. Instruction diversity drives generalization to unseen tasks, 2024. URL https://arxiv.org/abs/2402.10891. arXiv:2402.10891. Zhenyu Zhang, Tianyi Chen, Weiran Xu, Alex Pentland, and Jiaxin Pei. ReCAP: Recursive context-aware reasoning and planning for large language model agents. In Advances in Neural Information Processing Systems, volume 38, 2025. URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/ 88af3540325dd0b70617a9ab605f294d-Abstract-Conference.html. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, Jingren Zhou, and Junyang Lin. Group sequence policy optimization, 2025. URL https://arxiv.org/abs/2507.18071. arXiv:2507.18071. Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning, acting, and planning in language models. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 62138–62160. PMLR, 2024. URL https://proceedings.mlr.press/v235/zhou24r.html. Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for alignment. In Advances in Neural Information Processing Systems, volume 36, pages 55006–55021. Curran Associates, Inc., 2023a. doi: 10.52202/075280-2400. URL https://proceedings.neurips.cc/ paper_files/paper/2023/hash/ac662d74829e4407ce1d126477f4a03a-Abstract-Conference.html. Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc V. Le, and Ed H. Chi. Least-to-most prompting enables complex reasoning in large language models. In International Conference on Learning Representations, 2023b. URL https: //openreview.net/forum?id=WZH7099tgfM.

18

A

Exact Metric Definitions

This appendix gives the exact classification rules behind the metrics in Table 5. All patterns are regular expressions applied case-insensitively to individual assistant shell commands. Long patterns are wrapped across lines for readability; the implemented patterns contain no whitespace at the wrap points.

A.1

Retrieval calls

A command is classified as a retrieval call in three steps. Occurrences of >/dev/null are ignored; any other output redirection disqualifies the command. A command matching the exclusion pattern below is disqualified. The remaining commands qualify if they match the inclusion pattern. Inclusion pattern: (?:^|[;&|()]\s*)(?: ls|find|fd|rg|grep|egrep|fgrep|cat|head|tail|wc|stat|file|tree|pwd| readlink|realpath|which|whereis|type|jq|awk|cut|sort|uniq| git\s+(?:show|log|grep|ls-files|rev-parse|branch|remote|blame) )\b Exclusion pattern: \b(?:rm|mv|cp|mkdir|rmdir|touch|chmod|chown|ln|tee|truncate|patch)\b| \bgit\s+(?:add|commit|apply|checkout|restore|reset|clean|am| cherry-pick|rebase|merge)\b| \bsed\s+-[^\n;|&]*i\b|\bperl\s+-[^\n;|&]*i\b| \b(?:python|python3|node|ruby|php)\b| \b(?:pytest|unittest|jest|vitest|mocha|rspec|go\s+test|cargo\s+test| npm\s+test|yarn\s+test|pnpm\s+test)\b| \b(?:make|cmake|ninja|npm\s+install|yarn\s+install|pnpm\s+install| pip\s+install)\b| << Although git branch and git remote can mutate repository state with other arguments, all matched uses in this corpus were read-only: git branch appeared only in listing or inspection forms such as -a, --contains, and --show-current, and git remote appeared only as git remote -v. No branch creation, branch deletion, or remote mutation was counted as retrieval.

A.2

Information spans

Retrieval-call outputs are processed as follows: ANSI escape sequences are removed, the text is tokenized with the pattern [A-Za-z_][A-Za-z0-9_]*|\d+(?:\.\d+)?|[^\s] and lowercased, and every overlapping window of 12 consecutive tokens forms one span; an output shorter than 12 tokens forms a single span. Distinct retrieved information is the number of unique spans in a run. Repeated retrieved information is the share of spans that already occurred in an earlier output of the same run.

A.3

Formal tests and mutations

A command contains a formal test if it matches any of: (?:^|[;&|]\s*|\s)(?:python\d*\s+-m\s+)?pytest(?:\s|$) (?:^|[;&|]\s*|\s)py\.test(?:\s|$) \bgo\s+test\b \bcargo\s+test\b \b(?:npm|pnpm|yarn)\s+(?:run\s+)?test(?:\s|$|:) \b(?:npx\s+)?(?:jest|vitest|mocha)(?:\s|$) \bmake\s+(?:[\w.-]*test[\w.-]*|check)(?:\s|$) 19

(?:^|[;&|]\s*|\s)(?:tox|nosetests|unittest)(?:\s|$) \bpython\d*\s+-m\s+unittest\b \bbazel\s+test\b \bgradle\w*\s+test\b \bmvn\w*\s+(?:test|verify)\b \bdotnet\s+test\b \brspec(?:\s|$) A test counts as passing when it returns code zero, is not part of a status-masking pipeline or fallback, and its output contains no explicit failure marker. A command counts as a mutation if it matches any of: \bapply_patch\b \b(?:sed|perl)\b[^\n]*(?:\s-i\b|--in-place) \bgit\s+(?:apply|checkout|restore|reset|cherry-pick)\b \b(?:cp|mv|rm|touch)\s+(?![^\n]*(?:/tmp/|/dev/)) \.(?:write_text|write_bytes)\s*\( \bopen\s*\([^\n]{0,180},\s*['"](?:w|a|x|w\+|a\+)['"] \b(?:cat|printf|echo)\b[^\n]*(?:>>|>)\s+(?!/tmp/|/dev/)[^\s;&|]+ \btee\s+(?:-a\s+)?(?!/tmp/|/dev/)[^\s;&|]+ First-test position is the ordinal position of the first formal-test command among all of a run’s assistant shell commands, normalized to the range zero to one.

A.4

Patch metrics

The agent patch is the final submitted diff for a run. The reference patch is the task’s gold code patch; the task’s separate test patch is not included. File paths are read from diff --git headers. Reference-patch files touched is the number of distinct files appearing in both the agent patch and the reference patch. Added lines is the number of lines beginning with +, excluding +++ file headers, summed over all files in the agent patch.

20

Related documents

• Empirical Software Engineering in Practice: Insights from Google#627764
• Online Behavioral Experiments with jsPsych: A Workflow Guide#403378
• SynthCoder: Anti-pattern identification and model training for FIM mode code completion#1129102
• Library Drift: Diagnosing and Fixing a Silent Failure Mode in Self-Evolving LLM Skill Libraries#204848
• RISE - Reuse im Software Engineering : Schlussbericht des Fraunhofer IESE zum Projekt RISE#1137026