Preprint, May 2026
Multi-agent Collaboration with State Management Mengyang Liu1,2 , Taozhi Chen3 , Zhenhua Xu4 , Xue Jiang4 , Yihong Dong4 Shanghai Jiaotong University 2 Cortices AI 3 Emory University 4 Peking University {mengyangliu912, chentaozhi313, EthanDongyh}@gmail.com 1
arXiv:2605.20563v1 [cs.MA] 19 May 2026
Abstract Recent advances in multi-agent systems have shown great potential for solving complex tasks. However, when multiple agents edit a shared codebase concurrently, their changes can silently conflict and inconsistent views lead to integration failures. Existing multi-agent systems address this through workspace isolation (e.g., one git worktree per agent), but this defers conflict resolution to a post-hoc merge step where recovery is expensive. In this paper, we propose STORM, i.e., STate-ORiented Management for multi-agent collaboration. Specifically, STORM manages agent states by mediating their interactions with the shared workspace, ensuring that each agent operates on a consistent view of the codebase and that conflicting edits are detected and resolved at write time. We evaluate STORM on Commit0 and PaperBench across multiple LLMs. STORM outperforms the git-worktree-based multi-agent baseline by +18.7 on Commit0-Lite and +1.4 on PaperBench, while achieving comparable or better cost efficiency. Combined with single-agent runs, STORM reaches highest scores of 87.6 and 78.2 on the two benchmarks respectively, suggesting that explicit state management is a more effective foundation for multi-agent collaboration than workspace isolation. STORM can also be plugged into any multi-agent system seamlessly.1 .
1
Introduction
Multiple LLM agents working in parallel can solve tasks that are too large for any single agent to finish within its iteration budget (Dong et al., 2024; Qian et al., 2024; Hong et al., 2024; Geng and Neubig, 2026; Qu et al., 2026). In software engineering, agents can implement different modules concurrently; in scientific research, they parallelize experimental setups. But running agents in parallel on a shared workspace raises a question: when one agent edits a file, how do we know that its assumptions about the rest of the codebase are still valid? We treat this as a state management problem. An agent interacts with its workspace through file reads and writes. When it modifies a file, its reasoning depends not just on that file but on context from other files it has read (dependencies, interfaces, specifications). The edit is only safe when those context files have not changed since the agent last read them. This is a local consistency requirement: the agent does not need a frozen snapshot of the entire workspace, just assurance that the specific files informing its current edit are up to date. Existing multi-agent systems mostly avoid this problem by giving each agent its own workspace (e.g., a git worktree) and merging afterward (Geng and Neubig, 2026; Qu et al., 2026; Qin and Xu, 2026). Isolation prevents interference during editing but pushes all conflict resolution to the merge step, after agents have already committed to potentially incompatible designs. Textual merge conflicts are easy to spot; semantic conflicts, where both sides compile individually but break when combined, are harder, and current tooling cannot resolve them automatically. In this paper, we propose STORM (STate-ORiented Management), a state management framework for multi-agent collaboration. STORM mediates each agent’s file reads and writes. Before accepting a write, it checks whether the agent’s view of the target file and its context dependencies is still 1
Our code is available at https://github.com/dreamyang-liu/STORM
1
Preprint, May 2026
current. If another agent has modified any of those files in the interim, the write is rejected and the agent receives the updated content so it can retry from a correct baseline. Our main contributions can be attributed as: 1. A formulation of multi-agent state management as file-level context consistency: an agent’s write is valid only if the target file and its read dependencies have not been modified since the agent last observed them. 2. STORM, an architecture-agnostic state management framework for multi-agent collaboration that enforces local state consistency at write time, detecting conflicts immediately and enabling agents to retry from a correct baseline without workspace isolation. 3. Empirical validation on Commit0-Lite(Zhao et al., 2025) and PaperBench(Starace et al., 2025) with Sonnet 4.6. On Commit0-Lite, STORM achieves 82.5% macro pass rate and 46.2% weighted pass rate, outperforming single-agent (66.4% / 20.7%) and GitWorktree (63.8% / 24.6%) and we observe similar gain on Deepseek and Qwen model. On PaperBench, STORM scores 74.1 vs. 72.7 (GitWorktree) and 68.7 (single-agent).
2
STORM
Mainstream multi-agent systems give each agent its own git worktree (Geng and Neubig, 2026; Qu et al., 2026; Qin and Xu, 2026) so that agents cannot interfere with each other while working. The cost is paid later: once agents finish their individual task, their branches must be merged back together. If two agents edit the same file, or made incompatible design choices about a shared interface, the merge fails. Worse, because each agent worked in isolation without seeing what others were doing, these conflicts tend to compound. Agent A writes a helper assuming a certain signature; Agent B changes that signature in its own branch; code that depends on both is now broken in a way that neither branch exhibits alone. We avoid this by putting all agents in the same workspace. The key insight is that an agent does not need a globally consistent view of the entire repository to produce a correct edit. It only needs the files it has actually read to remain unchanged while it reasons. We call this property local state consistency and formalize it in Section 2.1. In practice, even with disjoint task assignments, agents sometimes need to edit the same file (e.g., two agents each implementing different functions in a shared module). Most of their edits do not interact, but at the boundaries where they do, agents need a way to exchange information. STORM addresses this with two mechanisms: write-time conflict control (Section 2.2), which rejects a write whenever the agent’s local view has gone stale and lets it retry with fresh context, and intent annotations (Section 2.3), structured comments that agents leave in the code so that when another agent reads the same file, it can see not just the raw code but the intent behind it, enabling coordination at these shared boundaries without explicit messaging. 2.1
Local State Consistency
An LLM agent does not need a frozen snapshot of the entire workspace to produce a correct edit. It only needs the files it has actually read to remain unchanged while it reasons. We formalize this as local state consistency. Workspace and agents. Let the workspace be a set of versioned files W = {(f, vf ) | f ∈ F}, where vf ∈ N is the current version of file f . A manager agent M decomposes a task T into sub-tasks {τ1 , . . . , τk } and assigns each to an engineer agent ai together with a primary file set Fi ⊆ F: M : T −→ {(τi , Fi , ai )}ki=1 ,
where Fi ∩ Fj = ∅ for i ̸= j.
(1)
The disjoint assignment reduces but does not eliminate conflicts: agents may still read or edit files outside their primary set (e.g., a shared utility or a common import). Agent local state. As agent ai works on τi , it accumulates a read snapshot Si recording every file it has observed and the version at observation time: Si = {(g, vgobs ) | ai has read g}. 2
(2)
Preprint, May 2026
When ai issues a write to file f producing new content c′ , its generation depends only on Si , the local context the LLM has seen, not on the full workspace state. This is the key asymmetry we exploit: correctness requires consistency of Si , not of W. In practice, each task τi requires accessing (reading or modifying) a set of files Ai ⊆ F that may extend beyond the assigned Fi . When two agents’ access sets overlap (Ai ∩ Aj ̸= ∅), the shared files form a boundary where conflicts may arise. STORM only needs to coordinate at these boundaries, leaving the non-overlapping majority of work fully parallel. Write validity. A write (ai , f, c′ ) is valid if and only if the agent’s local state is still consistent with the current workspace: ∀ (g, vgobs ) ∈ Si : vgobs = vgcur . (3) That is, no file that ai has read has been modified since its observation. A valid write is applied atomically: vf ← vf + 1 and the content of f is updated to c′ . Conflict.
A write is conflicting when Eq. 3 is violated. Two cases arise:
• Direct conflict: the target file itself was updated (vfobs < vfcur ), meaning another agent wrote to f concurrently. • Stale dependency: a dependency file g ̸= f was updated (vgobs < vgcur ), meaning ai ’s reasoning may rest on outdated context. In both cases, STORM rejects the write and returns the current state, enabling ai to refresh Si and retry from a correct baseline. Section 2.2 details the mechanism. Figure 1 shows the architecture. A single manager agent reads the repository, partitions work into sub-tasks scoped to disjoint file sets, assigns each to an engineer, reviews diffs after engineers finish, runs tests, and commits accepted changes. Only the manager commits. Each engineer receives a scoped task (e.g., “implement all functions in tensor_ops.py”) and accesses the workspace only through the STORM-mediated file_editor. Engineers do not communicate directly; coordination happens through the shared codebase and intent annotations (Section 2.3). 2.2
Write-time Conflict Control
STORM enforces the validity condition in Eq. 3 via a mechanism inspired by optimistic concurrency control (Kung and Robinson, 1981). The key observation is that in a well-decomposed task, most concurrent edits touch different files; the system lets all operations proceed without blocking and only intervenes when a conflict actually occurs. Implementation. Each file maintains a monotonically increasing version counter (vf , starting at 1). Every file_editor read returns the content together with vf ; every write must declare the expected version. The STORM layer validates the write against Eq. 3 by comparing the agent’s full read snapshot Si to the current workspace state. If validation passes, the write is applied atomically (vf ← vf + 1). If it fails, the write is rejected. Rejection payload. On rejection, STORM returns: (1) the current content of the target file, (2) a unified diff showing what changed since the agent’s last read (for direct conflicts), and (3) a list of stale dependencies with their version deltas. This gives the LLM enough context to re-plan from the current baseline without needing to re-read every file. Reservation. After a rejection, a short reservation is granted to the rejected agent on the target file. This prevents repeated alternating conflicts where two agents each invalidate the other in a tight loop. 2.3
Intent annotations
Version tracking catches file-level conflicts but not semantic ones. Two agents might implement the same helper with different signatures, or make incompatible assumptions about a shared data structure. To reduce this, engineers annotate their code with structured intent comments: 3
Preprint, May 2026
Parallel Engineer Agents
Manager Agent
Manager Agent
Work on the same shared workspace concurrently
Engineer 2
Engineer 1
Engineer N
Task Delegation
Results
… Implement Task
Implement Task
Implement Task
Read / Write via STORM
Read / Write via STORM
Read / Write via STORM
l Scan repository
l Review changes
l Create plan
l Integrate
l Assign tasks
l Commit
l Review & commit Read / Write Request
Read / Write Request
Response
Read / Write Request
Response
Response
STORM Manager Versioning
Validation (Write) Reservation Check
On Success Version Check (Direct) expected
Each file has a monotonic version (v1, v2, v3, …)
Agent’s Snapshot (at read time)
current
Current Versions (at write time)
Check if the file is reserved by another agent (up to 30s) Reservation = exclusive lock to prevent concurrent writes.
Compare expected_version (with agent) with current_version (on disk). Mismatch = CONFLICT.
Check all files in the agent’s snapshot (dependencies). If any version changed by others → STALE DEPENDENCY
Shared Workspace (Same for All Agents) utils.py def add(a, b): return a + b ...
v12
core.py
On Failure
Snapshot Validation (Transitive)
models.py
v8
class Engine: def run(self): ... ...
Write to workspace, bump version (v+1). Return success.
Return to agent: • Target file current content + diff • Stale dependency list (path, version, last modifier) Agent resolves and retries with the latest info.
Read / Write Request (file, expected_version, snapshot) v6
class Model: def predict(self): ... ...
…
Response (content, version, diff or latest content) File / Version Info
Figure 1: System architecture. The manager analyzes the repository, delegates tasks to parallel engineers, and commits their work. All engineers share one workspace; the STORM manager mediates file operations to detect and resolve conflicts.
# engineer_1: validate numeric inputs before summing def add(a, b): if not isinstance(a, (int, float)): raise TypeError("a must be numeric") return a + b Each comment identifies the author and describes what the block accomplishes. Engineers preserve annotations left by other agents unless their task requires changing the annotated block. When another agent reads the file, these comments provide a lightweight channel for semantic coordination: agents can see what others have done and avoid duplicate or conflicting work. The convention is injected into each engineer’s system prompt automatically.
3
Experiments
We evaluate on two agent benchmarks: Commit0-Lite(Zhao et al., 2025), and PaperBench(Starace et al., 2025). We compare five configurations across three LLMs (Claude Sonnet 4.6, Qwen 3.6 Plus, DeepSeek V4 Pro): (1)Single-agent with 100 iterations; (2)GitWorktree(Geng and Neubig, 2026), engineers in isolated worktrees merged after completion; (3)STORM, a manager with engineers sharing one workspace; and Combined variants that take the per-task best of single-agent and multiagent runs. Commit0-Lite uses 4 engineers; PaperBench uses 2 engineers with 2 rounds of delegation each. Evaluation. For Commit0-Lite, we run pytest on the final workspace and report Scorew (total tests passed / total tests) and Score (mean per-repository pass rate). For PaperBench, we use the Code-Dev subset following Geng and Neubig (2026) due to cost constraints: the LLM judge (Sonnet 4.6) grades only “Code Development” nodes in the rubric tree, evaluating whether the submitted source code correctly implements each criterion without requiring experiment execution. We report Score (mean per-paper judge score ×100). For both benchmarks, we report Costeff (total cost / Score, lower is better) and Timeeff (total wall-clock minutes across all tasks / Score, lower is better). 4
Preprint, May 2026
Table 1: Aggregated results across Commit0-Lite and PaperBench Code-Dev. Best per model in bold, second best underlined. Commit0-Lite Claude Sonnet 4.6 Single-Agent GitWorktree GitWorktree-Combined STORM STORM-Combined Qwen 3.6 Plus Single-Agent GitWorktree GitWorktree-Combined STORM STORM-Combined DeepSeek V4 Pro Single-Agent GitWorktree GitWorktree-Combined STORM STORM-Combined
PaperBench
Scorew ↑
Score ↑
Costeff ↓
Timeeff ↓
Score ↑
Costeff ↓
Timeeff ↓
20.7 24.6 31.4 46.2 49.2
66.4 63.8 78.6 82.5 87.6
3.2 8.6 8.1 6.3 6.3
18.8 20.9 25.7 16.8 21.1
68.7 72.7 76.6 74.1 78.2
12.5 17.2 22.7 12.6 17.3
11.6 22.5 30.8 24.1 31.4
34.0 16.7 36.3 61.4 76.2
75.3 57.4 83.0 70.5 88.2
1.3 6.9 3.7 2.5 2.1
13.0 56.0 31.1 12.5 13.2
47.7 51.6 55.4 55.0 57.0
4.9 13.4 12.1 8.2 10.7
16.1 36.0 43.1 21.0 29.6
26.8 18.5 30.9 32.3 41.3
65.2 44.0 75.2 63.2 77.7
1.8 3.8 3.8 3.0 3.4
31.3 45.1 53.9 39.0 49.0
62.9 55.8 60.8 66.5 68.3
4.1 8.8 9.1 9.7 10.9
16.5 35.7 41.3 35.5 41.7
Implementation. All agents run on OpenHands (Wang et al., 2025). Multi-agent runs follow a two-round protocol: in the first round the manager decomposes the task and dispatches engineers in parallel (each with 80 iterations); in the second round it reviews outputs, runs tests, and may reassign failed sub-tasks for one retry. The manager gets 50 iterations total. Detailed setup is in Appendix A. 3.1
Main results
Table 1 reports results on both Commit0-Lite (4 engineers) and PaperBench Code-Dev (2 engineers) across all three models. STORM achieves the highest weighted score on Commit0-Lite across all models (46.2, 61.4, 32.3 for Sonnet, Qwen, DeepSeek), with gains concentrated on large repositories with cross-file dependencies where shared-workspace coordination matters most. GitWorktree performs worst across the board, dropping up to 18 points below single-agent on Commit0-Lite. On PaperBench, STORM consistently outperforms both single-agent and GitWorktree for all three models: 74.1 vs. 68.7 single-agent (Sonnet), 55.0 vs. 47.7 (Qwen), and 66.5 vs. 62.9 (DeepSeek), demonstrating that the manager’s task decomposition and priority-driven delegation effectively covers more rubric criteria than a single agent working alone. The single agent wins on cost-efficiency across all models due to zero coordination overhead, but STORM closes this gap while achieving substantially higher scores. STORM-Combined (perpaper best of single-agent and STORM) is the top-scoring configuration for every model on both benchmarks, reaching 78.2 on PaperBench (Sonnet), 57.0 (Qwen), and 68.3 (DeepSeek), confirming that the two approaches are complementary: single-agent excels on papers where one focused agent can cover most criteria, while STORM’s parallel delegation captures broader coverage on complex multi-component papers. STORM also achieves better time-efficiency than GitWorktree on all models (e.g., 16.8 vs. 20.9 on Sonnet Commit0; 21.0 vs. 36.0 on Qwen PaperBench) because conflicts are resolved incrementally rather than in a costly merge step. Per-paper results are in Appendix C. 3.2
Scaling to More Engineers
A common finding in prior multi-agent systems is that performance degrades as more agents are added (Geng and Neubig, 2026; Lin, 2026). The reason is straightforward: for a fixed-size task, splitting work among more agents means each agent’s sub-task becomes smaller, but the file coupling between sub-tasks increases. More agents need to touch shared interfaces, read overlapping files, and make mutually consistent design choices. Under worktree isolation, this coupling manifests as merge conflicts that grow combinatorially with the number of branches. STORM inherenetly avoid it 5
Preprint, May 2026
Figure 2: Scaling engineers with Sonnet 4.6 on Commit0-Lite. (a) Both test-based and repo-based scores improve monotonically from 2 to 8 max engineers, and the line shows the average number actually deployed. (b) Cost scales linearly with engineer count. (c) Wall-clock time remains roughly constant due to parallel execution. given conflicts are detected and resolved at write time, higher coupling leads to more frequent but individually cheap rejections rather than a catastrophic merge failure at the end. Each conflict is resolved in isolation while the agent’s reasoning context is still fresh, so scaling up agents does not increase the difficulty of final integration. Figure 2 shows the effect of increasing the maximum number of engineers from 2 to 8 on Sonnet 4.6 with STORM. Both the test-based score (overall pass rate) and the repo-based score (macro-average per-repository pass rate) improve: from 38.2% to 69.7% (+31.5) and 71.3% to 87.1% (+15.8), respectively. The gains are not uniform across the two transitions. Moving from 2 to 4 engineers yields +12.0 macro points but only +8.2 overall points, because the improvement concentrates on medium-sized repositories (cookiecutter 40.9→98.6, imapclient 16.5→89.1, jinja 0.0→47.7). Moving from 4 to 8 engineers yields a smaller macro gain (+3.8) but a much larger overall gain (+23.3), driven almost entirely by babel (20.2→57.5) and jinja (47.7→66.5), repositories with thousands of tests and deep cross-file dependencies that benefit from aggressive parallelization. Notably, the manager does not always use all available engineers. The average number of engineers actually deployed is 2.0, 3.6, and 6.4 for max settings of 2, 4, and 8 respectively. Simple repositories (cachetools, deprecated, wcwidth) consistently receive only 2 engineers regardless of the maximum, avoiding unnecessary coordination overhead. The number of repositories solved at ≥99% pass rate grows from 7 (max=2) to 8 (max=4) to 10 (max=8) out of 16. Cost scales approximately linearly with the number of engineers ($199→$292→$429), while wallclock time remains roughly constant (∼13 hours total across all 16 repositories) because engineer tasks execute in parallel. The cost-efficiency ratio (dollars per percentage point of overall score) is stable at $5–6 per point across all configurations, indicating that additional engineers provide proportional value rather than diminishing returns at this scale. 3.3
Isolation Strategy Table 2: Effect of isolation strategy on Commit0-Lite with Claude Sonnet 4.6. Method Single-Agent Soft Isolation GitWorktree STORM
Scorew ↑
Score ↑
Costeff ↓
Timeeff ↓
20.7 24.0 24.6 46.2
66.4 65.4 63.8 82.5
3.2 7.7 8.6 6.3
18.8 29.1 20.9 16.8
In Table 2, we compare isolation strategies on Commit0-Lite with Claude Sonnet 4.6 using 4 engineers. Prompt based soft isolation and git worktree isolation achieve similar results (Scorew 24.0 vs. 24.6, Score 65.4 vs. 63.8), both improving over the single-agent baseline in weighted score (20.7) while 6
Preprint, May 2026
remaining comparable in unweighted score. This suggests that delegation alone provides gains on larger repositories where multiple agents can cover more test cases in parallel, but the choice between instruction-level constraints and physical branch separation has limited impact on final pass rate. STORM outperforms both isolation strategies (Scorew 46.2, Score 82.5) despite using the same number of engineers. First, STORM detects conflicts at write time rather than deferring them: when an edit violates local state consistency (Eq. 3), it is rejected immediately, letting the agent re-plan while its reasoning context is still fresh. Soft isolation instead relies on instruction-level constraints that are frequently violated, causing silent overwrites, while worktree isolation defers conflicts to a merge step where multi-file resolution often fails. Second, because all engineers share a single workspace, any read returns the latest committed state, including other engineers’ recent changes. Agents working on adjacent modules naturally observe each other’s implementations and can adapt interfaces accordingly, whereas worktree isolation keeps engineers blind to concurrent progress until merge time, by which point incompatible decisions may have already propagated. (b) Pass rate by coupling stratum
4
Pre-commit conflicts Post-commit conflicts Pass rate
2
40
70
81.6
77.7 70.9
70 59.5
60 50 40
0.94
1
Pass rate (%)
60
80 Pass rate (%)
2.94
64.8
(c) Coupling grows with parallelism
STORM
94.4 89.0
82.1
80 3
GitWorktree
97.7
90
83.4
82.2
Conflict events / run
Single-agent
100
100
36.3
20 30
Share of tasks affected (%)
(a) Late-caught conflicts discard STORM runs
Overlap Dependency
60 50 40 30 20 10
0.12
0
GitWorktree
STORM
STORM (k=8)
0
20
Low
Medium Repo coupling stratum
High
0
2 agents 4 agents 8 agents Number of parallel agents
Figure 3: Analysis summary on Sonnet 4.6. (a) STORM surfaces conflicts pre-commit, while GitWorktree leaves most conflicts for late (post-commit/merge) resolution; late-caught conflicts are associated with lower final pass rates (blue overlay). (b) STORM’s pass-rate advantage over the single-agent and GitWorktree baselines grows with cross-file coupling. (c) Narrow task scopes are not always independent scopes: first-round overlap and dependency signals remain common, and rise with k. Quantities in (a) and (c) are proxy measurements derived from manager-review events and declared task scopes. 3.4
Further Analysis
STORM helps by exposing invalid parallel work early. STORM’s benefit is converting hidden integration errors into explicit write-time feedback. As Figure 3(a) shows, on Sonnet 4.6 GitWorktree leaves most conflict events until post-commit merge while STORM surfaces them pre-commit at both k=4 and k=8, and the configurations that catch conflicts early also achieve higher final pass rates. The effect is not that STORM eliminates conflicts, but that it relocates them from a fragile post-hoc merge to a cheap write-time check. GitWorktree produces 3.81 reviewed write attempts per run at 91.8% acceptance, versus 6.00 at 81.2% for STORM (k=4) and 10.25 at 67.1% for STORM (k=8); the lower acceptance reflects a stricter consistency filter rather than weaker engineering, and STORM still outperforms GitWorktree in final repository quality. STORM remains robust on high-coupling repositories where alternatives collapse. STORM’s advantage is sharpest where parallel coordination matters most. Stratifying repositories by a proxy coupling score (from task overlap, dependency signals, multi-file scope, and rejection evidence), STORM’s lead over GitWorktree widens from +15.6 points on low-coupling and +5.4 on mediumcoupling subsets to +34.6 points on the high-coupling stratum (Figure 3(b)), because GitWorktree collapses outright there (36.3% vs. STORM 70.9%). STORM’s margin over the single-agent baseline remains positive across all strata (+20.0, +12.8, and +11.4 points from low to high); it narrows in the high-coupling regime only because the absolute ceiling falls for every method, not because STORM loses its edge. Concretely, as coupling scales from low to high, STORM degrades gracefully (97.7% → 94.4% → 70.9%), whereas single-agent falls from 77.7% to 59.5% and GitWorktree plummets from 82.1% and 89.0% to 36.3%: STORM is the only method that does not break down under coupling. The repository-level pattern agrees: marshmallow rises 0.0% → 82.3% and imapclient 9.7% → 89.1% (Table 3), precisely where branch-level isolation is most brittle. As a 7
Preprint, May 2026
result, STORM more than doubles the Sonnet weighted score of the single agent (46.2 vs. 20.7) and nearly doubles that of GitWorktree (46.2 vs. 24.6). Scaling is limited more by decomposition quality than by STORM itself. The diminishing returns of additional engineers come from decomposition, not STORM. As Figure 3(c) illustrates, STORM already produces narrow scopes: 91.4% of k=4 tasks are single-file, yet 50.0% still carry dependency signals and 21.7% of first-round tasks overlap another’s file scope. At k=8, single-file share stays high (85.1%) while first-round overlap rises to 35.1%. First-round overlap correlates with rejected-review rate (r = 0.28 overall, r = 0.78 within k=8): once the manager can no longer partition the repository into disjoint units, additional engineers create contention faster than useful parallel work. Case study: pre-hoc coordination versus post-hoc recovery. A paired run on jinja makes the aggregate pattern in Figure 3(a) concrete at the level of a single repository. Under GitWorktree (Figure 4a), four engineers work on isolated branches and the coupling around utils.py stays invisible until merge: the manager rejects the focus diff with an explicit merge conflict reason (red diamond) and the same engineer reworks the task in a second round inside the shaded recovery window before acceptance. Under STORM (Figure 4b, restricted to the coupling task set), the manager reasons about the same coupling at decomposition and instruction time: it packs utils.py and async_utils.py into a single assignment and explicitly sequences downstream consumers against it (gold-outlined bars carry this interdependence reasoning); the focus task then passes on the first review and no rejection appears in the coupled task set. The two timelines describe the same mechanism from opposite sides, GitWorktree lets coupling surface post hoc at the merge boundary, while STORM converts the identical signal into a pre hoc coordination decision baked into how tasks are defined and ordered. In summary, the analysis suggests a clear interpretation of STORM. Its main contribution is not cheaper computation or perfect recovery from every rejected edit. Its contribution is to replace delayed, fragile integration with immediate consistency checks that make parallel collaboration substantially more reliable on the repositories where parallelism is most useful. The residual failure modes of STORM are further analyzed in Appendix D.
4
Related Work
Multi-Agent Collaboration Multi-agent collaboration decomposes complex tasks into specialized roles (Dong et al., 2024; Qian et al., 2024; Hong et al., 2024; Wu et al., 2023). Compared with a singleagent workflow, a multi-agent system can separate planning, implementation, testing, debugging, and review (Tao et al., 2024; Li et al., 2025; Kumar et al., 2026). Existing work mainly studies role-based software development, repository-level issue resolution, and workspace-isolated parallel development. Self-Collaboration (Dong et al., 2024) first introduced the multi-agent framework for code generation (?). Subsequently, ChatDev and MetaGPT model software development as a structured workflow among specialized agents (Qian et al., 2024; Hong et al., 2024). MAGIS and SWE-Debate study multi-agent issue resolution on SWE-style benchmarks (Tao et al., 2024; Li et al., 2025). AgentForge adds execution feedback through Docker-based validation (Kumar et al., 2026). Git worktree is a common mechanism for workspace-isolated parallel agents. Systems such as Git Worktree, CORAL, and StatsClaw use git worktree to separate agent workspaces (Geng and Neubig, 2026; Qu et al., 2026; Qin and Xu, 2026). Each agent works in a separate branch or worktree, while shared memory or a central coordinator supports information exchange. Workspace isolation enables parallel exploration and reduces direct interference between agents’ code changes. However, git worktree only provides low-level workspace isolation. It does not solve task decomposition, dependency tracking, semantic conflicts, or merge selection. Different agents may edit related files under incompatible assumptions. Some errors may only appear after integration. Industry systems also report that optimistic concurrency control improves over lock-based shared-state coordination (Lin, 2026), but remains insufficient without hierarchical task decomposition. Therefore, worktree-based systems still require higher-level coordination, shared memory, review, and execution-based verification. State Management in Agentic Systems Agentic systems require persistent state across multi-step interaction. The state may include conversation history, plans, tool outputs, execution feedback, 8
Preprint, May 2026
Assignment
Coupling-aware assignment
Engineer work
Focus task
Rejected review
15 20 25 Minutes Since First Relevant Event
30
Accepted review
Manager E1
Focus R1
E2
R1
R2
E3
R1
E4
R1
0
R2
5
10
35
40
(a) GitWorktree: post-hoc detection. Manager E2
Focus R1
E1
R1
E3
R1
E4
R1
0
5
10 15 Minutes Since First Relevant Event
20
(b) STORM: pre-hoc coordination.
Figure 4: Paired run timelines on jinja; the legend in (a) applies to both panels. (a) GitWorktree exposes the utils.py coupling only at merge review: the red diamond marks the rejected review, the shaded band the rework window, and Focus R2 the retry that is eventually accepted. (b) STORM detects the same coupling at decomposition time and co-assigns or sequences the dependent tasks (gold-outlined manager instructions carry explicit interdependence reasoning); the focus task passes on its first review and no rejection remains within the coupled task set.
repository changes, and intermediate artifacts. Existing work manages agent state through reflection, memory, skill libraries, and execution traces (Shinn et al., 2023; Park et al., 2023; Packer et al., 2023; Wang et al., 2024; Hu et al., 2025). Reflexion and Generative Agents use reflection and episodic memory to improve later decisions (Shinn et al., 2023; Park et al., 2023). MemGPT manages long-term context through memory tiers (Packer et al., 2023). Voyager stores reusable skills as executable code, while HiAgent manages working memory with subgoals (Wang et al., 2024; Hu et al., 2025). However, state management remains difficult in multi-agent software development. Most existing methods focus on a single agent or a single memory store. Multi-agent systems must also manage local states, shared states, workspace states, and execution states. Git Worktree uses isolated workspaces and branch-and-merge coordination for asynchronous software engineering agents (Geng and Neubig, 2026). CORAL and StatsClaw use shared memory and isolated worktrees to support parallel execution (Qu et al., 2026; Qin and Xu, 2026). CodeCRDT instead coordinates agents through observable shared state with deterministic convergence (Pugachev, 2025). However, workspace isolation and shared memory do not guarantee state consistency, dependency tracking, or safe integration. Reliable multi-agent software development still requires explicit control over what state is stored, shared, updated, and used.
5
Conclusion
We presented STORM, a state management framework that replaces workspace isolation with local state consistency for multi-agent collaboration. All agents share one workspace; writes are accepted only if the agent’s observed files remain unchanged, and intent annotations coordinate semantics 9
Preprint, May 2026
at shared boundaries. On Commit0-Lite, STORM achieves 82.5% macro / 46.2% weighted pass rate with Sonnet 4.6 (vs. 66.4% / 20.7% single-agent, 63.8% / 24.6% GitWorktree). On PaperBench Code-Dev, STORM scores 74.1 vs. 72.7 and 68.7. Results generalize across Qwen and DeepSeek, with STORM-Combined reaching 88.2 / 76.2 on Qwen. Scaling to 8 engineers improves score monotonically with constant wall-clock time. These results suggest that explicit state management is a more effective foundation for multi-agent collaboration than workspace isolation.
References Cortices. Mission control for ai agents. https://cortices.io/. Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. Self-collaboration code generation via chatgpt. ACM Trans. Softw. Eng. Methodol., 33(7):189:1–189:38, 2024. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Communicative agents for software development. In ACL (1), pages 15174–15186. Association for Computational Linguistics, 2024. Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collaborative framework. In ICLR. OpenReview.net, 2024. Jiayi Geng and Graham Neubig. Effective strategies for asynchronous software engineering agents, 2026. URL https://arxiv.org/abs/2603.21489. Ao Qu, Han Zheng, Zijian Zhou, Yihao Yan, Yihong Tang, Shao Yong Ong, Fenglu Hong, Kaichen Zhou, Chonghe Jiang, Minwei Kong, Jiacheng Zhu, Xuan Jiang, Sirui Li, Cathy Wu, Bryan Kian Hsiang Low, Jinhua Zhao, and Paul Pu Liang. Coral: Towards autonomous multi-agent evolution for open-ended discovery, 2026. URL https://arxiv.org/abs/2604.01658. Tianzhu Qin and Yiqing Xu. Statsclaw: An ai-collaborative workflow for statistical software development, 2026. URL https://arxiv.org/abs/2604.04871. Wenting Zhao, Nan Jiang, Celine Lee, Justin T. Chiu, Claire Cardie, Matthias Gallé, and Alexander M. Rush. Commit0: Library generation from scratch. In ICLR. OpenReview.net, 2025. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai’s ability to replicate AI research. In ICML, Proceedings of Machine Learning Research. PMLR / OpenReview.net, 2025. H. T. Kung and John T. Robinson. On optimistic methods for concurrency control. ACM Trans. Database Syst., 6(2):213–226, 1981. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, and et al. Openhands: An open platform for AI software developers as generalist agents. In ICLR. OpenReview.net, 2025. Wilson Lin. Scaling long-running autonomous coding. Cursor Blog, January 2026. URL https: //cursor.com/blog/scaling-agents. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversation framework. CoRR, abs/2308.08155, 2023. Wei Tao, Yucheng Zhou, Yanlin Wang, Wenqiang Zhang, Hongyu Zhang, and Yu Cheng. MAGIS: llm-based multi-agent framework for github issue resolution. In NeurIPS, 2024. 10
Preprint, May 2026
Han Li, Yuling Shi, Shaoxin Lin, Xiaodong Gu, Heng Lian, Xin Wang, Yantao Jia, Tao Huang, and Qianxiang Wang. Swe-debate: Competitive multi-agent debate for software issue resolution. CoRR, abs/2507.23348, 2025. Rajesh Kumar, Waqar Ali, Junaid Ahmed, Najma Imtiaz Ali, and Shaban Usman. Agentforge: Execution-grounded multi-agent llm framework for autonomous software engineering, 2026. URL https://arxiv.org/abs/2604.13120. Yihong Dong, Jiaru Qian, Haoran Zhang, Peixu Wang, Binhua Li, Zhi Jin, Yongbin Li, Ge Li, Xiaokang Yang, and Xue Jiang. From i/o to code with discovery agent, 2026. URL https: //arxiv.org/abs/2605.15334. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: language agents with verbal reinforcement learning. In NeurIPS, 2023. Joon Sung Park, Joseph C. O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. In UIST, pages 2:1–2:22. ACM, 2023. Charles Packer, Vivian Fang, Shishir G. Patil, Kevin Lin, Sarah Wooders, and Joseph E. Gonzalez. Memgpt: Towards llms as operating systems. CoRR, abs/2310.08560, 2023. Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. Trans. Mach. Learn. Res., 2024, 2024. Mengkang Hu, Tianxing Chen, Qiguang Chen, Yao Mu, Wenqi Shao, and Ping Luo. Hiagent: Hierarchical working memory management for solving long-horizon agent tasks with large language model. In ACL (1), pages 32779–32798. Association for Computational Linguistics, 2025. Sergey Pugachev. Codecrdt: Observation-driven coordination for multi-agent llm code generation, 2025. URL https://arxiv.org/abs/2510.18893. Xue Jiang, Yihong Dong, Lecheng Wang, Zheng Fang, Qiwei Shang, Ge Li, Zhi Jin, and Wenpin Jiao. Self-planning code generation with large language models. ACM Trans. Softw. Eng. Methodol., 33 (7):182:1–182:30, 2024. Xue Jiang, Yihong Dong, Yongding Tao, Huanyu Liu, Zhi Jin, and Ge Li. ROCODE: integrating backtracking mechanism and program analysis in large language models for code generation. In ICSE, pages 334–346. IEEE, 2025. Yingwei Ma, Yongbin Li, Yihong Dong, Xue Jiang, Yanhao Li, Yue Liu, Rongyu Cao, Jue Chen, Fei Huang, and Binhua Li. Thinking longer, not larger: Enhancing software engineering agents via scaling test-time compute. In ASE, pages 3730–3741. IEEE, 2025. Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. Repocoder: Repository-level code completion through iterative retrieval and generation. In EMNLP, pages 2471–2484. Association for Computational Linguistics, 2023. Disha Shrivastava, Denis Kocetkov, Harm de Vries, Dzmitry Bahdanau, and Torsten Scholak. Repofusion: Training code models to understand your repository. CoRR, abs/2306.10998, 2023. Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In NeurIPS, 2023. Jia Li, Ge Li, Yunfei Zhao, Yongmin Li, Huanyu Liu, Hao Zhu, Lecheng Wang, Kaibo Liu, Zheng Fang, Lanshen Wang, Jiazheng Ding, Xuanming Zhang, Yuqi Zhu, Yihong Dong, Zhi Jin, Binhua Li, Fei Huang, Yongbin Li, Bin Gu, and Mengfei Yang. Deveval: A manually-annotated code generation benchmark aligned with real-world code repositories. In ACL (Findings), Findings of ACL, pages 3603–3614. Association for Computational Linguistics, 2024. 11
Preprint, May 2026
Xue Jiang, Tianyu Zhang, Ge Li, Mengyang Liu, Taozhi Chen, Zhenhua Xu, Binhua Li, Wenpin Jiao, Zhi Jin, Yongbin Li, and Yihong Dong. Think anywhere in code generation. CoRR, abs/2603.29957, 2026. Mofei Li, Taozhi Chen, Guowei Yang, and Jia Li. Memcoder: Multi-dimensional evolving memory for private-library-oriented code generation, 2026. URL https://arxiv.org/abs/2604. 24222. Yihong Dong, Jiazheng Ding, Xue Jiang, Ge Li, Zhuo Li, and Zhi Jin. Codescore: Evaluating code generation by learning code execution. ACM Trans. Softw. Eng. Methodol., 34(3):77:1–77:22, 2025.
12
Preprint, May 2026
A
Detailed Experiment Setup
We evaluate on Commit0 (Zhao et al., 2025), a repository-level code implementation benchmark. Each instance is a real Python repository with its test suite intact but source implementations removed (replaced with stubs or pass statements). The task is to implement the missing code so that the tests pass. We use the Commit0 Lite subset: 16 repositories spanning a range of domains and sizes. Several repositories require cross-file reasoning, making them difficult for a single agent and a natural fit for studying multi-agent collaboration. We also evaluate on PaperBench (Starace et al., 2025), a benchmark for evaluating AI agents on end-to-end research replication. Each instance is based on an ICML 2024 Spotlight or Oral paper. The task is to understand the paper, implement the required codebase from scratch, run experiments, and reproduce the key results. PaperBench uses hierarchical rubrics to evaluate fine-grained replication progress rather than only final execution success. These tasks require long-horizon planning, cross-file implementation, experiment execution, and result verification, making them a natural fit for studying multi-agent collaboration. Configurations. We compare five configurations across three LLMs (Claude Sonnet 4.6, Qwen 3.6 Plus, DeepSeek V4 Pro): (1) Single-agent with 100 LLM iterations; (2) GitWorktree (Geng and Neubig, 2026) with 4 engineers in isolated worktrees, merged after completion; (3) STORM with a manager and 4 engineers sharing one workspace; (4) GitWorktree-Combined; and (5) STORMCombined. The Combined variants run both single-agent and multi-agent, keeping the per-test union of passed tests, but only invoking multi-agent on repositories where single-agent did not already achieve 100%. Implementation. All agents run on the OpenHands SDK (Wang et al., 2025) inside Docker containers. The manager gets up to 50 LLM iterations; each engineer gets up to 80 per task and can be reassigned once. During analysis, the manager recommends how many engineers to use based on repository complexity; for simple repositories it may use as few as 2. Evaluation uses pytest with --json-report; if the JSON report is unavailable we fall back to parsing terminal output. The total number of tests per repository is taken from the ground-truth counts in the Commit0 paper (Zhao et al., 2025). Metrics. We report Scorew (weighted pass rate: total tests passed / total tests, dominated by large repositories) and Score (macro: mean of per-repository pass rates). For efficiency we report Costeff (total dollars / Scorew ) and Timeeff (total minutes / Scorew ), both lower-is-better. PaperBench. We use the PaperBench Code-Dev subset, which evaluates code development only and skips the reproduction execution step. The judge grades only “Code Development” nodes in the rubric. We use Sonnet 4.6 as the LLM judge. Due to the high per-run cost (20 papers per configuration), we report PaperBench results on Sonnet 4.6 agents.
B
Prompt templates
We summarize the key prompts used in the manager-engineer protocol. The full prompt text is available in the released code. Manager scan instruction (excerpt): Start by exploring the repository structure to understand what your team needs to work on and identify which functions have top priority to be implemented (i.e., those with pass statements). To explore the repository structure and dependencies: 1. Check the imports and the actual functions used in the files with pass statements and review the relevant tests to understand the EXPECTED BEHAVIOR of the functions and the dependencies between the files. 2. Run pytest -collect-only -continue-on-collection-errors to have a better understanding of the scope and the dependencies of this codebase. Collect all the undefined functions from the test collection errors and add them with clear docstrings and pass statements into the files and make a local commit. DO NOT make any changes to the functions with pass statements since those need to be implemented by the engineers.
13
Preprint, May 2026
Manager delegation. Manager delegation instruction (excerpt): Split the overall implementation work into up to k major tasks, balancing complexity and estimated effort as evenly as possible. Make sure the high dependent files are in the same major task. Try to split the major tasks at file level first; if a single file contains a disproportionately large amount of functions with pass statements, you can delegate at the function level and assign non-overlapping sets of functions to multiple engineers. For each engineer, assign the first task that has the highest priority within their major task. When you provide instruction to each engineer, briefly summarize the relevant repository structure and the dependencies so they don’t need to re-explore the repository. Then clearly specify which file/functions to implement and explain the purpose of this task. If the assigned functions depend on other stub functions, include a brief description of what each dependency does.
Engineer task prompt. Each engineer receives the repository structure summary, its assigned functions, dependency descriptions, the test command, and shared-workspace rules. The key constraint is shown below: Engineer task prompt (excerpt): You are a software engineer working on implementing a python code repository in a group. You are responsible for implementing the functions instructed by your manager (i.e., the functions with pass statements) and passing the unit tests. Shared Workspace: Your workspace is {worktree_path}. ALL engineers on your team share this SAME directory. DO NOT modify files that belong to other engineers. Only edit the specific files and functions assigned to you. Check before modifying any file to make sure another engineer hasn’t already changed it. Scope Discipline: You are ONLY responsible for implementing the functions listed below. If you see test failures caused by functions in OTHER files or functions NOT assigned to you, DO NOT attempt to fix them. Report the failure to your manager and move on. Multi-agent coordination: Every edit you make MUST include a one-line comment in the form # {engineer_id}: <short intent> IMMEDIATELY above the block you added or changed. If you see # <other-engineer-id>: ... comments, preserve both the comment and the block below it unless your task explicitly requires changing them. You are assigned to implement the following functions in the file: {file_path}: {functions}. After you finished the implementation, make sure it will not cause any hanging issues. Do NOT commit; the manager will review your changes and commit on your behalf.
Manager review and reassignment. After each round, the manager checks commit status. For successful commits, it assigns the next task. For failed commits, it reassigns the same task. In the final review, the manager merges all engineers’ work, resolves integration issues (import mismatches, naming inconsistencies), and checks for hanging code (infinite loops, input() calls). Manager delegation output format (excerpt): { "engineer_id": "engineer_1", "file_path": "path/to/file.py", "functions_to_implement": ["func1", "func2"], "instruction": "Implement the tensor storage layout. The TensorData class uses ..." }
C
Full experimental results
Tables 3–6 report per-instance results for all models. We highlight several patterns. 14
Preprint, May 2026
Table 3: Per-repository pass rate, cost, and time (seconds) for Claude Sonnet 4.6. Best pass rate per row in bold. GitWorktree (4 Eng)
STORM (4 Eng)
Repository
Rate
Single-Agent Cost
Time
Rate
Cost
Time
Rate
Cost
Time
babel cachetools chardet cookiecutter deprecated imapclient jinja marshmallow minitorch parsel portalocker pyjwt simpy tinydb voluptuous wcwidth
0.6 100.0 99.7 74.9 100.0 9.7 0.0 0.0 70.4 100.0 100.0 100.0 58.6 99.5 49.7 100.0
4.3 0.7 7.0 7.2 1.1 2.6 4.7 5.9 6.8 6.3 7.2 3.5 0.5 2.5 4.2 1.5
934 249 768 1958 698 3141 665 1150 2093 1568 2273 716 2109 940 3621 442
1.3 100.0 0.3 94.3 94.7 37.8 5.8 60.9 53.0 93.2 100.0 5.8 78.6 99.5 96.0 100.0
30.8 2.6 3.2 24.2 1.4 6.3 29.2 31.7 16.1 4.3 8.8 2.2 6.2 10.3 32.3 2.4
3301 657 817 2548 528 1698 3332 3162 2337 1202 1741 639 1627 1353 4928 956
20.2 100.0 22.1 98.6 100.0 89.1 47.1 82.3 70.4 100.0 100.0 100.0 100.0 99.5 90.6 100.0
7.9 1.5 18.7 24.8 3.4 24.9 39.3 32.8 35.9 11.0 11.1 18.6 21.9 11.1 25.9 2.9
1678 400 2932 5234 898 3736 3733 3896 4439 1974 1727 1783 2655 1474 9181 869
Table 4: Per-repository pass rate, cost, and time (seconds) for Qwen 3.6 Plus. Best pass rate per row in bold. Single-Agent
GitWorktree (4 Eng)
STORM (4 Eng)
Repository
Rate
Cost
Time
Rate
Cost
Time
Rate
Cost
Time
babel cachetools chardet cookiecutter deprecated imapclient jinja marshmallow minitorch parsel portalocker pyjwt simpy tinydb voluptuous wcwidth
3.6 100.0 99.7 38.7 100.0 100.0 98.8 28.5 33.9 100.0 83.3 97.3 76.4 88.1 55.7 100.0
3.7 0.6 3.0 3.0 1.0 2.3 0.6 3.9 4.3 1.4 0.7 4.3 3.8 3.0 4.5 3.9
1709 487 1159 1485 1135 1193 366 2714 1664 591 3623 1866 3132 1365 1346 2599
0.2 100.0 99.7 36.5 100.0 0.0 0.0 0.0 52.2 6.3 100.0 89.2 51.4 99.5 83.2 100.0
2.0 3.2 3.2 8.6 3.8 12.1 2.6 15.1 8.0 1.1 4.0 14.8 12.8 9.4 10.9 4.5
557 966 966 4234 2369 4801 744 3106 2463 546 8099 6453 8926 2940 2491 6393
74.2 100.0 38.8 56.4 100.0 44.6 0.0 23.7 30.4 82.0 100.0 100.0 81.4 96.0 100.0 100.0
30.1 2.2 12.1 9.2 2.1 19.2 2.6 7.7 5.4 10.7 5.9 13.9 8.7 7.0 12.2 4.0
5502 927 2350 3733 1124 4441 744 2386 1551 3239 4074 3814 3528 3550 2685 2544
STORM excels on cross-file repositories. The largest gains over both baselines come from repositories with heavy inter-file dependencies. On Sonnet, marshmallow jumps from 0.0% (single) and 60.9% (GitWorktree) to 82.3% under STORM; imapclient from 9.7% / 37.8% to 89.1%; and jinja from 0.0% / 5.8% to 47.1%. On Qwen, babel shows the most dramatic improvement: 3.6% (single) and 0.2% (GitWorktree) to 74.2% under STORM, demonstrating that shared-workspace coordination is critical for large, tightly coupled codebases. Single-agent strengths. The single agent remains competitive on small, self-contained repositories (chardet, deprecated, parsel) where decomposition overhead outweighs parallelism benefits. On Sonnet, chardet scores 99.7% single vs. 22.1% STORM, showing that STORM’s decomposition can occasionally hurt when the repository is better solved monolithically. PaperBench patterns. On PaperBench Code-Dev (Table 6), STORM leads on 11 of 20 papers, GitWorktree on 6, and single-agent on 3. STORM’s largest wins come on papers requiring substantial 15
Preprint, May 2026
Table 5: Per-repository pass rate, cost, and time (seconds) for DeepSeek V4 Pro. Best pass rate per row in bold. GitWorktree (4 Eng)
STORM (4 Eng)
Repository
Rate
Single-Agent Cost
Time
Rate
Cost
Time
Rate
Cost
Time
babel cachetools chardet cookiecutter deprecated imapclient jinja marshmallow minitorch parsel portalocker pyjwt simpy tinydb voluptuous wcwidth
1.2 71.2 99.7 69.5 100.0 16.9 29.3 38.6 36.1 97.6 36.1 88.8 73.6 99.5 85.9 100.0
3.9 0.1 2.6 4.3 0.5 3.5 5.0 5.2 3.4 3.3 0.1 4.4 3.2 2.4 4.3 0.7
3078 1275 2044 3627 3308 3622 3347 3282 3636 2861 3621 3625 3625 2188 3621 3513
0.7 100.0 0.3 34.3 9.9 9.7 3.3 49.0 51.7 100.0 97.2 91.9 52.9 100.0 0.0 2.6
7.3 1.1 0.9 4.1 0.3 3.2 13.6 12.8 3.9 4.7 2.5 4.2 6.6 4.5 0.3
6153 2016 987 2927 399 4337 6531 4745 2613 3434 5894 2214 4507 3086 277
19.3 100.0 1.3 35.1 100.0 49.1 5.8 47.9 44.3 100.0 97.2 81.1 66.4 96.5 67.8 100.0
4.9 1.3 3.9 3.0 1.6 6.4 11.7 9.3 4.4 12.5 3.1 6.3 6.9 4.2 15.2 2.1
6344 1561 3824 2569 2148 6320 6778 5402 3018 8645 3708 6668 4990 4873 6271 2487
code organization (what-will-my-model-forget: 99.8 vs. 82.9 single, lbcs: 95.6 vs. 84.1 single). GitWorktree wins on papers where independent sub-tasks map cleanly to separate files (sample-specific-masks: 98.2 vs. 72.8 STORM), suggesting that when task boundaries align perfectly with file boundaries, isolation incurs no penalty. Table 6: Per-paper scores, cost, and wall-clock time for Claude Sonnet 4.6 on PaperBench Code-Dev. Best score per row in bold. Single-Agent (100 iter)
GitWorktree (2 Eng)
Paper
Score
Cost
Time
Score
Cost
Time
Score
Cost
Time
sample-specific-masks bam what-will-my-model-forget pinn sequential-neural-score-estimation mechanistic-understanding test-time-model-adaptation robust-clip fre stochastic-interpolants adaptive-pruning all-in-one stay-on-topic-with-classifier-free-guidance bbox lca-on-the-line ftrl rice bridging-data-gaps lbcs sapg
93.9 97.4 82.9 81.0 92.6 94.4 58.5 51.1 72.5 78.4 43.6 87.3 67.5 43.8 72.0 53.5 41.0 42.1 84.1 35.7
28.40 5.34 207.31 42.22 28.33 15.69 5.75 21.03 68.11 13.66 34.99 27.88 22.92 34.22 62.13 37.82 44.08 5.37 137.75 16.09
2721 1832 3601 3600 2060 2558 2084 1843 1438 1902 2863 2991 2651 2061 1791 1965 1934 1719 3369 2710
98.2 97.7 94.7 91.1 88.7 81.9 81.4 78.4 78.0 76.4 72.7 72.0 70.4 68.0 66.6 63.4 59.7 52.9 46.7 14.6
38.81 110.97 215.28 20.61 35.29 17.66 36.87 34.59 79.55 18.47 45.88 46.90 29.61 46.35 147.78 51.09 76.31 35.81 145.55 15.48
5555 7067 3149 6459 4412 4404 3694 3677 2971 2423 6724 5894 5604 3438 5730 5506 7122 5958 6729 1567
72.8 97.9 99.8 69.6 91.0 61.9 80.8 82.6 78.4 79.3 72.8 72.6 69.2 50.5 75.7 66.1 61.9 67.9 95.6 34.8
22.64 14.27 315.79 10.58 39.91 0.00 40.42 45.45 20.49 22.63 26.60 20.66 38.31 12.44 25.93 26.62 70.25 43.24 19.75 28.80
6156 3806 3937 5328 6699 6246 4666 4029 5448 4888 5688 6389 4933 5443 4899 5302 6730 4865 6491 6434
D
STORM (2 Eng)
Failure analysis details
The failure analysis clarifies the boundary of what file-level state management can and cannot guarantee, restricted to STORM runs at k=4 and k=8. Failed tests are categorized from pytest traceback symptoms into coarse buckets: assertion or semantic failures, missing API or symbol failures, type or contract errors, not-implemented errors, and other runtime failures (Figure 5, left). These categories are directly derived from tracebacks, but they remain symptom labels rather than manually adjudicated root causes. 16
Preprint, May 2026
Table 7: Per-paper scores, cost, and wall-clock time for Qwen3.6-Plus on PaperBench Code-Dev. Best score per row in bold. Single-Agent (100 iter)
GitWorktree (2 Eng)
Paper
Score
Cost
Time
Score
Cost
Time
Score
STORM (2 Eng) Cost
Time
pinn lbcs sample-specific-masks stochastic-interpolants what-will-my-model-forget mechanistic-understanding bridging-data-gaps lca-on-the-line sequential-neural-score-estimation stay-on-topic-with-classifier-free-guidance bam fre ftrl adaptive-pruning test-time-model-adaptation rice robust-clip all-in-one bbox sapg
72.5 43.3 78.7 74.7 27.4 70.6 30.0 52.1 67.8 58.8 76.7 31.2 25.2 23.4 54.4 33.4 13.5 59.5 21.4 39.3
1.64 2.66 2.35 1.63 1.84 2.36 1.88 2.90 3.26 2.05 2.72 1.59 2.31 3.21 2.66 2.29 2.03 2.84 1.96 2.60
1860 2138 2499 1553 2785 2536 2366 2712 3345 3007 1565 1326 2491 1912 3600 1708 1266 2614 1715 3220
77.4 77.2 75.9 71.7 69.2 65.0 62.9 59.2 56.6 50.6 50.5 50.4 48.7 43.1 40.6 36.7 30.5 28.1 20.1 18.3
5.99 17.72 4.89 4.79 4.17 6.39 7.84 17.85 2.82 6.43 5.40 5.27 11.43 5.85 4.15 5.17 5.04 5.33 5.65 5.63
4671 6564 4847 3313 5456 6377 6070 8140 8542 4746 4819 3624 6813 5563 4191 4094 4202 6073 6254 7000
81.2 88.5 68.7 76.8 55.2 84.4 50.2 42.9 71.6 52.9 76.9 53.0 43.7 31.6 64.7 22.4 29.7 47.1 42.4 15.2
4.92 2.90 3.20 2.47 3.39 3.86 5.96 5.08 3.42 3.13 9.87 6.73 7.18 4.28 7.11 3.96 4.15 2.60 2.41 3.42
6043 4631 4604 2525 2079 6889 3078 3180 2344 3420 6708 3556 3204 3885 3368 2147 2035 1480 1248 2967
Table 8: Per-paper scores, cost, and wall-clock time for DeepSeek-V4-Pro on PaperBench Code-Dev. Best score per row in bold. Single-Agent (100 iter)
GitWorktree (2 Eng)
STORM (2 Eng)
Paper
Score
Cost
Time
Score
Cost
Time
Score
Cost
Time
sample-specific-masks pinn what-will-my-model-forget lbcs stochastic-interpolants sequential-neural-score-estimation bbox bam fre all-in-one test-time-model-adaptation adaptive-pruning bridging-data-gaps ftrl mechanistic-understanding lca-on-the-line robust-clip stay-on-topic-with-classifier-free-guidance rice sapg
82.8 87.4 73.1 72.5 81.3 82.5 39.2 91.9 65.0 87.8 60.2 37.6 45.7 41.5 76.1 47.1 37.0 62.7 51.8 34.1
1.79 2.07 2.69 2.01 1.22 2.48 2.42 4.37 2.58 1.87 2.34 3.28 3.54 2.80 2.42 2.80 2.13 3.31 3.30 2.18
3600 3536 3600 3601 2513 1873 3600 3600 2112 3570 1875 3600 3600 3600 3226 3408 2633 2579 2954 3225
90.8 88.4 82.3 78.0 69.0 68.7 65.6 65.3 64.4 63.8 62.9 47.1 45.7 45.1 42.8 36.5 33.6 29.7 28.1 8.6
7.22 6.79 2.65 3.73 3.57 4.30 4.33 5.57 8.38 9.28 2.86 6.79 6.15 4.81 2.53 4.44 5.40 3.25 2.82 3.24
8200 4730 5533 4596 4813 5973 4136 7399 8058 6804 4533 8182 6034 5377 5204 6418 6135 7393 5696 4474
82.8 88.9 79.7 95.8 60.2 84.2 53.3 79.1 55.8 97.6 57.2 53.5 58.1 43.6 88.3 75.8 65.1 72.2 20.3 19.2
7.71 4.66 6.37 5.64 4.51 5.52 4.71 6.48 8.77 8.63 3.61 4.98 5.80 8.77 8.05 8.53 11.68 4.62 6.45 3.51
7715 3745 8477 6477 5717 8880 6289 7407 8355 8035 5615 5644 6667 6679 6821 6330 8752 6227 7823 9851
Run-level cause tags are likewise heuristic proxies (Figure 5, right). Incomplete API is inferred from missing-symbol or not-implemented failures. Scope drift is inferred when accepted writes modify files outside the task’s declared scope. Budget/runtime is inferred from explicit agent errors such as max-iteration or timeout failures. Accepted same-file overlap is inferred when multiple task ids have accepted writes to the same file in a failed run. Failed tests themselves are dominated by assertion mismatches, missing APIs, and type or contract errors, indicating that unsuccessful runs still produce substantial but behaviorally incorrect implementations. Among failed STORM runs, cross-module scope drift and accepted same-file overlap appear in nearly all failed cases, and budget or runtime failures remain common at both k=4 and k=8. These proxies explain why STORM is not sufficient by itself: many remaining failures arise after writes have already been accepted as file-version consistent. STORM can ensure that accepted writes are based on a current workspace state, but 17
Preprint, May 2026
it cannot guarantee that the chosen task boundaries are semantically correct, that independently accepted edits compose cleanly, or that agents complete within budget.
Average Failed Tests per Run
Assertion/semantic Missing API/symbol Type/contract error
Not implemented Other runtime
STORM-4
Failed-Test Symptom Mix
Failed-Run Cause Proxies
43.3
41.8
STORM-8
Incomplete API
40
Scope drift Budget/runtime
20
Accepted same-file overlap 0
STORM (k=4)
0
STORM (k=8)
20
40 60 80 100 Share of Failed Runs (%)
Figure 5: Failure analysis for STORM runs at k=4 and k=8. Left: average failed-test symptom mix per run, decomposed into assertion/semantic, missing API/symbol, type/contract, not-implemented, and other runtime failures. Right: share of failed runs in which each run-level cause proxy fires, covering incomplete API, scope drift, budget/runtime, and accepted same-file overlap. Table 9: Effect of intent annotation on Commit0-Lite with Claude Sonnet 4.6 (STORM, 4 engineers). Method STORM (with annotation) STORM (no annotation)
E
Scorew ↑
Score ↑
Costeff ↓
Timeeff ↓
46.2 26.6
82.5 70.9
6.3 8.0
16.8 24.3
Limitations
Terminal bypass. STORM mediates the file_editor tool but not direct filesystem writes through bash (sed, echo >, Python scripts). A post-hoc diff mechanism detects these but cannot reject them preventively. No command coordination. Concurrent shell commands (e.g., two agents running formatters on overlapping files) are not serialized. Extending concurrency control to arbitrary terminal side effects remains open. File-level granularity. Two agents editing different functions in the same file trigger a false-positive rejection. Heavily shared files (e.g., __init__.py) become serialization bottlenecks. Line-level or hunk-level tracking would reduce this at the cost of managing shifting offsets after each edit.
F
Extended Related Work
Code Generation for Software Development LLM-based code generation has moved from function-level synthesis to broader software development tasks. Recent code generation agents can decompose tasks, write code, use tools, run tests, and debug failures (Jiang et al., 2024; 2025; Ma et al., 2025). Repository-level methods further retrieve and use cross-file context from existing projects, including APIs, dependencies, tests, and coding conventions (Zhang et al., 2023; Shrivastava et al., 2023). Benchmarks such as CrossCodeEval and DevEval show that realistic code generation requires cross-file reasoning and repository-level dependency understanding (Ding et al., 2023; Li et al., 2024). However, code generation alone is not enough for reliable software development. Real tasks often require long-horizon interaction, repeated testing, debugging, and revision. Existing methods mainly improve planning, retrieval, or generation quality (Jiang et al., 2026; Li et al., 2026). They provide limited support for multi-step collaboration, state control, and conflict management across agents. 18
Preprint, May 2026
Execution-grounded agents add test feedback to the generation loop (Kumar et al., 2026; Dong et al., 2025), but reliable coordination across long software development workflows remains an open problem.
19