Take It From Here: Evaluating Coding Agents On Project Takeover

Preprint

TAKE I T FROM H ERE : E VALUATING C ODING AGENTS ON P ROJECT TAKEOVER Dingxian Wang1,∗ Zhe Fu2,∗ Jiaqi Deng2,∗ Chao Yin1 Ganesh Prabhu1 Karthik Gomadam Rajagopal1 Darwin Yi3 Huan Huo2,† Guandong Xu2,4 Xianzhi Wang2 Andrew Rabinovich1 Zhao Chen1 1

arXiv:2611.07307v1 [cs.SE] 5 Nov 2026

3

Upwork Inc 2 University of Technology Sydney Scale AI 4 The Education University of Hong Kong

A BSTRACT Coding agents increasingly collaborate on long-term projects. When taking over a project, they need to recover currently applicable requirements from the interaction history and update them in response to new requests. However, most coding benchmarks do not include long interaction histories, and some evaluations that consider history still do not cover the full process from requirement reconstruction to project delivery. We therefore provide agents with the interaction history, current request, and pre-request project environment, ask them to make the requested changes, and assess takeover performance through delivery tests. Across six model–harness configurations and 37 targets eligible for executable validation, the mean complete-delivery rate under Full History is only 4.95%. We then use requirement atoms to independently diagnose a larger set of takeover targets, examining relevant information identification, pre-request state reconstruction, and current request processing. We further supply correct pre-request or target states to examine the remaining difficulties in requirement processing and project modification. The diagnostics show that none of the six configurations produces fully correct post-request requirement states, and a case illustrates omission of previously recovered constraints during updating. With correct target states supplied, the mean complete-delivery rate is 8.11%, with build and regression pass rates of 96.40% and 90.99%, but target-test success remains 9.01%. These results show difficulties in both forming target requirement states and implementing supplied targets in existing projects. Based on these findings, we introduce ReqMemBench, which combines requirement diagnostics with delivery validation to support systematic evaluation of project takeover.

1

I NTRODUCTION

Coding agents have progressed from generating standalone code snippets (Chen et al., 2021) to browsing files, modifying implementations, and running tests in existing codebases (Yang et al., 2024; Wang et al., 2025). When serving as assistants on long-term projects, they need to understand new user requests and build on existing work to move the project forward (Le et al., 2025; Orlanski et al., 2026). However, the information needed to interpret the current request is often scattered across earlier interactions (Wu et al., 2025; Laban et al., 2026). Therefore, Coding agents must extract task-relevant requirements from long interaction histories, determine how the new request changes those requirements, and then produce the required deliverables. Existing coding benchmarks mainly assess code generation and repository-level tasks, typically without long interaction histories (Jimenez et al., 2024; Miserendino et al., 2025). Although some studies examine history use, requirement updates, or intermediate steps, they do not cover the full process at a specified takeover point: reconstructing the current requirement state from history and ∗ †

These authors contributed equally to this work. Corresponding author: Huan Huo.

1

Preprint

Figure 1: Project takeover experiments. (a) Direct delivery from history, the current request, and the initial project, followed by three delivery checks. (b) Independent requirement diagnostics (RQ1: C1 only). (c) Oracle Pre-State predicts the target from the Gold pre-state, whereas Oracle PostState supplies the Gold target directly, followed by delivery evaluation. Diagnostic responses are not inputs to the main experiment. delivering the requested work (Wu et al., 2025; Hu et al., 2025; Liu et al., 2026; Zhou et al., 2026). We evaluate this process across different deliverable types. We construct takeover tasks from 44 projects’ long-term interactions and test direct delivery on 37 targets from 7 of them (Figure 1). Agents receive the history, current request, and pre-request project environment and submit modified projects for testing. We compare Full History (C1) with shorter, filtered Relevant History (C2) to examine whether shortening context improves delivery. Across six model–harness configurations, mean complete-delivery rates are only 4.95% and 2.70%, respectively. To investigate takeover difficulties, we link multi-turn descriptions through requirement atoms: semantically complete requirements that can evolve independently. On 349 targets, three diagnostics (RQ1–RQ3) assess relevant information identification, pre-request state reconstruction, and current request updating. For requests with sufficient information to proceed, no configuration produces a fully correct target state under either history condition. A case further shows that updating can omit previously recovered constraints that still apply, highlighting the difficulty of forming complete and accurate updated states. We then use Oracle Pre-State and Oracle Post-State to examine whether target requirements become actual project changes. Pre-State supplies the correct pre-request state for the agent to update and implement; Post-State directly supplies the correct target. Under Post-State, build and regression pass rates are 96.40% and 90.99%, respectively, but target-test success is only 9.01%. Thus, correct targets still do not ensure implementation of the requested changes. These findings motivate ReqMemBench, which links requirement diagnostics with project delivery in two phases. Agents first produce and freeze requirement responses from the history and current request. For execution-eligible targets judged actionable, they then modify the initial project using those responses and undergo delivery validation. Our contributions are: • We identify difficulties in both requirement understanding and project modification during takeover: agents struggle to form complete and accurate target requirement states from long interaction histories, and directly supplying correct targets still does not ensure implementation of the requested changes.

2

Preprint

• We establish a project-takeover evaluation framework combining direct delivery, independent requirement diagnostics, and Oracle state comparisons to assess historical requirement understanding, target-state formation, and implementation in existing projects. • We instantiate this evaluation design as ReqMemBench across multiple deliverable types, with 349 takeover targets from 44 projects and initial project environments and delivery validators for 37 targets, supporting joint evaluation of requirement understanding and project delivery.

2

R ELATED W ORK

Existing coding benchmarks assess both function generation from specifications (Chen et al., 2021; Austin et al., 2021; Zhuo et al., 2025) and code completion using information across multiple files (Ding et al., 2023; Liu et al., 2024). At the project level, SWE-bench and SWE-Lancer examine agents’ ability to solve real engineering tasks (Jimenez et al., 2024; Miserendino et al., 2025), while SWE-EVO and ProjDevBench further examine continuous evolution of existing software and complete project development (Le et al., 2025; Lu et al., 2026). These evaluations typically start from a given task description and code environment. In project takeover, however, the requirements governing the current task are scattered across earlier interactions, so agents must first determine which historical information still applies. Long-term memory benchmarks also examine recall and reasoning over conversational histories (Maharana et al., 2024). LongMemEval tests temporal reasoning, knowledge updating, and abstention when information is insufficient through multi-session question answering (Wu et al., 2025). MemoryAgentBench incrementally supplies information to assess retrieval, learning, understanding, and discarding superseded facts (Hu et al., 2025). LongMemEval-V2 examines inference of environmental changes and workflows from past web interactions (Wu et al., 2026). These studies evaluate historical retrieval and updating, whereas takeover also requires integrating that information into requirements for the current delivery. In coding, interactive benchmarks assess how agents revise code through multi-turn feedback or stepwise requirement refinement (Han et al., 2025; Miao et al., 2026; Zhan et al., 2025). Sequentialtask benchmarks examine reuse of prior experience and code quality under evolving requirements (Joshi et al., 2025; Zhu et al., 2026; Orlanski et al., 2026). LoCoEval introduces conflicts and updates in long conversations, then separately evaluates topic identification, information extraction, and function generation after the complete conversation (Liu et al., 2026). SWE-RPG diagnoses requirement and planning errors against reference answers, while RigorBench evaluates planning, verification, and recovery in execution traces (Zhou et al., 2026; Madiraju & Madiraju, 2026). These studies cover parts of the process. We evaluate project takeover at a specified point: reconstructing and updating applicable requirements from long histories, then delivering the requested changes. Combining requirement-state diagnostics and delivery validation also lets us examine whether agents implement explicit target requirements in an ongoing project.

3

M ETHODS AND DATA C ONSTRUCTION

We use requests made midway through a project as evaluation units. This section defines the takeover task and delivery evaluation, introduces the representation and scoring for independent requirement diagnostics, and describes data construction. 3.1

P ROJECT TAKEOVER TASK AND D ELIVERY E VALUATION

In direct delivery, agents receive the prior history, current request, and pre-request project environment, then submit the modified project (Figure 1(a); history conditions in Section 4.1). Three checks evaluate the result. The build check assesses code buildability or basic project-file validity. The target test checks implementation of the requested change, while the regression check assesses preservation of specified historical constraints that remain valid. Complete delivery requires all three checks to pass:

3

Preprint

P DeliveryRate =

i∈Texec 1[Bi ∧ Ti ∧ Ri ]

|Texec |

,

(1)

Here, Texec is the predefined executable target set, and Bi , Ti , Ri indicate the three check outcomes. Incomplete runs and missing deliverables count as failures. Infrastructure or validator errors remain pending until resolved, after which rates use the full target set. We also report individual check pass rates to identify unmet delivery requirements. Decision rules and error handling are in Appendix C.2. 3.2

R EQUIREMENT U NDERSTANDING D IAGNOSTICS

Requirement understanding affects agents’ delivery performance (Mu et al., 2024; Zhou et al., 2026), yet delivery checks do not directly reveal omissions or errors in that understanding. We therefore conduct independent requirement diagnostics (Figure 1(b)). Because one message may concern several requirements and a requirement may change across interactions, we link these descriptions through requirement atoms: semantically complete requirements that can evolve independently. This follows the requirements engineering practice of tracing requirement changes and their sources (Nuseibeh & Easterbrook, 2000; Gotel & Finkelstein, 1994). Each requirement atom has five state components. Attributes record its settings, while lifecycle status and scope indicate whether it still applies. Ambiguity records unresolved questions, and implementation evidence records earlier implementation claims or validation results. For requirements + involved in the current request, G− q denotes their pre-request state and Gq the target state, which incorporates current changes while retaining valid earlier content. New requirements appear only in G+ q ; neither state uses subsequent implementation or runtime feedback. Definitions, fields, and replay rules are in Appendix A.1. Three progressive diagnostics assess relevant requirement and evidence identification (RQ1), prerequest state reconstruction (RQ2), and current request updating or clarification (RQ3). In RQ3, when information is sufficient to determine target requirements, the agent selects ACT and outputs a complete post-request state. If critical information is missing, it selects CLARIFY, identifies the missing information, and asks clarification questions. These decisions apply to diagnostics, while the main experiment uses direct delivery (Section 3.1). Table 1 lists the evaluation objects and metrics; Appendix A.2 gives a state-update example. Table 1: What the requirement diagnostics evaluate. Diagnostic

What is evaluated

Core metrics

RQ1: Relevant Historical requirements and information evidence relevant to the current identification task RQ2: Pre-request state Pre-request states of relevant reconstruction requirements, G− q RQ3: Request update or ACT/CLARIFY decision, and G+ q clarification or material missing information and clarification questions

Precision, Recall, and F1 computed separately for requirements and evidence Attribute Reconstruction Score for matched requirements; Requirement Coverage Decision Accuracy, Exact Post-State Accuracy, Post-State Score, Blocking-Issue F1, and Question Validity

Predictions may express the same meaning as the gold-standard answer (Gold) in different words. We therefore use a semantic judge to determine their relationships, after which the program performs one-to-one requirement matching and compares field values. RQ1 and RQ2 match requirements independently: RQ1 evaluates requirement identification and scores evidence selection independently of requirement matching, while RQ2 reports attribute quality on matched requirements and Requirement Coverage to distinguish reconstruction errors from omissions. The program detects conflicting relation labels across diagnostics, updates matching after human review, and recomputes affected metrics (Appendix C.3). Matching rules and judge settings are given in Appendix C.1 and Appendix D.1, respectively.

4

Preprint

b q and Rq denote the predicted and Gold requirement For requests that should be acted upon, let R sets, respectively, and Mq+ their one-to-one matching. For each matched requirement r, sr ∈ [0, 1] is its mean score over applicable state dimensions. The Post-State Score is P 2 r∈Mq+ sr PostStateq = . (2) b q | + |Rq | |R This metric gives partial credit for partially correct states and penalizes missing or extra requirements. We also report Exact Post-State Accuracy, which requires a correct ACT decision, no missing or extra requirements, correct values for all applicable state fields with no extra state dimensions, and a list of attribute names to remove that matches Gold. For requests that require clarification, Blocking-Issue F1 measures the identification of critical missing information. Question Validity is evaluated only for blocking issues matched to Gold and measures whether the proposed questions can obtain the needed information. An incorrect action decision gives the Gold branch a content score of 0. Field scoring and null-value rules are given in Appendix C.1, and aggregation rules in Appendix C.2. 3.3

DATA C ONSTRUCTION AND VALIDATION

The materials cover 44 ongoing collaborative projects involving software, documents, and PCB design. Counting each normalized message as one turn, complete project histories range from 18 to 1,075 turns (mean 181.4; median 110). We annotate requirement atoms, change events, and source messages, replay events chronologically to construct each atom’s state graph, and select history-dependent requests with evaluable requirement changes, yielding 349 diagnostic targets. Each target includes the current request, preceding history, and task-relevant Gold pre-request and post-request states or clarification annotations. Histories end before the current request, and Gold excludes subsequent implementation and runtime feedback. Diagnostic inputs average 189.8 turns under C1 (median 118) and 9.7 under C2 (median 6). The construction process and complete length distributions are provided in Appendix B.2 and Appendix B.1, respectively. Programs check message references, event order, state transitions, and history boundaries, and an agent reviews RQ2/RQ3 Gold structure and semantics. Four human reviewers in two pairs cover all 44 projects, with two independent reviews per project. Agreements are accepted; disagreements are reviewed again. Gold review and semantic-score verification are described in Appendix B.3 and Appendix C.3, respectively. Delivery targets additionally require reproducible pre-request environments and executable acceptance tests. Initial environments must pass build and regression checks but fail the target test. These criteria yield 37 delivery targets from 7 projects; selection and validation scope are detailed in Appendix B.3.

4

E XPERIMENTAL D ESIGN

The three experiments use the same six model–harness configurations to evaluate direct delivery, requirement understanding, and project modification with correct-state inputs. The six configurations comprise GPT-6 Astra, GPT-6 Sol, GPT-5.6 Sol, and GPT-5.6 Terra with Codex CLI, and Claude Opus 5.5 and Claude Sonnet 5 with Claude Code, all using high reasoning effort. Each configuration, target, and condition has one independent run in a new process, and delivery experiments also use separate project environments. Requirement diagnostics have a 900second time limit, and direct delivery has a 1,800-second limit. Specific versions, tool permissions, prompts, and semantic judge settings are given in Appendix D.1 and Appendix D.3.

5

Preprint

4.1

P ROJECT TAKEOVER E XPERIMENT

The main experiment follows the direct-delivery procedure in Section 3.1, using Full History (C1) as the main condition and Relevant History (C2) as the comparison. C1 contains all eligible messages before the takeover point. Using requirement annotations, C2 retains the historical descriptions, earlier revisions, and supporting messages for requirements involved in the current request, removes unrelated messages, and preserves the original order. Its construction is described in Appendix B.2. Across the 37 delivery targets, the mean history lengths are 186.9 turns under C1 and 8.7 under C2. Both conditions share targets, current requests, initial environments, tool permissions, budgets, and validators within each configuration; only visible history changes. Motivated by difficulties reported in long-context evaluations (Bai et al., 2024; Hsieh et al., 2024), C2 examines whether shortening context improves delivery. Since it also changes information composition, the comparison measures the filter’s overall effect. Rates use all 37 targets (Section 3.3) per configuration and condition, and cross-configuration results average the six configuration-level rates. 4.2

R EQUIREMENT U NDERSTANDING D IAGNOSTIC E XPERIMENT

The requirement diagnostics use 349 targets from 44 projects, providing only the history, current request, and response specification, without project environment access. The model submits a single structured response following Section 3.2. This response is frozen and scored independently and is not supplied as an additional input to the main experiment. RQ1 uses C1 only, while RQ2 and RQ3 compare C1 and C2. For every configuration and condition, RQ3 decision metrics use all 349 targets, Exact Post-State Accuracy and Post-State Score both use all 312 Gold-ACT targets, and clarification metrics use the 37 Gold-CLARIFY targets. To distinguish action-selection errors from difficulties in constructing the target state, we also report Post-State Score conditional on a correct ACT decision. Because the two classes are imbalanced, decision evaluation reports accuracy, recall for each class, balanced accuracy, and an always-ACT reference. Content metrics are computed per target and then averaged over their applicable sets. Results are aggregated separately for each configuration before taking an equal-weight average across the six configurations. Matching, metric applicability, and failure scoring rules are given in Appendices C.1–C.2. 4.3

O RACLE S TATE C OMPARISON E XPERIMENT

Oracle comparisons examine project modification from correct-state inputs on the same six configurations and 37 targets (Figure 1(c)), with initial environments, tool permissions, coding budgets, and validators held fixed within each configuration. Oracle Pre-State supplies G− q and the current request, bypassing historical reconstruction. The model updates requirements or requests clarification. For ACT, its predicted target state is frozen and passed with the initial environment to coding for project modification and submission; CLARIFY does not enter coding. Oracle Post-State directly supplies Gold G+ q and the initial environment for delivery. In both conditions, the requirement input to coding contains only references, summaries, and final state fields, from predicted states in Pre-State and Gold states in Post-State (Appendix D.2). Within each configuration, we compare Post-State with direct C1 delivery and examine both Oracle conditions through the checks in Section 3.1. Post-State also changes the coding input relative to C1, while Pre-State and Post-State differ in state sources and whether an action decision precedes coding. Rates for complete delivery and each check use all 37 targets; non-coding runs and missing submissions count as failures.

5

R ESULTS AND A NALYSIS

5.1

C AN AGENTS TAKE OVER O NGOING P ROJECTS ?

Table 2 shows low complete-delivery rates under both history conditions. Across the six configurations, the mean rate is 4.95% under Full History (C1) and 2.70% under Relevant History (C2). On

6

Preprint

these 37 targets, the current filtering scheme yields a lower observed mean complete-delivery rate than Full History. We next use independent requirement diagnostics to investigate these difficulties. We examine whether agents can identify relevant requirements and evidence, reconstruct their pre-request states, and determine how to act on the current request. Table 2: Complete-delivery rates (%) under C1/C2 on the same 37 targets. The mean weights all six configurations equally; aggregation rules are in Appendix C.2. Harness Codex

Claude Code Overall

5.2

Model / summary

Full History (C1)

Relevant History (C2)

GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Opus 5.5 Sonnet 5 Mean across all configurations

10.81% 5.41% 2.70% 5.41% 5.41% 0.00% 4.95%

5.41% 2.70% 5.41% 0.00% 2.70% 0.00% 2.70%

W HERE D O AGENTS S TRUGGLE WITH R EQUIREMENT U NDERSTANDING ?

Table 3 shows that selecting relevant historical information and reconstructing pre-request states both remain difficult. Under Full History, GPT-6 Astra has an evidence Recall of 0.9211 but a Precision of only 0.2254. Although it identifies much of the Gold evidence, it also selects many messages outside that set. In RQ2, GPT-6 Sol has a Requirement Coverage of 0.6777 but an Attribute Reconstruction Score of only 0.2671, indicating that identifying a requirement does not ensure accurate recovery of its specific settings. Complete diagnostic results are in Appendix E.2. Table 3: Selected RQ1–RQ2 configuration means under C1 (0–1 scale). Scoring rules are in Appendix C.1, with full results and applicable target counts in Appendix E.2. Configuration GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Claude Opus 5.5 Claude Sonnet 5

Attribute Evidence Evidence Requirement Reconstruction Precision Recall Coverage Score 0.2254 0.3395 0.3162 0.4254 0.3189 0.3060

0.9211 0.8620 0.8182 0.6670 0.8364 0.7461

0.6689 0.6777 0.6157 0.4950 0.5934 0.5704

0.2813 0.2671 0.2451 0.2424 0.2811 0.2548

The agents also struggle to form complete and accurate target states after processing the current request. Under both C1 and C2, Exact Post-State Accuracy is 0/312 for every configuration. To examine partially correct predictions, Figure 2 reports Post-State Scores for different target subsets. For example, under Full History, Opus 5.5 scores only 0.2036 on 256 targets where it correctly selects ACT. Even on the 79 targets within this group whose requirement sets fully match Gold, the score is only 0.3656. The other configurations show similar gaps. Thus, a correct action decision and complete requirement identification do not ensure accurate states for those requirements. Beyond inaccurate state fields, updating can also result in the loss of requirements that the agent has already recovered. In a PCB task, the client asks to move the battery connector to the back of the board, opposite the USB-C port. Opus 5.5 records layout requirements including eight mounting holes and the M2.5 specification in its pre-request state, but omits the entire mechanical-layout requirement from the post-request state in the same response. The current request does not withdraw these requirements, yet they are not retained in the updated target state. Details are in Appendix E.3. Together, these results show that forming a complete and accurate target remains difficult: agents can produce incorrect state fields or omit requirements that still apply. We next supply correct pre-

7

Preprint

All Gold-ACT

Correct ACT

Correct ACT + complete req. set

GPT-6 Astra

0.0741 (n=312)

0.1298 (n=178)

0.2882 (n=50)

GPT-6 Sol

0.0535 (n=312)

0.1129 (n=148)

0.2422 (n=36)

GPT-5.6 Sol

0.0538 (n=312)

0.1043 (n=161)

0.2299 (n=39)

GPT-5.6 Terra

0.0653 (n=312)

0.1273 (n=160)

0.2899 (n=46)

Claude Opus 5.5

0.1670 (n=312)

0.2036 (n=256)

0.3656 (n=79)

Claude Sonnet 5

0.1041 (n=312)

0.1665 (n=195)

0.3320 (n=54)

Configuration

Post-State Score

0.0

0.2

0.4

0.6

0.8

1.0

Figure 2: Post-request state diagnostics under Full History. Post-State Scores are reported over all Gold-ACT targets, the correct-ACT subset, and the subset with a complete requirement-set match, with target counts shown. The three series describe different evaluation subsets. request and target states separately to examine the remaining difficulties in request processing and project modification. 5.3

H OW D OES P ROVIDING C ORRECT R EQUIREMENT S TATES A FFECT D ELIVERY ?

With correct target states supplied, Oracle Post-State achieves a mean complete-delivery rate of 8.11% (Table 4), above the 4.95% under C1 in the main experiment (Table 2), but still below 10%. To examine which aspects of delivery change, we compare the three checks under main-experiment C1 and Post-State. The component results in Table 4 show that, compared with C1 in the main experiment, Post-State increases the build pass rate from 58.56% to 96.40% and the regression pass rate from 51.35% to 90.99%, whereas target-test success rises only from 6.31% to 9.01%. This indicates that, given the correct target, coding agents can identify which requirements should remain unchanged, and that this information also helps them establish a functioning project environment. However, target-test success remains low, indicating that agents still struggle to implement the corresponding changes in code even when given the correct target state. Table 4: Pass rates for individual checks and complete delivery (%; 37 targets per configuration and condition). C1/C2 denote Full/Relevant History, and Pre/Post denote Oracle Pre-State/Post-State. Means weight their configurations equally; counts and aggregation rules are in Appendix E.1. Build

Configuration C1 GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Codex mean

C2

Pre

56.76 43.24 48.65 48.65 32.43 43.24 59.46 45.95 62.16 54.05 37.84 51.35 54.73 39.86 51.35

Target test Post

C1 C2

Pre Post

Regression C1

C2

Pre

94.59 16.22 8.11 5.41 13.51 45.95 37.84 40.54 94.59 8.11 2.70 5.41 16.22 45.95 24.32 35.14 97.30 2.70 5.41 10.81 13.51 51.35 37.84 56.76 91.89 5.41 0.00 2.70 5.41 48.65 27.03 45.95 94.59 8.11 4.05 6.08 12.16 47.97 31.76 44.59

Complete delivery Post

C1 C2

Pre Post

81.08 10.81 5.41 5.41 10.81 81.08 5.41 2.70 2.70 13.51 94.59 2.70 5.41 10.81 13.51 89.19 5.41 0.00 2.70 5.41 86.49 6.08 3.38 5.41 10.81

Opus 5.5 75.68 72.97 81.08 100.00 5.41 2.70 5.41 5.41 64.86 70.27 75.68 100.00 5.41 2.70 5.41 5.41 Sonnet 5 56.76 51.35 75.68 100.00 0.00 0.00 0.00 0.00 51.35 45.95 70.27 100.00 0.00 0.00 0.00 0.00 Claude Code mean 66.22 62.16 78.38 100.00 2.70 1.35 2.70 2.70 58.11 58.11 72.97 100.00 2.70 1.35 2.70 2.70 Overall mean

58.56 47.30 60.36 96.40 6.31 3.15 4.95 9.01 51.35 40.54 54.05 90.99 4.95 2.70 4.50 8.11

One possible explanation concerns how agents modify existing implementations. LLMs Get Lost finds that models in multi-turn conversations can rely excessively on prior answers and fail to revise earlier assumptions when new information arrives (Laban et al., 2026). SlopCodeBench also reports increasing structural complexity and redundant code across iterative extensions (Orlanski et al., 2026). Drawing on these observations, we hypothesize that, even with an explicit target, agents may 8

Preprint

retain earlier assumptions or make only local edits, leaving parts of the implementation inconsistent with the updated requirements. As a complementary comparison, Oracle Pre-State has a complete-delivery rate of 4.50%, showing that a correct pre-request state also does not ensure successful subsequent delivery. Together with Section 5.2, these results indicate difficulties in both forming complete and accurate updated requirement states and implementing explicit targets as actual changes to existing code or project files.

6

R EQ M EM B ENCH

6.1

M OTIVATION AND B ENCHMARK C OMPONENTS

On the 37 main-experiment targets, the current history filter does not yield a higher observed mean complete-delivery rate than Full History, while the requirement diagnostics reveal omissions and errors in target states. Yet even when correct target states are supplied, build and regression pass rates are high while the requested changes often fail acceptance tests. Project takeover thus requires both understanding requirements and implementing the updates in an existing project. We introduce ReqMemBench to link requirement-state annotations with delivery validation on shared takeover targets, evaluating both target-state formation and actual project modification. The benchmark contains 349 takeover targets from 44 projects, covering software, documents, and PCB deliverables. Each target includes the current request, preceding history, and evidence-linked Gold pre-request and post-request states or clarification annotations. For 37 targets from 7 projects, pre-request project environments and hidden validators additionally support delivery evaluation. 6.2

T WO -P HASE E VALUATION P ROTOCOL

Phase A follows the requirement diagnostics in Section 3.2 on all 349 targets. Agents answer using only the current request and visible history, without project environment access. Responses are frozen before implementation. Only execution-eligible runs with valid Phase A responses selecting ACT enter Phase B. Agents receive the initial environment, current request, visible history, and frozen response, then modify the project and undergo the delivery checks in Section 3.1. Rates use all 37 delivery targets per condition, counting non-coding runs and missing submissions as failures.

7

L IMITATIONS

Executable validation currently covers only 37 tasks from 7 projects, limiting the range of projects and takeover scenarios represented in the delivery experiments. In addition, our experiments cover six model–harness configurations using Codex CLI and Claude Code, which limits their representativeness for other agent systems. Future work can expand the set of projects and takeover points with recoverable historical environments and include more models and harnesses to examine whether these findings hold across settings. Requirement boundaries, state values, and free-text matching still depend on semantic judgment. Annotation and semantic-scoring uncertainty therefore remain relevant when interpreting the diagnostic results.

8

C ONCLUSION

On the 37 evaluated targets, complete-delivery rates are low under both history conditions, and the current history filter yields no increase in the observed mean rate. None of the six configurations produces a fully correct updated requirement state, and updating can omit valid historical constraints. Under Post-State, the mean complete-delivery rate is 8.11%, compared with 4.95% under C1, but target-test success remains 9.01% despite high build and regression pass rates. These findings reveal difficulties in both forming updated requirement states and implementing them in existing projects. ReqMemBench combines requirement diagnostics with delivery validation to evaluate both.

9

Preprint

E THICS S TATEMENT We construct the benchmark from project interaction materials that have been processed to protect privacy, while preserving task-relevant message order and requirement evolution. In using and presenting these materials, we protect participants’ identities and sensitive project information. We do not disclose raw private conversations or access credentials, and we do not release project assets without authorization. The benchmark evaluates coding agents’ project takeover capabilities and is not intended to identify or evaluate the original participants.

AI S TATEMENT We used large language models to assist with requirement annotation, review, and semantic scoring. During annotation, the models helped identify requirement atoms in interaction histories, extract requirement changes and their source evidence, and review pre-request and post-request requirement states. During scoring, the models assessed semantic relations between predictions and Gold, while programs performed requirement matching and score calculation. Settings for the semantic judge are provided in Appendix D.1. The authors are responsible for the annotation and scoring procedures, experimental results, and conclusions.

R EFERENCES Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. URL https://arxiv.org/ab s/2108.07732. MBPP. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3119–3137, 2024. doi: 10.18653/v1/2024.acl-long.172. URL https://aclanthology.org/2024.acl-l ong.172/. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. HumanEval. Yangruibo Ding, Zijian Wang, Wasi Uddin Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. CrossCodeEval: A diverse and multilingual benchmark for cross-file code completion. In Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, volume 36, 2023. doi: 10.52202/075280-2023. URL https://proceedings.neurips.cc/paper _files/paper/2023/hash/920f2dced7d32ab2ba2f1970bc306af6-Abstrac t-Datasets_and_Benchmarks.html. O. C. Z. Gotel and Anthony Finkelstein. An analysis of the requirements traceability problem. In Proceedings of the First IEEE International Conference on Requirements Engineering, pp. 94– 101. IEEE Computer Society, 1994. doi: 10.1109/ICRE.1994.292398. Hojae Han, Seung-won Hwang, Rajhans Samdani, and Yuxiong He. ConvCodeWorld: Benchmarking conversational code generation in reproducible feedback environments. In International 10

Preprint

Conference on Learning Representations (ICLR), 2025. URL https://proceedings.ic lr.cc/paper_files/paper/2025/hash/6091f2bb355e960600f62566ac0e28 62-Abstract-Conference.html. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. RULER: What’s the real context size of your long-context language models? In Conference on Language Modeling, 2024. URL https://openreview.net/f orum?id=kIoBbc76Sy. Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremental multi-turn interactions. arXiv preprint arXiv:2507.05257, 2025. URL https://arxiv.or g/abs/2507.05257. MemoryAgentBench. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR), 2024. URL https://proceeding s.iclr.cc/paper_files/paper/2024/hash/edac78c3e300629acfe6cbe9ca 88fb84-Abstract-Conference.html. Thomas Joshi, Shayan Chowdhury, and Fatih Uysal. SWE-Bench-CL: Continual learning for coding agents. arXiv preprint arXiv:2507.00014, 2025. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. LLMs get lost in multiturn conversation. In International Conference on Learning Representations (ICLR), pp. 54738– 54778, 2026. URL https://proceedings.iclr.cc/paper_files/paper/2026 /hash/59f6421e64707225fdf5b28840679a07-Abstract-Conference.html. Tue Le, Minh Vu Thai Pham, Dung Nguyen Manh, Huy Nhat Phan, and Nghi D. Q. Bui. SWEEVO: Benchmarking coding agents in long-horizon software evolution scenarios. arXiv preprint arXiv:2512.18470, 2025. v6 revised May 2026 – CHECK: set year to 2026 if you cite the revision or a subsequent venue version. Tianyang Liu, Canwen Xu, and Julian McAuley. RepoBench: Benchmarking repository-level code auto-completion systems. In International Conference on Learning Representations (ICLR), 2024. URL https://proceedings.iclr.cc/paper_files/paper/2024/h ash/d191ba4c8923ed8fd8935b7c98658b5f-Abstract-Conference.html. Yang Liu, Li Zhang, Fang Liu, Ping Lin, and Xinyi Li. A scalable benchmark for repositoryoriented long-horizon conversational context management. arXiv preprint arXiv:2603.06358, 2026. LoCoEval. Pengrui Lu, Shiqi Zhang, Yunzhong Hou, Lyumanshan Ye, Chaoyi Huang, Zixi Chen, Ji Zeng, Hantao Jiang, Pengfei Liu, Yiwei Wang, and Ming-Hsuan Yang. ProjDevBench: Benchmarking AI coding agents on end-to-end project development. arXiv preprint arXiv:2602.01655, 2026. Meher Bhaskar Madiraju and Meher Sai Preetam Madiraju. RigorBench: Benchmarking engineering process discipline in autonomous AI coding agents. arXiv preprint arXiv:2606.22678, 2026. Unrefereed preprint, 30 tasks – see notes on citation weight. Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. Evaluating very long-term conversational memory of LLM agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13851–13870. Association for Computational Linguistics, 2024. doi: 10.18653/v1/2024.acl -long.747. URL https://aclanthology.org/2024.acl-long.747/. Chunyu Miao, Henry Peng Zou, Yangning Li, Yankai Chen, Yibo Wang, et al. RECODE-H: A benchmark for research code development with interactive human feedback. In International Conference on Learning Representations (ICLR), 2026. arXiv:2510.06186. Samuel Miserendino, Michele Wang, Tejal Patwardhan, and Johannes Heidecke. SWE-Lancer: Can frontier LLMs earn $1 million from real-world freelance software engineering? In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pp. 44412–44450. PMLR, 2025. URL https://proceedings.mlr. press/v267/miserendino25a.html. 11

Preprint

Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. ClarifyGPT: A framework for enhancing LLM-based code generation via requirements clarification. Proceedings of the ACM on Software Engineering, 1(FSE):2332–2354, 2024. doi: 10.1145/3660810. URL https://doi.org/10.1145/3660810. Bashar Nuseibeh and Steve M. Easterbrook. Requirements engineering: A roadmap. In Proceedings of the 22nd International Conference on Software Engineering, Future of Software Engineering Track, pp. 35–46. ACM, 2000. doi: 10.1145/336512.336523. Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Nicholas Roberts, Frederic Sala, and Aws Albarghouthi. SlopCodeBench: Benchmarking how coding agents degrade over long-horizon iterative tasks. arXiv preprint arXiv:2603.24755, 2026. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations (ICLR), pp. 65882–65919, 2025. URL https://proceedings.iclr.cc/paper_files/pape r/2025/file/a4b6ad6b48850c0c331d1259fc66a69c-Paper-Conference.pd f. Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. LongMemEval: Benchmarking chat assistants on long-term interactive memory. In International Conference on Learning Representations (ICLR), 2025. URL https://proceedings.iclr.cc/pape r_files/paper/2025/hash/d813d324dbf0598bbdc9c8e79740ed01-Abstrac t-Conference.html. Di Wu, Zixiang Ji, Asmi Kawatkar, Bryan Kwan, Jia-Chen Gu, Nanyun Peng, and Kai-Wei Chang. LongMemEval-V2: Evaluating long-term agent memory toward experienced colleagues. arXiv preprint arXiv:2605.12493, 2026. URL https://arxiv.org/abs/2605.12493. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems, volume 37, pp. 50528–50652. Curran Associates, Inc., 2024. doi: 10.52202/079017-1601. URL https://papers.nips.cc/p aper/2024/hash/5a7c947568c1b1328ccc5230172e1e7c-Abstract-Confere nce.html. Zexun Zhan, Shuzheng Gao, Ruida Hu, and Cuiyun Gao. SR-Eval: Evaluating LLMs on code generation under stepwise requirement refinement. arXiv preprint arXiv:2509.18808, 2025. Xin Zhou, Chun Yong Chong, Kisub Kim, Yun Peng, Rui Shu, Zihan Wu, Xu Han, Guowen Yuan, Zeyang Zhuang, Jounghoon Kim, Jeongjin Ju, Seongmin Ju, Taein Yoon, and David Lo. A unified issue resolution benchmark for requirement clarification, planning, and code generation for coding agents. arXiv preprint arXiv:2608.09072, 2026. URL https://arxiv.org/ab s/2608.09072. Jiayuan Zhu, Junde Wu, Minhao Hu, Shengda Zhu, Jiazhen Pan, Weixiang Shen, Yijun Yang, Fenglin Liu, Jianye Hao, Yueming Jin, Qirong Ho, and Min Xu. SWE Context Bench: A benchmark for context learning in coding. arXiv preprint arXiv:2602.08316, 2026. Official name is ”SWE Context Bench”; not the same paper as ContextBench (arXiv:2602.05892). Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, et al. BigCodeBench: Benchmarking code generation with diverse function calls and complex instructions. In International Conference on Learning Representations (ICLR), 2025. arXiv:2406.15877.

12

Preprint

A

R EQUIREMENT R EPRESENTATION AND S TATE E XAMPLE

A.1

R EQUIREMENT ATOMS , S TATES , AND R EPLAY

A requirement atom is a semantically complete requirement that can be independently modified, deferred, withdrawn, or verified. Specific settings of the same mechanism are attributes. Optional families group atoms without owning states or automatically propagating changes. Each atom maintains five types of information.

Table 5: Requirement-state dimensions. Field Attributes, attributes Lifecycle status, lifecycle_status

Scope, scope Ambiguity, ambiguity Implementation evidence, execution

Representation and meaning Store currently established parameters, behavior, and constraints. Each event records only what it establishes or changes; replay carries forward other attributes that remain valid. ACTIVE means currently applicable, DEFERRED means implementation is postponed, and REMOVED means explicitly withdrawn. The value remains null when history is insufficient to determine it. Contains persistence, components, and contexts, which specify duration, applicable components, and contexts. A modification overwrites only dimensions explicitly changed. Records unresolved issues by source event, including the VALUE, SCOPE, or LIFECYCLE dimension, a description, and provenance. The value is null when no issue remains open. Stores the latest concrete implementation claim, failure, or successful validation, labeled CLAIMED WORKING, FAILED, or VERIFIED WORKING. The value is null when no evidence has been established.

Lifecycle and implementation evidence are separate: an ACTIVE requirement can have FAILED implementation evidence. When history is incomplete, replay starts from an incomplete baseline with empty attributes and null unknown dimensions, without inventing an INTRODUCE event. Events follow normalized message order and their annotated order within each message. Unchanged fields are inherited. The keys in value removals are deleted before updates are applied; a key cannot be removed and updated in the same event. A null scope updates dimension means unchanged. An ambiguity is resolved only through an explicit resolves ambiguity event ids link from a later event of the same atom; subsequent modifications or successful verification do not clear other open issues. Each atom has at most one INTRODUCE. The program checks payloads, sources, order, and ambiguity links. Deterministic replay builds a separate requirement state graph for each atom. After each event is applied, a new state node stores the five state dimensions and the supporting_event_ids that directly establish the current snapshot. An edge connects the previous node to the node produced by the event and records its event_id, event_type, source_message_id, and supporti ng_message_ids. The first edge has a null from_state_id, indicating that no earlier state node is stored. Each atom therefore has an ordered trajectory of state snapshots. The project-level requirement_graphs field holds the collection of these graphs; it does not add dependency or conflict edges between atoms. One takeover target corresponds to one client message. The pre-request state is the last state before that message; the post-request state applies all events triggered by it and stops before later implementation feedback. Scoring uses requirements directly involved in the request: new atoms appear only after the request, while withdrawn atoms remain with lifecycle REMOVED. The implementa-

13

Preprint

Table 6: Event operations used in state replay. Main replay operation

Event INTRODUCE MODIFY DEFER RESUME REMOVE AMBIGUOUS IMPLEMENTATION CLAIM RUNTIME FAILURE RUNTIME VERIFICATION

Write initial attributes and scope, and set lifecycle status to ACTIVE. If only an incomplete observation baseline existed, clear its old implementation evidence. Remove specified attributes, then apply attribute and scope updates. Carry forward unchanged content and clear implementation evidence to null. Set lifecycle status to DEFERRED, preserving existing attributes, scope, and implementation evidence. Set lifecycle status to ACTIVE without attribute or scope modifications. Set lifecycle status to REMOVED, preserving historical attributes, scope, and implementation evidence. Add an open issue, retaining the last established requirement without guessing alternative values or lifecycle status. Update implementation evidence to CLAIMED WORKING. Update implementation evidence to FAILED. Update implementation evidence to VERIFIED WORKING.

tion rejects reintroduction, modification, deferral, or resumption of the same atom after REMOVE, but permits later implementation claims, failures, or verification concerning the withdrawal. These events update only implementation evidence, while lifecycle status remains REMOVED. A.2

A P RE - REQUEST AND P OST- REQUEST S TATE E XAMPLE

Word-template target 43255761_T010 illustrates how a request maps to a specific transition in the state graph. In message 154, the client accepts the calendar-dropdown input proposed in message 153, then requests day/month/year formatting in message 268. Current request 270 explicitly asks to add regular Enter input. For the date-field-control atom, REQ_DATE_FIELD_CONTROL, this request produces MODIFY event REQ_DATE_FIELD_CONTROL_E004, which connects pre-request node REQ_DATE_FIELD_CONTROL_S003 to post-request node REQ_DATE_F IELD_CONTROL_S004. The event updates only date_entry_method, so the unchanged date_format is retained. The following excerpt comes from construction_gold.affected_requirement_t ransitions.REQ_DATE_FIELD_CONTROL in this target’s RQ3 file. It shows the event, state-node references, and two attributes rather than the complete state or a new response format. { "event_ids": [ "REQ_DATE_FIELD_CONTROL_E004" ], "event_types": [ "MODIFY" ], "state_provenance": { "before_state_id": "REQ_DATE_FIELD_CONTROL_S003", "after_state_id": "REQ_DATE_FIELD_CONTROL_S004" }, "before": { "attributes": { "date_entry_method": "calendar dropdown for date selection", "date_format": "day/month/year" } 14

Preprint

}, "after": { "attributes": { "date_entry_method": "calendar dropdown or regular Enter input", "date_format": "day/month/year" } } } The atom remains ACTIVE with unchanged scope and null ambiguity and implementation evidence. Messages 153, 154, and 268 are visible under both C1 and C2, and both conditions have Gold ACT. C1 also includes message 269 asking about the Enter option, while C2 omits it; current request 270 itself explicitly asks to add Enter input to the date field. The same request also affects REQ_MULTILINE_FIELD_INPUT, so the complete Gold contains two atoms, with their pre-request and post-request states and corresponding transitions. This example illustrates requirement updating and is outside the 37 delivery targets.

B

DATA C ONSTRUCTION AND Q UALITY C HECKS

B.1

DATA S COPE AND S TATISTICS

The dataset contains interaction records from 44 real projects, covering software, Word templates and reports, and PCB deliverables. We report the size and history length of the requirement diagnostic set and the subset with delivery validation separately.

Table 7: Dataset scope and target counts. Set

Current count

Projects contributing formal requirement-understanding targets

44 projects

Interpretation

Each contributes at least one complete requirement-understanding target. After target selection, before filtering for 350 targets Output of the target-point selection the complete diagnostic sequence procedure. Formal requirement-understanding 349 targets Each target has RQ1, RQ2, and RQ3 targets files, with 349 files for each RQ. Gold-ACT / Gold-CLARIFY 312 / 37 targets Request-update and clarification branches after review and adjudication. Formal delivery-validation targets 37 targets from 7 The same target set is used under projects each history condition.

Selected target 42204309 T036 introduces only new requirements and has no relevant pre-request requirements. Excluding it from the complete diagnostic sequence leaves 349 targets. One turn is one normalized message record, including retained empty-text records. History length excludes the current request and future messages. Project statistics weight the 44 projects equally; takeover-history statistics weight targets equally. Overlapping histories do not constitute independent message counts. The complete projects contain 7,983 messages. Q1/Q3 use linear interpolation at (n-1)p. Lengths are calculated from normalized project messages and the history-message counts recorded for RQ2, then checked against RQ3 and actual inputs.

15

Preprint

Table 8: Interaction and history lengths, measured in messages. Statistical unit

Count

Complete project interactions All takeover targets, C1 All takeover targets, C2 Delivery targets, C1 Delivery targets, C2

B.2

Mean Median

44 projects 181.43 349 targets 189.80 349 targets 9.74 37 targets 186.86 37 targets 8.70

Q1

Q3 Min–max

110 74.25 204.50 18–1,075 118 54 216 2–909 6 3 10 1–90 95 55 257 5–745 7 4 12 1–40

C ONSTRUCTION AND H ISTORY I NPUTS

Project interactions Normalization

Requirement annotation

Atoms Source messages Change events

Annotation review

Structural checks Semantic checks Source links

Input isolation

Chronological state replay State trajectories

Takeover points and Gold

History-dependent requests Pre/post-request states or clarification Gold

349 diagnostic targets; 37 also support delivery validation

Agent-visible inputs

Scoring-only assets

Current request + preceding history C1: Full History / C2: Relevant History Diagnostics: no project environment 37 delivery targets: initial project environment

Requirement and evidence annotations Pre/post-request states or clarification Gold 37 delivery targets: hidden validators Build / target / regression checks

Figure 3: Data construction and input isolation. Requirement atoms, change events, and source evidence are annotated from project interactions, reviewed, and replayed chronologically to obtain requirement states. Selected takeover points yield 349 diagnostic targets, of which 37 additionally have initial project environments and executable validation. Public inputs are separated from Gold and hidden validators. C2 provides relevant historical messages rather than correct requirement states.

Construction normalizes message order, then uses a language model to identify atoms, extract source-linked events, and check boundaries and cross-requirement effects. The program validates message references, roles, payloads, and order before deterministic replay under A.1. Candidates are grouped by client message; selection chooses evaluation points without regenerating Gold. Candidate selection uses GPT-5.6 Sol, high reasoning, and AI SCORE THRESHOLD mode with threshold 6, yielding 350 selected targets, of which 349 enter the complete requirement-diagnostic set. Prompt excerpts are in D.3. C1 contains all eligible messages before the request. C2 collects complete earlier event trajectories of directly affected, pre-existing requirements, deduplicates their source and supporting messages, and preserves order; it does not automatically expand the whole family. Evidence scoring requires groups that still support the current state, whereas C2 retains the relevant requirements’ full evolution. The diagnostic workspace contains only four public files, covering the current task, visible history, instructions, and response schema. The task description records the current request and links it to the history and response specification; Gold, selection metadata, internal requirement identifiers, and validator configurations remain on the research side. Input materialization checks that selected messages exist, preserve their original order, and exclude the current request. It also checks that

16

Preprint

the request, history, message counts, and fingerprints agree across the RQ1–RQ3 views. Public files are checked for forbidden fields and internal identifiers before being copied into a fresh diagnostic workspace. Structural validation additionally checks that C1 matches the full history pool, C2 is an orderpreserving subset of C1, and history_message_count matches the number of selected messages. Temporal boundaries use positions in the normalized message sequence rather than the numerical values of message IDs. Direct delivery uses separate project environments, whose archive and state-check records are described in B.3. The release separately stores requirement events, RQ123 Gold, selection records, and review records, and records the initial environment and validator associated with each of the 37 executable targets. B.3

G OLD R EVIEW AND E XECUTABLE TARGETS

All 44 formal projects pass Gold-state structural validation. RQ2/RQ3 for all 349 targets also undergo structural review, semantic review, and adjudication sequentially by the same agent. The recorded method is SINGLE_AGENT_ROLE_SEPARATED_PANEL, with independent_ag ent_panel set to false. RQ2 records 7 ordered-list and 27 empty-container rule adjustments. RQ3 records 9 ordered-list adjustments and 8 branch revisions, changing the distribution from 306 ACT/43 CLARIFY to 312/37. These adjustments are summarized from programmatic checks and offline agent-review records. Four human reviewers work in two pairs to review all 44 projects. Each project is assessed separately by both reviewers in its assigned pair. Agreeing judgments are accepted, and disagreements are reviewed again. The table reports the size of formal Gold; annotations shared by C1 and C2 are counted once.

Table 9: Gold items covered by human verification. Review item

Counting unit

Requirement boundaries Evidence assignment and coverage Pre-request states

Requirement–target item Required evidence group Requirement–target state item Requirement–target state item Takeover target Blocking issue

ACT post-request states ACT/CLARIFY decisions Missing information and clarification

Gold items 733 2,108 733 763 349 43

Execution selection starts with 123 targets from 7 projects. Removing 24 Gold-CLARIFY targets leaves 99 Gold-ACT targets: initial screening identifies 81 potentially verifiable targets, 16 requiring subjective review, and 2 without deterministic observables. Environment-based screening leaves 40 validator candidates. Initial checks must yield Build PASS, Target FAIL, and Regression PASS. Word targets T004, T005, and T007 already pass Target and are excluded, leaving 37 targets: 22 from four software/website projects, 9 ESP32 PCB targets, 4 Word-template targets, and 2 securityreport targets. Checks cover specified local behavior or artifact constraints, without certifying physical PCB performance or all visual experiences. Initial validator checks are described in C.3. The registry links each of the 37 executable environments to its target and pre-request message boundary. Their manifests classify them as synthetic-executable-pre-state, covering local domain services, mobile state services, and KiCad, OOXML, and report generators. The manifests record successful pre-state Gold, archive, and leakage checks. Initial validation results are Build PASS, Target FAIL, and Regression PASS.

17

Preprint

C

S CORING AND VALIDATION

C.1

R EQUIREMENT AND S TATE S CORING

RQ1, RQ2, and RQ3 separately align predicted and Gold requirements using semantic labels SAME ATOM, MERGED ATOMS, SUBPART OF ATOM, RELATED DIFFERENT ATOM, UNRELATED, or UNCERTAIN. Only SAME ATOM creates edges for maximum-cardinality one-to-one matching. Among maximum-cardinality matchings, RQ1 maximizes the sum of fixed edge weights based on summary similarity, evidence similarity, and Gold-evidence coverage. RQ1/RQ2 do not share relation matrices; an observed label inconsistency is documented in E.2. RQ1 is evaluated only under C1. Matched requirements are TP; unmatched predictions and Gold items are FP and FN, yielding Precision, Recall, and F1. With no predictions, Precision/F1 are zero. Evidence is scored independently: Gold units are requirement–required-evidence groups and predictions are requirement-reference–message-ID pairs. Any acceptable message covers a group; each predicted evidence item matches at most one group. Additional acceptable messages in covered groups and neutral messages receive neither additional credit nor FP penalties. Other messages are FP and uncovered groups are FN. Evidence scoring does not require a requirement match. RQ2 compares states of matched requirements. Fields first align by exact path, then by SAME STATE VARIABLE edges in maximum-cardinality one-to-one matching; correspondence does not imply a correct value. Numbers and booleans use typed exact comparison, enums and short values use normalized comparison, free text uses equivalence labels, unordered sets use set F1, ordered lists use longest-common-subsequence F1, records match under specified keys, and objects are compared recursively. Extra closed-world fields enter the prediction denominator; UNCERTAIN receives no equivalence credit. Let Fbr , Fr be the applicable predicted and Gold field sets, MrF their matching, − vij the value score, R− q the pre-request Gold requirement set, and Mq its matching: 2 ar =

P

(i,j)∈MrF vij

|Fbr | + |Fr |

,

Coverageq =

|Mq− | |R− q |

.

(3)

Attribute scores average over matched requirements with applicable fields, while coverage uses the full Gold requirement set. Unmatched extra requirements do not enter the conditional attributequality denominator. With no requirement matches, coverage is zero and conditional attribute quality is N/A. NULL EXACT requires explicit null; omission scores zero. SKIP or score=false items are excluded, and mutually empty collections do not provide full credit. No applicable fields on either side yields N/A; adding fields to empty Gold yields zero. The Gold specification determines applicability of the five state dimensions. RQ3 Decision Accuracy uses all 349 targets; ACT/CLARIFY Recall use 312/37 Gold targets, and Balanced Accuracy is their mean. Gold-ACT scoring includes newly introduced requirements. Each matched requirement receives the mean over applicable dimensions Dr , used in the Post-State Score defined in Section 3.2: sr =

1 X srd . |Dr |

(4)

d∈Dr

Exact post-request correctness requires ACT, no missing or extra requirements, correct applicable fields, no extra dimensions, and removed attribute keys matching Gold. This removal list and change type do not independently add to the continuous score. Gold-CLARIFY scoring first aligns requirements, blocker dimensions, and applicable fields, then matches equivalent missing information one to one to compute Blocking-Issue F1. Question Validity is the proportion of matched blockers whose questions elicit at least one Gold-acceptable fact, scoring zero without matches. Strict clarification additionally requires the complete blocker set and all questions valid. A wrong branch scores zero on its Gold-branch content metrics; the other branch is N/A. Implementation details for matching and comparison. Scoring proceeds through requirementrelation classification, one-to-one matching, attribute-field alignment, value comparison, branch scoring, and target aggregation. Requirement candidates include all prediction–Gold pairs; field 18

Preprint

candidates stay within the same matched requirement. RQ1 evidence scoring remains independent of successful requirement matching.

Table 10: Semantic relation labels for requirement matching. Label

Meaning

SAME ATOM

The same independently evolving requirement, possibly worded differently. The prediction combines this Gold requirement with other requirements. The prediction expresses only an attribute or rule fragment of the Gold requirement. Related content, but different independently evolving requirements. No requirement-level correspondence. Insufficient information for a reliable judgment.

MERGED ATOMS SUBPART OF ATOM RELATED DIFFERENT ATOM UNRELATED UNCERTAIN

RQ1 maximizes total edge weight among maximum-cardinality matchings. The weight is 106 round(106 Js ) + 103 round(103 Je ) + round(103 Ce ). Here, Js is summary-token Jaccard similarity using case-folded Unicode word-character tokens; Je is the Jaccard similarity between predicted message IDs and the union of Gold-acceptable IDs; and Ce is their intersection divided by that Gold union. Stable node and edge insertion resolves exact ties. RQ2/RQ3 use augmenting-path maximum-cardinality matching in prediction and Gold input order without these weights. Attribute alignment first fixes equal full paths and sends only remaining fields for semantic judgment. SAME STATE VARIABLE denotes the same variable, DIFFERENT STATE VARIABLE denotes different variables, and UNCERTAIN denotes insufficient information. Stable maximum-cardinality matching follows; value agreement does not determine field identity. The formula for ar applies specifically to attributes, rather than to all five state dimensions. The Gold specification recursively identifies applicable fields, excluding SKIP or score=false paths and descendants. Empty top-level attributes yield no fields; a nonexcluded nested empty object remains one predicted field. Unmatched extra attribute fields enter the prediction denominator, and missing fields remain in the Gold denominator. Other dimensions use their own comparators before averaging over applicable dimensions. Except for excluded items, missing or comparator-incompatible values score zero. Mutually empty collections do not automatically score one. A state with no applicable dimensions is N/A, contributes zero to the RQ3 continuous numerator, and is not exactly correct. The exact removal check takes Gold delta.removed paths entries beginning with attributes., removes that prefix, and compares sorted lists. The response retains change type, but the current implementation neither compares its semantic label nor adds it to continuous credit. Clarification edges require matching requirements and dimensions, plus exact fields when Gold specifies a field. Blocker matching follows prediction order and Gold indices without reoptimizing for question validity. Scoring illustrations, not experimental results. If Gold has two attribute fields and the prediction has three, with one matched field scoring one and all remaining fields unmatched or scoring zero, ar = 2/(2 + 3) = 0.4. In a separate one-requirement example, all applicable state values may be correct and Post-State Score may equal one. If the Gold removal list is [”old option”] but the prediction is [], exact correctness is still zero. These examples distinguish the continuous penalty for extra fields from the separate exact check of removals. The cross-diagnostic consistency check and human-review procedure are described in C.3, with a detected disagreement illustrated in E.2.

19

Preprint

Table 11: Field comparators and scoring rules. Comparator

English rule

NULL EXACT BOOLEAN EXACT NUMBER EXACT NORMALIZED EXACT

Requires explicit null; omission scores zero. Requires the same Boolean value and type. Requires an equal numeric value and excludes Booleans. Case-folds strings and collapses whitespace; other values use canonical JSON with sorted keys. Only EQUIVALENT between text values scores one. Uses F1 over normalized unique elements; mutually empty sets are N/A. Uses longest-common-subsequence F1, preserving order and repetitions; mutually empty lists are N/A. Averages applicable children; each extra field under closed world adds a zero-score item. Compares item fields and requires exact matching key fields under their comparators. Maximizes total credit over a one-to-one record matching, then computes soft F1; indices resolve ties. Excludes the item without awarding full credit.

SEMANTIC FACT SET F1 ORDERED LIST RECURSIVE FIELDS UNORDERED RECORD F1

SKIP or score=false

C.2

D ELIVERY D ECISIONS AND AGGREGATION

Delivery PASS requires Build, Target, and Regression to pass under DETERMINISTIC THREE GATE, without an LLM decision. Direct delivery has no ACT gate; the two-phase benchmark and Oracle Pre-State retain their action gates. Submitted projects use the same decision rules:

Table 12: Treatment of run outcomes in delivery scoring. Run outcome

Formal score

Handling

Agent incomplete, no valid deliverable submitted, or coding process failed Build, task-specific test, and regression checks all PASS Any validly executed check returns FAIL, with no HARNESS ERROR A check reports HARNESS ERROR

0

Record an agent delivery failure and retain the target in the denominator.

1

Record delivery PASS.

Checks unfinished or components incomplete

0

Record the primary failure in build, target, then regression order. No Delivery Pass Resolve the infrastructure or evaluator Rate reported yet error, then reevaluate the frozen submission. Pending evaluation Retain pending status; do not count as PASS or reduce the denominator without justification.

HARNESS ERROR is checked first, followed by build, target, and regression failures. Exit codes 0/1 indicate PASS/FAIL; other codes, startup errors, and check timeouts indicate HARNESS ERROR. After infrastructure recovery, the frozen submission is reevaluated; the invalid evaluation attempt is not a model failure. RQ1–RQ3 are computed per target and macro-averaged. RQ2 excludes legitimate N/A values but retains agent-failure zeros; coverage uses the full applicable set. RQ3 state and clarification content average over 312 and 37 Gold targets. Agent noncompletion, interruption, or unstarted runs score zero

20

Preprint

on higher-is-better metrics, while unsupported autonomy rate and unnecessary clarification rate receive one and therefore include failure penalties. Frozen responses awaiting judge evaluation remain pending, and the corresponding complete aggregate is withheld. Delivery uses a fixed denominator of 37 targets per configuration and condition. In gated conditions, CLARIFY, invalid responses, and missing deliverables score zero. Complete results require all pending items resolved without reducing the denominator. C1/C2 comparisons pair identical targets, and cross-configuration means weight all six configurations equally. Each target–condition– configuration has one planned run; technical retries do not add independent samples. Component pass rates and unexecuted checks. Build, Target, and Regression pass rates each use all 37 targets for the same configuration and condition. Only a PASS status adds one to the corresponding numerator: PassRatek =

1 X 1[statusik = PASS], 37

k ∈ {Build, Target, Regression}.

(5)

i∈Texec

Not entering coding, submitting no valid artifact, or failing during coding contributes no component PASS but remains in the denominator. These outcomes are not executed component FAILs. After Build FAIL, the validator marks the other checks SKIPPED: complete delivery scores zero, and the skipped checks contribute no PASS without being relabeled FAIL. Build HARNESS ERROR also skips the remaining checks, but the evaluation stays pending until the frozen submission is reevaluated after recovery. With no valid FAIL and an incomplete component set, evaluation remains pending rather than becoming a completed zero score. HARNESS ERROR takes precedence over valid check failures. Component rates therefore reflect execution/submission coverage as well as check outcomes, and changes in rates alone do not isolate artifact quality. Main-experiment C1 and C2 use this common reporting rule. C.3

J UDGE AND VALIDATOR V ERIFICATION

The semantic judge returns relation labels, while programs validate responses, match items, and calculate scores. Missing, duplicate, out-of-range, or invalid labels are rejected. Human verification follows the four-reviewer paired procedure in B.3 and examines requirement identity, state-variable identity, text equivalence, clarification blockers, and question validity. Two reviewers assess each project separately; agreeing judgments are accepted, and disagreements are reviewed again. E.2 presents a case of label disagreement across judge calls. Finally, we checked consistency across diagnostics. For each frozen response, the program aligned prediction–Gold requirement pairs in RQ1 and RQ2, verified that their descriptions and historical evidence agreed, and compared relation labels. It recorded differences in matching eligibility for individual requirement pairs and differences in the number of matches for each target. Human reviewers then resolved the flagged pairs using the requirement-atom definition and historical evidence, after which the program updated matching and recalculated affected metrics. Initial executable-validator checks show that all 37 retained targets among the 40 candidates have Build PASS, Target FAIL, and Regression PASS, indicating that the initial project is usable and the requested change remains incomplete. Oracle rescoring uses the validator, initial environment, and frozen submission associated with each target.

D

E XPERIMENTAL S ETTINGS AND P ROMPTS

D.1

M ODELS , I NPUTS , AND RUNTIME S ETTINGS

All six configurations use high reasoning and a fresh process for each target, condition, and configuration. Wall-clock limits are 900 seconds for diagnostics and 1,800 seconds for coding. The configurations are listed below. Diagnostic prompts embed the current task, visible history, instructions, and response schema in order, without runner-side truncation or summarization. Tool use is prohibited, and responses are frozen after contract validation. Two-phase coding receives the request, history, frozen response, and 21

Preprint

Table 13: Evaluated configurations and per-target time budgets.

Agent Harness

Harness Version

Model Identifier

Requirement Diagnostic Budget

Claude Code Claude Code Codex CLI Codex CLI Codex CLI Codex CLI

2.1.283 2.1.283 0.155.1 0.155.1 0.155.1 0.155.1

claude-opus-5-5 claude-sonnet-5 gpt-5.6-sol gpt-5.6-terra gpt-6-astra gpt-6-sol

900 s 900 s 900 s 900 s 900 s 900 s

Coding Budget 1,800 s 1,800 s 1,800 s 1,800 s 1,800 s 1,800 s

initial environment; direct delivery receives the request, history, and initial environment. Harness isolation and tool switches are listed below. Coding permits editing the current project and running local tests, while prohibiting network access, inspection of other runs, and changes to frozen responses. These are tool-use restrictions, without operating-system network isolation. All configurations share a gpt-5.6-sol, high semantic judge through stage1-chat-completions-v1, with concurrency 4, a 900-second timeout, and up to 3 retries. Service or parsing recovery reevaluates unscored frozen responses without regenerating answers or adding independent repetitions. Existing two-phase and recovery records cover October 15–25, 2026 (UTC). Beyond the reasoning levels and runtime budgets specified above, parameters follow harness and service defaults. We do not override output limits, temperature, random seeds, or automatic context compaction and truncation settings. Effective process commands. The original Codex coding configuration specifies workspace-write, while inspected processes use danger-full-access. The table lists effective settings based on the saved process records.

Table 14: Harness switches for diagnostic and coding runs. Harness and stage Codex, requirement diagnostics Codex, coding Claude, requirement diagnostics

Claude, coding

Key switches --ephemeral; --ignore-user-config; --ignore-rules; --skip-git-repo-check; --sandbox read-only --ephemeral; --ignore-user-config; --ignore-rules; --skip-git-repo-check; --sandbox danger-full-access --restricted; --no-session-persistence; --disable-slash-commands; --permission-mode dontAsk; --permission-prompts none; --tools ""; --disallowedTools mcp *; --strict-mcp-config; --no-chrome --restricted; --no-session-persistence; --permission-mode dontAsk; --tools and --allowedTools: Read,Edit,Write,Glob,Grep,PowerShell,Bash; --disallowedTools: mcp *,WebFetch,WebSearch; --strict-mcp-config; --no-chrome

22

Preprint

D.2

O RACLE I NPUTS AND S HARED TARGETS

Within a configuration, the three conditions share 37 unique targets, initial environments, tool permissions, a 1,800-second coding budget, and validators. Pre-State additionally has 900 seconds for requirement processing.

Table 15: Inputs supplied before and during coding.

Condition

Requirement Processing Before Coding

Inputs Visible During Coding

Direct delivery under No separate C1 requirement-assessment stage Oracle Pre-State Correct pre-request requirement state G− q + current request; history is empty Oracle Post-State

Current request + Full History + initial project environment Restricted projection of the predicted post-request requirement state + initial project environment, after ACT is selected Correct post-request requirement Restricted projection of the gold state G+ post-request requirement state + initial q supplied by the researchers project environment

Pre-State supplies correct pre-request states with neutral references and empty history. The model processes the request and its prediction is frozen; CLARIFY prevents coding and scores zero under the fixed denominator of 37. Both Oracle coding stages receive only requirement references, summaries, and final state fields from the frozen target state. The projection removes the original request, history, pre-states, decision, change type, and removed attribute keys. The full response is frozen separately and checked against the public projection. Post-State uses researcher-supplied Gold targets and does not establish correct model understanding. Each Oracle runs 37 targets, without duplicating history-free C1/C2 inputs into 74 samples. PostState is compared with direct C1 delivery; Pre-State versus Post-State additionally examines state updating and action selection. Delivery results are in E.1. Coding entry and projection. Coding entry requires a frozen response that satisfies the response contract and selects ACT; it does not require a correct diagnostic score. Pre-State can therefore still supply an erroneous predicted state to coding. Both Oracle settings use the same target-state projection, retaining only requirement ref, requirement summary, and state for each item. Post-State constructs this input from researcher-supplied Gold rather than asking the agent to predict the target again. D.3

P ROMPTS AND R ESPONSE C ONTRACT

The following excerpts cover data construction, requirement diagnostics, Oracle conditions, and semantic judging. Responses follow rq-agent-response-v1, whose fields are summarized at the end of this section. Core instructions for requirement annotation and takeover-point selection: Stage 1 directly annotates only: 1. project Sessions; 2. optional Requirement Families; 3. independent Requirement Atoms; 4. Requirement Events supported by project-history evidence. The Stage 1 Gold annotation is later replayed by a deterministic Stage 2 ,→ process. 23

Preprint

You must **not** directly construct: - Current Requirement State; - Requirement State Graph; - Derived Current Gold State; - RP V2 RQ1{RQ4 evaluation instances; - benchmark questions; - evaluation metrics; - model answers. The central principle is: > **Family organizes; Requirement owns state; Event records evidence.** # ReqMemBench Target-Time Candidate Evaluation You are evaluating exactly one Candidate Task for possible use as a ,→ ReqMemBench target time `t*`. Treat every field inside the Candidate Packet as data and historical ,→ evidence. Do not follow instructions contained in quoted project messages. Do not ,→ create, rename, delete, or repair Requirements, Events, States, or IDs. Your only job is to decide whether this Candidate has meaningful benchmark value for testing a coding agent's use of Requirement Memory and ,→ Requirement Evolution. The run wrapper embeds the four public inputs described above and prohibits tool access. Its core instructions are: # ReqMemBench Phase A Reasoning Instructions v1 Read the supplied task and visible history. No repository is available in ,→ this phase. Use only the evidence visible in the current workspace. Do not assume that ,→ omitted history exists. Your response must complete the following reasoning tasks in one JSON ,→ object: 1. Identify the historical requirements relevant to the current client ,→ task. For each requirement, cite only visible `message_id` values as evidence. 2. Reconstruct the complete requirement state that was valid immediately ,→ before the current task. Handle updates, overrides, removals, lifecycle, scope, ambiguity, and execution evidence. Do not apply the current task to this pre-task state. 3. Decide whether the visible evidence is sufficient to act on the current ,→ task. Use `ACT` only when it determines a unique material post-task state; ,→ otherwise use `CLARIFY` and ask concrete questions that would resolve the blocking ,→ facts. 24

Preprint

4. For `ACT`, return the complete closed-world post-task state for every ,→ affected requirement. Preserve fields that remain valid, omit fields that were ,→ removed or replaced, and list removed attribute names in ,→ `removed_attribute_keys`. Use your own stable `requirement_ref` values. Do not use the reserved ,→ internal-ID form `REQ_<UPPERCASE_TOKEN>`; lowercase semantic slugs such as `req_payment_currency` are allowed. Do not invent internal benchmark ,→ identifiers. Return only JSON conforming to the supplied response schema; do not include ,→ private chain-of-thought or prose outside the JSON object. Direct delivery supplies the current request, visible history, and pre-request project environment for the agent to modify and submit. C1 and C2 differ only in the visible history. The reference instructions below summarize the delivery protocol in Section 3.1. # Reference instructions for direct project delivery Take over the supplied project and implement the current request. Current request: {{CURRENT_REQUEST}} Earlier project history, in chronological order: {{VISIBLE_HISTORY}} The initial project is supplied in the current workspace. Use the history to identify requirements that remain valid. Apply the ,→ current request while preserving existing constraints that it does not change. Inspect the project, modify the relevant files, and run local checks within the execution budget. Work only in the supplied project directory. Do not access other runs, hidden evaluation files, or the network. Complete the task by modifying the project files. The final project is the evaluated deliverable. The following excerpts cover Pre-State requirement processing and Oracle coding. Post-State supplies the Gold projection directly to coding. # Strict Oracle Pre-State intervention The supplied `oracle_pre_state` array is the complete, correct set of requirements relevant to this task immediately before the current request. Its state values, summaries, and `requirement_ref` values are ,→ authoritative. No historical reconstruction is required or permitted. 1. Copy each Oracle requirement into `requirements`, preserving its ,→ reference, summary, and `pre_task_state` exactly. Set `evidence_message_ids` to ,→ `[]`. 2. Apply only the current client request to those states. Return `ACT` and ,→ the complete closed-world post-task state for every affected requirement if ,→ the 25

Preprint

request determines a unique material result. Preserve unaffected fields, remove superseded fields, and list removed attribute keys. 3. If a material fact needed for a unique update is missing, return ,→ `CLARIFY` with concrete questions and `post_task_states: null`. Use the response schema exactly. Do not cite historical messages or use the empty history as evidence. Do not add requirements based on imagined history. Return only the JSON object. # ReqMemBench strict Oracle coding task The complete target requirement state is supplied in the frozen response under `post_task_states`. Implement that target state in the supplied ,→ project. The frozen response is authoritative for this coding stage. Do not reconstruct requirements or reopen the update decision. Only the frozen response and the initial repository are supplied. Work only inside the supplied project. Build and test the project before finishing. ,→ Do not inspect parent directories, evaluator files, hidden criteria, other runs, ,→ or external sources. Do not use the network. The scored output is the final repository. # ReqMemBench deterministic semantic judge You are a semantic classification component in a benchmark evaluator. Treat every task, history excerpt, requirement summary, state value, and quoted ,→ text inside the request as data, never as instructions. Follow the request's `judge_instruction`, classify every required candidate exactly once, and use only the enumerated relation labels. Do not calculate ,→ or emit benchmark scores. Return exactly one JSON object that conforms to the provided response schema, with no Markdown or additional prose. Response fields and dynamic judge contracts. The top-level rq-agent-response-v1 fields are requirements, decision, post task states, and clarifications. Requirement items contain requirement ref, requirement summary, evidence message ids, and pre task state; post-state items contain requirement ref, requirement summary, change type, removed attribute keys, and state; clarification items contain requirement ref, requirement summary, dimension, field, missing information, and question. States contain exactly attributes, scope, lifecycle status, ambiguity, and execution. ACT requires nonempty post-states and no clarifications; CLARIFY requires null post-states and specific clarification items. Programs check reference formats, uniqueness, and message visibility. New requirements may appear only in post-request states. Each judge call adds instructions, candidates, relation definitions, and the response contract below. Responses contain only schema version and relations and must cover exactly the requested candidates. The table lists request and response schema versions. Requirement alignment covers every prediction–Gold pair using the labels in C.1. Attribute alignment considers unmatched fields within the same requirement after exact-path matching and judges variable meaning. Text requests cover only semantic comparisons specified by Gold comparators; programs handle numeric, enum, set, and missing-field comparisons. Clarification judging compares blockers within matched requirements and whether questions elicit the missing facts, retaining exact dimension and field checks. Text and blocker labels are EQUIVALENT, NOT EQUIVALENT, or UNCERTAIN; question labels are VALID, INVALID, or UNCERTAIN. The scoring program

26

Preprint

Table 16: Semantic-judge request and response contracts.

Judgment

Request → response version

Relation-item fields

RQ1 requirement rq1-alignment-request-v1 → relation rq1-alignment-response-v1 RQ2/RQ3 requirement requirement-alignment-request-v1 → relation requirement-alignment-response-v1 Attribute-field identity state-field-alignment-request-v1 → state-field-alignment-response-v1 Text equivalence state-semantic-request-v1 → state-semantic-response-v1 Clarification blocker rq3-clarification-semantic-request-v1 → and question rq3-clarification-semantic-response-v1

prediction ref, gold ref, relation prediction ref, gold ref, relation prediction field ref, gold field ref, relation fact id, relation candidate id, issue relation, question validity

constructs these requests, checks versions, fields, scope, duplicates, and coverage, then calculates scores. Experimental prompts and scoring code remain unchanged after the formal experiments. Diagnostic runs record prompt, instruction, and response-schema fingerprints and retain the four public inputs. Judge calls also retain requests, responses, and call records, with candidates, relation definitions, and response contracts in each request. Version checks use the saved records for the corresponding run.

E

S UPPLEMENTARY R ESULTS AND FAILURE C ASE

E.1

C OMPLETE D ELIVERY R ESULTS

Each configuration and condition uses the same 37 targets, and means weight the six configurations equally. The C1/C2 results used in the Oracle comparisons are the same main-experiment results, with the same history conditions and input-output procedure. Pre-State and Post-State scores use a common reviewed validator version and combine harness-level scores with independent rescoring, including GPT-5.6 Sol results not copied back to the original summary. Of the 222 Post-State configuration–target evaluations, 219 submit artifacts; 184 of these submissions pass build and regression checks but fail the target test, accounting for 84.02% of submitted artifacts. Counts corresponding to the history conditions in Table 4. The table expresses the pass rates in Table 4 as passing-target counts out of 37 for each configuration, using the same C1/C2 conditions as the main experiment. Oracle process records and frozen submissions. The table summarizes saved process and submission-freeze records for each run. Each configuration and condition uses 37 targets, totaling 222 target–configuration observations. Under Pre-State, 86 runs do not enter coding because the agent selects CLARIFY. Post-State has 219 process records and frozen submissions, while three runs have no frozen submission.

27

Preprint

Table 17: Complete-delivery counts and rates. C1 (Full C2 (Relevant Oracle Oracle History) History) Pre-State Post-State

Configuration

4/37 (10.81%) 2/37 (5.41%) 1/37 (2.70%) 2/37 (5.41%) 2/37 (5.41%) 0/37 (0.00%) 4.95%

Codex / GPT-6 Astra Codex / GPT-6 Sol Codex / GPT-5.6 Sol Codex / GPT-5.6 Terra Claude Code / Opus 5.5 Claude Code / Sonnet 5 Mean

2/37 2/37 4/37 (5.41%) (5.41%) (10.81%) 1/37 1/37 5/37 (2.70%) (2.70%) (13.51%) 2/37 4/37 5/37 (5.41%) (10.81%) (13.51%) 0/37 1/37 2/37 (0.00%) (2.70%) (5.41%) 1/37 2/37 2/37 (2.70%) (5.41%) (5.41%) 0/37 0/37 0/37 (0.00%) (0.00%) (0.00%) 2.70% 4.50% 8.11%

Table 18: Oracle component counts and rates by configuration. Configuration

Condition

GPT-6 Astra

Pre

GPT-6 Astra

Post

GPT-6 Sol

Pre

GPT-6 Sol

Post

GPT-5.6 Sol

Pre

GPT-5.6 Sol

Post

GPT-5.6 Terra

Pre

GPT-5.6 Terra

Post

Opus 5.5

Pre

Opus 5.5

Post

Sonnet 5

Pre

Sonnet 5

Post

Build

Target Regression All three

18/37 2/37 15/37 2/37 (48.65%) (5.41%) (40.54%) (5.41%) 35/37 5/37 30/37 4/37 (94.59%) (13.51%) (81.08%) (10.81%) 16/37 2/37 13/37 1/37 (43.24%) (5.41%) (35.14%) (2.70%) 35/37 6/37 30/37 5/37 (94.59%) (16.22%) (81.08%) (13.51%) 23/37 4/37 21/37 4/37 (62.16%) (10.81%) (56.76%) (10.81%) 36/37 5/37 35/37 5/37 (97.30%) (13.51%) (94.59%) (13.51%) 19/37 1/37 17/37 1/37 (51.35%) (2.70%) (45.95%) (2.70%) 34/37 2/37 33/37 2/37 (91.89%) (5.41%) (89.19%) (5.41%) 30/37 2/37 28/37 2/37 (81.08%) (5.41%) (75.68%) (5.41%) 37/37 2/37 37/37 2/37 (100.00%) (5.41%) (100.00%) (5.41%) 28/37 0/37 26/37 0/37 (75.68%) (0.00%) (70.27%) (0.00%) 37/37 0/37 37/37 0/37 (100.00%) (0.00%) (100.00%) (0.00%)

28

Preprint

Table 19: Component pass counts under C1 and C2 (37 targets per condition). Configuration

Condition Build Target Regression All three

GPT-6 Astra GPT-6 Astra GPT-6 Sol GPT-6 Sol GPT-5.6 Sol GPT-5.6 Sol GPT-5.6 Terra GPT-5.6 Terra Opus 5.5 Opus 5.5 Sonnet 5 Sonnet 5

C1 C2 C1 C2 C1 C2 C1 C2 C1 C2 C1 C2

21 16 18 12 22 17 20 14 28 27 21 19

6 3 3 1 1 2 2 0 2 1 0 0

17 14 17 9 19 14 18 10 24 26 19 17

4 2 2 1 1 2 2 0 2 1 0 0

Table 20: Oracle process and submission counts. Configuration GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Opus 5.5 Sonnet 5 Total

Pre Pre Pre no Post Post Post no records submissions submission records submissions submission 19 17 23 19 30 28 136

19 17 23 19 30 28 136

29

18 20 14 18 7 9 86

37 37 37 34 37 37 219

37 37 37 34 37 37 219

0 0 0 3 0 0 3

Preprint

E.2

S UPPLEMENTARY R EQUIREMENT D IAGNOSTIC R ESULTS

Table 3 in the main text presents selected RQ1–RQ2 metrics under C1; the complete diagnostic results are reported below. RQ1 is evaluated only under Full History (C1). Requirement and evidence Precision, Recall, and F1 are computed per target and macro-averaged over 349 targets per configuration. RQ2 compares C1 and C2: Requirement Coverage is averaged over all 349 targets, whereas Attribute Reconstruction Scores exclude legitimate N/A cases and retain zero scores for agent failures. N denotes the number of targets included in each attribute-score mean. RQ1 and RQ2 use the same historical requirement sets. We addressed relation-label disagreements from separate judge calls through automated detection, human adjudication, and metric recalculation, as described in C.3. A case recorded before adjudication is shown below. Decisions. Values are C1/C2 pairs. With 312 Gold-ACT and 37 Gold-CLARIFY targets, always ACT yields accuracy 0.8940 and balanced accuracy 0.5. Branch content. Post-State uses 312 Gold-ACT targets; blocker, question, and strict-clarification metrics use 37 Gold-CLARIFY targets, with wrong branches scoring zero. Exact Post-State Accuracy is 0/312 for all six configurations under C1/C2. State quality given correct ACT. The following reports conditional means and counts for GoldACT targets predicted ACT. Set and content decomposition. A requirement set fully matches when predicted, Gold, and matched counts satisfy p = g = m > 0; Post-State then equals the mean matched-state score. To control target changes caused by C1/C2 action selection, the paired comparison uses the intersection of correct-ACT targets. Within the correct-ACT subsets, the fully matched-set proportions averaged across configurations are 27.32%/32.29%, with subset Post-State means 0.2913/0.3843 and 304/285 target×configuration observations. The shared correct-ACT subset contains 792 observations, with the configuration mean rising from 0.1503 to 0.1923. Statistics are computed within configurations before equalweight averaging; pooled observations are not independent targets. Cross-RQ label disagreement before human adjudication. For 42204309 T007, two judge calls assigned different labels to the same prediction–Gold requirement pair. The table retains the labels and scores before adjudication to illustrate the detected conflict. Recall and Coverage are target-level metrics. After the program detected this disagreement, human reviewers resolved it using the requirementatom definition and source evidence. The program then updated matching and recalculated the affected metrics. Decomposition of Exact Post-State Accuracy. Using only saved frozen scores, we assign each Gold-ACT target to the first unmet condition in the order of action, requirement set, state fields, and removed-attribute list. Categories are disjoint and sum to 312 under each history condition. The first category includes wrong action decisions and incomplete responses. The removal-list category applies only after all earlier conditions pass, so zero there does not imply correct lists on all other targets. Each cell reports C1 / C2 target counts. All 589 saved scores with fully matched requirement sets fail at least one strict state-correctness condition; no target fails only the final removal-list condition. This explains the zero strict accuracy, while the existing continuous scores retain partial credit. The audit generated no new responses and changed neither semantic labels nor existing scores.

30

Preprint

Table 21: Historical information identification and pre-request state reconstruction. (a) Requirement and evidence identification under Full History (C1). (b) Attribute reconstruction and requirement coverage; within each numeric cell, C1 is above C2. Coverage uses 349 tasks per configuration. Attribute scores exclude legitimate N/A cases and retain zero scores for agent failures. Attributescore sample sizes are given in Table 22. (a) Historical information identification Requirement

Configuration

Evidence

P

R

F1

P

R

F1

GPT-6 Astra

0.2577

0.6821

0.3423

0.2254

0.9211

0.3166

GPT-6 Sol

0.3196

0.6704

0.3980

0.3395

0.8620

0.4207

GPT-5.6 Sol

0.3768

0.6302

0.4331

0.3162

0.8182

0.3939

GPT-5.6 Terra

0.4428

0.5232

0.4507

0.4254

0.6670

0.4416

Claude Opus 5.5

0.3567

0.6232

0.4193

0.3189

0.8364

0.4048

Claude Sonnet 5

0.4027

0.5835

0.4462

0.3060

0.7461

0.3766

(b) Pre-request state reconstruction Configuration

Attribute

Requirement

score

coverage

0.2813 0.3019 0.2671 0.3062 0.2451 0.3127 0.2424 0.2918 0.2811 0.3342 0.2548 0.2615

0.6689 0.6607 0.6777 0.6271 0.6157 0.5992 0.4950 0.5834 0.5934 0.5878 0.5704 0.5715

GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Claude Opus 5.5 Claude Sonnet 5

Table 22: Numbers of targets included in the Attribute Reconstruction Score means. Configuration

N C1 N C2

GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Claude Opus 5.5 Claude Sonnet 5

265 277 251 221 253 238

31

266 252 249 251 246 241

Preprint

Table 23: Action-selection performance under C1/C2. Paired entries show C1 above C2. Configuration GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Claude Opus 5.5 Claude Sonnet 5

Decision Balanced Accuracy Accuracy ACT Recall CLARIFY Recall 0.5931 0.4670 0.5043 0.3954 0.5358 0.4327 0.5415 0.4413 0.7880 0.7335 0.6361 0.5673

0.6771 0.6066 0.6156 0.5666 0.6094 0.5874 0.6483 0.6041 0.6670 0.6961 0.6774 0.6270

0.5705 0.4295 0.4744 0.3494 0.5160 0.3910 0.5128 0.3974 0.8205 0.7436 0.6250 0.5513

0.7838 0.7838 0.7568 0.7838 0.7027 0.7838 0.7838 0.8108 0.5135 0.6486 0.7297 0.7027

Table 24: Post-request state and clarification scores under C1/C2. Paired entries show C1 above C2.

Configuration GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Claude Opus 5.5 Claude Sonnet 5

Strict Blocking-Issue Question Clarification Post-State F1 Validity Success Rate 0.0741 0.0744 0.0535 0.0600 0.0538 0.0569 0.0653 0.0870 0.1670 0.1710 0.1041 0.1064

0.0874 0.0667 0.1051 0.0667 0.0739 0.0802 0.0676 0.0901 0.0423 0.1523 0.0694 0.0874

0.1622 0.1351 0.2162 0.1351 0.1622 0.1622 0.1351 0.1351 0.0811 0.2703 0.1081 0.1351

0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0270 0.0000 0.0000 0.0270 0.0270

Table 25: Post-State Scores conditional on correct ACT decisions.

Configuration GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Claude Opus 5.5 Claude Sonnet 5

Correct ACT Conditional Correct ACT Conditional Post-State Post-State count count C2 C2 C1 C1 178 148 161 160 256 195

0.1298 0.1129 0.1043 0.1273 0.2036 0.1665

32

134 109 122 124 232 172

0.1733 0.1719 0.1456 0.2189 0.2300 0.1929

Preprint

Table 26: Requirement-set matching and shared-target Post-State Scores under C1/C2. Paired entries show C1 above C2. Matched-set Matched-set Shared ACT Shared-target count C1 / C2 Post C1 / C2 count Post C1 / C2

Configuration GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Claude Opus 5.5 Claude Sonnet 5

50 41 36 38 39 40 46 41 79 68 54 57

0.2882 0.3715 0.2422 0.3492 0.2299 0.3215 0.2899 0.4563 0.3656 0.4468 0.3320 0.3604

127 99 107 100 221 138

0.1384 0.1710 0.1317 0.1591 0.1051 0.1528 0.1471 0.2358 0.2082 0.2320 0.1709 0.2029

Table 27: Cross-diagnostic label disagreement before human adjudication. Evaluation RQ1 RQ2

Prediction reference

Gold requirement

reward_pay Prize Claim Flow out reward_pay Prize Claim Flow out

Relation

Target metric

MERGED ATOMS

Requirement Recall =0 Requirement Coverage = 0.25

SAME ATOM

Table 28: Decomposition of exact post-request state failures under C1/C2. Configuration

Set State not Removal list No correct ACT mismatch fully exact only Exact success

GPT-6 Astra GPT-6 Sol GPT-5.6 Sol GPT-5.6 Terra Claude Opus 5.5 Claude Sonnet 5

134 / 178 128 / 93 164 / 203 112 / 71 151 / 190 122 / 82 152 / 188 114 / 83 56 / 80 177 / 164 117 / 140 141 / 115

33

50 / 41 36 / 38 39 / 40 46 / 41 79 / 68 54 / 57

0/0 0/0 0/0 0/0 0/0 0/0

0/0 0/0 0/0 0/0 0/0 0/0

Preprint

E.3

PCB FAILURE C ASE

PCB case: Relevant requirements are found, but the target state remains incomplete. Under C1 on 43214420_T004, the client asks to move the battery connector to the back of the board, on the side opposite the USB-C connector. Opus 5.5 identifies both relevant historical requirements, achieving requirement Precision and Recall of 1, evidence Recall of 1, evidence Precision of 0.625, and RQ2 Requirement Coverage of 1. Its pre-request state already includes layout constraints such as 8 mounting holes and the M2.5 specification. However, its post-request response retains only the battery connector and omits the entire mechanical-layout requirement, yielding a Post-State Score of 0.3259. The final delivery passes the build and regression checks but fails the task-specific test. This record shows the co-occurrence of historical coverage, target-state omission, and delivery failure; the case alone does not establish causality. The continuous score is also affected by comparisons of other battery-connector fields, whereas omission of the entire mechanical-layout requirement can be checked directly from the prediction structure. This case comes from the benchmark’s linked two-phase evaluation and illustrates omission of a valid constraint within one response, together with the corresponding delivery checks. Oracle artifact-implementation case. GPT-5.6 Sol’s 43214420 T004 comes from a separate Oracle Post-State run, not coding continued from the Opus diagnostic response above. The structured artifact model (artifact_model) and design summary update battery-connector placement to the back opposite USB-C, but the delivered PCB file is byte-identical to the initial archive. Build and regression pass; the target test reports that the connector-edge normals are not opposite. State and artifact comparison. The tables correspond to the two independent runs described above. The first examines one requirement response; the second examines the Oracle deliverable.

Table 29: Requirement-state omission in the Opus diagnostic case. Opus diagnostic record Pre-request mechanical requirement Post-request references Identification and coverage Post-request state Checks Target error

Saved field and result requirement ref = req power board outline; mounting holes = ”8 at the corners”; mounting hole size = ”M2.5” only req battery connector placement; no req power board outline Requirement Precision = 1; Recall = 1; RQ2 Coverage = 1 Post-State Score = 0.325926; mechanical requirement absent Build PASS; Target FAIL; Regression PASS connector edge normals are not opposite (dot=0.000000)

34

Preprint

Table 30: Artifact changes and delivery checks in the Oracle PCB case. GPT-5.6 Sol Oracle artifact

File comparison and checks

Changed files

Design summary, artifact model, current-state representation, and test files Byte-identical to initial archive Build PASS; Target FAIL; Regression PASS connector edge normals are not opposite (dot=1.000000)

Delivered PCB Checks Target error

The omission in the first run can be checked directly from its output items. Comparing the initial and final archives in the second run distinguishes changes to structured descriptions from changes to the delivered PCB artifact.

35

Related documents

• Do Personalized Skills Help Coding Agents? An Empirical Study of Developer Interaction Histories#610038
• Do Personalized Skills Help Coding Agents? An Empirical Study of Developer Interaction Histories#608976
• Do Personalized Skills Help Coding Agents? An Empirical Study of Developer Interaction Histories#444707
• Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025#635698
• Professional Software Developers Don't Vibe, They Control: AI Agent Use for Coding in 2025#633349