CodeSpec: Dual Executable Specifications for Agentic Long-Horizon Feature Development Peiding Wang1 , Li Zhang1 , Fang Liu1∗ , Taichuan Li1 , Yinghao Zhu2 1
State Key Laboratory of Complex & Critical Software Environment, School of Computer Science and Engineering, Beihang University, China 2 School of Computing and Data Science, The University of Hong Kong {wangpeiding, fangliu}@buaa.edu.cn Feature Instruction + Existing Repository
arXiv:2607.26777v1 [cs.SE] 29 Jul 2026
Abstract LLM-based code agents have advanced repository-level software development through iterative interaction with codebases and tools. However, feature development requires integrating new behaviors into existing architectures through coherent cross-component functional chains. Existing agents typically derive such chains through free-form reasoning, often producing unreliable feature designs with incomplete functional chains. Moreover, textual designs are difficult to verify and enforce, making it challenging to maintain design–implementation consistency throughout long-horizon development. We propose CodeSpec, a dual executable specification method for repository-level feature development. It builds reliable functional chains from evidence pairing subrequirement semantics with repository architectures, then compiles them into complementary architecture and behavior specifications that check chain completeness and correctness while preserving design–implementation consistency over long interactions. On FeatureBench, which targets feature development in existing repositories, CodeSpec achieves 70.7%, 55.0%, and 49.9% pass rates under DeepSeek-V4-Pro, outperforming representative baselines such as Claude Code. Results on the repository generation benchmark NL2Repo-Bench further demonstrate its generalizability. The code and data are available at https://github.com/zhu-zhu-ding/CodeSpec.
Introduction With the rapid advancement of large language models, automated software development has evolved from standalone function generation (Chen 2021; Austin et al. 2021) to repository-level tasks (Li et al. 2024; Jimenez et al. 2024). Feature development is particularly important because realworld software systems are continuously extended to satisfy evolving requirements (Du et al. 2025; Li et al. 2025; Zhou et al. 2026). Unlike standalone generation, implementing a new feature requires integrating new behaviors into an existing architecture. Agents must determine not only which code entities should be created or modified, but also how these entities should interact across components. Feature development therefore commonly requires the design stage that orga∗
Corresponding author. Copyright © 2027, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
Task Statement • Implement MLflow Distributed Tracing System Core Functionalities • Create and manage distributed tracing ...... • Support both live and immutable span ......
Repository-Constrained Design # base_mlflow.py
Class BaseMlflow SpanProcessor()
# span.py import
dispatch
LiveSpan/Span/...
APIs Files Modules Constraints Dependencies
Long-Horizon Constraint Drift Turn 1
create_mlflow_span()
invoke ?
MLFlow Repository
• • • • •
Turn 50
Turn 150
Processor → invoke → Factory
propagate ?
# provider.py
# trace_data.py
# constant.py
?
# trace_manager.py
Factory → dispatch → Span types
TraceData
# fluent.py
Turn 300
serialize # utils.py
# otel.py
?
TraceData → serialize → OTLP
OTLP
Functional-chain constraints gradually fade
Locally Plausible Code, Incomplete Feature Realization Required Factory Call class BaseMlflowSpanProcessor(...): def on_start(self, span, parent_context): mlflow_span = create_mlflow_span( span, trace_id, span_type )
Generated Local Call class BaseMlflowSpanProcessor(...): def on_start(self, span, parent_context): mlflow_span = LiveSpan( span, trace_id, span_type ) bypasses the required factory relation
Figure 1: Feature development requires agents to derive repository-consistent functional chains and preserve them throughout implementation. Missing a critical architectural relation, such as invoking factory method create_mlflow_span(), may break the end-to-end feature behavior. nizes these entities and interactions into a coherent functional chain from feature entry points to observable behaviors. The repository-level nature of feature development makes LLM-based coding agents a promising paradigm. Representative agents, including Claude Code (Anthropic 2026), Codex (OpenAI 2025), SWE-agent (Yang et al. 2024), and OpenHands (Wang et al. 2025c), rely on the agent’s reasoning capabilities to implicitly plan development through naturallanguage interactions while iteratively generating and repairing code. Other approaches introduce explicit design processes: MetaGPT (Hong et al. 2024) and ChatDev (Qian et al. 2024) produce natural-language artifacts describing module responsibilities and collaboration workflows, while RTADev (Liu et al. 2025) performs architecture planning before implementation. However, both free-form implicit planning and explicit textual design provide limited support for deriving reliable repository-grounded functional chains and
enforcing them throughout long-horizon development. As illustrated in Figure 1, existing agents face two closely related challenges: • Unreliable functional-chain design. Existing agents typically design features through free-form reasoning, which can deviate from repository architecture constraints and overlook the call and dependency relations among code entities. Consequently, they may derive functional chains that are incomplete or inconsistent with the repository. • Weak long-horizon enforcement. Functional designs are typically represented as free-form textual plans, which are difficult to verify and enforce during iterative exploration, editing, and repair. As a result, the implementation may gradually diverge from the intended functional chain, leading to inconsistencies between design and realization. To address these challenges, we propose CodeSpec, a dual executable specification method for agentic long-horizon feature development. It constructs repository-grounded functional-chains for feature sub-requirements and compiles each chain into complementary architecture and behavior executable specifications. CodeSpec consists of two key components: 1) Evidence-Grounded Functional Chain derives traceable feature realization paths by identifying relevant functional units and connecting them with paired requirement semantics and repository as evidence, such as design patterns, call relations, and dependencies. This reduces free-form architectural reasoning and grounds each transition in the existing repository to improve the design reliability. 2) Dual Executable Specification Compilation converts each functional chain into two executable forms: an architecture specification that represents its architecture realization and a behavior specification that represents its expected execution. They complementarily verify whether the chain is complete and whether it behaves correctly, thereby preserving design–implementation consistency throughout long-horizon development. Experiments on FeatureBench (Zhou et al. 2026) show that CodeSpec achieves 70.7%, 55.0%, and 49.9% %Passed on the Lite, Fast, and Full splits, consistently outperforming coding-agent and architecture-planning baselines under the same DeepSeek-V4-Pro backbone. Results with GPT5.4-mini and NL2Repo-Bench (Ding et al. 2026) which evaluates greenfield repository development from naturallanguage requirements further demonstrate robustness across models and generalization to greenfield repository generation. Ablation and further analyses confirm the effectiveness of evidence-grounded functional chains and the complementary executable specifications, particularly on complex and long-horizon tasks, with moderate additional cost. Our main contributions are summarized as follows: • We propose evidence-grounded functional-chain reasoning that pairs task semantics with repository architecture evidence at each transition, enabling agents to derive reliable cross-component realization paths from feature entry points to observable behaviors. • We introduce dual executable specifications that represent the architecture realization and expected execution
of each functional chain, jointly verifying whether the chain is complete and behaves correctly while maintaining design–implementation consistency throughout longhorizon development. • We conduct extensive experiments on FeatureBench and NL2Repo with different backbone models, demonstrating the effectiveness, robustness, generalizability, and favorable quality–cost trade-off of CodeSpec for long-horizon feature development.
Related Work Repository-level Feature Development Benchmarks Code generation research has evolved from standalone function synthesis (Chen 2021; Austin et al. 2021) to repositorylevel software development, where models need to understand project context and generate patches in real codebases. Benchmarks such as DevEval (Li et al. 2024) and SWEbench (Jimenez et al. 2024) evaluate such repository-level capabilities through realistic software engineering tasks. More recent feature-level benchmarks, including FEA-Bench (Li et al. 2025), and FeatureBench (Zhou et al. 2026), further focus on practical feature development, where a high-level instruction may require coordinated changes and generation across multiple functions, files, classes, and interfaces. These tasks usually involve a full development process of design, implementation, testing, execution, and repair, making oneshot generation insufficient and motivating coding agents that interact with repositories and development environments over multiple steps.
LLM-based Coding Agents LLM-based coding agents have become a popular paradigm for repository-level software development (Wang et al. 2026, 2025b,a). Unlike one-shot generation methods, they can autonomously invoke tools, inspect files, text plan, edit programs, and iteratively repair failures. Representative coding agents and assistants include Claude Code (Anthropic 2026), Codex (OpenAI 2025), SWE-agent (Yang et al. 2024), and OpenHands (Wang et al. 2025c). In addition, developmentoriented agent frameworks such as MetaGPT (Hong et al. 2024), ChatDev (Qian et al. 2024) and RTADev (Liu et al. 2025) organize software development into structured stages and use textual architecture designs to coordinate software development. Despite their progress, existing agents still mainly maintain design information as free-form textual plans, which makes it difficult to explicitly preserve functional chains and architecture constraints during longhorizon feature development.
Method Task Definition In this section, we define the feature development task. Given a natural-language feature instruction I and an existing repository R, an LLM-based coding agent generates a patch P to get the updated repository R′ which preserving existing functionality and satisfying the instruction I: P = LLM(I, R),
R′ = R ⊕ P.
(1)
1 Functional Chain Construction
2 Dual Executable Specification Compilation
Evidence-Grounded Functional Chain Requirements Semantics
Processor
invoke
< ρ1> <e1>
<d1>
Inputs
Call Relations
<u2>
< ρ2>
Factory
dispatch
<e2>
<u3> SpanType
propagate
<d3>
<d2>
...
<uk>
< ρk-1> <ek-1>
...
data flow
<dk>
Repository Grounding Requirements
Feature Instruction
Repository Architecture
Paired Evidence
Design Patterns
<u1>
Executable Architecture Specification
Create span... Propagate...
span.py
io.py
...
util.py
Behavior Analysis
all checks pass
Execute Specifications
types.py
trace.py
Executable Behavior Specification 1 def test_subfunction_chain_i(): # requirement: <required feature behavior i> 2 # chain: u1 -> u2 -> ... -> uk 3 # evidence: desgin patterns, call relations, ... 4 # behaviors: b1, b2, b3, ..., bn 5 6 check_output("<π1>", "<boutput1>") 7 check_boundary("<π1>", "<bboundary1>") 8 check_state("<π1>", "<bstate1>") 9 ... 10 11 check_output("<πn>", "<boutputn>") check_boundary("<πn>", "<bboundaryn>") 12 check_state("<πn>", "<bstaten>") 13 14 return feedback(info) 15
3 Specification Guided Code Generation
Code Units
Export OTLP...
Existing Repository
1 def spec_subfunction_i(repo): # requirement: <required feature behavior i> 2 # chain: u1 -> u2 -> ... -> uk 3 # evidence: desgin patterns, call relations, ... 4 # data flow: invoke, dispatch,propagate, ... 5 6 check_unit(repo, "<u1>", "<u2>") 7 check_relation("<u1>", "<u2>", "< ρ1>") 8 check_data_flow("<u1>", "<u2>", "<d1>") 9 ... 10 check_unit(repo, "<uk-1>", "<uk>") 11 check_relation("<uk-1>", "<uk>", "<ρk-1>") 12 check_data_flow("<uk-1>", "<uk>", "<dk>") 13 14 return feedback(info) 15
Edit Patch
base.py
Architecture Checks
Path Integrity
Behavior Checks
Runtime Correctness
Verified Patch
Arch Feedback
Chain Violation Behavior Feedback
Revise
Behavior Failure
long-horizon constraint preservation
Figure 2: The overview of CodeSpec.
Overview As shown in Figure 2, CodeSpec constructs reliable functional chains using evidence that pairs sub-requirement semantics with the corresponding repository architecture. It then compiles each chain into dual executable specifications: an architecture specification representing its structural realization and a behavior specification representing its expected execution. These specifications complementarily verify whether the chain is complete and behaves correctly, thereby maintaining design–implementation consistency throughout long-horizon feature development.
These units provide candidate building blocks, but selecting relevant units alone does not yield a valid feature design; the key is determining how they should be connected to realize each obligation. CodeSpec therefore prompts the LLM to perform evidence-driven chain reasoning by pairing requirement evidence, which explains why a transition is needed from requirements, with repository evidence, which shows how it is supported by existing design patterns, call relations, or repository architectures. The resulting functional chain is an ordered sequence in which each adjacent pair of units uki and uk+1 is linked according to its supporting evidence eki : i ci = {u1i , e1i , u2i , . . . , uki , eki , uk+1 }. i
Evidence-Grounded Functional Chain A feature is realized through coordinated interactions among multiple repository components rather than isolated code modifications. However, feature instructions usually describe desired behaviors without specifying which components should participate or how they should interact. To bridge this gap, CodeSpec constructs an evidence-grounded functional chain for each sub-function by pairing requirement semantics with repository architecture evidence at every reasoning step. Given a feature instruction I, the agent first decomposes it into a set of sub-requirements:
(4)
Here, uki denotes a functional unit involved in realizing requirements ri , and eki denotes the evidence supporting the transition from uki to uk+1 . Each evidence eki pairs rei quirement semantics with repository evidence, ensuring that adjacent units are connected through traceable, repositorysupported reasoning rather than free code relations.
Dual Executable Specification Compilation
Each requirement ri represents a specific behavior that the feature should achieve. For each requirement, the agent identifies candidate implementation units that need to be modified or introduced:
The constructed functional chains explicitly describe how individual feature requirement should be realized across repository components. However, retaining these chains as textual design artifacts is unsuitable for long-horizon development, because agents may gradually overlook pre-designed key information during long-term interactions. Therefore, we compile each functional chain into two complementary executable specifications: the architecture specification that verifies path integrity and the behavior specification that verifies runtime correctness along the path.
Uri = {u1 , u2 , . . . , un }.
Executable Architecture Specification Given a subrequirement ri and its functional chain ci , CodeSpec instan-
LLM(I) = {r1 , r2 , . . . , rm }.
(2)
(3)
tiates an executable architecture specification sarch using the i template τarch shown in Figure 2: sarch = LLMarch (R, ri , ci ; τarch ). i
(5)
Inspired by architecture conformance checking and program analysis (Pruijt et al. 2017; Murphy, Notkin, and Sullivan 2001; Kildall 1973), we operationalize path integrity using three representative dimensions: functional units, architectural relations, and data flows. The template maps the units, relations, and data flows in ci into three types of checks: sarch = i
oki n CheckUnit(uji )
which may cover a functional unit, a transition, or a subpath. The checks examine observable outcomes, including returned values and external effects, boundary and exceptional conditions, and state transitions at multiple positions along the functional chain. Thus, the behavior specification examines multiple intermediate and final behaviors rather than only the final output. During implementation, violated expectations are reported as localized behavior violations. In this way, executable behavior specifications complement architecture specifications by preserving runtime correctness along each functional chain.
Specification Guided Code Generation
j=1
oki −1 n j ) , ρ ∪ CheckRelation(uji , uj+1 i i
j=1 oki −1
n , dji ) ∪ CheckDataFlow(uji , uj+1 i
(6) .
S arch =
j=1
Here, ρji and dji are jointly inferred from the requirement and repository evidence in eji , representing the intended architectural relation and data-flow operation between uji and uj+1 , respectively. CheckUnit verifies that each required i unit exists, CheckRelation checks whether adjacent units preserve the intended relation ρji , and CheckDataFlow verifies the required data state dji between them. During implementation, the specification is executed against the updated repository. Missing units, broken relations, or interrupted data flows are reported as localized architecture violations. In this way, executable architecture specifications preserve the path integrity of each functional chain. Executable Behavior Specification Given the subrequirement ri , its functional chain ci , and the corresponding architecture specification sarch , CodeSpec jointly analyzes i the intended path structure and architectural constraints to identify behavior-observation subpaths, and instantiates an executable behavior specification using the template τbeh shown in Figure 2: sbeh = LLMbeh (ri , ci , sarch ; τbeh ). i i
(7)
Inspired by established property-based, categorypartition, and model-based testing techniques (Claessen and Hughes 2000; Ostrand and Balcer 1988; Chow 1978), we operationalize behavioral obligations using three representative dimensions: observable outputs, boundary conditions, and state transitions. The template derives three types of behavioral checks at multiple observation scopes along the functional chain: sbeh = {CheckOutput(πiq , boutput ), i i,q CheckBoundary(πiq , bboundary ), i,q
For all sub-requirements, the generated architecture and behavior specifications are aggregated as:
(8)
ni CheckState(πiq , bstate i,q )}q=1 .
Here, ni denotes the number of behavioral observation scopes derived for ci , and πiq ⊆ ci denotes the q-th scope,
m [
sarch , i
i=1
S beh =
m [
sbeh i .
(9)
i=1
The agent implements the feature under the joint constraints of S arch and S beh . Architecture feedback identifies missing units, broken relations, or interrupted data flows, while behavior feedback reports incorrect runtime behaviors along the functional chains. The agent incorporates both signals into subsequent edits until all specifications pass or the interaction budget K is exhausted. Algorithm 1: Specification Guided Code Generation Input: Instruction I, repository R, specifications S arch and S beh , budget K Output: Patch P 1: P, Farch , Fbeh ← ∅ 2: k ← 0 3: while k < K do 4: ∆P ← LLMcode (I, R, S arch , S beh , P, Farch , Fbeh ) 5: P ← P ⊕ ∆P 6: R′ ← R ⊕ P 7: Farch ← ExecArch(R′ , S arch ) 8: Fbeh ← ExecBeh(R′ , S beh ) 9: if Pass(Farch ) ∧ Pass(Fbeh ) then 10: break 11: end if 12: k ← InteractCount() 13: end while 14: return P Here, ExecArch verifies the integrity of the functional chains, whereas ExecBeh verifies their runtime behaviors. Their complementary feedback jointly preserves path integrity and behavioral correctness throughout long-horizon implementation.
Experimental Setups Research Questions We evaluate CodeSpec through the following research questions:
Lite
Agent Scaffold OpenHands Claude Code Mini-SWE-Agent RTADev CodeSpec
Fast
Full
%Passed
#Resolved
Avg. Cost ($)
%Passed
#Resolved
Avg. Cost ($)
%Passed
#Resolved
Avg. Cost ($)
60.0±5.7 59.8±5.4 62.6±6.0 65.3±5.6 70.7±5.1
6 6 7 6 9
0.15 0.11 0.09 0.17 0.18
41.4±3.6 47.0±3.8 49.6±4.0 49.7±3.8 55.0±3.8
10 14 24 15 17
0.13 0.12 0.09 0.16 0.19
37.1±2.7 41.1±2.8 43.4±2.8 46.1±2.8 49.9±2.8
18 23 25 26 28
0.11 0.10 0.09 0.15 0.16
Table 1: Comparison under the DeepSeek-V4-Pro (max thinking) backbone model on FeatureBench. The Lite, Fast, and Full splits contain 30, 100, and 200 tasks, respectively. Bold numbers indicate the best performance.
Agent Scaffold
%Passed
#Resolved
Avg. Cost ($)
Mini-SWE-Agent OpenHands RTADev Codex CodeSpec
47.5±6.0 48.1±5.9 45.9±5.7 51.7±6.2 54.7±5.8
2 3 3 4 5
0.7 1.2 1.3 0.7 1.3
Table 2: Comparison of different coding agents with the same GPT-5.4-mini backbone (medium thinking) on the FeatureBench Lite split. • RQ1: Overall Performance. How does CodeSpec compare with existing coding agents on repository-level feature development tasks? • RQ2: Ablation Study. How does each component contribute to the CodeSpec’s performance? • RQ3: Executable vs. Textual Specifications. Do executable specifications preserve design constraints more effectively than textual specifications? • RQ4: Greenfield Repository Development Generalization. Can CodeSpec generalize from repository evolution to greenfield repository generation?
Benchmarks and Metrics We evaluate CodeSpec on FeatureBench (Zhou et al. 2026), a repository-level feature development benchmark requiring coordinated modifications across mutli-interfaces. Each problem averages 4.8k words and requires about 800 lines of valid code, compared with 0.2k words and about 32 lines in SWE-Bench(Jimenez et al. 2024). It contains three splits: Lite (30), Fast (100), and Full (200). Following FeatureBench, we report %Passed, the average proportion of fail-to-pass tests passed across tasks, and Resolved #, the number of tasks for which all required tests pass. The former measures partial implementation progress, while the latter measures complete task resolution. We also reporte the average cost metric for each task: Avg.Cost ($).
Baselines and Implementation Details We compare CodeSpec with five representative coding-agent scaffolds:
• OpenHands (Wang et al. 2025c), a general-purpose agent supporting repository exploration, code editing, execution, and iterative repair. • Mini-SWE-Agent (Yang et al. 2024), a lightweight agent based on a minimal repository interaction and repair loop. • Claude Code (Anthropic 2026), a widely adopted industrial coding agent for autonomous repository editing, execution, and debugging. • Codex (OpenAI 2025), an OpenAI coding agent supporting repository-level reasoning, tool use, code modification, and iterative refinement. • RTADev (Liu et al. 2025), an architecture-first agent that uses textual architecture designs to guide development; For controlled comparison, all methods (except for Codex) use DeepSeek-V4-Pro as the backbone model. We additionally evaluate GPT-5.4-mini to examine model-level generalization. All methods share the same repository environments, evaluation scripts, and interaction budgets. We bootstrap task-level results 10,000 times and report the resulting mean and standard deviation.
Experimental Results RQ1: Overall Performance As shown in Table 1, CodeSpec consistently achieves the best performance across all FeatureBench splits under the same DeepSeek-V4-Pro backbone. On the Lite split, CodeSpec reaches 70.7% %Passed and resolves 9 tasks, outperforming the strongest baseline RTADev by 5.4 percentage points in %Passed and 3 resolved tasks. Similar improvements are observed on the Fast and Full splits, where CodeSpec improves %Passed over RTADev by 5.3 and 3.8 percentage points, respectively, while achieving the highest number of resolved tasks. Compared with general-purpose coding agents such as OpenHands and Mini-SWE-Agent, CodeSpec obtains larger gains across all splits. Unlike textual plans, which remain passive context and can be overlooked during extended interactions, executable specifications continuously expose whether the intended functional chains remain complete and correct. Their lightweight feedback also helps agents localize violations without repeatedly reconsidering the entire design. These results demonstrate that explicitly modeling functional chains and enforcing them through executable specifications effectively improves repository-level feature development.
%Passed
#Resolved
Avg. Cost ($)
Specification Ablations w/o All Specification w/o Behavior Specification w/o Architecture Specification
62.6±6.0 64.0±5.2 66.6±5.7
7 6 7
0.09 0.13 0.16
6
0.16
9
0.18
Evidence-Grounded Functional Chain Ablations w/o Evidence 64.8±5.7 CodeSpec
70.7±5.1
100%
Mean pass rate
Variant
RQ2: Ablation Study Table 3 reports the contribution of executable specifications and evidence-guided functional-chain construction. Removing all specifications decreases %Passed from 70.7% to 62.6% and reduces the number of resolved tasks from 9 to 7, confirming that persistent executable constraints are important for long-horizon feature implementation. The two specification types provide complementary benefits. Removing the behavior specification causes a substantial drop to 64.0% Passed and 6 resolved tasks, indicating that architecture connectivity alone cannot ensure correct outputs, boundary handling, or state transitions. Removing the architecture specification reduces %Passed to 66.6% and resolves 7 tasks, showing that behavior checks alone may still permit incomplete or unintended realization paths. Their combination therefore improves both path integrity and runtime correctness. Evidence-guided functional-chain is also critical. Without explicit evidence grounding, %Passed falls from 70.7% to 64.8%, while the number of resolved tasks decreases from 9 to 6. This suggests that requirement semantics and repository evidence help identify more reliable functional units and relations, thereby reducing unsupported architectural reasoning. Although the complete method incurs the highest average cost at $0.18 per task, the increase is modest relative to the consistent gains in both partial progress and complete task resolution.
69.5
CodeSpec
RTADev
77.2 74.1
71.8 68.9
60% 43.8
40%
36.3 38.3 17.2
20%
Table 3: Ablation study on the FeatureBench Lite split. All variants use DeepSeek-V4-Pro as the base model.
0%
<= 3,000 words
3,000-5,000 words
> 5,000 words
Task description length (words)
(a) Pass-rate comparison across different instruction-length groups.
100%
Mean pass rate
The additional architecture design and specification execution introduce moderate overhead. Compared with RTADev, CodeSpec increases the average cost by only $0.01–$0.03 per task, while consistently improving feature completion performance across different task scales. The additional interactions mainly come from executing functional-chain specifications and leveraging their violation feedback for iterative refinement, which provides more precise guidance during implementation. To further examine robustness across different backbone models, we additionally compare agent scaffolds using GPT-5.4-mini on the Lite split (Table 2). CodeSpec maintains the best performance, achieving 54.7% %Passed and resolving 5 tasks under the same backbone, even surpassing Codex, a dedicated coding agent developed by OpenAI for GPT series LLMs. These results suggest that the effectiveness of CodeSpec is robust across different backbone models rather than relying on a specific LLM.
80%
CodeSpec (Text)
CodeSpec (Text)
RTADev
78.8 78.6
80% 60%
CodeSpec
64.8
69.0
65.2
65.2 55.2
51.2
50.1
40% 20% 0%
<= 120 turns
120-200 turns
Agent interaction turns
> 200 turns
(b) Pass-rate comparison across different agent interaction-length groups.
Figure 3: Pass-rate comparison among textual specifications, executable specifications, and RTADev across tasks grouped by instruction length and agent interaction turns.
RQ3: Executable vs. Textual Specifications We further investigate whether executable specifications preserve design constraints more effectively than textual specifications. CodeSpec (Text) retains the same evidencegrounded functional-chain construction as CodeSpec, but represents the resulting design as textual information rather than executable specifications. We compare it with CodeSpec and RTADev, another code agent based on textual design planning. We compared the three methods described above on the FeatureBench Lite split. As shown in Figure 3, the three methods perform relatively similarly on simpler tasks with instructions below 3,000 words or interactions under 120 turns. The gaps widen as task complexity increases: CodeSpec achieves 71.8% on instructions of 3,000–5,000 words, compared with 43.8% for CodeSpec (Text), and reaches 65.2% beyond 200 turns, outperforming the two textual approaches by 14.0 and 15.1 points. Although textual plans also provide useful design information, placing all details in the context increases input size and the risk of overlooking requirements and design constraints. In contrast, CodeSpec provides lightweight executable feedback throughout interaction, offering more reliable guidance for requirement-intensive and long-horizon tasks.
Agent Scaffold OpenHands Claude Code Mini-SWE-Agent RTADev CodeSpec
Easy
Medium
Hard
Overall
%Passed
%Passed
%Passed
%Passed
#Resolved
Avg.Cost ($)
48.6±6.5 62.0±5.6 58.0±6.3 61.3±6.3 70.0±5.6
26.2±4.4 45.7±4.6 35.0±4.2 47.4±4.7 51.7±4.3
9.4±2.2 19.3±3.7 18.7±4.2 16.3±4.8 20.4±4.7
26.4±2.9 41.6±3.2 35.5±3.1 41.3±3.5 46.6±3.3
2 5 4 7 8
0.10 0.11 0.08 0.14 0.17
Table 4: Comparison on NL2Repo using DeepSeek-V4-Pro (max thinking). Easy, Medium, and Hard correspond to repositories with <1.5K, 1.5K–4K, and >4K lines of code (LOC). Bold indicates the best result.
RQ4: Greenfield Repository Development Generalization To assess generalization beyond repository evolution, we evaluate CodeSpec on NL2Repo-Bench (Ding et al. 2026), which contains 104 greenfield tasks that require generating a Python repository from a requirements document and an empty workspace. CodeSpec first creates a repository skeleton and functional units, after which the evolving codebase provides repository evidence for functional-chain construction and specification compilation. As shown in Table 4, CodeSpec achieves 70.0%, 51.7%, and 20.4% %Passed on the Easy, Medium, and Hard splits, outperforming the strongest baselines by 8.0, 4.3, and 1.1 percentage points, respectively. It also obtains the best overall %Passed (46.6%) and #Resolved (8), demonstrating its generalizability to greenfield development. The larger gains on Easy and Medium tasks suggest that CodeSpec effectively organizes requirements into coherent implementations for repositories within approximately 4K lines of code, while the smaller gain on Hard tasks reflects the remaining difficulty of coordinating larger requirements for current agents. Although its additional specification and feedback steps slightly increase the average cost to 0.17, the performance gains indicate a favorable quality–cost trade-off.
Discussion Case Study Figure 4 presents a representative MLflow featuredevelopment task in Featurebench that requires the agent to preserve a complete span lifecycle across multiple modules. The RTADev and CodeSpec (Text) identify the major entities, but fail to preserve several cross-module relations during implementation. In particular, it dispatches spans based on is_recording() rather than the required OTel span type, misses a required import between modules, and leaves the OTLP conversion path incomplete. Although many individual interfaces are implemented, these broken connections lead to an incomplete feature and a pass rate of 56.7%. In contrast, CodeSpec compiles the functional chain into complementary executable architecture and behavior specifications. Architecture feedback reveals whether the chain remains complete and connected, while behavior feedback identifies whether the connected path realizes the intended behavior correctly. This lightweight, targeted feedback al-
Implementation Comparison
RTADev && CodeSpec (Text)
× create_mlflow_span() × Processor missing cross-module import
Pass Rate: 56.7% LiveSpan / Span / NoOpSpan
wrong dispatch
def create_mlflow_span(): #... if isinstance(otel_span, OTelReadableSpan): if otel_span.is_recording(): return LiveSpan(otel_span, trace_id, span_type or SpanType.UNKNOWN) else: return Span(otel_span)
CodeSpec ✓ Processor
✓
×
TraceData
OTLP
serialization incomplete
Broken Relations
× Processor → Factory: invoke broken by missing import × Factory → Span Types: wrong dispatch × TraceData → OTLP: serialization incomplete
Pass Rate: 100% create_mlflow_span()
✓
LiveSpan / Span / NoOpSpan
def create_mlflow_span(): #... if not otel_span or isinstance(otel_span, NonRecordingSpan): return NoOpSpan() if isinstance(otel_span, OTelSpan): return LiveSpan(otel_span, trace_id, span_type) if isinstance(otel_span, OTelReadableSpan): return Span(otel_span)
✓
TraceData
✓
OTLP
Executable Checks
✓ check_runtime_imports() cross-module import ✓ check_create_mlflow_span_runtime() NoOpSpan · LiveSpan · Span dispatch ✓ TraceData → OTLP serialization ✓ ......
Figure 4: Case study of MLflow distributed tracing in Featurebench by using DeepSeek-V4-Pro. lows the agent to localize and repair structural or behavioral violations during implementation. The case therefore shows that executable specifications go beyond textual design descriptions by continuously enforcing the functional chain and maintaining design–implementation consistency across long-horizon development.
Conclusion We propose CodeSpec, a dual-specification approach for agentic long-horizon feature development. It constructs evidence-grounded functional chains from feature requirements and repository evidence, and compiles each chain into complementary executable architecture and behavior specifications. Architecture specifications preserve the integrity of feature realization paths, while behavior specifications verify runtime correctness along those paths. Together, they provide fine-grained and persistent feedback throughout implementation. Experiments on FeatureBench show that CodeSpec consistently outperforms representative coding agents, particularly on complex and long-horizon tasks. Moreover, results on NL2Repo-Bench further demonstrate its generalizability across repository-level development settings.
References Anthropic. 2026. Claude Code: An Agentic Coding Tool. https://docs.anthropic.com/en/docs/claude-code/ overview. Accessed: 2026-07-11. Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732. Chen, M. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374. Chow, T. S. 1978. Testing software design modeled by finitestate machines. IEEE transactions on software engineering, (3): 178–187. Claessen, K.; and Hughes, J. 2000. QuickCheck: a lightweight tool for random testing of Haskell programs. In Proceedings of the fifth ACM SIGPLAN international conference on Functional programming, 268–279. Ding, J.; Long, S.; Pu, C.; Zhou, H.; Gao, H.; Gao, X.; He, C.; Hou, Y.; Hu, F.; Li, Z.; Shi, W.; Wang, Z.; Zan, D.; Zhang, C.; Zhang, X.; Chen, Q.; Cheng, X.; Deng, B.; Gu, Q.; Hua, K.; Lin, J.; Liu, P.; Li, M.; Pan, X.; Peng, Z.; Qin, Y.; Shan, Y.; Tan, Z.; Xie, W.; Wang, Z.; Yuan, Y.; Zhang, J.; Zhao, E.; Zhao, Y.; Zhu, H.; Zhu, L.; Zou, C.; Ding, M.; Jiao, J.; Liu, J.; Liu, M.; Liu, Q.; Tao, C.; Yang, J.; Yang, T.; Zhang, Z.; Chen, X.; Huang, W.; and Zhang, G. 2026. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. arXiv:2512.12730. Du, Y.; Cai, Y.; Zhou, Y.; Wang, C.; Qian, Y.; Pang, X.; Liu, Q.; Hu, Y.; and Chen, S. 2025. Swe-dev: Evaluating and training autonomous feature-driven software development. arXiv preprint arXiv:2505.16975. Hong, S.; Zhuge, M.; Chen, J.; Zheng, X.; Cheng, Y.; Wang, J.; Zhang, C.; Yau, S.; Lin, Z.; Zhou, L.; et al. 2024. MetaGPT: Meta programming for a multi-agent collaborative framework. In International Conference on Learning Representations, volume 2024, 23247–23275. Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K. 2024. Swe-bench: Can language models resolve real-world github issues? In International Conference on Learning Representations, volume 2024, 54107–54157. Kildall, G. A. 1973. A unified approach to global program optimization. In Proceedings of the 1st annual ACM SIGACTSIGPLAN symposium on Principles of programming languages, 194–206. Li, J.; Li, G.; Zhao, Y.; Li, Y.; Liu, H.; Zhu, H.; Wang, L.; Liu, K.; Fang, Z.; Wang, L.; Ding, J.; Zhang, X.; Zhu, Y.; Dong, Y.; Jin, Z.; Li, B.; Huang, F.; Li, Y.; Gu, B.; and Yang, M. 2024. DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Findings of the Association for Computational Linguistics: ACL 2024, 3603–3614. Bangkok, Thailand: Association for Computational Linguistics. Li, W.; Zhang, X.; Guo, Z.; Mao, S.; Luo, W.; Peng, G.; Huang, Y.; Wang, H.; and Li, S. 2025. Fea-bench: A bench-
mark for evaluating repository-level code generation for feature implementation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 17160–17176. Liu, J.; Wang, G.; Yang, R.; Zeng, J.; Zhao, M.; and Cai, Y. 2025. RTADev: Intention Aligned Multi-Agent Framework for Software Development. In Che, W.; Nabende, J.; Shutova, E.; and Pilehvar, M. T., eds., Findings of the Association for Computational Linguistics: ACL 2025, 1548–1581. Vienna, Austria: Association for Computational Linguistics. ISBN 979-8-89176-256-5. Murphy, G. C.; Notkin, D.; and Sullivan, K. J. 2001. Software reflexion models: Bridging the gap between design and implementation. IEEE Transactions on Software Engineering, 27(4): 364–380. OpenAI. 2025. Introducing Codex. https://openai.com/ index/introducing-codex/. Accessed: 2026-07-11. Ostrand, T. J.; and Balcer, M. J. 1988. The category-partition method for specifying and generating functional tests. Communications of the ACM, 31(6): 676–686. Pruijt, L.; Köppe, C.; van der Werf, J. M.; and Brinkkemper, S. 2017. The accuracy of dependency analysis in static architecture compliance checking. Software: practice and Experience, 47(2): 273–309. Qian, C.; Liu, W.; Liu, H.; Chen, N.; Dang, Y.; Li, J.; Yang, C.; Chen, W.; Su, Y.; Cong, X.; et al. 2024. Chatdev: Communicative agents for software development. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers), 15174–15186. Wang, P.; Zhang, L.; Liu, F.; Shi, L.; Li, M.; Shen, B.; and Fu, A. 2025a. CodeIF-Bench: Evaluating Instruction-Following Capabilities of Large Language Models in Interactive Code Generation. arXiv:2503.22688. Wang, P.; Zhang, L.; Liu, F.; Tao, C.; and Zhu, Y. 2026. CodeMEM: AST-Guided Adaptive Memory for Repository-Level Iterative Code Generation. arXiv preprint arXiv:2601.02868. Wang, P.; Zhang, L.; Liu, F.; Zhu, Y.; Xu, W.; Shi, L.; Lian, X.; Li, M.; Shen, B.; and Fu, A. 2025b. EfficientEdit: Accelerating Code Editing via Edit-Oriented Speculative Decoding. arXiv:2506.02780. Wang, X.; Li, B.; Song, Y.; Xu, F. F.; Tang, X.; Zhuge, M.; Pan, J.; Song, Y.; Li, B.; Singh, J.; et al. 2025c. Openhands: An open platform for ai software developers as generalist agents. In International Conference on Learning Representations, volume 2025, 65882–65919. Yang, J.; Jimenez, C.; Wettig, A.; Lieret, K.; Yao, S.; Narasimhan, K.; and Press, O. 2024. Swe-agent: Agentcomputer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37: 50528–50652. Zhou, Q.; Zhang, J.; Wang, H.; Hao, R.; Wang, J.; Han, M.; Yang, Y.; Wu, S.; Pan, F.; Fan, L.; et al. 2026. Featurebench: Benchmarking agentic coding for complex feature development. arXiv preprint arXiv:2602.10975.