EvoCodeAgent: Project-Level Code Generation via Architecture-Code Co-Evolution and Experience-Guided Repair

E VO C ODE AGENT: Project-Level Code Generation via Architecture-Code Co-Evolution and Experience-Guided Repair

arXiv:2611.06328v1 [cs.SE] 4 Nov 2026

Qianhui Zhao, Li Zhang, Fang Liu∗ School of Computer Science and Engineering, State Key Laboratory of Complex & Critical Software Environment, Beihang University {zhaoqianhui, fangliu}@buaa.edu.cn

Abstract—Project-level code generation aims to automatically synthesize complete software repositories from high-level requirements, producing coherent multi-file systems with well-structured modules and consistent cross-file dependencies. Despite the strong capabilities of large language models (LLMs) in specification understanding and code synthesis, existing approaches still exhibit several critical limitations. First, many methods rely on static architectural designs that cannot adapt during iterative generation, leading to a growing misalignment between architecture and implementation. Second, they lack mechanisms to accumulate and reuse experience across iterations, resulting in recurring errors and reduced efficiency. Third, they often depend on standalone LLM-based evaluation of intermediate artifacts, which suffer from limited transparency and potential unreliability under long-context reasoning. To address these challenges, we propose E VO C ODE AGENT, a state-centric framework for project-level code generation. E VO C ODE AGENT maintains an evolving architectural state initialized from structured project specifications and continuously reconciles predicted relations from architecture and skeletons with realized relations extracted from generated code. It further uses tool-grounded verification evidence to support reliable state updates and abstracts reusable repair experiences from previous revision attempts to guide subsequent generation. Extensive experiments on project-level benchmarks of different scales, including DevBench and CodeProjectEval, demonstrate that E VO C ODE AGENT consistently outperforms baseline approaches. Under the GPT-5 setting, it achieves unit test pass rates of 60.48% and 27.73%, respectively, yielding up to a 53.5% improvement over the strongest baseline. Index Terms—Project-Level Code Generation, Multi-Agent Framework, Large Language Models

I. I NTRODUCTION Project-level code generation aims to automatically synthesize an entire project from high-level requirement descriptions, including its file organization and executable implementations [1, 2, 3]. In real-world software engineering, developers typically build complete software repositories consisting of multiple files, modules, and interdependent components rather than isolated code snippets. As software development increasingly relies on automated assistance to improve productivity and reduce engineering effort [4, 5, 6, 7, 8, 9], enabling large language models (LLMs) to generate project-scale systems has become an important research goal. Compared with traditional

code generation tasks that focus on producing individual functions or small code fragments based on localized contexts [10, 11, 12, 13, 14], project-level code generation requires LLMs to construct coherent multi-file systems with wellorganized modules and consistent cross-file dependencies. Consequently, it represents a substantially more challenging yet practically important problem in automated software engineering. Recent research has made progress in project-level code generation. As state-of-the-art LLMs such as Claude [15], DeepSeek [16, 17, 18], Gemini [19, 20], and GPT [21, 22] demonstrate strong capabilities in understanding natural language specifications and generating high-quality code, LLMbased approaches [2, 23, 24, 25, 26, 27, 28, 29] have become the main force in this task. Despite these encouraging results, existing approaches still face several important limitations: ❶ Static architectural representations cannot capture architectural evolution throughout the implementation process. Many existing approaches [2, 23, 24, 26] directly generate code from requirement descriptions without explicitly introducing an architectural design stage, which may suffer from the semantic gap between natural language requirements and executable code. To mitigate this issue, some recent methods [25, 27] incorporate structured intermediate representations as architectural guidance. However, these representations are typically constructed at an early planning stage and are mainly used as fixed blueprints during subsequent generation. Such a design is inherently limited, as architectural specifications produced before implementation may be incomplete or inaccurate due to ambiguous requirements and missing low-level details. As generation proceeds, skeletons may introduce new imports, implementations may revise interfaces, and execution feedback may reveal missing or inconsistent components. These implementation-derived facts naturally require the architectural representation to be revisited and updated. Without continuously reconciling predicted architectural relations with realized implementation relations, later generation steps may rely on outdated dependencies or incomplete interfaces, leading to architecture-code drift. ❷ Iterative feedback lacks reusable experience abstrac-

tion. Project-level code generation typically involves multiple rounds of iterative refinement, during which LLMs may repeatedly encounter similar issues, such as missing dependencies, inconsistent interfaces, import failures, or incomplete implementations. However, existing approaches [25] mainly use feedback, local diffs, or recent revision histories as contextual memory for the current iteration. Although such information can support local fixes, it is often tightly coupled with specific files, error traces, and code contexts. As a result, when similar issues reappear in later stages or different files, LLMs may struggle to transfer prior repair knowledge and may repeatedly make the same mistakes. Without mechanisms to abstract concrete refinement trajectories into context-independent repair scenarios and strategies, iterative feedback may not effectively guide subsequent generations beyond the immediate local fix. ❸ Standalone LLM-based judgments provide limited reliability for intermediate artifact validation. Multi-stage code generation frameworks typically produce a series of intermediate artifacts that should be validated before proceeding to subsequent steps. In many existing approaches [23, 24, 25, 29], this validation is predominantly handled by LLM-based judges in a standalone manner, without sufficient support from external tools or structured verification mechanisms. In practice, these evaluations often require LLMs to assess large-scale natural language descriptions or code artifacts under complex long-context settings. Due to inherent limitations in long-context reasoning, such as context dilution and attention degradation, LLMs may fail to consistently incorporate all relevant information [30, 31], leading to unstable or inconsistent judgments. Moreover, standalone judgments usually provide limited verifiable evidence about which files, interfaces, dependencies, or requirements are missing or inconsistent. This lack of actionable evidence makes the results difficult to verify and limits their usefulness for state updates and targeted repair. To address these limitations, we propose E VO C ODE AGENT, a state-centric project-level code generation framework. Instead of treating artifact as a static intermediate representation, E VO C ODE AGENT maintains an evolving project state throughout generation. The state is initialized from project requirements and continuously updated throughout generation using verification feedback, allowing architecture to guide code generation while enabling generated artifacts to refine architectural decisions. In addition, E VO C ODE AGENT grounds intermediate evaluation with specialized verification tools, producing explicit evidence for state reconciliation and repair. Finally, it abstracts reusable repair experiences from refinement trajectories, allowing recurring error patterns to be addressed with accumulated knowledge rather than rediscovered from scratch. Experimental results demonstrate that E VO C ODE AGENT achieves state-of-the-art performance on project-level code generation tasks. Under the GPT-5 setting, E VO C ODE AGENT attains a unit test pass rate of 60.48% on the small-scale DevBench benchmark and 27.73% on the largerscale CodeProjectEval benchmark, yielding up to a 53.5% relative improvement over the strongest baseline.

Our contributions can be summarized as follows: • We identify key limitations in existing project-level code generation frameworks, including static architecture-code misalignment, limited reuse of iterative repair experience, and unreliable standalone LLM-based evaluation. • We propose E VO C ODE AGENT , a state-centric projectlevel code generation framework. It maintains an evolving project state to support architecture-code co-evolution, tool-grounded verification, and experience-guided repair. • We conduct extensive experiments on project-level benchmarks of different scales, including DevBench and CodeProjectEval. Results show that E VO C ODE AGENT consistently and significantly outperforms strong baselines under various experimental settings. Our code can be found at https://github.com/whisperzqh/EvoCodeAgent.git. II. R ELATED W ORK A. Project-level Code Generation Benchmarks Project-level code generation aims to synthesize a complete multi-file software project from high-level requirements [1], requiring the construction of interdependent components while maintaining architectural consistency and cross-file dependencies. This differs from repository-level function generation, where models generate individual functions given repository context [13, 14, 32, 33, 34]. Early datasets such as SRDD [24], SRLCG [35], CASSD [36], and SoftwareDev [23], provide only software requirements without ground-truth code, leading to evaluations based on heuristic metrics or human judgment, rather than direct comparison with reference implementations. Later benchmarks introduce ground-truth repositories to enable more objective evaluation. SketchEval [2] measures project-level similarity via SketchBLEU, but such similarity-based evaluation cannot fully capture functional correctness. DevBench [1] incorporates executable test cases across multiple languages, though its projects remain relatively small (2.2 files and 276 LOC on average). Commit0 [37] and JavaBench [38] further provide executable evaluation settings with starter code or code skeletons. Commit0 covers 54 Python libraries, while JavaBench extends project-level evaluation to Java and objectoriented programming with four Java projects, 106 classes, and 389 methods. However, both focus on completing missing implementations within provided code structures, rather than generating complete repositories from scratch. More recent benchmarks emphasize larger scale and realism. CodeProjectEval [25] offers 18 fully runnable repositories with richer structures (12.7 files and 2,388 LOC on average) and comprehensive tests, enabling reliable evaluation. NL2Repo-Bench [39] further extends to long-horizon generation with 104 tasks across diverse domains, varying difficulty levels. B. LLM-based Project-level Code Generation Approaches Existing approaches to project-level code generation can be broadly categorized into pipeline-based methods and agentbased frameworks.

Generation Agent

Generation Judge Agent Phases

Architecture Design

Evolving Project State

Inputs Product Requirement Document

UML Diagrams

Initialize SSAT-derived state

Outputs

Structural Nodes

Predicted Relations

modules / files / classes / functions / global code

design & skeleton dependencies

Artifact Snapshots architecture / skeleton / code

(class / sequence)

Final Codebase

Realized Relations imports / calls / signatures / interfaces

Evolved Project State

Verification Feedback judge agent output

④ State Reconciliation Architecture Design Document

Code Implementation

Skeleton Generation

① State-guided Retrieval

structure / relation / snapshots / feedback

Experience kind (from success / failure) scenario, experience

③ Tool-grounded Verification structure / interface / execution checks

upstream & downstream context

② Artifact Generation / Refinement architecture / skeleton / code

Fig. 1: Overview of E VO C ODE AGENT. Pipeline-based approaches break the generation into sequential stages. CodeS [2] adopts a hierarchical strategy that progressively generates directory structures, file-level sketches, and function implementations. SRLCG [35] follows a similar paradigm, utilizing a multidimensional chain-of-thought and self-rectification, and integrating generated files through dynamic backtracking. While these methods produce structured outputs, they largely rely on one-pass generation with limited intermediate validation, making them prone to error propagation across stages. Agent-based frameworks coordinate multiple LLM-driven agents to simulate software development workflows. MetaGPT [23] organizes role-based agents within a global memory pool, whereas ChatDev [24] enables multi-turn collaboration among specialized agents through dialogue-based collaboration. However, both primarily target small-scale tasks and suffer from limited long-context memory. AgileCoder [29] introduces rolebased iterative development with a dynamically maintained dependency graph for context retrieval and testing, but this graph depends on already generated code and thus provides limited guidance in early stages. ZeroRepo [27] constructs a Repository Planning Graph (RPG) for graph-guided repository generation. Although RPG is progressively constructed during planning, it does not explicitly reconcile planned relations with implementation relations extracted from generated code during generation. ProjectGen [25] further structures generation into architecture, skeleton, and code stages, introducing the Semantic Software Architecture Tree (SSAT) to model modules and dependencies. Similarly, the SSAT remains static after construction, and its reliance on solely LLM-based judgment may lead to unreliable evaluation. In contrast, E VO C ODE AGENT treats the architectural rep-

resentation as an evolving project state rather than a static planning artifact. It continuously reconciles predicted relations inferred from architecture and skeletons with realized relations extracted from generated code, while using tool-grounded evidence and reusable repair experiences to guide state updates and subsequent generation. III. A PPROACH To support adaptive and reliable project-level code generation, we propose E VO C ODE AGENT, a state-centric framework that combines staged multi-agent generation with continuous state evolution. As shown in Figure 1, E VO C ODE AGENT proceeds through three phases: architecture design, skeleton generation, and code implementation. Each phase is managed by a generation–judge agent pair, where the generation agent produces or revises phase-specific artifacts and the judge agent evaluates them with feedback, enabling iterative refinement within the phase. Rather than following a one-way staged pipeline, E VO C ODE AGENT connects the three phases through an evolving project state initialized from the input artifacts, including the product requirement document, UML diagrams, and architecture design document. During generation, this state is updated through state-guided retrieval, artifact generation, tool-grounded verification, and state reconciliation. Thus, architecture guides downstream skeleton and code generation, while generated artifacts and verification feedback are written back to refine earlier architectural assumptions. In addition, execution feedback from the code implementation stage is distilled into reusable experience memory, providing guidance for subsequent iterations. The following subsections describe these mechanisms in detail.

A. Evolving Project State

Algorithm 1 Architecture-Code Co-evolution

The evolving project state serves as the core representation in E VO C ODE AGENT. It extends the SSAT [25], which provides a machine-parsable hierarchy of modules, files, classes, functions, and global code blocks, into a richer dynamic representation. Beyond the static structural hierarchy encoded by SSAT, the project state additionally maintains dependency relations, generated artifacts, and verification feedback, and is continuously updated throughout generation. The reusable experience set is maintained alongside the project state and is described in Section III-D. Formally, the project state at iteration t is defined as:

1: Input: Initial architecture state Pa , max skeleton iterations Ts , max code iterations Tc (T ) 2: Output: Final codebase C (Tc ) and evolved project state Pc c 3: Skeleton Generation Phase (0) 4: Ps ← Pa 5: for ts = 1 to Ts do 6: Retrieve repair targets from verification feedback 7: Generate or refine partial skeleton ∆S (ts ) 8: Update project state:

Pt = (Nt , Rtp , Rtr , At , Vt ),

B. Architecture-Code Co-Evolution E VO C ODE AGENT first initializes the architecture state Pa by analyzing the input artifacts. This produces the structural nodes Na , architecture artifacts Aa , and architecture-level verification feedback Va . At the end of the architecture design phase, both predicted and realized relations remain empty, since only structural containment information is available before skeleton generation. To bridge the gap between architectural design and implementation in conventional staged generation, E VO C ODE AGENT performs architecture-code co-evolution over the evolving project state. Specifically, dependency relations are progressively constructed and refined during skeleton generation and code implementation, allowing architectural guidance to remain aligned with the evolving codebase. Algorithm 1 summarizes this process. During the skeleton generation phase, the agent iteratively generates and refines skeleton artifacts based on the current skeleton state. After each refinement step, judge agents perform structural verification to identify missing interfaces, inconsistent declarations, and other structural issues. The refined

, ∆S (ts ) , ∆V (ts ) )

9: end for (T ) 10: Infer predicted relations: Rp ← finfer (Ps s ) 11: Initialize code generation state: (0)

Pc

(1)

where Nt denotes structural nodes, Rtp denotes predicted relations, Rtr denotes realized relations, At denotes generated artifact snapshots, and Vt denotes verification feedback. Specifically, Nt records the SSAT-derived hierarchical structure of the project, including modules, files, classes, functions, global code blocks, and their containment relations. Rtp and Rtr denote predicted and realized dependencies, respectively, providing architectural priors before implementation and updated dependency information during code implementation. By continuously reconciling predicted and realized relations, E VO C ODE AGENT keeps architectural guidance aligned with the evolving codebase. At stores snapshots of generated architecture, skeleton, and code artifacts, enabling comparison across iterations. Vt stores structured verification feedback produced by judge agents based on tool checks and test execution, such as missing files, uncovered interfaces, failed check tests, and repair targets. Together, these components provide a unified foundation for dependency-aware retrieval, state reconciliation, and iterative refinement.

(t −1)

(t )

Ps s ← ReconcileState(Ps s

(T )

← UpdateRelations(Ps s , Rp )

12: Code Implementation Phase 13: for tc = 1 to Tc do (t −1) (t ) 14: Pc c ← Pc c file,(tc ) do 15: for each file node ni ∈ Nc 16: Retrieve dependency-aware context: (t )

Cctx (ni ) ← RetrieveContext(ni , Pc c ) 17:

Generate or refine code: C (tc ) (ni ) ← fgen (Cctx (ni ))

18:

Extract realized structures: ∆N , ∆Rr ← fextract (C (tc ) (ni ))

19:

Reconcile project state: (t )

(t )

Pc c ← ReconcileState(Pc c , ∆N , ∆Rr , ∆V) 20: end for 21: end for (T ) 22: return C (Tc ) , Pc c

skeleton together with the newly produced verification feedback is reconciled into the project state, enabling progressive correction of structural inconsistencies. After the final skeleton iteration, the system analyzes the complete skeleton to infer a set of predicted relations, derived from multiple signals such as import statements, cross-file symbol matching, and semantic similarity between code elements. These predicted relations capture potential dependencies not explicitly encoded in the initial structural nodes and provide dependency priors for subsequent code generation. During code implementation, the evolving project state guides context retrieval for each target file, enabling dependency-aware and structure-consistent generation. For a target file node ni , incoming and outgoing dependencies are identified based on the current relation state. In the first code iteration, context retrieval primarily relies on predicted relations Rp , since implementation-derived dependencies are not yet available. In later iterations, realized relations Rr extracted from generated code progressively dominate dependency retrieval, while predicted relations serve as supplementary priors. The retrieved artifacts are aggregated into context Cctx (ni ) and provided to the generation agent for implementation. This dependency-aware retrieval ensures that both upstream depen-

TABLE I: Task-oriented tools in E VO C ODE AGENT. Tool Name

Description

Technical Basis

Tools for Generation Agents Requirement Access Access requirement artifacts from PRD, UML, and architecture documents Symbol Extractor Extract all classes, functions, and variable symbols from project state nodes Artifacts Retriever Retrieve project artifacts (architecture nodes, skeletons, or implemented code) for a given file path

Hierarchical document access Structure-aware traversal, symbol extraction Path-based indexing

Tools for Judge Agents Requirement Access Artifacts Retriever Documentation Searcher Structure Verifier Interface Verifier Missing File Detector Skeleton Coverage Verifier

Hierarchical document access Path-based indexing Keyword search, semantic relevance ranking Rule-based hierarchical validation UML parsing, node extraction and matching nodes-guided file existence verification AST-based code extraction, architecture node matching

Access requirement artifacts from PRD, UML, and architecture documents Retrieve project artifacts (architecture nodes, skeletons, or implemented code) for a given file path Retrieve the most relevant blocks from project documentation based on keyword queries Verify artifacts against structure node specifications for structural and property completeness Verify architecture completeness against UML-defined classes, functions, attributes, and global blocks Detect code files defined in the architecture but absent in architecture nodes Verify skeleton code includes all architecture-specified files, classes, and functions

dencies (e.g., imported modules) and downstream usages (e.g., calling sites) are considered, improving interface consistency and cross-file interaction correctness. After code generation, the implementation is evaluated through test execution to obtain structured feedback, such as interface mismatches and failed check tests. Static analysis further extracts realized relations and structural updates that reflect actual dependencies in the generated code. These signals are reconciled into the project state, continuously aligning architectural assumptions with implementation reality. Through iterative state reconciliation, the co-evolution mechanism maintains a bidirectional linkage between architecture and code, providing a unified and adaptive backbone for project-level code generation. C. Tool-Grounded Verification E VO C ODE AGENT equips both generation and judge agents with task-oriented tools to enable selective context retrieval and structured artifact verification, as summarized in Table I. Rather than relying solely on LLM reasoning, E VO C ODE AGENT adopts a hybrid verification paradigm in which judge agents combine structured tool outputs with LLM-based reasoning to produce more reliable and interpretable feedback. Generation agents primarily use tools to retrieve compact and relevant context, including requirement artifacts, symbols from the current project state, and previously generated artifacts, thereby reducing irrelevant context and improving reasoning efficiency. Judge agents use tools to collect explicit verification evidence, such as missing architectural elements, uncovered interfaces, and structural inconsistencies, which serves as the basis for iterative refinement. The role of tools varies across different artifact granularities. At the architecture level, verification focuses on structural completeness and consistency with input artifacts, enabling the judge agent to identify missing nodes, incomplete interfaces, and architectural inconsistencies. At the skeleton level, verification ensures that the generated skeleton faithfully covers all files, classes, and functions specified in the current project state while maintaining syntactic correctness. At the implementation level, verification is primarily driven by executable feedback from predefined check tests, which provides direct evidence of functional correctness. The structured evidence collected from all stages is written into the verification component Vt of the evolving project

state. These verification signals are subsequently used during state reconciliation and future generation, making tool-based verification not only a mechanism for evaluation but also a source of actionable guidance for iterative refinement. By grounding agent decisions in explicit evidence rather than relying solely on implicit LLM reasoning, this design improves verification reliability, reduces error propagation across stages, and enables more robust project-level code generation under long-context settings. D. Experience Abstraction and Reuse E VO C ODE AGENT accumulates reusable repair experiences from iterative refinement trajectories to support long-horizon self-improvement. Project-level code generation frequently encounters recurring issues, such as incompatible imports, inconsistent public APIs, incorrect serialization behavior, or incomplete exception handling. Although these issues may arise in different files or at different iterations, they often share similar failure patterns and repair strategies. If the system only retains raw feedback from previous iterations, similar errors may still need to be rediscovered and repaired repeatedly. To address this, E VO C ODE AGENT abstracts concrete modification trajectories into reusable repair experiences. After each refinement iteration, E VO C ODE AGENT compares artifact snapshots and verification outcomes before and after modification. Let At−1 and At denote artifact snapshots before and after a refinement step, and let Vt−1 and Vt denote the corresponding verification evidence. These differences capture what was modified, which issue the modification attempted to resolve, and whether the resulting verification outcome improved. Based on these trajectory differences, E VO C ODE AGENT extracts an experience: ei = (si , ai , yi ),

(2)

where si denotes an abstract scenario, ai denotes the corresponding corrective strategy, and yi denotes the observed outcome. The scenario summarizes the conditions under which the experience is applicable, such as an import-time compatibility failure or an interface mismatch between method signatures and calling sites. The strategy describes the corresponding repair principle, such as preserving optional parameters. The outcome records whether the strategy improved verifi-

cation results, produced no observable effect, or introduced regressions. The experience set is updated incrementally: Et = Et−1 ∪ Abstract(At−1 , At , Vt−1 , Vt ).

(3)

During subsequent generation or repair in code implementation stage, E VO C ODE AGENT retrieves experiences relevant to the current context. Experiences retrieved by the error messages are injected into the generation context as highlevel guidance rather than direct code patches, allowing the generation agent to reuse repair principles while adapting them to the current implementation. This mechanism differs fundamentally from storing local historical feedback. Raw feedback and code diffs are often tied to specific files or iterations, whereas experiences in E VO C ODE AGENT are abstracted into reusable knowledge associated with explicit outcomes. As a result, they can generalize across files and later iterations when similar scenarios arise. By transforming past refinement trajectories into reusable guidance, E VO C ODE AGENT reduces repeated error discovery, improves repair efficiency, and enables more effective iterative self-correction during project-level code generation. IV. E VALUATION To evaluate the effectiveness of E VO C ODE AGENT, we address the following research questions: RQ1: Overall Performance. How does the generation performance of E VO C ODE AGENT compare against state-of-theart approaches on project-level code generation benchmarks? RQ2: Ablation Study. What are the contributions of each component of E VO C ODE AGENT to the overall performance? RQ3: Efficiency Analysis. How does E VO C ODE AGENT balance quality, cost, and token efficiency against existing approaches?

A. Datasets and Backbone Models To support automated test-based evaluation, we use two datasets, DevBench [1] and CodeProjectEval [25], both providing PRDs, UML diagrams, and architectural specifications for each task. • DevBench: It is designed to evaluate LLM performance across the full software development lifecycle. We use the Python subset of DevBench, which contains 10 tasks with an average of 2.2 files and 276 lines of code each. • CodeProjectEval: It is designed to support realistic evaluation through executable test cases and structured project artifacts, containing 18 Python tasks across diverse domains with an average of 12.7 files and 2,388.6 LOC. We conduct experiments using two backbone LLMs, DeepSeek-v3.2 [40] and GPT-5 [41], both of which have demonstrated strong capabilities in code generation. B. Baselines We compare E VO C ODE AGENT with three state-of-the-art project-level code generation approaches. All of these approaches take user requirements as input and directly generate

complete project-level code as output. Detailed information can be seen as follows: CodeS [2]: It introduces a hierarchical sketching framework that decomposes project-level code generation into multi-granularity sub-tasks through a structured sketching and filling process. • MetaGPT [23]: It models software development as a multi-agent collaboration workflow guided by predefined standard operating procedures, enabling controllable task decomposition and information exchange among specialized agents. • ProjectGen [25]: It is a multi-agent framework for project-level code generation that decomposes the process into architecture design, skeleton generation, and code completion, using a semantic software architecture tree and iterative refinement with memory-based context management to bridge requirements and implementation. •

C. Metrics We evaluate the generated repositories using the following metrics, covering functional correctness, architectural alignment, and code complexity. Pass rate (%Pass): We use unit test pass rate as the primary metric to evaluate code generation performance, as it directly reflects the functional correctness of the generated repository. It is computed as the percentage of unit tests successfully passed by the generated code. • Architecture consistency: We define architecture consistency as a metric for measuring structural alignment between the generated and ground-truth repository. Specifically, we extract all root-to-leaf paths from the structural nodes Na at the end of architecture design phase and reconstruct the corresponding paths in the ground-truth repository via AST parsing. The metric is defined as: •

ArchConsistency =

2|Pn ∩ Pg | |Pn | + |Pg |

(4)

where Pn and Pg denote the path sets of the generated and ground-truth repositories, respectively. Since each path captures the hierarchical dependency chain of a software component, the F1 score over path sets provides a balanced measure. • Number of generated files (#Files): We record the number of generated files to assess the model’s ability to accurately reconstruct the project structure. • Lines of code (LOC): We compute the total number of lines of generated code to estimate the scale of produced implementations and analyze differences in generation completeness across methods. • Code complexity: We measure the cyclomatic complexity of the generated code to assess its structural quality and maintainability.

TABLE II: Comparison between E VO C ODE AGENT and baseline approaches on DevBench and CodeProjectEval dataset. %Pass represents the overall pass rate on the dataset, while the other metrics are average / median values across individual tasks. Model

Approach

#Files Avg. Mid.

Avg.

LOC Mid.

Complexity Avg. Mid.

Arch Consistency Avg. Mid.

%Pass

DevBench DeepSeek-v3.2

CodeS MetaGPT ProjectGen E VO C ODE AGENT

3.1 1.7 3.0 3.0

2.5 1.0 2.0 2.0

563.3 243.0 303.2 331.0

372.0 260.0 256.5 361.5

5.56 2.48 3.40 3.91

5.82 2.90 3.20 3.63

/ / 58.80 78.92

/ / 57.91 100.0

25.00% 29.84% 42.74% 50.81%

GPT-5

CodeS MetaGPT ProjectGen E VO C ODE AGENT

2.4 1.8 3.0 3.0

2.0 1.5 2.0 2.0

277.9 226.3 460.3 454.2

255.0 219.0 459.0 431.5

4.51 2.95 8.17 8.28

4.15 3.50 7.16 6.52

/ / 61.94 84.92

/ / 61.82 98.65

4.03% 32.26% 57.26% 60.48%

DeepSeek-v3.2

CodeS MetaGPT ProjectGen E VO C ODE AGENT

1.9 10.5 5.5 9.3

0.0 9.5 5.5 8.5

201.3 1838.0 643.1 1367.4

0.0 988.0 619.0 1280.0

0.93 4.30 4.32 3.30

0.00 2.20 2.55 2.80

/ / 32.70 46.69

/ / 31.65 43.19

0.00% 1.04% 3.16% 7.02%

GPT-5

CodeS MetaGPT ProjectGen E VO C ODE AGENT

6.9 9.0 12.7 13.3

5.5 7.5 11.5 11.5

1249.7 1559.9 2468.2 2352.4

966.5 1439.0 2274.5 2399.0

3.52 2.30 5.55 5.70

3.73 2.52 4.60 5.19

/ / 55.78 86.04

/ / 56.03 92.45

1.51% 4.00% 18.07% 27.73%

CodeProjectEval

D. Implementation Details For E VO C ODE AGENT, we implement the multi-agent system using LangChain1 , a framework for building applications with large language models. To retrieve relevant experience, we employ BM25 [42] to measure similarity between feedback texts. Since experience records preserve task-specific terminology and structural cues, lexical overlap provides an effective retrieval signal, making BM25 a lightweight yet effective choice that has also been shown to outperform embeddingbased retrieval in software engineering scenarios [43]. Following Zhao et al. [25], we set the maximum number of iterations to 3 for architecture design and skeleton generation stages, and to 5 for DevBench and 10 for CodeProjectEval in code implementation stage. For both E VO C ODE AGENT and baseline approaches, we ensure a fair comparison by providing the same set of inputs, including the PRD, UML diagrams, and architectural design documents. The PRD is a structured requirements document describing the system’s functional and non-functional aspects. UML diagrams (including class, package, and sequence diagrams) capture the structural relationships within the repository in Mermaid format. The architecture design document provides a high-level representation of the repository structure, including a source-code directory tree and concise descriptions of each file’s responsibilities, classes, and functions. To ensure consistency and reproducibility, all experiments are conducted with temperature=0, while the agent responsible for generating code fix suggestions operates with temperature=0.8. The maximum context length is set to the upper limit of each LLM, with context windows of 128K tokens for DeepSeek-V3.2 and 400K tokens for GPT-5. 1 https://www.langchain.com

V. R ESULTS A. RQ1: Overall Performance We compare the performance of E VO C ODE AGENT against state-of-the-art approaches on DevBench and CodeProjectEval. The results are presented in Table II. We further analyze the results from three perspectives: functional correctness, architecture consistency, as well as code scale and complexity. 1) Functional correctness: As measured by the unit test pass rate (%Pass), E VO C ODE AGENT consistently achieves the best performance across all settings. On DevBench, it attains 50.81% with DeepSeek-v3.2 and 60.48% with GPT-5, outperforming all baselines by a clear margin. When evaluated on the more challenging CodeProjectEval dataset, the performance gap becomes even more pronounced. While CodeS and MetaGPT achieve competitive results on DevBench, their performance drops substantially on CodeProjectEval, indicating limited capability in handling complex project-level tasks. In contrast, E VO C ODE AGENT remains the top performer, achieving 7.02% and 27.73%, demonstrating superior robustness and adaptability in more demanding scenarios. The degradation of the baselines can be attributed to their design limitations. CodeS relies on a single-pass generation paradigm without iterative refinement or explicit quality control, making it susceptible to error propagation. Errors introduced in early stages cascade throughout the generation process, often resulting in file name resolution errors, incorrect code insertion, and unannotated natural language in generated files. MetaGPT directly generates code from natural language requirements without an explicit architectural design phase, leading to incomplete project structures, missing components, and partially defined function attributes. Although ProjectGen introduces an architecture design stage, its architecture is static and only serves as initialization for downstream generation. It

also lacks mechanisms for accumulating abstract knowledge across iterations, limiting the refinement effectiveness. 2) Architecture Consistency: As CodeS and MetaGPT do not provide an explicit structural representation, we do not compute architecture consistency for them. The architecture consistency results indicate that E VO C ODE AGENT consistently produces repository structures that are more closely aligned with the ground-truth architecture than ProjectGen across different datasets and backbone models. ProjectGen relies solely on LLM-based judgment to evaluate architectural correctness, requiring the model to simultaneously reason over input specifications and generated architectural artifacts. Such long-context reasoning may reduce evaluation accuracy, particularly when complex structural dependencies need to be identified and compared. In contrast, E VO C ODE AGENT augments LLM-based judgment with specialized tools that provide explicit structural evidence, enabling more accurate architecture verification and refinement. This improvement is particularly important as architecture establishes the structural foundation for subsequent code generation. Higher architectural consistency provides stronger dependency priors and more reliable intermediate states, thereby facilitating more efficient iterative refinement and ultimately improving repository-level generation quality. 3) Code scale and complexity: From this perspective, larger code size or higher complexity does not necessarily imply better performance. A more desirable property is the ability to adapt the scale of generated code to the difficulty of the task. In this regard, E VO C ODE AGENT produces solutions on DevBench with moderate lines of code and complexity, achieving strong performance while avoiding unnecessary overhead. On the more challenging CodeProjectEval dataset, E VO C ODE AGENT generates substantially larger and more complex codebases, aligning with the increased structural demands of project-level tasks. Moreover, the scale of its generated codebases closely aligns with that of the groundtruth implementations across datasets. This behavior suggests that E VO C ODE AGENT can effectively adapt the granularity and richness of generated programs to task difficulty. By comparison, CodeS exhibits clear limitations in handling complex scenarios. On CodeProjectEval, it produces significantly smaller codebases in terms of both lines of code and number of files, which likely leads to insufficient functional coverage and contributes to its lower pass rates. This indicates that CodeS tends to under-generate for complex tasks and lacks the ability to appropriately scale its outputs. Notably, under the DeepSeek-v3.2 backbone on CodeProjectEval, the median performance of CodeS is 0, indicating that fewer than half of the generated projects can even be successfully parsed. On the DevBench dataset, where CodeS generates the largest and most complex codebases, it still achieves the lowest pass rates, suggesting that merely increasing code scale and complexity does not translate into better functional correctness.

TABLE III: Abaltion results of E VO C ODE AGENT on CodeProjectEval under GPT-5. %Pass represents the overall pass rate on the dataset, while the other metrics are averaged across individual tasks. w/o tools w/o co-evolution w/o experience E VO C ODE AGENT

#Files

LOC

Complexity

Arch Cons.

%Pass

12.3 12.5 13.3 13.3

2371.7 2298.7 2470.1 2352.4

5.80 5.35 5.91 5.70

81.44 82.28 85.90 86.04

10.88% 21.08% 17.23% 27.73%

Experiences Kind: SuccessExtracted from Successful Fixes Scenario: Import-time errors due to using language features unavailable in the target runtime version block test collection. Experience: Use version-conditional imports and graceful fallbacks for newer typing features or APIs. Employ try/except to import from a backport package and provide a final safe fallback (e.g., a generic type) to avoid hard failures. When feasible, enable postponed evaluation of annotations to reduce reliance on runtime availability of types. This approach restores compatibility and allows tests to run, revealing deeper issues. Experiences Kind: SuccessExtracted from Failed Fixes Scenario: Method signatures diverge from established interface contracts by making optional parameters required or removing support for forwarded keyword arguments. Experience: Preserve canonical API shapes: keep optional parameters optional and support passthrough **kwargs in both leaf and delegating methods. Changing signatures to require arguments or to fixed parameters breaks existing call sites and tests, especially in libraries with widely used public interfaces.

Fig. 2: Examples of experience accumulated during iterative code generation. Answer to RQ1: E VO C ODE AGENT achieves state-of-theart performance across small- and large-scale project-level code generation tasks, demonstrating robust effectiveness and generalization across diverse backbones and datasets. B. RQ2: Ablation Study We conducted ablation experiments to evaluate the contributions of key components in E VO C ODE AGENT. For the tools and experience modules, we completely removed the corresponding functionalities. For the architecture-code coevolution mechanism, we limited the context construction to only the structural nodes generated during the initial architecture stage and the predicted relations produced in the skeleton stage, omitting the iterative co-evolution. As shown in Table III, removing any single component leads to a clear decline in functional correctness as measured by pass rate, highlighting the complementary roles of these components. Specifically, removing task-oriented tools results in a substantial decrease in pass rate from 27.73% to 10.88%, accompanied by the largest drop in architecture consistency. This indicates that tool support is critical not only for providing structured signals and reducing error propagation during generation, but also for enabling more accurate architecture verification and precise retrieval of relevant information necessary for code construction. Similarly, eliminating the architecture-code co-evolution mechanism reduces the pass rate to 21.08%, while causing only a moderate drop in architecture consistency. This suggests that the primary benefit of co-evolution lies not in improving the initial architecture itself, but in dynamically identifying relevant contextual dependencies and filtering out unrelated information during code generation, which is crucial for producing better outputs. Based on our calculations, the average

TABLE IV: Statistics of experience on CodeProjectEval.

(a) DevBench

Category

Metric

Value

Usage

Average experiences per task Retrieval rate

26.4 31.9%

Creation Stage

Early-stage experiences Middle-stage experiences Late-stage experiences

47.4% 31.1% 21.5%

Retrieval Outcome

Improvement after retrieval No observable effect Regression after retrieval

35.3% 61.8% 2.9%

(b) CodeProjectEval

Fig. 3: Comparison of E VO C ODE AGENT and baselines on token consumption and performance under GPT-5. project size on CodeProjectEval under GPT-5 is 19,376.06 tokens, while the context retrieved via the co-evolution mechanism has an average length of 6,560.93 tokens, accounting for only 33.9% of the total. This demonstrates that co-evolution effectively focuses on the most relevant portions of the project, providing concise yet informative context that is sufficient to guide accurate code generation without the overhead of processing the entire repository. The removal of experience-based memory also leads to a clear degradation, with the pass rate dropping to 17.23%, while architecture consistency remains largely unchanged. This suggests that it mainly contributes to iterative refinement in detail rather than broad architecture construction. As illustrated in Figure 2, the retained experience consists of abstract and reusable insights that provide guidance during iterative refinement and help avoid repeating prior mistakes. Additionally, the consistent code scale and complexity across all variants indicate that the overall architecture of E VO C ODE AGENT is well-suited for generating substantial project code, while the individual components primarily contribute to improving functional correctness rather than altering code size or structural richness. Answer to RQ2: Each component of E VO C ODE AGENT contributes to performance, with task-oriented tools providing the largest gains, while co-evolution and experiencebased memory further improve robustness and reliability.

CodeS and MetaGPT consume the most tokens, their pass rates remain quite low. In contrast, E VO C ODE AGENT achieves approximately 1.5 times the functional performance of ProjectGen with only a modest increase in token expenditure, highlighting its superior ability to scale efficiently to more complex projects. Notably, CodeS generates code on a perfunction basis, providing all context at each step, which is impractical for large-scale projects, as it leads to excessively long contexts from which the LLM struggles to extract useful information. Across both datasets, E VO C ODE AGENT consistently delivers strong generation performance at a competitive token cost. On CodeProjectEval, end-to-end project generation takes 70.6 minutes on average and incurs an estimated API cost of $3.17 per repository. These results demonstrate that E VO C ODE AGENT’s evolutionary agent-based framework effectively produces reliable and functionally correct code across a wide range of project-level scenarios, with its advantages becoming more pronounced in challenging large-scale tasks. Answer to RQ3: E VO C ODE AGENT achieves strong generation performance at a competitive token cost across both small- and large-scale datasets, with efficiency gains more pronounced on large-scale projects. VI. D ISCUSSION A. Statistics and Effectiveness of Experience

C. RQ3: Efficiency Analysis Figure 3 presents an analysis of E VO C ODE AGENT and baseline approaches, evaluating the trade-off between token consumption and generation performance under GPT-5. On the small-scale project-level code generation dataset DevBench, E VO C ODE AGENT consumes approximately 1.2 times the tokens of MetaGPT, yet achieves nearly twice the unit test pass rate. Due to its use of tool calls and experience summarization, E VO C ODE AGENT incurs additional LLM invocations and a slightly higher token cost compared to ProjectGen. Nevertheless, it converts this computational overhead into higher functional code quality. This advantage of E VO C ODE AGENT becomes more pronounced on the larger-scale project-level code generation dataset CodeProjectEval, which imposes stricter demands on long-context modeling and systematic code generation. While

To better understand the role of abstracted experience, we analyze its usage statistics on CodeProjectEval, as summarized in Table IV. A substantial portion of accumulated experiences are successfully retrieved, with nearly half of reusable experiences originating from early development stages. This suggests that useful development patterns are often discovered early and remain valuable throughout later iterations. Retrieved experiences generally provide effective and safe auxiliary guidance. A considerable fraction of retrieval events leads to measurable improvements, while regression remains rare, indicating that experience reuse is unlikely to mislead the generation process. The effectiveness of retrieval largely depends on the similarity between current failures and previously encountered cases. Qualitatively, retrieved experiences often capture recurring software engineering patterns, such as data validation, API

4% 4%

1%

3%

2%

2%

1%

Artifacts Retriever (code)

2%

4%

Artifacts Retriever (skeleton)

2%

4%

31%

TABLE V: Comparison between E VO C ODE AGENT and Claude Code on deprecated project under DeepSeek-v3.2.

Artifacts Retriever (arch)

10%

35%

2%

Symbol Extractor

Approach

%Pass

Tokens

ClaudeCodeself-testing ClaudeCodeexternal-feedback E VO C ODE AGENT

11.2% 18.8% 51.4%

2,244,863 2,203,084 685,589

Documentation Searcher

10%

10%

Requirement Access Structure Verifier Interface Verifier

12%

26%

DevBench

35%

Missing File Detector Skeleton Coverage Verifier

CodeProjectEval

Fig. 4: Distribution of tool invocation frequencies of E VO C ODE AGENT under GPT-5. consistency, serialization, and exception handling. These issues frequently involve cross-file dependencies and repeated implementation logic, making them particularly suitable for reuse across iterative refinement.

if inspect.isclass(wrapped): EvoCodeAgent entity_type = "class" else: if isinstance(wrapped, staticmethod): … elif isinstance(wrapped, classmethod): … elif instance is not None: … elif hasattr(wrapped, '__self__') and wrapped.__self__ is not None: … elif inspect.ismethod(wrapped) or inspect.ismethoddescriptor(wrapped): … elif hasattr(wrapped, '__func__’): …

Claude Code if isinstance(wrapped, type): entity_type = "class" elif instance is None: entity_type = "function" else: entity_type = "method"

B. Tool Usage Analysis

Fig. 5: Comparison of callable type identification for deprecation message generation.

To investigate tool usage across projects of different scales, we counted tool invocations on DevBench and CodeProjectEval, as shown in Figure 4. Across both datasets, artifact retrieval tools dominate overall usage, with code and skeleton retrieval accounting for the largest proportion, highlighting the importance of structured access to intermediate artifacts for maintaining cross-file consistency. Symbol extraction and architecture retrieval also remain stable, indicating that structure-aware reasoning is essential regardless of project scale. As task scale increases, tool usage becomes more concentrated. On the smaller-scale DevBench dataset, tool usage is more diverse, with verification and auxiliary tools contributing noticeably. In contrast, on CodeProjectEval, code and skeleton retrieval occupy a much larger proportion, suggesting that large-scale generation depends increasingly on efficient context retrieval, while the relative role of verification diminishes.

enabling more efficient feedback-driven refinement. These results highlight a trade-off between the flexibility of unified long-context generation and the efficiency of modular workflow-based software engineering. Figure 5 illustrates a representative implementation difference. Claude Code adopts a simpler invocation-based strategy for callable classification, which offers lower implementation complexity and is often sufficiently robust for common cases. In contrast, E VO C ODE AGENT performs finer-grained analysis through Python introspection. In this example, execution feedback enables E VO C ODE AGENT to identify and correct the mismatch during subsequent iterations, illustrating how structured decomposition improves error localization and recovery. This comparison indicates the trade-off between lightweight heuristic-based generation and more structured but analysisintensive workflow design.

C. Case Study To better understand the behavioral differences between workflow-based software engineering agents and monolithic CLI-based coding agents, we conduct a case study using Claude Code2 as a representative CLI-based baseline on the deprecated project from CodeProjectEval. Claude Code receives the same structured inputs as E VO C ODE AGENT, including the PRD, UML diagrams, and architectural design documents. We evaluate it under two settings: self-testing, where it autonomously generates and validates the entire project, and external-feedback, where it generates only source code and receives check-test feedback after each iteration. Table V summarizes the results. Claude Code demonstrates strong end-to-end generation capability, benefiting from a unified conversational workflow that enables rapid development and flexible refinement. However, this design also leads to steadily growing context and higher token consumption during iterative revisions. In contrast, E VO C ODE AGENT decomposes development into specialized stages with independent contexts, 2 https://www.claude.com/product/claude-code

D. Threats to Validity The primary threats to validity stem from dataset coverage, backbone model selection, and input quality. First, our experiments are conducted on a limited set of datasets, which may constrain the generalizability of the results. To mitigate this threat, we evaluate E VO C ODE AGENT on two benchmarks of different scales, covering both small-scale and largescale project-level code generation, where E VO C ODE AGENT consistently outperforms baseline methods. Second, modelspecific bias may affect performance. To reduce this risk, we evaluate E VO C ODE AGENT with two representative LLMs and observe consistent performance gains across both settings. A further threat arises from input quality, as incomplete or contradictory requirements may degrade generation quality. To alleviate this issue, E VO C ODE AGENT employs closed-loop feedback with iterative project state refinement and leverages multiple input artifacts to compensate for missing information. Finally, E VO C ODE AGENT is inherently language-agnostic, and its core mechanisms can be extended to other programming languages in future work.

VII. C ONCLUSION In conclusion, this work presents E VO C ODE AGENT, a multi-agent framework for project-level code generation that effectively addresses key limitations of existing approaches. By combining an evolving project state, toolgrounded verification, and iterative experience accumulation, E VO C ODE AGENT enables reliable state reconciliation and continuous self-improvement. Experimental results on diverse benchmarks, including DevBench and CodeProjectEval, show that E VO C ODE AGENT consistently surpasses state-of-the-art methods, achieving substantial improvements in unit test pass rates and demonstrating robust generalization across projects of varying complexity. These findings highlight the potential of combining iterative, experience-aware generation with multi-agent collaboration and dynamic architecture-code coevolution to enhance the reliability, consistency, and scalability of automated software synthesis. R EFERENCES [1] B. Li, W. Wu, Z. Tang, L. Shi, J. Yang, J. Li, S. Yao, C. Qian, B. Hui, Q. Zhang et al., “Prompting large language models to tackle the full software development lifecycle: A case study,” in COLING, 2025. [2] D. Zan, A. Yu, W. Liu, D. Chen, B. Shen, Y. Yao, W. Li, X. Chen, Y. Gong, B. Guan et al., “Codes: Natural language to code repository via multi-layer sketch,” ACM Transactions on Software Engineering and Methodology, 2025. [3] D. Zan, B. Chen, F. Zhang, D. Lu, B. Wu, B. Guan, W. Yongji, and J.-G. Lou, “Large language models meet nl2code: A survey,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 7443–7464. [4] J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022. [5] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” in The eleventh international conference on learning representations, 2022. [6] P. Gao, Z. Tian, X. Meng, X. Wang, R. Hu, Y. Xiao, Y. Liu, Z. Zhang, J. Chen, C. Gao et al., “Trae agent: An llm-based agent for software engineering with test-time scaling,” arXiv preprint arXiv:2507.23370, 2025. [7] X. Wang, B. Li, Y. Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y. Song, B. Li, J. Singh et al., “Openhands: An open platform for ai software developers as generalist agents,” in The Thirteenth International Conference on Learning Representations, 2025. [8] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [9] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: Language agents with verbal reinforcement learning,” Advances in neural information processing systems, vol. 36, pp. 8634–8652, 2023. [10] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, and et al, “Evaluating large language models trained on code,” 2021. [11] J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. [12] T. Liu, C. Xu, and J. McAuley, “Repobench: Benchmarking repositorylevel code auto-completion systems,” in The Twelfth International Conference on Learning Representations, 2024. [13] F. Zhang, B. Chen, Y. Zhang, J. Keung, J. Liu, D. Zan, Y. Mao, J.G. Lou, and W. Chen, “Repocoder: Repository-level code completion through iterative retrieval and generation,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 2471–2484. [14] H. Yu, B. Shen, D. Ran, J. Zhang, Q. Zhang, Y. Ma, G. Liang, Y. Li, Q. Wang, and T. Xie, “Codereval: A benchmark of pragmatic code generation with generative pre-trained models,” in Proceedings of the

46th IEEE/ACM International Conference on Software Engineering, 2024, pp. 1–12. [15] Anthropic, “Claude 4 Sonnet,” may 2025, released May 22, 2025; hybrid reasoning model with enhanced coding, multimodal support, and extended-thinking capabilities. Accessed: 2025-07-18. [Online]. Available: https://www.anthropic.com/news/introducing-claude-4 [16] D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li et al., “Deepseek-coder: when the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024. [17] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan et al., “Deepseek-v3 technical report,” arXiv preprint arXiv:2412.19437, 2024. [18] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” arXiv preprint arXiv:2501.12948, 2025. [19] G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805, 2023. [20] G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang et al., “Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context,” arXiv preprint arXiv:2403.05530, 2024. [21] A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276, 2024. [22] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023. [23] S. Hong, M. Zhuge, J. Chen, X. Zheng, Y. Cheng, J. Wang, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin et al., “Metagpt: Meta programming for a multi-agent collaborative framework,” in The Twelfth International Conference on Learning Representations, 2023. [24] C. Qian, W. Liu, H. Liu, N. Chen, Y. Dang, J. Li, C. Yang, W. Chen, Y. Su, X. Cong et al., “Chatdev: Communicative agents for software development,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15 174–15 186. [25] Q. Zhao, L. Zhang, F. Liu, J. Cheng, C. Wu, J. Ai, Q. Meng, L. Zhang, X. Lian, S. Song et al., “Towards realistic project-level code generation via multi-agent collaboration and semantic architecture modeling,” ACM Transactions on Software Engineering and Methodology, 2025. [26] M. Seo, J. Baek, S. Lee, and S. J. Hwang, “Paper2code: Automating code generation from scientific papers in machine learning,” arXiv preprint arXiv:2504.17192, 2025. [27] J. Luo, X. Zhang, S. Liu, J. Wu, J. Liu, Y. Huang, Y. Huang, C. Yin, Y. Xin, Y. Zhan et al., “Rpg: A repository planning graph for unified and scalable codebase generation,” in The Fourteenth International Conference on Learning Representations, 2026. [28] J. Liu, C. Xu, C. Wang, T. Bai, W. Chen, K. Wong, Y. Lou, and X. Peng, “Evodev: An iterative feature-driven framework for end-to-end software development with llm-based agents,” arXiv preprint arXiv:2511.02399, 2025. [29] M. H. Nguyen, T. P. Chau, P. X. Nguyen, and N. D. Bui, “Agilecoder: Dynamic collaborative agents for software development based on agile methodology,” in 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering (Forge). IEEE, 2025, pp. 156–167. [30] F. Liu, Y. Liu, L. Shi, Z. Yang, L. Zhang, X. Lian, Z. Li, and Y. Ma, “Beyond functional correctness: Exploring hallucinations in llmgenerated code,” IEEE Transactions on Software Engineering, 2026. [31] Z. Zhang, C. Wang, Y. Wang, E. Shi, Y. Ma, W. Zhong, J. Chen, M. Mao, and Z. Zheng, “Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation,” Proceedings of the ACM on Software Engineering, vol. 2, no. ISSTA, pp. 481–503, 2025. [32] Y. Wang, Y. Wang, D. Guo, J. Chen, R. Zhang, Y. Ma, and Z. Zheng, “Rlcoder: Reinforcement learning for repository-level code completion,” in 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024, pp. 165–177. [33] K. Zhang, J. Li, G. Li, X. Shi, and Z. Jin, “Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges,” in Proceedings of the 62nd Annual Meeting of the

Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 13 643–13 658. [34] D. Wu, W. U. Ahmad, D. Zhang, M. K. Ramanathan, and X. Ma, “Repoformer: selective retrieval for repository-level code completion,” in Proceedings of the 41st International Conference on Machine Learning, 2024, pp. 53 270–53 290. [35] H. Ma, Y. Liang, J. Si, W. Zhang, H. Guan, C. Zheng, B. Xu, and W. Lu, “Srlcg: Self-rectified large-scale code generation with multidimensional chain-of-thought and dynamic backtracking,” arXiv preprint arXiv:2504.00532, 2025. [36] S. Zhang, J. Wang, G. Dong, J. Sun, Y. Zhang, and G. Pu, “Experimenting a new programming practice with llms,” arXiv preprint arXiv:2401.01062, 2024. [37] W. Zhao, N. Jiang, C. Lee, J. T. Chiu, C. Cardie, M. Gallé, and A. M. Rush, “Commit0: Library generation from scratch,” in The Thirteenth International Conference on Learning Representations, 2025. [38] J. Cao, Z. Chen, J. Wu, S.-C. Cheung, and C. Xu, “Javabench: A benchmark of object-oriented code generation for evaluating large language models,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 870–882. [39] J. Ding, S. Long, C. Pu, H. Zhou, H. Gao, X. Gao, C. He, Y. Hou, F. Hu, Z. Li et al., “Nl2repo-bench: Towards long-horizon repository generation evaluation of coding agents,” arXiv preprint arXiv:2512.12730, 2025. [40] A. Liu, A. Mei, B. Lin, B. Xue, B. Wang, B. Xu, B. Wu, B. Zhang, C. Lin, C. Dong et al., “Deepseek-v3. 2: Pushing the frontier of open large language models,” arXiv preprint arXiv:2512.02556, 2025. [41] A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram et al., “Openai gpt-5 system card,” arXiv preprint arXiv:2601.03267, 2025. [42] S. Robertson, H. Zaragoza et al., “The probabilistic relevance framework: Bm25 and beyond,” Foundations and Trends® in Information Retrieval, vol. 3, no. 4, pp. 333–389, 2009. [43] Z. Yang, S. Chen, C. Gao, Z. Li, X. Hu, K. Liu, and X. Xia, “An empirical study of retrieval-augmented code generation: Challenges and opportunities,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 7, pp. 1–28, 2025.

Related documents

• Code2LoRA: Hypernetwork-Generated Adapters for Code Language Models under Software Evolution#1488388
• Beyond Repository Boundaries: Cross-Repository Graph Retrieval for Code Generation#996791
• CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation#429448
• CodeArena: A Collective Evaluation Platform for LLM Code Generation#1490348
• OS-Bench: Evaluating SWE Agents on Cross-OS Code Migration#1345191