arXiv CSopen access

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

arXiv:2606.22082v1 [cs.SE] 20 Jun 2026

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation YIFEI WANG, School of Computer Science, Wuhan University, China RUIYIN LI, School of Computer Science, Wuhan University, China PENG LIANG, School of Computer Science, Wuhan University, China QIONG FENG, School of Computer Science, Nanjing University of Science and Technology, China ZENGYANG LI, School of Computer Science, Central China Normal University, China MOJTABA SHAHIN, School of Computing Technologies, RMIT University, Australia ARIF ALI KHAN, M3S Research Group, SEIS Unit, University of Oulu, Finland Natural language to repository generation (NL2Repo) requires a system to construct an entire software repository from a natural-language requirements document. Compared with function-level code generation, this task demands longer planning horizons, stable interfaces across files, and iterative debugging of cross-file inconsistencies. To address these challenges, we propose CodeTeam, an LLM-based multi-agent framework that separates planning, decision making, and implementation into distinct, coordinated stages. In the planning stage, multiple Architect agents draft competing software design sketches (SDSs), optionally grounded by retrieved design references. A cto (Chief Technical Officer) agent then evaluates, selects, and normalizes the most promising SDS into a machine-checkable contract that specifies file ownership, public interfaces, and dependency constraints. In the implementation stage, Developer agents generate code under a dependencyaware scheduler with bounded context and lightweight Git-based coordination, while a qa agent runs tests and drives iterative repairs. On the synthesis-based SketchEval benchmark, we explicitly compare CodeTeam’s prompt-engineering (PE) and supervised fine-tuning (SFT) variants with the corresponding CodeS variants, where CodeTeam improves the overall SketchBLEU by 4.1 and 2.9 absolute points, respectively. On the execution-based NL2Repo-Bench benchmark, used as an external validation protocol, CodeTeam achieves the highest average test pass rate in both settings (34.6% PE, 42.3% SFT), confirming that the sketch-improvements extend to functional correctness under upstream test suites. Ablation results show that project-specific developer allocation and retrieval-augmented planning each contribute substantially to the SketchBLEU improvement (9.9% and 8.1% relative, respectively), while Git-based coordination adds a smaller but consistent complementary improvement. CodeTeam and the experimental results are available at [35]. CCS Concepts: • Software and its engineering → Software development techniques. Additional Key Words and Phrases: Large Language Model, Multi-Agent System, Repository-Level Code Generation, Software Engineering Automation, NL2Repo Authors’ Contact Information: Yifei Wang, [email protected], School of Computer Science, Wuhan University, Wuhan, China; Ruiyin Li, School of Computer Science, Wuhan University, Wuhan, China, [email protected]; Peng Liang, School of Computer Science, Wuhan University, Wuhan, China, [email protected]; Qiong Feng, School of Computer Science, Nanjing University of Science and Technology, Nanjing, China, [email protected]; Zengyang Li, School of Computer Science, Central China Normal University, Wuhan, China, [email protected]; Mojtaba Shahin, School of Computing Technologies, RMIT University, Melbourne, Australia, [email protected]; Arif Ali Khan, M3S Research Group, SEIS Unit, University of Oulu, Oulu, Finland, [email protected]. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1557-7392/2026/0-ART0 https://doi.org/10.1145/nnnnnnn.nnnnnnn ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:2

Wang et al.

ACM Reference Format: Yifei Wang, Ruiyin Li, Peng Liang, Qiong Feng, Zengyang Li, Mojtaba Shahin, and Arif Ali Khan. 2026. CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation. ACM Trans. Softw. Eng. Methodol. 0, 0, Article 0 ( 2026), 36 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn

1

Introduction

Large Language Models (LLMs) have demonstrated strong capabilities in code generation, but prevailing evaluation paradigms remain largely confined to short, self-contained tasks such as single-function synthesis in HumanEval-style settings [3]. In contrast, generating a complete software repository from natural-language requirements is a distinct task rather than a direct extension of function-level code generation, introducing challenges in file structures, modules, and interfaces [42]. Recent execution-based NL2Repo-Bench results further show that from-scratch repository generation requires architectural design, dependency management, multi-module implementation, and packaging, and that current agents exhibit failures such as loss of global architectural consistency and brittle cross-file dependencies [8]. Consequently, generating a complete repository from a natural-language specification demands more than localized code completion. It requires a system that can design a complete repository structure, maintain consistencies among cross-file interfaces, and iteratively detect and repair errors that manifest throughout the subsequent lifecycle. Recent repository-level benchmarks show that multi-file tasks are heterogeneous. RepoBench [23] and CrossCodeEval [9] focus on repository-level completion within an existing codebase. The FEA-Bench benchmark [22] evaluates incremental feature implementation, requiring the model to add new functionality while editing related parts of an existing repository. Moreover, CodeS [42] and the more recent NL2Repo-Bench [8] push one step further: the model starts from an empty workspace and must generate a complete repository solely from natural-language requirements. Across these settings, recurring challenges include managing long-range dependencies, reasoning under partial or underspecified information, and addressing errors that only manifest during systemlevel integration. These challenges indicate that repository-level code generation is not solely a modeling problem, but also a workflow design problem. Multi-agent software development systems, such as ChatDev [28] and MetaGPT [16], demonstrate that role specialization and structured interaction can enhance multi-step generation. Similarly, repository-interaction agents such as CodeAgent [44], SWE-agent [40], and OpenHands [34] further underscore the importance of tool use, environment interaction, and iterative execution for achieving reliable outcomes. Despite these advances, a critical gap remains in the natural language-to-repository (NL2Repo) [42] generation. Existing approaches each address part of the NL2Repo pipeline, but none simultaneously tackles the full set of challenges involved in end-to-end repository construction. ChatDev [28] and MetaGPT [16] emphasize collaborative development workflows; SWE-agent [40] focuses on issue resolution in existing repositories; OpenHands [34] provides an open platform for AI developers based on agents with integrated benchmarking; and CodeAgent [44] targets repo-level code generation with tool support for information retrieval, implementation, and testing. More closely related efforts, such as ProjectGen [45], incorporate architecture design and skeleton construction. However, no prior approach simultaneously addresses all three challenges that arise when generating a repository from an empty workspace: (1) high-level design planning that produces a machinecheckable contract, (2) cross-file interface consistency maintained throughout implementation, and (3) coordinating bounded-context implementation against that contract with dependency-aware scheduling and iterative repair. To address this gap, we propose CodeTeam, an LLM-based multi-agent framework tailored to NL2Repo. Our approach extends the sketch-based paradigm of CodeS into a complete end-toend workflow. Specifically, multiple Architect agents first generate competing software design ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:3

sketches (SDSs), optionally supported by retrieved design references. A dedicated cto (Chief Technical Officer) agent subsequently selects the most promising design and normalizes it into a machine-checkable contract before code implementation. Then, Developer agents implement files under a dependency-aware scheduler, exchange interface updates through a lightweight Git-based workflow. Throughout this process, a qa (Quality Assurance) agent is activated under certain conditions, such as when generation tasks are complete, or when test failures are detected. Detected failures are then fed back to the Developer agents for repair. The main contributions of this study are as follows: ● We propose CodeTeam, a multi-agent framework for repo-level code generation that combines SDS competition, optional RAG (Retrieval-Augmented Generation) grounding, ctoguided contract normalization, project-specific developer allocation, dependency-aware execution, Git-based coordination, and qa-driven repair. ● We conduct a comprehensive evaluation of CodeTeam on SketchEval benchmark with SketchBLEU [42] metrics comparing both a prompt-engineering (PE) variant and a supervised fine-tuning (SFT) variant against the corresponding CodeS [42] variants, as well as a single-model baseline and multiple agent baselines. ● We introduce an external execution-based validation protocol on NL2Repo-Bench to enable more rigorous end-to-end assessment. Organization: The rest of this paper is structured as follows: Section 2 presents the related work. Section 3 describes the study design, including the overall framework of CodeTeam, its core modules, and implementation details. Section 4 presents the results, which are further analyzed and discussed in Section 5. The potential threats to validity are clarified in Section 6. Finally, Section 7 concludes this work with future work directions. 2 Related Work 2.1 From NL2Code to Long-Context Code Generation Previous work on natural language to code (NL2Code) mainly focuses on generating small, self-contained programs. Benchmarks such as CoNaLa [41], CONCODE [18], and APPS [15] pair natural-language descriptions with target programs and evaluate NL2Code performance using metrics ranging from string-based similarity to execution-based tests. These datasets also drove the development of code-pretrained models. For example, CodeT5 introduces identifier-aware pre-training to improve both code understanding and code generation [36], while the HumanEval framework established an influential test-based evaluation for synthesized programs [3]. Opensource models then explore richer generation capabilities. InCoder explores bidirectional context and code infilling [12], whereas CodeGen emphasizes multi-turn program synthesis [26]. Despite these advances, such settings largely underrepresent the complexity of real-world software development. In practice, software implementation activities must follow extensive surrounding context, including dependencies across files, shared utilities, build conventions, and repository-specific APIs. Although CONCODE can partially address this by providing the rest of the Java class as context [18], most early NL2Code benchmarks still abstract away cross-file calls, configuration, and repository structure. Consequently, strong performance on isolated code snippets does not necessarily translate into the ability to generate coherent and consistent code at the repository level. Recent work has begun to address this limitation by shifting attention from isolated generation to effective context utilization. Retrieval-augmented generation (RAG) introduces mechanisms for dynamically selecting relevant information as external memory [21], and this idea has been adapted to code generation by retrieving relevant documentation at generation time [46]. Meanwhile, ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:4

Wang et al.

long-context adaptation methods such as LongLoRA extend usable context length at lower cost [5]. However, for repository-level code generation, increasing context length alone is insufficient. Effective systems must also determine which contextual information (e.g., design constraints and interface definitions) should be exposed at each step and ensure that such information remains consistent across multiple interdependent generation steps. 2.2

Repository-Level Code Completion and NL2Repo

Repository-level code generation can be divided into three related yet distinct settings, primarily differentiated by the extent to which the repository context is predefined versus constructed during generation. The first is repository-level completion, where the repository already exists and the main challenge is to retrieve and use relevant cross-file context. For example, RepoBench [23] formalizes this setting and covers retrieval, completion, and end-to-end pipelines. CrossCodeEval [9] broadens evaluations to multiple programming languages and shows that cross-file reasoning is beyond a single ecosystem. Methods such as RepoCoder [43] and GraphCoder [24] tailor retrieval to this setting by iteratively retrieving relevant repository snippets or code-context graphs before generation. The second setting is repository-level incremental development, in which a system extends an existing repository by implementing a new feature. Compared to completion, this setting requires not only generating new code but also coordinating edits across multiple existing files. For example, FEA-Bench [22] formalizes this feature-implementation setting and shows that current models struggle when required to add new components and modify existing ones under executable tests. Although this setting is closer to real-world maintenance scenarios than snippet generation, it still assumes that the repository structure, build configuration, and many interfaces are already in place. The third setting is from-scratch repository generation from natural language requirements (NL2Repo). CodeS [42] formalizes this NL2Repo task and proposes a multi-layer sketch decomposition (RepoSketcher, FileSketcher, and SketchFiller), together with the SketchEval benchmark and the SketchBLEU metric. More recently, NL2Repo-Bench extends the from-scratch setting to 104 Python library tasks and evaluates generated repositories using the original upstream pytest suites, making the benchmark much stricter about end-to-end executability [8]. The three settings above form a spectrum defined by the degree of available structure: from fully specified repositories (completion), to partially specified ones (incremental development), to entirely unconstrained generation (NL2Repo). While closely connected, they are not interchangeable. Methods that perform well when a repository structure and dependencies are given often fail when required to generate file trees, public APIs, and the dependency structure. 2.3

LLM-Based Agents for Software Development

A growing body of work investigates LLM-based agents that address complex tasks through multi-step interaction, tool use, and collaboration. In the context of software development, systems such as ChatDev [28] organize specialized agents for design, coding, and testing, using structured communication protocols to improve end-to-end coherence. Similarly, MetaGPT [16] incorporates standardized operating procedures into the prompting, reducing error propagation across development stages. More recent systems such as MapCoder [17] further decompose multi-step code generation through explicit planning and retrieval-augmented agents. Therefore, these systems show that role specialization and explicit coordination can improve the stability of long-horizon generation over single-pass generation tasks. Recent studies evaluate LLM agents in realistic software engineering environments grounded in existing repositories. SWE-bench [19] constructs tasks from real GitHub issues and the corresponding pull requests, requiring models to resolve issues end-to-end by repository edits. Building ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:5

on this benchmark, SWE-agent demonstrates that designing stronger agent-computer interfaces enables more effective navigation, file editing, and test execution [40]. OpenHands [34] further generalizes this paradigm by providing an open platform for agent evaluation. While these studies significantly advance agent evaluation, they predominantly assume an existing codebase as the starting point, which differs fundamentally from the NL2Repo setting. More closely related to our work are approaches that explicitly target repository- or project-level generation from scratch. CodeAgent introduces a tool-integrated framework for repo-level code generation and a dedicated benchmark, CodAgentBench [44]. ProjectGen [45] studies project-level generation from requirements through architecture design, skeleton construction, and code completion guided by a structured architecture representation. These systems are more relevant to the goal of our work, but CodeTeam differs in a key aspect: its software design sketch (SDS) serves not merely as a planning artifact, but as a machine-checkable contract that specifies file ownership, public interfaces, and dependency constraints, and that actively governs every subsequent implementation step. Even in the absence of multi-agent architectures, iterative repository-level generation with intermediate feedback has been shown to improve performance on generation tasks. For example, AlphaCodium [31] is a test-driven multi-stage pipeline that outperforms single-pass prompting on code generation tasks. Similarly, Self-Debugging [4] enables LLMs to refine their own generated code by explaining and re-executing it, and Reflexion [32] equips agents with verbal self-reflection to improve decision-making across development stages. These observations reinforce an important finding for NL2Repo: when tasks involve many dependent decisions, task decomposition and iterative verification are not just implementation conveniences, but central to generating correct repositories.

2.4

Conclusive Summary

Existing literature has addressed several complementary aspects of repository-level generation under varying operating conditions. Early work on NL2Code and long-context modeling primarily improves local synthesis and context utilization. Studies on repository-level completion and featureimplementation benchmarks focus on generation within existing codebases. Meanwhile, agentbased frameworks demonstrate the benefits of role specialization, tool integration, and iterative verification in managing complex generation workflows. These studies suggest that high-quality software generation depends not only on model capacity, but also on how planning, context, and execution are organized across the end-to-end workflow. While CodeS [42] and NL2Repo-Bench [8] establish NL2Repo as a meaningful setting for repository-level code generation from natural-language requirements, the broader methodological space for this setting remains relatively underexplored. In particular, unlike tasks defined over an existing repository, NL2Repo introduces additional challenges, including repository planning, cross-file coordination, and consistency maintenance throughout the generation process. To better address these challenges, we propose CodeTeam, a multi-agent framework for end-to-end repository construction in the NL2Repo setting. Built on sketch-based planning and agent collaboration, CodeTeam further strengthens coordination and iterative refinement through SDS competition, cto-guided contract normalization, dependency-aware scheduling, lightweight Git coordination, and qa-driven repair mechanisms.

ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:6

Wang et al.

3

Study Design

3.1

Research Questions

We investigate whether a workflow-oriented multi-agent system improves NL2Repo generation, and we identify which mechanisms contribute most to any observed SketchBLEU mectric [42] improvements. To this end, we formulate three research questions (RQs) that progressively evaluate the system at the end-to-end, planning, and implementation levels. RQ1 evaluates controlled end-toend performance on SketchEval [42] and uses NL2Repo-Bench [8] as an external execution-based validation protocol, running all 104 tasks under the same controlled conditions to verify that sketch-level improvements translate into executable correctness. RQ2 focuses on the planning stage, isolating the impact of retrieval-augmented grounding on architectural design quality, and RQ3 analyzes how coordination mechanisms affect repository construction during implementation by scheduling and Git-based communication. RQ1: How do CodeTeam’s PE and SFT versions perform on repository-level code generation tasks compared with CodeS and other baseline methods? To answer RQ1, we evaluate two instantiations of CodeTeam: a prompt-engineering (PE) variant and a supervised fine-tuning (SFT) variant that adapts the same backbone model (Qwen2.5-72BInstruct) to follow the SDS-centric workflow more reliably. We compare CodeTeam’s PE/SFT variants against those of CodeS and representative agent-based baselines adapted to NL2Repo. We report sketch-based results on SketchEval and complement them with an execution-based validation protocol on NL2Repo-Bench. RQ2: Does retrieval-augmented grounding improve architectural planning in NL2Repo tasks? Planning mistakes in NL2Repo (e.g., missing modules or inconsistent interfaces) often propagate to downstream implementation. CodeTeam incorporates an optional RAG subsystem in the architect stage, combined with a competition-and-selection workflow. To isolate its effect, we remove retrieval while keeping the rest of the workflow intact, and evaluate both end-to-end quality and planningstage diagnostics. RQ3: Can dynamic developer allocation and lightweight Git-based coordination improve repository generation quality? Repository generation requires managing inter-file dependencies and frequent interface updates. CodeTeam addresses these challenges with Architect-guided developer allocation, dependencyaware execution, and a Git-based coordination that propagates interface changes under a bounded context. For answering RQ3, we ablate these coordination mechanisms and analyze their impact on repository generation quality under the same backbone model and computational budget. 3.2

Design and Implementation of CodeTeam

Figure 1 illustrates the end-to-end workflow of CodeTeam. Starting from a README-style requirements document (the input format used in SketchEval benchmark [42] and NL2RepoBench benchmark [8]), the system first performs rule-based preprocessing (Step 0 ) to produce a normalized requirements document. The workflow then enters a planning-and-selection phase: in ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:7

Step ❶, 𝑁 Architect agents independently propose alternative software design sketches (SDSs); in Step ❷, a cto agent evaluates these candidates, performs comparative analysis, and selects a single adopted solution. The adopted SDS explicitly determines the three design artifacts highlighted in Figure 1: the technology stack, the project file structure, and the developer allocation plan. In Step ❸, CodeTeam materializes the adopted SDS by initializing a Git repository, pre-constructing the file hierarchy, and instantiating exactly the Developer agents specified in the selected developer plan. Step ❹ corresponds to the implementation stage, during which the Developer agents generate code following the prescribed structure while coordinating through lightweight Git-based communication. In Step ❺, the qa agent evaluates newly completed dependency layers or small batches of files against the requirements and the SDS. In Step ❻, QA findings are routed back to the responsible Developer agents, and any impacted dependent files are requeued when interface changes propagate. Steps ❹–❻ repeat until one of the following termination conditions is met: no new issues are detected, a predefined iteration limit is reached, or resource constraints (e.g., wall-clock time or token budget) are exhausted. Finally, Step ❼ outputs the generated Git repository. Unless otherwise specified, the default configuration uses 𝑁 =4 Architect agents, one cto agent, and one qa agent. The number of Developer agents is not fixed globally. Instead, each Architect agent proposes a project-specific developer plan within its SDS, specifying the required team size of Developer agents, 𝐷, and a file-to-developer ownership map. After the cto agent selects an SDS, CodeTeam instantiates exactly those 𝐷 Developer agents for the run. The choice of agent multiplicity reflects a trade-off between performance and computational cost. Empirically, setting 𝑁 =4 Architect agents provides a favorable balance between design diversity and token efficiency in the current configuration of CodeTeam. Meanwhile, cto agent and qa agent are responsible for deterministic, single-pass operations (e.g., selecting among SDS candidates or generating and executing lightweight tests) that do not substantially benefit from parallelization. We expect that increasing the number of agents in these roles would yield negligible performance improvements while potentially introducing unnecessary stochasticity, since their tasks are inherently sequential. Therefore, the current configuration uses a single cto and a single qa agent. To ensure fair evaluation, all agents within a run share the same backbone model and decoding configuration (e.g., temperature, top-𝑝, and the remaining sampling parameters). This design isolates the effects of workflow structure and inter-agent coordination, ensuring that observed performance differences arise from the framework itself rather than variations in model behavior. 3.2.1 Requirements Preprocessing. We treat a repository’s README as the input of naturallanguage requirements. To convert this input into a structured and analysis-ready form, we apply a rule-based preprocessing pipeline. Specifically, we normalize section headings, flatten nested bullet lists, keep fenced code blocks only when they convey actionable information (e.g., commands, configuration keys, or API examples). Nonessential elements (e.g., badges, images, changelogs, release notes, and contributor sections) are removed. The output is a single requirements document with stable section markers. These markers allow subsequent system components to reference requirement spans by heading name rather than by raw line indices. Overall, this preprocessing step keeps requirements-relevant details while eliminating noise that is unlikely to contribute to downstream repository construction. 3.2.2 Software Design Sketch as an Executable Contract. CodeTeam represents each candidate plan as a structured software design sketch (SDS), a machine-checkable design contract that specifies the repository-level implementation plan before code generation begins. Each SDS records the three design artifacts highlighted in Figure 1: the technology stack, the project file structure, and the developer allocation plan. Here, the project structure is broader than a directory tree: it includes the ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:8

Wang et al.

．

ａｐｐ／

Ｄｊａｎｇｏ

ＰｏｓｇｒｅＳＱＬ

ＣＴＯ Ａｇｅｎｔ

Ｒｅｑｕｉｒｅｍｅｎｔ： Ｉ ｎｅｅｄ ａｎ

ａｐｐｌｉｃａｔｉｏｎ ｆｏｒ ｍｙ

◎

２

Ｔｅｃｈｎｏｌｏｇｙ Ｓｔａｃｋ

ＣＴＯ Ｄｅｔｅｒｍｉｎａｔｉｏｎ

Ｐｒｅｐｒｏｃｅｓｓｉｎｇ

．ＣＳＳ

ｐｅｒｓｏｎａｌ ｏｎｌｉｎｅ

ｓｈｏｐ．．

Ａｒｃｈｉｔｅｃｔ Ａｇｅｎｔｓ

Ｄｅｖｅｌｏｐｍｅｎｔ Ｓｏｌｕｔｉｏｎｓ

品 Ｓｔｒｕｃｔｕｒｅ

Ａｄｏｐｔｅｄ Ｓｏｌｕｔｉｏｎ

ｓｃｒｉｐｔｓ．ｊｓ

Ｐｒｏｊｅｃｔ Ｆｉｌｅ

ｉｍａｇｅｓ

Ｌｏｇｏ．ｐｎｇ

ｃｏｎｆｉｇ．ｐｙ

ｒｅｑｕｉｒｅｍｅｎｔｓ．ｔｘｔ ｒｕｎ．ｐｙ

Ｄｅｖｅｌｏｐｅｒ Ａｇｅｎｔｓ Ｎｅｅｄｅｄ

Ｔｅｓｔ ｎｅｗｌｙ ｃｏｍｐｌｅｔｅｄ ⑤ ｄｅｐｅｎｄｅｎｃｙ ｌａｙｅｒｓ ａｎｄ

Ｃｏｌｌａｂｏｒａｔｅ Ａｃｃｏｒｄｉｎｇ ｔｏ ｔｈｅ

Ｉｎｉｔｉａｔｉｎｇ

Ｓｔｒｕｃｕｒｅ

Ａｇｅｎｔｓ

ｓｍａｌｌ ｂａｔｃｈｅｓ ｏｆ ｆｉｌｅｓ

Ｄｅｖｅｌｏｐｅｒ Ａｇｅｎｔ ２

④

Ｇｉｔ Ｒｅｐｏｓｉｔｏｒｙ

Ｃｏｎｔａｉｎｉｎｇ Ｇｅｎｅｒａｔｅｄ Ｓｏｆｔｗａｒｅ

Ｃｏｌｌａｂｏｒａｔｅ Ｗｉｔｈ

⑦

Ｇｉｔ

Ｏｕｔｐｕｔ ＱＡ Ａｇｅｎｔ

Ｒｅｐｏｓｉｔｏｒｙ ③ Ｇｉｔ Ｉｎｉｔｉａｔｅｄ

Ｄｅｖｅｌｏｐｅｒ Ａｇｅｎｔ １

Ｄｅｖｅｌｏｐｅｒ

Ａｇｅｎｔ ３

Ｍｕｌｔｉｐｌｅ Ｒｏｕｎｄｓ ｏｆ Ｆｅｅｄｂａｃｋｓ ＆ Ｍｏｄｉｆｉｃａｔｉｏｎ

Fig. 1. CodeTeam workflow from requirements preprocessing ( 0 ) to architect planning (❶), cto selection (❷), solution materialization and Git-coordinated implementation (❸–❹), qa-driven repair (❺–❻), and final repository output (❼).

fixed repository file hierarchy, each file’s responsibility and public interface, and explicit dependency edges among files. Concretely, an SDS specifies: (1) the technology stack, including programming language, framework, testing framework, and key dependencies; (2) a fixed repository file tree that the system instantiates before implementation; (3) per-file responsibilities and public interfaces (e.g., classes or functions signatures); (4) explicit cross-file dependencies (e.g., imports or calls); (5) a developer allocation plan that determines how many Developer agents are needed and which files are owned by Developer 0 to Developer 𝐷−1; and (6) coarse workload estimates to guide execution. By embedding team size and ownership into the SDS, the system treats these factors as project-specific planning decisions rather than fixed global hyperparameters. From an operational perspective, the SDS must include four mandatory fields: tech_stack, repo_tree, developer_plan, and files. The repo_tree field records the file hierarchy, while each entry in files records the corresponding file-level contract, including its purpose, exported interface, owner, and dependencies. A candidate is considered parseable only if it can be loaded as JSON without manual repair. Structural validity then requires that every file listed in files appears in repo_tree, every owner identifier appears in developer_plan, every depends_on edge points to a declared file or exported symbol, and the resulting inter-file dependency graph is acyclic after collapsing same-file symbol references. Candidates that fail these hard checks are still logged for diagnostics, but excluded from cto selection. Following plan selection by the cto agent, CodeTeam initializes a Git repository, creates the full file tree, and instantiates the Developer agents declared in the chosen SDS. During implementation, developers are constrained to operate strictly within the predefined structure: no extra files outside the SDS can be created, and file ownership must remain consistent with the developer plan. This constraint is enforced in our main experiments to ensure that architectural decisions remain traceable to the planning stage and that comparisons across methods stay reproducible. Figure 1 shows a simplified (illustrative) SDS schema. Listing 1. Simplified SDS schema

ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:9

{ "tech_stack": {"language": "Python", "framework": "Flask", "test": "pytest"}, "repo_tree": ["src/core.py", "src/api.py", "tests/test_api.py"], "dependencies": ["flask>=2.0", "pytest"], "developer_plan": { "num_developers": 3, "owners": [ {"id": 0, "files": ["src/core.py"]}, {"id": 1, "files": ["src/api.py"]}, {"id": 2, "files": ["tests/test_api.py"]} ] }, "files": [ {"path": "src/core.py", "purpose": "domain objects", "public_api": ["class User: ..."], "owner": 0}, {"path": "src/api.py", "purpose": "HTTP API wrappers", "public_api": ["def fetch_user(user_id: str) -> User"], "depends_on": ["src/core.py::User"], "owner": 1}, {"path": "tests/test_api.py", "purpose": "API tests", "depends_on": ["src/api.py::fetch_user"], "owner": 2} ] }

3.2.3 Architect Stage: Diversified Planning with Optional Retrieval Grounding. Candidate SDSs are created in the Architect stage. Specifically, CodeTeam instantiates 𝑁 Architect agents, each of which independently drafts one candidate SDS. Each candidate specifies not only the repository structure and interfaces, but also a project-specific development plan, including the number of developers to instantiate and the corresponding file ownership assignments. This design enables different architects to explore trade-offs between parallelism and coordination cost. In subsequent implementations, we consider both token costs and SDS quality when empirically selecting and adopting (𝑁 =4) in our experiments as a balanced configuration. A common failure mode is design convergence, where different Architect agents produce highly similar designs, thereby reducing diversity and weakening competitive selection. To mitigate this issue, we introduce several mechanisms. First, the generation order of architects is randomized per seed. Second, each Architect agent is assigned a different design preference (e.g., minimal vs. layered architecture, or class-based vs. functional organization). Third, instead of exposing full prior SDS candidates, each Architect agent only receives a short summary of previously claimed top-level module names and high-level design intents. Consequently, the non-duplication constraint is enforced at the coarse module-boundary level rather than by exposing full candidate plans to downstream architects. When RAG is enabled, the system retrieves the top-𝑘 design references (𝑘=5 by default) from a curated corpus of public repositories. Each item is stored as a compact representation (including README summary, file tree, dependency patterns, file-role summaries, and exported-interface hints) rather than large raw code blocks. Architect agents recognize this information as design hints, such as module decompositions, dependency choices, typical file boundaries, and plausible team organization. Architect agents are instructed to produce an original SDS tailored to the current requirements. Before submission, each Architect agent performs a lightweight selfvalidation that mirrors the parsing and validity checks, as well as the proposed developer count and file ownership mapping. 3.2.4 cto Stage: Selection and Normalization. CTO selection consists of two steps: SDS candidate filtering and SDS candidate selection. First, an automatic filtering process removes invalid candidates. Specifically, candidates are discarded when they exhibit JSON parsing failures, missing ownership, unresolved file references, inconsistent symbol declarations, or cyclic dependencies at the file level. Second, the cto agent evaluates each remaining candidate on four criteria: (1) structural validity, (2) interface consistency, (3) implementability under the stated dependencies and resource constraints, ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:10

Wang et al.

and (4) the appropriateness of the proposed developer count and file ownership relative to the repository scale. Each criterion is scored from 0 to 2, yielding a total score in the range (︀0, 8⌋︀, and the scores are presented in cto agent’s JSON format output. In the case of ties (i.e., when multiple SDS candidates achieve the same total score), preference is given to candidates with fewer undeclared assumptions and lower cross-module dependency fan-out, as these properties indicate better modularity and robustness. The selected SDS is then normalized into a machine-actionable representation by canonicalizing paths, deduplicating symbols, resolving naming conflicts, materializing the developer-to-file ownership mapping, and constructing a file-level dependency graph for execution. If normalization fails after selection, the next highest-scoring valid SDS candidate is selected. The normalized SDS becomes the single source of truth for Developer agents and qa agents. 3.2.5 Development Stage: Architect-planned Allocation with Dependency-aware Execution. CodeTeam implements repositories at the granularity of file tasks, where the task pool is constrained by the developer plan selected in the SDS. Accordingly, Step ❹ in Figure 1 comprises two coupled processes: (1) Developer agents follow the adopted file structure and developer-to-file ownership mapping, and they synchronize changes through Git-based coordination. Each Developer agent is assigned ownership of a subset of files, and the number of developers varies across repositories. Once the cto agent accepts an SDS, that developer-to-file ownership mapping becomes the initial division of labor for the entire execution. To avoid sharing the entire repository with every Developer agent, CodeTeam constructs a compact task-specific context consisting of: (1) the SDS fragment relevant to the target file; (2) the current content of the target file; and (3) interface briefs for dependent files, including signatures, invariants, and the latest summarized interface changes. Developer agents may request at most two additional interface briefs per task, but they do not read the full content of files owned by other agents. Each brief exposes only the depended-upon file path, exported symbols, typed signatures when available, stated invariants, and the latest update_reason summary. This keeps calls focused and reduces long-context pressure while still enabling cross-file consistency. Algorithm 1 describes the execution logic. It maintains a ready queue based on dependency satisfaction and dispatches each ready file to its recorded owner. Files without prerequisites are prioritized early inside each Developer agent’s ownership set; if an interface change occurs, affected dependent files can be requeued for repair and routed back to their owners. The readyqueue priority is a deterministic ordering rather than a learned or tuned score. Among ready files owned by the same Developer agent, CodeTeam first chooses the file with the longest downstream dependency chain, since delaying such a file is most likely to block later tasks. If there is a tie, it chooses the file with more direct dependent files; if a tie still remains, it chooses the file that appears earlier in the normalized SDS file list. We denote this ordering compactly as prio(𝑓 ) = (depth(𝑓 ), fanout(𝑓 ), −order(𝑓 )). 3.2.6 Git-based Coordination, QA Feedback, and Termination. Steps ❺–❻ in Figure 1 form a qaguided repair loop on top of the Git-tracked workspace. To propagate interface updates without sharing full file contents, CodeTeam uses lightweight Git-based coordination. Each Developer agent works on an isolated branch and commits changes upon completing a file task. Each commit message includes a structured update_reason record with four elements: the target file, modified exported symbols, a compatibility note, and affected dependent files. These commit messages serve as the primary communication medium, enabling other agents to update dependent files by using summarized interface information. The qa agent evaluates intermediate states during the qa-guided repair loop without attempting to reconstruct hidden reference tests from the benchmarks (e.g., Nl2Repo-Bench [8]). Instead, ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:11

Algorithm 1: Architect-planned developer allocation with dependency-aware execution in CodeTeam. Input: Normalized SDS, ownership map 𝑂, file dependency graph 𝐺=(𝑉 , 𝐸) Output: Repository implementation 1 initialize Developer agents from 𝑂 2 𝐶 ←∅ // completed files 3 𝑅 ← {𝑣 ∈ 𝑉 ⋃︀ deps(𝑣) = ∅} 4 compute prio(𝑣) for each 𝑣 ∈ 𝑉 5 while 𝑅 ≠ ∅ or running tasks exist do 6 foreach idle Developer agent 𝑑𝑖 do 7 if 𝑅 ∩ 𝑂(𝑑𝑖 ) ≠ ∅ then 8 𝑓 ← arg max𝑔∈𝑅∩𝑂(𝑑𝑖 ) prio(𝑔) 9 assign 𝑓 to developer 𝑑𝑖 with compact context 10 mark 𝑓 as running 11 12 13 14 15 16 17 18

if a developer finishes 𝑓 then commit changes with a structured message including update_reason update the interface index and refresh dependent briefs remove 𝑓 from running and from 𝑅 𝐶 ← 𝐶 ∪ {𝑓 } add newly-ready files into 𝑅 based on 𝐶 if QA trigger holds then run tests; route failures to the owners of suspected source files and re-queue affected files if needed

after completing each dependency layer or a small batch of ready files, the qa agent generates lightweight, temporary tests from the SDS and specified requirements. These tests are generated within a temporary QA directory, executed, and removed before final evaluation, ensuring that they do not affect SketchBLEU, repository-size statistics, or the final delivered file tree. QA feedback is normalized into issue records containing a symptom category, traceback or assertion message, suspected source files, and the minimal repair scope. The scheduler requeues the smallest affected set of files: by default, only the likely source file is requeued and assigned back to its recorded owner for repair, and dependent files are requeued only when the issue impacts a public API, import path, or shared configuration artifact. To identify the true error source when failures involve multiple dependent files, the qa agent applies four diagnostic criteria in sequence: (1) interface consistency: the suspected source file must declare all symbols and types referenced by the failing dependent file in its public interface; (2) dependency directionality: if the failure traceback originates from a dependent file but the root cause is an undeclared or changed interface in a source file, the source file is flagged as the repair target; (3) structural validity: the source file must pass syntactic checks independently before the dependent file is re-evaluated; and (4) minimal repair scope: when multiple files exhibit cascading failures, only the earliest file in the dependency chain that violates (1) or (3) is requeued by default, and downstream dependents are requeued only when the issue propagates to a public API, import path, or shared configuration artifact. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:12

Wang et al.

Execution terminates under one of three conditions: (1) the qa agent reports no failures over a complete validation pass, (2) the configured QA-iteration limit is reached, or (3) a global wall-clock or token budget is exceeded. This corresponds to the transition from the iterative loop in Steps ❹–❻ to the final output in Step ❼. 3.3

Benchmark Experiment Design

We evaluate our approach on SketchEval, a benchmark introduced by CodeS [42]. SketchEval comprises 19 real-world Python repositories, each paired with requirements documents and reference implementations. Following the official SketchEval split, we use three difficulty groups: hard repositories contain more than 10 Python files or more than 2,500 Python code lines; medium repositories contain more than 5 Python files or more than 500 Python code lines but do not meet the hard threshold; and the remaining repositories are classified as easy. This yields 5 easy, 8 medium, and 6 hard tasks. For each task, the input is the processed requirements document, and the output is a complete repository generated in an empty workspace. SketchBLEU [42] serves as the main evaluation metric. SketchBLEU adapts CodeBLEU [30] for repository-level evaluation by comparing sketch representations of generated and reference repositories using four complementary signals: (1) n-gram overlap, (2) weighted n-gram overlap, (3) structural similarity, and (4) dataflow-level semantic similarity. Specifically, n-gram overlap is a BLEU-style precision score computed over short token sequences after the generated and reference repositories are linearized; it therefore measures whether local code fragments, such as import statements, function signatures, and assignment or return statements, are reproduced. Weighted n-gram overlap uses the same token-sequence comparison but gives larger weights to language keywords that mark definitions, imports, returns, and control structure, such as def, class, return, and import, reducing the influence of matches on punctuation or generic boilerplate. Structural similarity compares the repository structure encoded by the sketch, including the directory tree, module boundaries, and file-level syntax skeletons. Dataflow-level semantic similarity compares function-level dataflow relations across the two repositories, capturing whether dependencies among symbols, APIs, and imported components are preserved. We compare CodeTeam against four baselines that cover single-agent generation, multi-agent software development, generalist autonomous agents, and prior task-specific NL2Repo methods. ● Vanilla. The Vanilla baseline uses a single agent to generate the entire repository without explicit task decomposition, role collaboration, or cross-file iterative repair. At each round, the same backbone model receives the full processed requirements document and is given basic file creation and file read/write capabilities. When the context length approaches the model limit, the system automatically compresses the interaction history and continues generation by feeding the model both the full requirements document and the compressed history. This loop continues until the model determines that the requirements have been fully implemented or the global token budget is exhausted. ● ChatDev. ChatDev [28] is an academic multi-agent software development framework. We adapt its design, coding, and testing role decomposition to the NL2Repo setting while restricting its tools to workspace inspection, file editing, and local test execution. ChatDev serves as the primary academic multi-agent baseline for evaluating whether role-based software development workflows transfer to repository-level code generation from natural language requirements. ● AutoGPT. AutoGPT [13] is a representative open-source generalist autonomous agent framework. We instantiate it as a single Controller agent that iteratively plans, generates code, inspects the workspace, edits files, and executes local tests under the same resource ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:13

constraints. This baseline evaluates whether a generic long-horizon agent loop, without task-specific repository planning mechanisms, is sufficient for NL2Repo. ● CodeS. CodeS [42] is a task-specific NL2Repo baseline based on multi-layer sketch generation. We include both its prompt-engineering (PE) and supervised fine-tuning (SFT) variants when applicable. CodeS serves as the representative prior NL2Repo baseline and provides a direct comparison against methods designed specifically for repository-level generation. For a fair comparison, none of the baselines receives SDS generation, architect competition, retrieval hints, dependency-aware developer allocation, Git-style interface specifications, or the ownership-aware QA repair mechanism used by CodeTeam. All methods are evaluated with identical wall-clock limits, global token budgets, and iteration limits. Within each controlled comparison, all agents are instantiated with the same backbone model and decoding configuration. The main evaluation (RQ1) assesses the full CodeTeam system, including Architect-stage RAG when enabled. RQ1 is interpreted as a holistic system-level comparison rather than as a retrieval-controlled comparison. To assess the impact of retrieval under a controlled one-factor removal setting, RQ2 compares CodeTeam with and without Architect-stage RAG under identical downstream stages and resource constraints. 3.3.1 Backbone Model and Inference Settings. All agents use Qwen2.5-72B-Instruct as the backbone model [33]. We choose this model by jointly considering code-generation performance, long-context capability, open-weight availability, and the practical training and inference cost of running both PE and SFT experiments at scale. We adopt an open-weight model rather than a closed API-based service to ensure consistency across prompting-only and SFT settings, including the same model families and tokenizers. Unless otherwise specified, we set a temperature of 0.2, top-𝑝 of 0.95, and keep other sampling parameters fixed. We set the maximum input context length to 128k tokens per request and limit each generation to 8,192 new tokens. These limits are sufficient for the README input plus the compact SDS-based context packages used by CodeTeam. 3.3.2 SFT Setup. The supervised fine-tuning (SFT) variant adapts the same Qwen2.5-72B backbone with parameter-efficient fine-tuning (PEFT) [33]. Specifically, we use a QLoRA-based training configuration [7] combined with a LongLoRA-style long-context training setup [5]. The training set contains 18,240 training instances and the 960 validation instances (95/5 split), where each instance corresponds to a single workflow step. These steps include architect-level SDS generation, CTO normalization, development-level file implementation using SDS slices, and QA issue summarization and repair. The data are public Python projects filtered by language, documentation quality, and testability. To reduce contamination risk, we exclude any repository that matches benchmark targets by repository name, GitHub origin, package name, dependency manifests, README n-gram overlap, or package lineage. We also remove near-duplicate repositories by combining README similarity and file-tree similarity, and we keep the SFT source pool disjoint from the Architect-stage RAG corpus. Training is conducted for 3 epochs with AdamW, a learning rate of 1 × 10−4 , cosine decay, 3% warmup, weight decay 0.1, and gradient clipping at 1.0. The backbone is loaded in 4-bit NF4 with BF16 computation following QLoRA [7]; we use LoRA rank of 64, LoRA 𝛼=128, and dropout of 0.05 applied to attention and MLP projection layers. The maximum training sequence length is 32k tokens. Training runs on 4 NVIDIA A800 80GB GPUs with ZeRO-3 [29], gradient checkpointing, and FlashAttention-2 [6]. The effective global batch size is 64 (micro-batch size 1 per GPU, gradient accumulation 8). The final checkpoint is selected based on validation loss. 3.3.3 RAG Corpus and Retrieval Settings. The Architect-stage RAG corpus is built from GitHub Python repositories with more than 500 stars. We exclude benchmark repositories, as well as forks, ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:14

Wang et al.

mirrors, archived projects, boilerplate templates, and any repositories overlapping with training or test targets by name, origin, package lineage, dependency manifests, or high README similarity. After automatic filtering, a manual audit was conducted by the first author to further reduce leakage risk. For each candidate repository, the audit inspected its repository name, GitHub origin, README, package namespace, dependency manifests, top-level file tree, and project description against both the training pool and benchmark targets. Repositories were removed if they appeared to be forks, mirrors, renamed variants, tutorial templates, benchmark-derived projects, or near-duplicates of any target repository. Borderline cases were resolved conservatively by exclusion. For indexing, each repository is converted into chunks rather than raw code, including README summaries, top-level file trees, dependency manifests, and brief file-role and interface summaries. The resulting artifacts are segmented into 768-token chunks with 128-token overlap and embedded using BGE-M3 [2]. We index these embeddings using an HNSW [25] index implemented in FAISS [10]. At query time, the architect stage retrieves top-𝑘=5 chunks from distinct repositories based on cosine similarity. Additional metadata-based filtering removes near-duplicate chunks, exact file-tree matches, and chunks that are highly similar to the current benchmark task. This design prioritizes reusable architectural patterns while minimizing the risk of implementation-level copying. Each method is evaluated on each task using three random seeds, and we report the mean±standard deviation of SketchBLEU. To assess statistical significance, we apply the Wilcoxon signed-rank test [37] and bootstrap confidence intervals [11] to the task-level score differences between methods; details are reported alongside the results in Section 4.1. 3.4

Execution-Based Evaluation on NL2Repo-Bench

To assess whether the conclusions on SketchEval generalize to a stricter execution-based setting, we introduce an external evaluation-based protocol on NL2Repo-Bench [8]. NL2RepoBench contains 104 tasks spanning nine categories of Python libraries. Each task provides the agent with a single natural-language specification and an empty workspace, and the generated repository is evaluated by executing the original upstream pytest suite in a controlled environment [8]. The benchmark groups tasks by repository size into easy (26), medium (46), and hard (32), with an average input length of approximately 18.8k tokens [8]. Compared to SketchEval, NL2RepoBench emphasizes end-to-end executability: generated repositories must satisfy package layout, dependency specification, import consistency, and functional correctness as verified by the upstream test suites [8]. In our work, NL2Repo-Bench is used as an external validation protocol. We follow the documentonly setting of NL2Repo-Bench, where agents do not have access to the target repository, scaffolding, or test cases during generation. The reporting metrics are the standard benchmark metrics, including the average test pass rate and Pass@1, along with the easy/medium/hard breakdown [8]. Every method listed in Table 5 is evaluated on all 104 tasks using three random seeds (the same seed set as SketchEval); we report the mean pass rate across seeds for each task and then average over tasks. As NL2Repo-Bench is substantially larger and more expensive than SketchEval, we treat it as an external validation benchmark for RQ1 rather than repeating the full ablation study on all 104 tasks. A practical adaptation is needed for repository generation because the original NL2Repo-Bench protocol assumes that a coding agent receives a single natural-language specification in an empty workspace and then autonomously produces a complete, installable Python repository. This formulation does not directly expose an interface for CodeTeam’s multi-agent, multi-stage workflow, which decomposes repository construction into architect competition, CTO-based sketch selection, developer implementation, Git-mediated coordination, and QA-driven repair. We therefore keep ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:15

Table 1. Ablation variants of CodeTeam (PE) and what is disabled. Variant

Change compared with full CodeTeam

Full CodeTeam

RAG + architect competition + dynamic developer allocation + Git coordination + QA loop CodeTeam w/o RAG Disable retrieval hints; architects design SDS only from requirements CodeTeam w/o dy- Replace Architect-guided developer count/ownership with a fixed fournamic allocation developer round-robin assignment CodeTeam w/o Git co- Disable branch-based workflow and structured update_reason; no commitordination based briefs

the benchmark input condition unchanged, providing each task specification as the initial requirements document without revealing the target repository, scaffolding, or test cases, but we route the implementation process through CodeTeam’s internal coordination workflow. Crucially, this adaptation affects only how the repository is generated. After generation is complete, the produced repository is returned to the original NL2Repo-Bench evaluation framework, where it is packaged and evaluated using the benchmark’s standard execution-based procedure, including the original upstream pytest suite and the official pass-rate metrics. When a baseline can be rerun under the same document-only setting, we include the rerun results in our comparisons. Concretely, we rerun all eight methods listed in Table 5 (Vanilla, ChatDev, AutoGPT, CodeS, and CodeTeam in both PE and SFT variants) using the same backbone model (Qwen2.5-72B), identical decoding hyper-parameters, and the same computational budget as in SketchEval. For the agent-based baselines (ChatDev, AutoGPT), we apply the same adaptation procedure used in SketchEval: we replace each baseline’s original backbone with Qwen2.5-72B while preserving the baseline’s own prompting templates, role definitions, and interaction protocols; the only modification is to format the NL2Repo-Bench task specification as the initial user message instead of the SketchEval sketch prompt. This ensures that observed performance differences are attributable to the workflow design rather than to model or prompt discrepancies. For systems that cannot be directly reproduced, we cite published results as contextual references, but they are not integrated into controlled comparison tables. Overall, the role of the NL2Repo-Bench experiment is to provide an external validation of end-to-end functional correctness, complementing the controlled analysis conducted on SketchEval. 3.5

Ablation Study Design

To answer RQ2 and RQ3, we design a set of ablation variants in which individual mechanisms are removed while the rest of the workflow remains unchanged. Unless otherwise specified, all ablations are conducted under the CodeTeam-PE setting, using the same backbone model (Qwen2.572B), decoding settings, and computational constraints as the full system. Table 1 summarizes the variants and the mechanism disabled in each case. 3.5.1 Ablation for RQ2: Disabling RAG in the Architect Stage. To evaluate the role of retrieval, we construct CodeTeam without RAG (CodeTeam w/o RAG) by removing the retrieval subsystem entirely. Architect agents receive only the requirements document and the competition constraints (i.e., design-preference prompts and a non-duplication requirement), without access to external references. The cto agent still selects and normalizes an SDS, and the downstream Developer agents and qa-guided repair loop remain unchanged. Besides end-to-end SketchBLEU, we record planning-stage diagnostics that do not require executing reference repositories: (1) SDS parse success rate (whether the output is machine-checkable), ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:16

Wang et al.

(2) structural validity rate (measuring the absence of missing referenced files/interfaces in the SDS), and (3) plan diversity across 𝑁 candidates, measured by pairwise overlap of file-path sets and API name sets. These diagnostics help determine whether retrieval primarily improves final quality by stabilizing plan correctness, increasing candidate diversity, or both. 3.5.2 Ablation for RQ3: Disabling Dynamic Allocation in the Developer Stage. To assess the impact of dynamic developer allocation, we construct CodeTeam in which the developer plan is removed from the SDS and replaced with a fixed team of four Developer agents for each repository. After SDS normalization, files are assigned among the four agents using a deterministic round-robin rule over the SDS file list, regardless of repository scale or module coupling. Agents implement their assigned files sequentially, and the fixed developer-to-file ownership mapping remains fixed throughout execution. When QA reports failures, repair requests are routed to the original file owner. Dependent files are not automatically requeued unless explicitly specified by the QA report. This design preserves the multi-agent setting but removes project-specific team sizing and Architect-guided file ownership. In addition to SketchBLEU, we record (1) the number of QA iterations for convergence (or until token resource exhaustion), (2) the number of rework events triggered by interface mismatches (e.g., failing imports, missing attributes), and (3) the average context size per agent request as a proxy for long-context pressure. These records indicate whether dynamic developer allocation reduces the propagation of repairs across dependent files. 3.5.3 Ablation on Git-based Coordination. To assess the impact of commit-based coordination, we construct CodeTeam w/o Git-based workflows. Specifically, we disable the branch-per-agent workflow and remove structured commit messages. To avoid edit conflicts, we introduce a simple file-level locking scheme in which only one agent can edit a file at any given time. However, interface briefs can no longer be derived from update_reason commits. Instead, when an agent requests a brief from another file, the system returns only the SDS-declared signatures, which may become outdated as the implementation evolves. We focus on cross-file consistency metrics: (1) the number of interface-mismatch failures reported by the qa agent (e.g., missing symbols, wrong signature usage), (2) the frequency of cross-file brief requests by Developer agents for external files (as a proxy for coordination demand), and (3) end-to-end SketchBLEU and stability across random seeds. Across all ablations, we maintain the same architect/CTO prompts (except for retrieval removal), the same QA loop and termination criteria, and the same computational budgets. Each ablation is evaluated on all SketchEval tasks using three random seeds, and we report average performance along with the diagnostic metrics described above. These one-factor-at-a-time ablations estimate the conditional marginal contribution of each component relative to the full CodeTeam configuration; they are not intended to exhaustively characterize higher-order interactions among components. 4

Results

This section reports the experimental results corresponding to the three RQs formulated in Section 3.1, following the evaluation protocol described in Section 3.3 to Section 3.5. We first present the controlled sketch-based evaluation on SketchEval (RQ1), followed by executionbased validation on NL2Repo-Bench. We then analyze the ablation results that address planningstage retrieval (RQ2) and implementation-stage coordination (RQ3). Unless stated otherwise, all SketchEval results are averaged over three random seeds and measured by SketchBLEU [42]. For NL2Repo-Bench [8], every method is likewise evaluated on all 104 tasks using the same three random seeds; we report the average test pass rate and estimate Pass@1 as the average singlegeneration full-task success rate over the three seeds. In addition to aggregate scores, we examine ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:17

task-level differences across the 19 SketchEval repositories to assess whether improvements are broadly distributed rather than dominated by a few outlier tasks. 4.1

RQ1: SketchEval Performance of CodeTeam

Table 2 reports the main benchmark results on SketchEval, and Figure 2 provides a compact summary of the overall SketchBLEU scores for key methods. 4.1.1 Overall Performance in PE and SFT Settings. In the prompt-engineering (PE) setting, CodeTeam achieves an overall SketchBLEU of 51.7%, outperforming the strongest NL2Repo baseline, CodeS (PE), by 4.1 absolute points (8.6% relative). The improvement is more pronounced on medium and hard repositories (see Section 3.3), where cross-file dependencies are denser and iterative coordination becomes more important. Compared with the agent-based baselines (ChatDev [28] and AutoGPT [39]), CodeTeam consistently yields higher scores across all difficulty levels. This result suggests that merely introducing role specialization is insufficient for NL2Repo; effective repository generation additionally requires explicit planning contracts such as the SDS, dependency-aware implementation scheduling, and systematic cross-file repair mechanisms. A closer look at the per-difficulty breakdown reveals distinct failure patterns across baseline categories. On easy repositories (5 tasks, typically 4–7 files with shallow dependency graphs), all methods except Vanilla achieve reasonably high scores, and the margin between CodeTeam and the best NL2Repo baseline is modest (3.4 points over CodeS (PE)). This is expected: when cross-file constraints are sparse, even a simple role decomposition can produce a structurally adequate repository. On medium repositories (8 tasks, 8–15 files with moderate coupling), the agent-based baselines begin to degrade more sharply. ChatDev, for instance, drops from 55.0 (easy) to 46.1 (medium), a 16% relative decline, whereas CodeTeam drops only from 56.4 to 56.0 (1% relative). Manual inspection of ChatDev outputs on medium tasks suggests that the design– coding–testing pipeline lacks an explicit module-boundary contract: once the initial design omits a module, no downstream stage recovers it, and the implementation proceeds with a reduced file set. AutoGPT exhibits a related but more severe pattern: its single-controller loop tends to generate files sequentially without revisiting earlier decisions, so interface mismatches introduced early in the sequence accumulate without being corrected. On hard repositories (6 tasks, 14–22 files with dense inter-package dependencies), the gap widens further. CodeS (PE) outperforms the agent baselines by approximately 5–6 points (range: 4.8–6.3), confirming that sketch-based decomposition provides a meaningful structural foundation; however, CodeTeam extends this advantage by an additional 4.7 points (37.2 → 41.9). This gap reflects the end-to-end gain of the full workflow rather than a single isolated component. Table 7 provides conditional marginal estimates: on hard repositories, removing SDS-stage retrieval lowers performance by 5.0 points, removing SDS-derived dynamic allocation and dependent-file requeueing lowers performance by 5.4 points, and disabling Git-based coordination lowers performance by 1.7 points. The QA repair analysis in Section 4.4 further shows a 5.4-point improvement from the initial implementation pass to the final repaired output across all tasks. These estimates are not additive, but they suggest that SDS planning and SDS-guided allocation/scheduling are the largest contributors, while QA repair and Git coordination mainly improve late-stage cross-file consistency. In the fine-tuning (SFT) setting, CodeTeam achieves the highest overall score (60.9%), surpassing CodeS (SFT) by 2.9 absolute points (5.0% relative). The smaller relative margin compared with the PE setting indicates that fine-tuning partially internalizes the workflow conventions that CodeTeam enforces through its multi-agent structure; nevertheless, the structured multi-agent workflow still provides a measurable additional SketchBLEU benefit even when the backbone model has been adapted to the task. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:18

Wang et al.

Table 2. Performance on SketchEval measured by SketchBLEU (%). We report mean±std over three seeds. Best results within each block are in bold. “All” is the mean over all individual tasks, not a weighted average of the per-difficulty group means. Setting

Method

Easy (5)

Medium (8)

Hard (6)

All (19)

PE

Vanilla ChatDev AutoGPT CodeS (PE) CodeTeam (PE)

16.8±4.1 55.0±2.5 53.8±2.9 53.0±2.6 56.4±2.0

14.2±3.2 46.1±3.4 45.0±3.8 52.1±3.2 56.0±2.5

12.9±4.7 31.8±4.5 32.4±4.3 37.2±3.8 41.9±3.3

14.5±2.3 43.9±2.0 43.3±2.2 47.6±2.2 51.7±1.8

SFT

Vanilla (SFT) CodeS (SFT) CodeTeam (SFT)

25.9±2.7 61.0±2.1 64.1±1.8

23.4±2.5 58.2±2.1 61.0±1.8

22.2±3.2 55.3±3.1 58.1±2.6

23.7±1.7 58.0±1.6 60.9±1.4

SketchBLEU (%)

60.9

58

60

51.7 47.6

43.9

43.3

40 20

14.5

0

lla

i Van

Dev

at Ch

T

P toG Au

P eSCod

E Co

-PE

eam deT

T

-SF

eS Cod

T

-SF eam T e od

C

Fig. 2. Overall SketchBLEU on SketchEval (all 19 tasks).

4.1.2 SketchBLEU Sub-score Decomposition. Moving from PE to SFT, the n-gram (B.) and weighted n-gram (B.W.) sub-scores of CodeS improve by 12.3 and 13.0 absolute points, respectively, whereas the structural (M.S.) and dataflow (M.D.) sub-scores improve by 9.2 and 7.9 points. This asymmetry suggests that fine-tuning primarily helps the backbone model reproduce surface-level token patterns - such as idiomatic Python constructs, common import statements, and standard boilerplate - while structural and cross-file semantic alignment benefit comparatively less from parameter adaptation alone. By contrast, CodeTeam’s improvements over CodeS are more evenly distributed in the SFT setting: the structural gap narrows from 4.1 (PE) to 3.2 (SFT) points, while the dataflow gap narrows from 7.0 to 4.4 points. The persistence of a meaningful dataflow advantage even after SFT indicates that the dependency-aware scheduling and interface-brief mechanism address cross-file consistency challenges that fine-tuning alone does not fully internalize. In other words, SFT and workflow design improve partially overlapping but distinct aspects of repository quality: SFT strengthens within-file generation fidelity, while the multi-agent workflow primarily improves between-file coordination. To gain deeper insight into the nature of the improvements, Table 3 decomposes SketchBLEU into its four constituent sub-scores as defined in CodeS [42]. In both the PE and SFT settings, ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:19

Table 3. SketchBLEU decomposition on SketchEval (all 19 tasks, %). B. and B.W. denote n-gram and weighted n-gram overlap; M.S. and M.D. denote structural and dataflow matching [42]. Values are mean±std over three seeds. BLEU

BLEUweight

Matchstruc

Matchdf

CodeS (PE) CodeTeam (PE)

44.2±2.3 47.0±2.0

44.0±2.2 46.6±1.8

56.8±3.2 60.9±2.7

46.5±3.6 53.5±3.1

CodeS (SFT) CodeTeam (SFT)

56.5±1.6 59.0±1.3

57.0±1.4 59.4±1.3

66.0±2.2 69.2±1.8

54.4±2.5 58.8±2.2

6.1

MAE of #Files

6 4.7

4

2.9 1.8

2 0

a

ill Van

C

Dev hat

S

e Cod

Co

eam deT

(a) Repository file count error.

MAE of Python LOC

Method

1,900

2,000 1,500

1,050

1,000

680 420

500 0

lla

i Van

atD Ch

ev

eS am Cod odeTe C

(b) Repository LOC error.

Fig. 3. Repository size alignment on SketchEval (PE setting). Values are mean absolute errors against reference repositories.

CodeTeam exhibits a more pronounced advantage on the structural (M.S.) and dataflow (M.D.) components, which are specifically designed to capture repository-level integrity and semantic consistency beyond surface-level token overlap. This finding aligns with the design rationale of CodeTeam: the SDS contract and the dependency-aware development stage primarily target broken interfaces and cross-file inconsistencies - failure modes that are poorly reflected by n-gram-based overlap alone. 4.1.3 Repository Size and Task-level Distribution. Repository generation can fail without producing explicit error messages by producing outputs with an incorrect scale - for example, too few files or overly shallow implementations that omit required modules. To diagnose this failure mode, we compare the generated repository size against the reference repositories in SketchEval by measuring the mean absolute error (MAE) of both file count and Python lines of code (LOC). As shown in Figure 3, CodeTeam reduces the size error compared with the representative baselines shown, suggesting that (1) the planning stage tends to produce more realistic file decompositions that better approximate the reference structure, and (2) the development stage can more reliably sustain the generation of larger repositories without collapsing to a small subset of files. Finally, we examine whether the observed improvements are broadly distributed across tasks rather than driven by a few outliers. In a task-level comparison between CodeTeam (PE) and CodeS (PE), CodeTeam achieves a higher SketchBLEU on 15 out of 19 repositories, ties on 2 (difference within 0.5 points), and scores lower on 2. The wins are concentrated on medium and ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:20

Wang et al.

Table 4. Task-level comparison on SketchEval (19 repositories). “Tie” means absolute difference < 0.5 SketchBLEU points. Comparison

Win

Tie

Loss

CodeTeam (PE) vs CodeS (PE) CodeTeam (SFT) vs CodeS (SFT)

15 14

2 3

2 2

hard repositories, while easy repositories show smaller margins - an expected pattern, since easy repositories involve fewer cross-file constraints and thus present less opportunity for workflowlevel improvements to take effect. A similar distribution holds between CodeTeam (SFT) and CodeS (SFT), with wins on 14 repositories, 3 ties, and 2 losses. To assess statistical significance, we apply the Wilcoxon signed-rank test [37] to the 19 task-level SketchBLEU differences (each averaged over three seeds) between CodeTeam and CodeS. In the PE setting, the test yields 𝑝 = 0.0001, and a bootstrap 95% confidence interval for the mean per-task improvement is (︀2.8, 5.3⌋︀ points (10,000 resamples); in the SFT setting, the corresponding values are 𝑝 = 0.0005 and (︀1.9, 3.8⌋︀ points. Both intervals exclude zero, indicating that the improvements are statistically significant at the 0.05 level. A one-sided sign test on the 17 non-tied PE comparisons (15 wins vs. 2 losses) further corroborates this conclusion (𝑝 = 0.0012). To complement the aggregate statistics, we examine two representative repositories in detail. Case 1: django-tui (medium). This repository implements a Django management command that exposes Django command inspection and execution through a Textual/Trogon-based terminal user interface (TUI). Its core functionality spans the Django command package src/django_tui/management/comma the command-builder screen in tui.py, the interactive shell screen in ish.py, the accompanying trogon.scss stylesheet, and a small Django test project. CodeS (PE) produces a plausible Python package but does not preserve the management-command layout required by Django’s command discovery. In particular, it tends to place the TUI entry point in a generic module such as django_tui/tui.py and treats the interactive shell as an inline helper instead of a separate screen module. As a consequence, the generated project weakens the import path from management/commands/tui.py to InteractiveShellScreen, and the stylesheet path used by the Textual app is no longer aligned with the command module location. In contrast, CodeTeam’s SDS explicitly declares the nested management/commands package, separates tui.py, ish.py, and trogon.scss, and records the dependencies from the Django command class to the UI and shell components. The Developer agent assigned to the command module receives an interface brief that specifies the expected Command.handle entry point and the InteractiveShellScreen import, thereby preventing the mismatch. On this repository, CodeTeam (PE) achieves a SketchBLEU score of 56.8, compared with 49.2 for CodeS (PE), with the improvement concentrated in the structural (+6.2) and dataflow (+8.1) sub-scores. Case 2: sim-web-visualizer (hard). This repository is a browser-based 3D visualizer for simulation environments. It includes shared MeshCat visualization logic, simulator-specific adapters for IsaacGym and SAPIEN, parsers for URDF/MJCF/mesh assets, packaged axis geometry assets, and example scripts for multiple simulator backends. The primary challenge is that simulator clients depend on the shared MeshCatVisualizerBase and AssetResource contract, while the base client in turn dispatches to parser modules that must preserve mesh materials, poses, and resource paths. CodeS generates a flatter structure that merges much of the parser and visualization logic into a small number of modules, and it often treats simulator backends as independent scripts rather than adapters sharing a common resource-loading interface. This structural mismatch weakens the dataflow between base_visualizer_client.py, parser/mesh_parser.py, ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:21

parser/urdf.py, parser/mjcf.py, and the IsaacGym/SAPIEN clients. More importantly, the flattened implementation duplicates mesh-material conversion logic across backends and is more likely to omit package-data paths such as assets/axis_geom, reducing both repository-scale alignment and import consistency. CodeTeam avoids this structural merging because the SDS contract specifies the shared visualizer base, the parser package, the two simulator adapters, and the packaged assets separately, assigns tightly coupled parser files to the same Developer agent, and records the cross-package dependencies explicitly. When the qa agent detects a missing parser import or asset path declaration, it routes the repair to the responsible parser/package developer, who can fix the shared interface without rewriting the simulator adapters. The resulting SketchBLEU is 44.1 for CodeTeam versus 35.7 for CodeS on this task. These cases illustrate two recurring mechanisms behind the aggregate improvements: (1) the SDS enforces fine-grained file boundaries that prevent premature merging of logically distinct modules, and (2) the ownership-aware repair loop localizes fixes to the responsible Developer agent, avoiding cascading rework across unrelated files. 4.2

RQ1: Execution-Based Validation on NL2Repo-Bench

To examine whether the improvements observed on SketchEval generalize to a stricter evaluation setting, we additionally compare methods on NL2Repo-Bench [8]. In this benchmark, each generated repository is judged by executing the original upstream pytest suite rather than by structural overlap. Consequently, even minor mistakes in packaging, dependency specification, or cross-file API usage can sharply reduce the score [8], making NL2Repo-Bench a stringent complement to SketchEval for assessing end-to-end functional correctness. Table 5 reports the results. The overall pass rates are modest, underscoring the difficulty of end-to-end repository generation under upstream tests. In the PE setting, Vanilla prompting barely produces runnable repositories, achieving only a 5.4% overall pass rate, because a single-pass generation struggles with package layout and cross-file imports when constructing an entire repository at once. Among the agent-based baselines, ChatDev leads at 25.2%, followed by AutoGPT (23.5%). CodeS (PE) reaches 30.2%, benefiting from its sketch-guided decomposition. CodeTeam (PE) achieves the highest PE-setting score at 34.6% overall, improving over CodeS (PE) by 4.4 absolute points. Notably, the relative ranking of methods is consistent with the SketchEval results, but the performance gaps are slightly wider on medium and hard tasks, where cross-file coordination errors are more likely to manifest as test failures. In the SFT setting, fine-tuning improves all methods substantially. CodeTeam (SFT) reaches 42.3% overall pass rate with a Pass@1 of 6.1%, outperforming CodeS (SFT) by 4.3 absolute points. Compared with the PE setting, the gap between easy and hard tasks narrows under SFT, suggesting that fine-tuning helps the backbone model internalize dependency and packaging conventions that are otherwise fragile under prompting alone. To understand why generated repositories fail execution, we manually classify the root causes of test failures for a stratified sample of 30 tasks (10 per difficulty level) under the PE setting. For each sampled task, we inspect the pytest traceback of the best-seed run and assign each failure to one of four categories: (1) packaging and environment errors, where the repository cannot be installed or imported due to missing setup.py/pyproject.toml, incorrect package names, or undeclared third-party dependencies; (2) import and module-resolution errors, where ImportError or ModuleNotFoundError occurs because a file is missing from the generated tree or an internal import path is incorrect; (3) API-mismatch errors, where the repository is importable but test assertions fail because function signatures, return types, or class hierarchies differ from what the upstream tests expect; and (4) logic errors, where the code structure and API surface are correct but the implementation produces wrong results on specific test inputs. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:22

Wang et al.

Table 5. Average upstream test pass rate (%) and Pass@1 (%) for NL2Repo-Bench. All methods are evaluated on all 104 tasks using three random seeds; pass-rate scores are mean±std averaged over seeds per task and then over tasks. The Pass@1 column reports the average percentage of tasks fully solved by a single generation, averaged over three random seeds. “Overall” is the mean over all individual tasks, not a weighted average of the per-difficulty group means. All baselines are rerun under identical controlled conditions (same backbone, decoding settings, and document-only prompt format). Setting

Method

Overall (%)

Pass@1 (%)

Easy (26)

Medium (46)

Hard (32)

PE

Vanilla ChatDev AutoGPT CodeS (PE) CodeTeam (PE)

5.4±1.1 25.2±1.7 23.5±1.9 30.2±1.5 34.6±1.3

0.6 3.2 2.6 4.2 5.1

9.2±2.6 36.5±3.6 34.6±3.8 42.3±2.9 48.1±2.6

5.1±1.8 24.3±2.8 22.8±3.0 29.1±2.5 33.7±2.2

2.8±1.4 17.2±3.2 15.6±3.4 21.9±2.8 25.0±2.6

SFT

Vanilla (SFT) CodeS (SFT) CodeTeam (SFT)

11.9±1.4 38.0±1.2 42.3±1.1

0.6 4.5 6.1

18.5±3.0 51.9±2.4 56.2±2.2

11.3±2.4 37.0±2.0 41.3±1.8

7.5±2.2 28.1±2.1 32.5±2.0

Table 6. Distribution of root-cause failure categories on NL2Repo-Bench (PE setting, stratified sample of 30 tasks). Each failure is assigned to the earliest-stage category that causes the test to fail. Method Vanilla ChatDev CodeS (PE) CodeTeam (PE)

Packaging (%)

Import (%)

API mismatch (%)

Logic (%)

52.4 38.5 34.2 22.1

30.1 26.3 28.7 19.4

11.6 22.4 21.8 25.3

5.9 12.8 15.3 33.2

Table 6 reports the distribution. Across all methods, packaging and import errors together account for the majority of failures, confirming that structural and organizational mistakes - rather than algorithmic bugs - are the dominant bottleneck in repository-level generation. CodeTeam (PE) reduces packaging errors from 34.2% (CodeS) to 22.1% of sampled failures, primarily because the SDS pre-declares the full file tree and dependency manifest before implementation begins, eliminating a common failure mode in which the generator omits __init__.py files or misnames packages. Import errors decrease from 28.7% to 19.4%, reflecting the benefit of dependency-aware scheduling: files are implemented only after the files they depend on are complete, and interface briefs supply the exact import paths that dependent files should use. API-mismatch errors account for a slightly larger proportion, increasing from 21.8% to 25.3%, partly because the qa agent’s lightweight tests do not always detect subtle signature mismatches until the upstream suite exposes them. Logic errors constitute a larger proportion under CodeTeam than under CodeS (33.2% vs. 15.3%), but this is primarily because structural errors decrease substantially while logic errors remain a persistent residual - a proportional shift that suggests CodeTeam addresses many structural failure modes, shifting the dominant error category from organizational errors to semantic errors, where further improvements would likely require stronger execution-based feedback. While structural and packaging errors decrease as expected, the absolute count of logic failures does not fall proportionally. This is partly because logic errors are inherently harder to eliminate: they require not only correct module boundaries but also correct algorithmic behavior within each module. For instance, in several sampled hard tasks, the generated code correctly imports shared utilities and preserves class hierarchies, yet the implementation diverges in edge cases that the ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:23

lightweight QA tests do not cover (e.g., incorrect handling of empty inputs, off-by-one boundary conditions, or wrong coercion of return types). Additionally, the SDS’s file-boundary enforcement sometimes encourages developers to split logically coupled operations across modules, which can introduce subtle interaction bugs that do not appear until the upstream test suite exercises cross-module data flow. These cases suggest that logic errors under CodeTeam are not merely residual structural mistakes but represent a distinct failure mode that would benefit from richer execution feedback during generation. Overall, the NL2Repo-Bench results confirm that the architectural and coordination improvements in CodeTeam translate into measurable improvements not only in sketch-level similarity but also in executable correctness. The consistent method ranking across both benchmarks provides converging evidence that the observed gains reflect improvements in repository construction rather than artifacts of a particular evaluation metric. It is worth noting the relationship between the average pass rate and Pass@1 in Table 5. Pass@1 measures the expected probability that a single generation attempt fully solves a task, estimated by averaging full-task success over the three random seeds, whereas the average pass rate reflects the mean proportion of individual tests passed across all runs. For CodeTeam (PE), the average pass rate is 34.6% while Pass@1 is 5.1%, indicating that only a small fraction of single generation attempts fully solve an entire task, even though the typical run passes about one-third of its tests. Under SFT, CodeTeam achieves 42.3% average pass rate with a Pass@1 of 6.1%, suggesting that fine-tuning not only raises the average quality but also improves full-task success under singleattempt evaluation. This interpretation is corroborated by the standard deviations in Table 5, which decrease from ±1.3 (PE) to ±1.1 (SFT) for CodeTeam’s overall pass rate. By comparison, CodeS (SFT) achieves a Pass@1 of 4.5%, indicating that the stability benefit partly comes from the backbone adaptation and partly from the structured workflow that constrains the generation space. On this larger benchmark, we compute one pass-rate difference per task across the 104 tasks and apply the Wilcoxon signed-rank test [37]. The test yields 𝑝 < 0.001 for both the PE and SFT comparisons. The bootstrap 95% confidence intervals for the mean task-level pass-rate improvement are (︀4.0, 4.9⌋︀ and (︀4.1, 4.6⌋︀ percentage points, respectively, indicating that the average gains are consistently positive rather than driven by a small number of outlier tasks.

RQ1 Summary: CodeTeam consistently outperforms all baselines on both SketchEval and NL2Repo-Bench under both PE and SFT settings. On SketchEval, CodeTeam achieves 51.7% (PE) and 60.9% (SFT) overall SketchBLEU, with improvements concentrated in the structural and dataflow sub-scores and most pronounced on medium and hard repositories. On NL2RepoBench, CodeTeam reaches 34.6% (PE) and 42.3% (SFT) pass rates, confirming that sketch-level improvements translate into executable correctness. The method rankings are consistent across both benchmarks, with all improvements statistically significant (𝑝 < 0.05).

4.3

RQ2: Effect of Retrieval-Augmented Grounding on Architectural Planning

Table 7 reports the ablation results for both RQ2 and RQ3 under the PE setting. We restrict these component-level ablations to PE because their goal is to isolate the effects of workflow-level mechanisms under a fixed model-adaptation condition; the SFT setting mainly changes the backbone model’s format-following and generation fidelity, which is complementary to the coordination mechanisms being ablated. The final column indicates the workflow aspect most directly affected by each ablation, rather than implying a strict one-to-one mapping to a pipeline stage. Removing the RAG subsystem from the architect stage degrades the overall SketchBLEU from 51.7% to 47.5%, ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:24

Wang et al.

Table 7. Ablation results of CodeTeam (PE) on SketchEval measured by SketchBLEU (%). Each variant removes one component from the full system. “Rel. drop” is computed as (Full − Variant)⇑Full. Values are mean±std over three seeds. Variant

Easy (5) Medium (8) Hard (6) All (19) Rel. drop Primary affected workflow aspect

Full CodeTeam (PE) CodeTeam w/o RAG CodeTeam w/o Dynamic Alloc CodeTeam w/o Git

56.4±2.0 52.8±2.5 52.5±2.7 55.0±2.0

56.0±2.5 52.1±2.9 50.6±3.1 54.8±2.5

41.9±3.3 36.9±4.0 36.5±4.1 40.2±3.2

51.7±1.8 47.5±1.7 46.6±2.0 50.2±1.4

– 8.1% 9.9% 2.9%

– Planning Implementation Repair coordination / Crossfile consistency

corresponding to an 8.1% relative decrease. The degradation increases with task difficulty: removing RAG reduces SketchBLEU by 3.6 points on easy tasks (56.4→52.8), 3.9 points on medium tasks (56.0→52.1), and 5.0 points on hard tasks (41.9→36.9). Even on easy repositories, the drop is nontrivial, indicating that retrieval-augmented grounding contributes to planning quality regardless of repository scale. On medium and hard repositories, the impact is larger because the requirements space is substantially broader, and architects without retrieval are more likely to omit secondary modules (e.g., utility packages, configuration loaders, or test infrastructure) that are not explicitly mentioned in the requirements but are necessary for a complete and coherent repository. RAG mitigates this gap by surfacing reusable file-decomposition precedents from real-world repositories, which can remind architects of module boundaries that are easy to omit when requirements are underspecified. For example, for a command-line data-processing task, retrieved design summaries may suggest separating the command layer, core transformation logic, input/output adapters, configuration loading, validation, and tests, instead of collapsing the SDS into only a command entry point and one processing file. Such retrieval does not have to be purely abstract to be useful: if a retrieved project has a related architectural shape, its file organization can still serve as a valid design precedent, provided that the benchmark repositories are not included in the retrieval corpus and the retrieved artifacts are limited to design-level summaries rather than raw code. Moreover, the RAG corpus is extensible rather than fixed; when new architectural patterns are added to the corpus, the same retrieval-grounded planning mechanism can expose them to the architects. Therefore, the RAG ablation should be interpreted as measuring the practical value of retrieval-grounded design references for SDS generation, rather than as an isolated test of whether the model can abstract and transfer high-level file-decomposition patterns across tasks. To trace this end-to-end impact back to planning quality, Table 8 summarizes three planningstage diagnostics collected during SDS generation. With RAG enabled, the SDS parse success rate - indicating whether the generated SDS can be loaded as valid JSON without re-generation increases from 0.86 to 0.94, and structural validity - the proportion of candidates free from missing file references or undeclared interfaces - rises from 0.78 to 0.90. Furthermore, plan diversity across the 𝑁 =4 Architect agents, measured by average pairwise Jaccard distance of file-path and APIname sets, increases substantially from 0.25 to 0.42. These diagnostics support the intended role of RAG within the CodeTeam workflow: retrieval does not replace independent planning, but rather provides grounded design hints that help architects avoid missing modules and reduce the tendency for multiple agents to converge on nearly identical SDS candidates. Figure 4 visualizes these planning-stage signals. Although the absolute values depend on the specific implementation of the SDS validation checker, the relative gaps between the two variants are stable across seeds. Overall, RAG primarily improves what to build - that is, file decomposition and interface sketching at the planning stage - rather than how to implement individual files at ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:25

Table 8. Planning-stage diagnostics for RQ2 (PE setting). “Diversity” is the average pairwise Jaccard distance between file-path sets and API-name sets across Architect proposals (higher means more diverse). Variant CodeTeam (with RAG) CodeTeam w/o RAG

1

0.94 0.86

SDS parse success

Structural validity

Plan diversity

0.94 0.86

0.90 0.78

0.42 0.25

0.9

Score

0.78

CodeTeam CodeTeam w/o RAG

0.5

0.42 0.25

0 Parse

Valid

Diversity

Fig. 4. RAG improves planning-stage quality signals (PE setting).

the development stage. This interpretation is consistent with the observation that RAG yields larger gains in the structural component of SketchBLEU (Table 3), which specifically measures architectural alignment. RQ2 Summary: Removing RAG from the architect stage reduces overall SketchBLEU by 4.2 points (8.1% relative), with larger degradation on harder repositories. RAG improves planningstage diagnostics: SDS parse success rate rises from 0.86 to 0.94, structural validity from 0.78 to 0.90, and plan diversity from 0.25 to 0.42. RAG primarily enhances what to build (file decomposition and interface sketching) rather than how to implement individual files.

4.4

RQ3: Effect of Dynamic Developer Allocation and Git-Based Coordination

Table 7 shows that dynamic developer allocation is the most impactful individual mechanism in our ablation study. Replacing the Architect-guided developer count and file ownership with a fixed four-developer round-robin scheme reduces the overall SketchBLEU from 51.7% to 46.6%, corresponding to a 9.9% relative decrease. The degradation is largest on hard repositories, indicating that project-specific team sizing and ownership planning become especially critical when repositories contain many tightly coupled modules. By contrast, removing Git-based inter-agent coordination yields a comparatively smaller SketchBLEU decrease (2.9% relative). This result is consistent with our qualitative observations during development: structured commit messages primarily benefit scenarios in which interfaces evolve during the repair loop, but not all repositories trigger frequent interface-level revisions. Accordingly, we characterize Git-based coordination as a complementary sub-module that improves cross-file interface stability in scenarios where APIs evolve during the repair loop rather than a dominant contributor to overall SketchBLEU performance. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:26

Wang et al.

Table 9. Efficiency and consistency diagnostics for RQ3 (PE setting). “Mismatch failures” aggregates QAreported errors that indicate cross-file inconsistency (e.g., missing symbols, wrong signatures, broken imports). Variant Full CodeTeam (PE) CodeTeam w/o Dynamic Allocation CodeTeam w/o Git Coordination

QA rounds ↓

Mismatch failures ↓

Avg. context (k tokens) ↓

3.6 5.2 3.9

7.8 12.4 9.1

3.1 4.4 3.2

To provide further insight beyond the aggregate SketchBLEU scores, Table 9 reports diagnostic indicators related to convergence efficiency and cross-file consistency. Dynamic developer allocation reduces the number of QA rounds needed to reach a stable state (from 5.2 to 3.6 on average) and substantially lowers the count of interface-mismatch failures (from 12.4 to 7.8). It also reduces the average context size per LLM call from 4.4k to 3.1k tokens, because repository-specific ownership enables the system to construct more focused, file-level context packages while avoiding unnecessary cross-owner rework. Removing Git-based coordination slightly increases mismatch failures (from 7.8 to 9.1), but the observed increase is smaller than the one associated with removing dynamic developer allocation. The substantially larger effect of dynamic developer allocation can be explained by the different positions of the two mechanisms in the generation workflow. Dynamic allocation is an upstream and preventive mechanism: it determines the number of developers, file ownership, repair routing, and context construction before implementation begins. When this mechanism is replaced by a fixed round-robin assignment, tightly coupled files may be split across unrelated agents, while each agent receives broader and less focused context. This causes interface assumptions to diverge early and then propagate through later implementation and QA rounds. By contrast, Git-based coordination is a downstream and reactive mechanism. It helps agents communicate interface changes after files have already been implemented or repaired, but it cannot correct an unsuitable initial division of labor or reduce long-context pressure caused by poor ownership. Therefore, dynamic allocation prevents many cross-file inconsistencies from arising in the first place, whereas Git-based coordination mainly mitigates the smaller subset of inconsistencies caused by interface drift during the repair loop. Figure 5 visualizes these diagnostic indicators. Taken together, these diagnostics suggest a clear practical explanation for the observed SketchBLEU improvements: dynamic developer allocation reduces the volume of late-stage rework by aligning team size and file ownership with the repository’s actual modular structure, while Git-based coordination primarily mitigates a subset of interface-drift cases that arise when implementations diverge from the originally declared APIs. To further characterize the repair dynamics, we examine how SketchBLEU evolves across successive QA iterations in the full CodeTeam (PE) system. After the initial implementation pass (Iteration 0), the average SketchBLEU across all 19 tasks is 46.3%. The first QA round yields the largest single-round improvement, raising the score to 49.8% (+3.5 points), as the most obvious structural defects - such as missing __init__.py files, broken import paths, and undeclared symbols - are detected and repaired. The second round provides an additional 1.4-point improvement (to 51.2%), primarily addressing secondary inconsistencies that emerge after the first round of fixes, such as signature mismatches exposed when a previously missing module is restored. Subsequent rounds exhibit diminishing returns: rounds 3 and 4 together contribute only 0.8 points on average, and the system typically converges (no new QA issues) between rounds 3 and 4 for easy and medium tasks. Hard repositories occasionally require the full budget of 5–6 rounds, consistent with the higher QA-round count reported in Table 9 for the ablated variant. Note that because different tasks ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

15 12.4

10 Value

9.1

0:27

Full CodeTeam w/o Dynamic Allocation w/o Git Coordination

7.8

5

0

5.2 3.6

4.4

3.9

3.1

ds

atch

Roun

Mism

3.2

ext Cont

Fig. 5. Dynamic developer allocation improves convergence and reduces cross-file mismatch indicators (PE setting).

converge and terminate at different rounds, the per-round averages reported above are computed only over the tasks that remain active at each iteration; consequently, summing the per-round deltas does not exactly recover the final aggregate SketchBLEU in Table 2, which averages each task’s terminal score regardless of the round at which it converged. This convergence pattern has two practical implications. First, the majority of the cumulative SketchBLEU improvement contributed by qa-driven repair is captured within the first two QA rounds, suggesting that a resource-constrained deployment could limit QA iterations to 2–3 rounds with only modest degradation in repository quality. Second, the decreasing marginal improvement across successive QA rounds indicates that the qa agent’s lightweight generated tests are effective at catching structural and interface-level defects but have limited reach for deeper semantic bugs, which would require stronger execution-based feedback to detect and resolve. RQ3 Summary: Dynamic developer allocation has a larger effect than Git-based coordination because it acts earlier and more globally in the development workflow. By aligning developer count, file ownership, repair routing, and context construction with the repository’s modular structure, it prevents many cross-file inconsistencies before implementation begins. When removed, QA rounds increase from 3.6 to 5.2, interface-mismatch failures from 7.8 to 12.4, and average context size from 3.1k to 4.4k tokens per developer call, leading to a 5.1-point SketchBLEU drop (9.9% relative). Git-based coordination is still useful, but its role is narrower and more reactive: it mainly stabilizes interface changes that arise during the qa-driven repair loop, producing a smaller 2.9% relative drop when removed. Thus, dynamic developer allocation improves repository generation primarily by preventing coordination failures, while Git-based coordination mitigates residual interface drift.

5

Discussion

This section interprets the experimental findings, examines the mechanisms behind the observed improvements, and discusses the broader implications for researchers and practitioners. 5.1

Interpretation of the Results

We first identify the key reasons behind CodeTeam’s performance improvements and then discuss the role each evaluation benchmark plays in validating these results. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:28

Wang et al.

5.1.1 Reasons behind CodeTeam’s Performance Improvements. Multi-agent coordination and qa-driven repair complement sketch-based decomposition. A central observation from Table 2 is that CodeTeam improves over CodeS under both PE and SFT settings, with a larger margin in the PE setting. This finding suggests that a structured multi-agent workflow can compensate for crossfile consistency limitations that emerge when directly prompting a backbone model to produce a complete repository. The finding is consistent with the original motivation of CodeS, where multi-layer sketch decomposition reduces the difficulty of generating long structured outputs [42]. However, our results further indicate that sketch decomposition alone is not sufficient when code generation spans multiple interacting files; it needs to be coupled with inter-agent coordination and iterative qa-driven repair mechanisms that actively maintain cross-file consistency during implementation. Improved structural and dataflow quality through the SDS contract. The component-level decomposition of SketchBLEU in Table 3 shows that CodeTeam’s improvements over CodeS are concentrated in the repository-level components: in the PE setting, structural matching increases from 56.8 to 60.9 (+4.1 points) and dataflow matching increases from 46.5 to 53.5 (+7.0 points), compared with smaller improvements on BLEU (+2.8) and weighted BLEU (+2.6); in the SFT setting, the corresponding structural and dataflow improvements remain +3.2 and +4.4 points, again larger than the n-gram-based improvements (+2.5 and +2.4). These structural and dataflow components are specifically designed to capture repository-level integrity beyond surface-level token overlap [42]. The software design sketch (SDS) functions as an explicit, machine-checkable contract that binds file boundaries, public APIs, and inter-file dependencies into a coherent specification. Once such a contract is established, Developer agents can implement files with smaller, task-specific contexts, and the developer-to-file ownership mapping enables precise routing of repairs and re-queuing of dependent files when interfaces drift during iterative development. This observation also resonates with findings from repository-level completion benchmarks, where downstream performance depends heavily on retrieving and using appropriate cross-file context rather than generating code in isolation [23, 43]. Improvements concentrate on complex repositories. The improvements concentrate on medium and hard repositories (see Table 2), which typically involve more modules and tighter inter-module coupling. When cross-file dependencies are weak, as in easy repositories, multiple approaches can achieve comparable structural correctness. As dependency complexity increases, systems that fail to control interface consistency tend to degrade more sharply. A similar phenomenon was reported by CodeS, where prompt-based methods exhibit a stronger performance decline on harder repositories, while stronger decomposition and fine-tuning mitigate that decline [42]. In the case of CodeTeam, the dependency-aware scheduling and qa-driven repair mechanisms specifically target these complex scenarios, which explains why the relative SketchBLEU improvement becomes more visible as task difficulty increases. Alignment of multi-agent coordination with established software engineering practices. The effectiveness of these mechanisms can be understood through an analogy with human development teams. A repository is not built by a single developer writing all files in sequence; the process typically involves (1) an architecture review in which senior engineers agree on module boundaries, public interfaces, and technology choices; (2) a task assignment phase in which files or features are distributed to developers based on expertise and module ownership; and (3) iterative code review cycles in which reviewers verify interface contracts and cross-module consistency before changes are merged. CodeTeam instantiates this workflow at the agent level: the Architect competition and ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:29

CTO selection correspond to architecture review, the SDS-based developer allocation corresponds to task assignment with explicit ownership, and the qa-driven repair loop corresponds to code review. These coordination structures reduce the context burden on any single agent while preserving global coherence across the repository, which explains why the benefit is larger on complex repositories where cross-file dependencies are denser. More realistically scaled repositories from the SDS. Figure 3 shows that CodeTeam produces repository sizes closer to those of the reference repositories. This observation is consistent with an empirical pattern reported in CodeS, where agent-based baselines tend to generate only a small number of files or produce limited LOC, while sketch-based pipelines can sustain larger repository structures [42]. In CodeTeam, the SDS combined with the pre-construction of the full file tree reduces the likelihood that a generation run terminates with an underspecified or incomplete repository. Although size alignment should not be interpreted as evidence of correctness by itself, it serves as a basic validity indicator: generating a repository at a plausible scale is often a necessary precondition for capturing the required functionality.

5.1.2 Roles of Evaluation Benchmarks. Role of SketchEval. SketchEval rewards structural alignment with a reference repository sketch and captures architectural and dataflow quality. The SketchBLEU metric, motivated by CodeBLEU but extended to repository-level sketches with dataflow matching [30, 42], provides a scalable proxy for structural and semantic alignment when running every generated repository is computationally prohibitive. Our results show that CodeTeam’s improvements are concentrated in the structure- and semantics-oriented sub-scores (Table 3), reinforcing the value of evaluation metrics that go beyond purely token-based overlap. Role of NL2Repo-Bench. The NL2Repo-Bench experiment shifts the evaluation objective from repository resemblance to executable correctness: whether the generated repository can pass the original upstream pytest suite across 104 tasks [8, 42]. The benchmark is challenging: even state-of-the-art LLM-based coding agents achieve only 14~40% pass rates [8]. Crucially, the relative ranking of methods on NL2Repo-Bench mirrors the ranking on SketchEval, providing converging evidence that the improvements from structured planning and coordination are not artifacts of sketch-level evaluation but reflect improvements in repository construction quality. This convergence is consistent with the design intent of NL2Repo-Bench, which aims to expose long-horizon failure modes such as premature termination, loss of global coherence, and fragile cross-file dependencies [8] - precisely the failure modes that CodeTeam’s SDS-based planning, dependency-aware scheduling, and qa-driven repair are designed to address. Complementarity of the two benchmarks. Taken together, SketchEval and NL2Repo-Bench suggest that a robust repository-level evaluation strategy should combine proxy metrics with execution-based tests. SketchBLEU is useful during rapid system iteration because it is computationally inexpensive and straightforward to decompose into interpretable sub-scores, whereas NL2Repo-Bench provides a stricter assessment of functional completeness at larger scale [8, 42]. We view these two benchmarks as complementary rather than competing evaluation modalities. 5.2

Implications

5.2.1 Implications for researchers. Multi-agent workflow design remains valuable even with stronger backbone models. Although the SFT setting yields higher absolute performance than PE, ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:30

Wang et al.

CodeTeam still improves over CodeS by a noticeable margin in both settings (see Table 2). Finetuning and multi-agent workflow design are complementary rather than substitutive: fine-tuning helps the backbone model follow structured output formats and cope with longer input contexts, while the multi-agent workflow reduces the burden on any single generation call to maintain crossfile consistency, such as stable APIs, import paths, file ownership, and dependency assumptions across repository components [5, 42]. Future work should investigate whether the relative rankings and ablation patterns hold across backbone families (e.g., CodeLlama, DeepSeek-Coder, StarCoder2) and model parameter sizes (e.g., 7B, 14B, 32B, and 70B models) to strengthen the generalizability of these findings. Bridging NL2Repo generation with execution-based repository-level repair. The QA loop in CodeTeam is intentionally lightweight, yet the efficiency diagnostics (see Table 9) indicate that even targeted repository-level repair meaningfully reduces late-stage implementation rework. This finding aligns with the broader trend where multi-step workflows with intermediate feedback consistently outperform single-pass prompting [31]. Bridging NL2Repo generation with repositoryrepair benchmarks such as SWE-bench [19, 40] represents a promising direction: a system could first generate a repository from natural-language requirements and then apply repair-oriented agents to improve functional correctness under execution-based feedback. Incorporating richer static analysis feedback. The current RAG corpus provides design-level hints about file decomposition and dependency structure but does not explicitly encode architectural quality attributes such as modularity and testability. Incorporating architecture-specific retrieval criteria, for instance, filtering candidates by coupling metrics or documentation completeness, may further stabilize SDS generation quality. Similarly, static analysis tools [20], call-graph analysis [14], type-inference engines, and import-resolution tools could be integrated into the QA stage to detect interface mismatches earlier and more precisely than the current lightweight test-generation approach. Multi-language and large-scale repository support. All experiments are conducted on Python repositories. Extending CodeTeam to multi-language projects would require language-aware dependency analysis, heterogeneous build-system support, and cross-language interface specifications. At larger scales, the SDS can grow to thousands of tokens for repositories exceeding 40~50 files; a hierarchical SDS that first specifies package-level contracts and then refines file-level details within each package could mitigate this issue. Parallelizing QA across independent subgraphs of the dependency graph is a natural extension that could reduce the wall-clock time of the QA repair loop by allowing independent file groups to be checked and repaired concurrently. 5.2.2 Implications for Practitioners. Investing more tokens in planning reduces downstream implementation rework. A well-established principle in software engineering is that defects detected during design are substantially cheaper to fix than those discovered during integration or testing [1]. The multi-agent design of CodeTeam applies this principle to LLM-based code generation: it shifts computational effort toward up-front architectural planning (SDS generation and CTO evaluation) so that cross-file inconsistencies are resolved before individual files are implemented. The RQ3 efficiency diagnostics in Table 9, including QA rounds, interface-mismatch failures, and average context size per LLM call, provide guidance on when and how practitioners should deploy CodeTeam. The multi-agent workflow introduces additional cost, but this cost is most valuable for repository-level tasks where architectural planning, cross-file consistency, and iterative repair directly affect final code quality. Therefore, practitioners should not necessarily ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:31

use the full workflow for every programming request. Instead, they can reserve CodeTeam for complex or high-risk tasks, and use dynamic developer allocation and bounded QA rounds to keep the token budget manageable. These RQ3 diagnostics suggests a practical deployment strategy: spend additional tokens when coordination and validation are likely to prevent repository-level failures, but use a simpler single-agent workflow for small, localized edits. For deployment scenarios where token cost is a constraint, a two-stage budget allocation - investing more tokens in up-front planning and fewer in iterative repair - appears to be a favorable strategy. A small number of QA iterations suffice to capture the majority of repository-quality improvements. Iterative feedback loops are widely used in LLM-based code generation to progressively improve output quality [4, 32]. A common finding across these systems is that the improvement contributed by each additional iteration decreases rapidly: early rounds fix the most critical errors, while later rounds yield only incremental refinements. CodeTeam exhibits the same pattern. The convergence analysis (see Section 4.4) shows that the majority of the cumulative SketchBLEU improvement contributed by qa-driven repair is captured within the first two QA rounds. A resource-constrained deployment of CodeTeam could therefore limit the number of QA iterations to 2~3 rounds with only modest degradation in repository quality, making CodeTeam practical for cost-sensitive settings. This observation also suggests a practical deployment strategy: teams can start with a small number of QA rounds and increase it only for repositories whose complexity warrants additional QA iterations. Explicit interface contracts and file ownership enable incremental adoption of CodeTeam. Three design choices made in CodeTeam facilitate its incremental adoption in practice. First, the SDS serves as an explicit interface contract: each file’s public API (function signatures, input/output types, and cross-file call targets) is declared before implementation begins, so that downstream Developer agents can code against stable interfaces rather than evolving assumptions. Second, file-level ownership assigns every source file to exactly one Developer agent, eliminating concurrent edits to the same file and simplifying conflict resolution. Third, dependency-aware task scheduling (see Algorithm 1) topologically orders file-generation tasks according to their import dependencies, ensuring that a file’s dependencies are generated before the file itself. Together, these three design choices make it straightforward to parallelize file generation across independent subgraphs of the dependency graph, thereby scaling CodeTeam to larger repositories without sacrificing cross-file consistency. Explicit API interface specification and role-based partitioning of file ownership are also related to coordination mechanisms explored in multi-agent code-generation frameworks such as ChatDev [28] and MetaGPT [16]; however, CodeTeam differs by grounding these mechanisms in an SDS-centered, repository-level generation and QA workflow. Practitioners can adopt individual components of CodeTeam (e.g., SDS-based planning without the full QA loop) incrementally, scaling the multi-agent workflow to match the size of the target repository. 6

Threats to Validity

We discuss the main threats to the validity of our findings and the mitigation steps we adopted, following standard guidance for empirical software engineering studies [38]. 6.1 Construct Validity On SketchEval, the primary outcome measure is SketchBLEU [42], an automatic proxy for repository quality. SketchBLEU extends CodeBLEU [30] and therefore inherits a well-known limitation of n-gram-based metrics [27]: a higher score does not, by itself, guaranty that the generated repository is executable, secure, or maintainable. To mitigate this threat, we report the ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:32

Wang et al.

component-level decomposition (see Table 3), which emphasizes structure and dataflow matching [42], and include complementary indicators such as repository size errors (see Figure 3) and QA-reported mismatch failures (see Table 9). We additionally conduct an execution-based evaluation on NL2Repo-Bench [8]; the consistent method ranking across both benchmarks strengthens the evidence that CodeTeam produces more structurally coherent and functionally correct repositories. A second construct threat concerns the operationalization of the mechanisms utilized in CodeTeam. For instance, “dynamic developer allocation” is realized through a combination of Architect-guided team sizing, file ownership mapping, dependency-aware execution (Algorithm 1), and compact context packages. Part of the measured SketchBLEU improvement may stem from specific implementation-level design choices rather than solely from the high-level mechanism. Similarly, “RAG for design” depends on a particular retrieval corpus, embedding model, and top-𝑘 setting. To make these choices transparent, we describe the concrete implementation of each component in Section 3.2, and Table 7 isolates each mechanism through controlled one-factor removals. Still, one-factor-at-a-time ablations cannot fully capture all possible synergies or interference effects [38]. 6.2

Internal Validity

Internal validity concerns whether the observed performance differences can be attributed to the workflow mechanisms under study rather than to confounding factors. Although all methods share the same backbone model and decoding configuration, the prompt templates differ across systems. To mitigate this source of confounding, the ablation studies for both RQ2 and RQ3 compare variants of CodeTeam that differ only in the presence or absence of a single mechanism while keeping all prompts identical. Additionally, CodeTeam involves several hyperparameters (𝑁 =4, top-𝑘=5, QA iteration limit, CTO scoring rubric) selected based on preliminary experiments rather than exhaustive search. We report all chosen values transparently (see Section 3.2) and conducted limited sensitivity checks (𝑁 ∈ {2, 4, 6}, 𝑘 ∈ {3, 5, 7}) that showed consistent trends. 6.3

External Validity

Our evaluation covers SketchEval (19 Python repositories) [42] and NL2Repo-Bench (104 Python tasks) [8]. Despite this breadth, the study does not cover larger codebases, mixed-language systems, or projects with heavier build or test pipelines. We therefore position our findings as evidence within a clearly defined benchmark setting rather than as universal claims. A related threat is data contamination: benchmark repositories may appear in the backbone model’s training corpora. We follow the official CodeS evaluation protocol [42], and retrieved RAG artifacts are provided only as high-level design hints with explicit prohibitions on verbatim copying, but benchmark-specific bias cannot be fully excluded. 6.4

Reliability

LLM-based systems are inherently stochastic, and multi-agent workflows can amplify variability. To improve reliability, we evaluate each method with three random seeds and report mean±standard deviation. All core comparisons reach statistical significance at the 0.05 level via the Wilcoxon signed-rank test [37] and bootstrap confidence intervals (see Section 4.1). The method ranking is further confirmed on NL2Repo-Bench (𝑝 < 0.001 for both PE and SFT comparisons), where the larger sample size (104 tasks) provides substantially higher statistical power. To improve experimental traceability, our system records the SDS, agent messages, Git commits, and QA reports for each run, which can be regenerated by executing the released implementation. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

7

0:33

Conclusions

This paper presents CodeTeam, an LLM-based multi-agent framework for NL2Repo that separates planning, decision making, and implementation into distinct, coordinated stages. CodeTeam combines (1) an architecture competition stage in which multiple Architect agents propose competing software design sketches (SDSs), optionally grounded by retrieval-augmented design references, (2) a cto agent that selects and normalizes the winning SDS into a machine-checkable contract, and (3) a dependency-aware development stage with dynamic developer allocation, Gitbased coordination, and qa-driven iterative repair. On SketchEval, CodeTeam improves the overall SketchBLEU from 47.6 (CodeS) to 51.7 in the prompt-engineering (PE) setting and from 58.0 to 60.9 in the supervised fine-tuning (SFT) setting, with improvements concentrated in structural and dataflow sub-scores. Ablation results confirm that both RAG-assisted planning and dynamic developer allocation contribute substantially to the observed SketchBLEU improvements. On NL2Repo-Bench, CodeTeam achieves the highest pass rate in both settings (34.6% PE, 42.3% SFT), confirming that sketch-level improvements translate into executable correctness. Overall, the main implication of CodeTeam is that NL2Repo should be treated not merely as a long-context code generation task, but as a coordinated software engineering process. We proposed CodeTeam because from-scratch repository generation exposes failure modes that local code synthesis alone cannot reliably address: the system must first derive a coherent architecture from underspecified requirements, then preserve interface contracts, dependency constraints, and implementation responsibilities across many file-level generation steps. By making these contracts explicit and coupling them with role specialization, dependency-aware scheduling, and targeted QA feedback, CodeTeam provides a practical workflow pattern for using LLM agents as a coordinated development team rather than isolated code generators. Its contribution therefore lies not only in improving benchmark performance, but also in demonstrating how LLM-based repository generation can be made more structured, inspectable, and extensible. As future work, we plan to integrate execution-based feedback into the QA loop to bridge the gap between structural evaluation and functional correctness, incorporate stronger static analysis feedback (e.g., call-graph analysis, type inference) into the scheduling and repair stages, and extend the framework to multi-language and larger-scale repositories through hierarchical SDS planning and cross-language interface contracts. We discuss these directions in detail in Section 5.2. Data Availability The replication package of this study has been made available at [35]. Acknowledgments This work has been partially supported by the National Natural Science Foundation of China (NSFC) with Grant Nos. 92582203 and 62402348. References [1] Barry Boehm and Victor R. Basili. 2001. Software Defect Reduction Top 10 List. IEEE Computer 34, 1 (2001), 135–137. [2] Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. BGE M3-Embedding: MultiLingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv preprint arXiv:2402.03216 (2024). [3] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374 (2021). [4] Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. 2024. Teaching Large Language Models to Self-Debug. In Proceedings of the 12th International Conference on Learning Representations (ICLR). OpenReview.net, 1–23. ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:34

Wang et al.

[5] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. 2024. LongLoRA: Efficient Fine-tuning of Long-Context Large Language Models. In Proceedings of the 12th International Conference on Learning Representations (ICLR). OpenReview.net, 1–19. [6] Tri Dao. 2023. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning. arXiv preprint arXiv:2307.08691 (2023). [7] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized LLMs. In Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS Proceedings, 10088–10115. [8] Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Qizhi Chen, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Weihao Xie, Zihan Wang, Yishuo Yuan, Jiayu Zhang, Enduo Zhao, Yunfei Zhao, He Zhu, Liya Zhu, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyang Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, Wenhao Huang, and Ge Zhang. 2025. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. arXiv preprint arXiv:2512.12730 (2025). [9] Yangruibo Ding, Zijian Wang, Wasi Ahmad, Hantian Ding, Ming Tan, Nihal Jain, Murali Krishna Ramanathan, Ramesh Nallapati, Parminder Bhatia, Dan Roth, and Bing Xiang. 2023. CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion. In Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS Proceedings, 46701–46723. [10] Matthijs Douze, Anton Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss Library. arXiv preprint arXiv:2401.08281 (2024). [11] Bradley Efron. 1979. Bootstrap Methods: Another Look at the Jackknife. Annals of Statistics 7, 1 (1979), 1–26. [12] Daniel Fried, Armen Aghajanyan, Jessy Lin, Sida Wang, Eric Wallace, Freda Shi, Ruiqi Zhong, Wen-tau Yih, Luke Zettlemoyer, and Mike Lewis. 2022. InCoder: A Generative Model for Code Infilling and Synthesis. arXiv preprint arXiv:2204.05999 (2022). [13] Significant Gravitas. 2023. AutoGPT. https://github.com/Significant-Gravitas/AutoGPT. [14] Mary W. Hall and Ken Kennedy. 1992. Efficient Call Graph Analysis. ACM Letters on Programming Languages and Systems 1, 3 (1992), 227–242. [15] Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. arXiv preprint arXiv:2105.09938 (2021). [16] Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In Proceedings of the 12th International Conference on Learning Representations (ICLR). OpenReview.net, 1–29. [17] Md. Ashraful Islam, Mohammed Eunus Ali, and Md Rizwan Parvez. 2024. MapCoder: Multi-Agent Code Generation for Competitive Problem Solving. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 4917–4942. [18] Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, and Luke Zettlemoyer. 2018. Mapping Language to Code in Programmatic Context. In Proceedings of the 23rd Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 1643–1652. [19] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWEbench: Can Language Models Resolve Real-World GitHub Issues?. In Proceedings of the 12th International Conference on Learning Representations (ICLR). OpenReview.net, 1–51. [20] Syed Mohammad Kashif, Ruiyin Li, Peng Liang, Amjed Tahir, Qiong Feng, Zengyang Li, and Mojtaba Shahin. 2026. Beyond Functional Correctness: Design Issues in AI IDE-Generated Large-Scale Projects. arXiv preprint arXiv:2604.06373 (2026). [21] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv preprint arXiv:2005.11401 (2020). [22] Wei Li, Xin Zhang, Zhongxin Guo, Shaoguang Mao, Wen Luo, Guangyue Peng, Yangyu Huang, Houfeng Wang, and Scarlett Li. 2025. FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 17160–17176. [23] Tianyang Liu, Canwen Xu, and Julian McAuley. 2024. RepoBench: Benchmarking Repository-Level Code AutoCompletion Systems. In Proceedings of the 12th International Conference on Learning Representations (ICLR). OpenReview.net, 1–19.

ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation

0:35

[24] Wei Liu, Ailun Yu, Daoguang Zan, Bo Shen, Wei Zhang, Haiyan Zhao, Zhi Jin, and Qianxiang Wang. 2024. GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Model. arXiv preprint arXiv:2406.07003 (2024). [25] Yu. A. Malkov and D. A. Yashunin. 2016. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. arXiv preprint arXiv:1603.09320 (2016). [26] Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. 2022. CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis. arXiv preprint arXiv:2203.13474 (2022). [27] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 311–318. [28] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). ACL, 15174–15186. [29] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv preprint arXiv:1910.02054 (2020). [30] Shuo Ren, Daya Guo, Shuai Lu, Long Zhou, Shujie Liu, Duyu Tang, Neel Sundaresan, Ming Zhou, Ambrosio Blanco, and Shuai Ma. 2020. CodeBLEU: a Method for Automatic Evaluation of Code Synthesis. arXiv preprint arXiv:2009.10297 (2020). [31] Tal Ridnik, Dedy Kredo, and Itamar Friedman. 2024. Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering. arXiv preprint arXiv:2401.08500 (2024). [32] Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. 2023. Reflexion: Language Agents with Verbal Reinforcement Learning. In Proceedings of the 37th Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS Proceedings, 8634–8652. [33] Qwen Team, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. 2024. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115 (2024). [34] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In Proceedings of the 13th International Conference on Learning Representations (ICLR). OpenReview.net, 1–38. [35] Yifei Wang, Ruiyin Li, Peng Liang, Qiong Feng, Zengyang Li, Mojtaba Shahin, and Arif Ali Khan. 2026. Replication Package for the Paper: CodeTeam: An LLM-Powered Multi-Agent Framework for Repository-Level Code Generation. https://github.com/WhitenWhiten/CodeTeam. [36] Yue Wang, Weishi Wang, Shafiq Joty, and Steven C. H. Hoi. 2021. CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation. In Proceedings of the 26th Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 8696–8708. [37] Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods. Biometrics Bulletin 1, 6 (1945), 80–83. [38] Claes Wohlin, Per Runeson, Martin Höst, Magnus C. Ohlsson, Björn Regnell, and Anders Wesslén. 2012. Experimentation in Software Engineering. Springer. [39] Hui Yang, Sifu Yue, and Yunzhong He. 2023. Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions. arXiv preprint arXiv:2306.02224 (2023). [40] John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Proceedings of the 38th Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS Proceedings, 50528–50652. [41] Pengcheng Yin, Bowen Deng, Edgar Chen, Bogdan Vasilescu, and Graham Neubig. 2018. Learning to Mine Aligned Code and Natural Language Pairs from Stack Overflow. In Proceedings of the 15th International Conference on Mining Software Repositories (MSR). ACM, 476–486. [42] Daoguang Zan, Ailun Yu, Wei Liu, Dong Chen, Bo Shen, Yafen Yao, Wei Li, Xiaolin Chen, Yongshun Gong, Bei Guan, Zhiguang Yang, Yongji Wang, Lizhen Cui, and Qianxiang Wang. 2025. CodeS: Natural Language to Code Repository via Multi-Layer Sketch. ACM Transactions on Software Engineering and Methodology (2025). [43] Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generation. In Proceedings of the 28th Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 2471–2484. [44] Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. 2024. CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges. In Proceedings of the 62nd Annual Meeting of the Association

ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

0:36

Wang et al.

for Computational Linguistics (ACL). ACL, 13643–13658. [45] Qianhui Zhao, Li Zhang, Fang Liu, Junhang Cheng, Chengru Wu, Junchen Ai, Qiaoyuanhe Meng, Lichen Zhang, Xiaoli Lian, Shubin Song, and Yuanping Guo. 2025. Towards Realistic Project-Level Code Generation via Multi-Agent Collaboration and Semantic Architecture Modeling. arXiv preprint arXiv:2511.03404 (2025). [46] Shuyan Zhou, Uri Alon, Frank F. Xu, Zhiruo Wang, Zhengbao Jiang, and Graham Neubig. 2023. DocPrompting: Generating Code by Retrieving the Docs. In Proceedings of the 11th International Conference on Learning Representations (ICLR). OpenReview.net, 1–16.

ACM Trans. Softw. Eng. Methodol., Vol. 0, No. 0, Article 0. Publication date: 2026.

Related documents

• Experimental Settings in LLM-Based Program Repair: A Study of Inputs, Tool Access, Feedback, and Validation#965480
• A11yLTLNav: Automatic Detection of Accessibility Navigation Failures#965481
• Detecting Argument-Swap Bugs Using Context-Enhanced Code Representations#965482
• A Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI Agents#965483
• CARE: A Responsibility-Oriented Architecture for Domain-Scoped Resolution in Modular and Upgradeable Smart Contracts#965484