Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios Ruida Hu1 , Xinchen Wang1 , Chao Peng2 , Cuiyun Gao1† , David Lo3 1 Harbin Institute of Technology, Shenzhen, China 2 Independent Researcher, China
3 Singapore Management University, Singapore
arXiv:2604.06742v1 [cs.SE] 8 Apr 2026
[email protected],[email protected],[email protected],[email protected],[email protected]
Abstract
Keywords
The evolution of Large Language Models (LLMs) has catalyzed a paradigm shift towards intent-driven software development, where autonomous agents are expected to architect and deliver complete, runnable software systems entirely from scratch. However, existing benchmarks fail to adequately assess this promising 0-to-1 software generation capability due to two fundamental limitations. First, they predominantly rely on predefined structural scaffolds, regarding the task as mere file-filling and failing to evaluate the LLM’s crucial ability in repository structure planning. Second, they heavily depend on rigid white-box unit testing, which forces agents to conform to specific internal implementations and lacks scalable, end-to-end behavioral validation from a user-centric perspective. To bridge this gap, we introduce CLI-Tool-Bench, a novel, structureagnostic benchmark designed to evaluate the ground-up generation of Command-Line Interface (CLI) tools. Powered by a black-box differential testing framework, CLI-Tool-Bench comprises 100 highquality, real-world repositories spanning diverse difficulty levels, programming languages, and application domains. For each repository, our automated pipeline synthesizes a robust suite of endto-end test cases. We evaluate the agent-generated software by executing it in isolated sandboxes and comparing its system-level side effects and terminal outputs against human-written oracles using a rigorous, multi-tiered equivalence metric. Leveraging CLI-Tool-Bench, we conduct an extensive evaluation of seven state-of-the-art LLMs deployed within prominent agent frameworks. Our experiments reveal that top-tier models achieve less than 43% overall success, highlighting that 0-to-1 generation remains a highly challenging frontier. Furthermore, we discover that higher token consumption does not necessarily yield better performance, and agents exhibit a strong tendency to generate monolithic code structures.
LLM Agent, Software Generation, Evaluation
CCS Concepts • Software and its engineering → Software verification and validation. † Corresponding author.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference acronym ’XX, Woodstock, NY © 2018 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-1-4503-XXXX-X/2018/06 https://doi.org/XXXXXXX.XXXXXXX
ACM Reference Format: Ruida Hu1 , Xinchen Wang1 , Chao Peng2 , Cuiyun Gao1† , David Lo3 . 2018. Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios. In Proceedings of Make sure to enter the correct conference title from your rights confirmation email (Conference acronym ’XX). ACM, New York, NY, USA, 11 pages. https://doi.org/XXXXXXX.XXXXXXX
1
Introduction
The advent of Large Language Models (LLMs) has catalyzed a paradigm shift in automated software engineering [1, 5, 11, 16]. Moving beyond simple code completion, recent advancements have spurred the development of autonomous LLM-based agents [3, 4, 14, 24] capable of tackling complex programming tasks. This evolution has given rise to the era of “Vibe Coding” or intent-driven development [15, 18, 21], where users, ranging from professional developers to non-technical individuals, can generate functional software simply by expressing their high-level requirements in natural language. In this new paradigm, autonomous agents are expected to take a high-level natural language requirement and generate a complete, runnable software repository entirely from scratch. To measure the capabilities of these agents, robust evaluation benchmarks are indispensable. However, existing benchmarks fall short of evaluating the true potential of agents in real-world software creation. Traditional benchmarks, such as HumanEval [6] and MBPP [2], are confined to function-level or snippet-level generation. While recent efforts like SWE-bench [12] have elevated the evaluation to the repository level, they predominantly focus on software maintenance tasks—such as resolving issues or adding features to existing codebases—rather than the creation of software from scratch. More recently, attempts like NL2Repo-Bench [7] have explored repository-level generation from scratch; however, their evaluation methodologies reveal critical limitations that hinder the accurate assessment of modern LLM agents. Specifically, we identify three major challenges in the current evaluation landscape: Challenge 1: Reliance on Predefined Repository Structures. Despite rapid advancements, evaluating the true 0-to-1 generation capabilities of LLM agents remains an open challenge. Existing repository-level benchmarks predominantly focus on issue resolution or feature addition within already established repositories [12]. Even in generation-focused benchmarks, the evaluation heavily relies on a fixed, predefined repository structure [7]. These benchmarks typically provide agents with pre-built file skeletons and
Ruida Hu1 , Xinchen Wang1 , Chao Peng2 , Cuiyun Gao1† , David Lo3
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
directory scaffolds, reducing the complex task of software generation to mere code-filling. This structure-dependent paradigm fundamentally bypasses a critical step in software creation: repository structure planning. In real-world 0-to-1 development, developers must autonomously decide how to organize directories, modularize files, and manage dependency configurations. By constraining agents to predefined structures, current benchmarks fail to assess whether LLMs can independently plan and construct a coherent repository from scratch. Challenge 2: Absence of End-to-End Black-Box Testing. Furthermore, existing evaluations heavily rely on white-box unit testing. These tests are tightly coupled with the internal implementation details of the software, forcing the generated code to conform to specific function signatures or class definitions. However, from a user-centric perspective, software utilities, such as Command-Line Interface (CLI) tools, are always consumed as black boxes. Users care about whether the tool correctly parses command-line arguments, produces the expected terminal outputs, and executes the correct system-level side effects (e.g., modifying the file system), regardless of how the internal code is structured. The reliance on rigid white-box testing not only stifles the structural autonomy of LLMs but also fails to provide a realistic, end-to-end validation of the software’s functional correctness. To address these limitations, we introduce CLI-Tool-Bench, a novel benchmark specifically designed to evaluate the 0-to-1 generation of CLI tools. CLI-Tool-Bench shifts the evaluation paradigm from structure-dependent white-box testing to structureagnostic black-box differential testing. We curate a high-quality dataset of 100 real-world CLI repositories across three programming languages (Python, JavaScript, Go) and complexity levels. To overcome the reliance on predefined structures (Challenge 1), the agent is provided only with a natural language requirement and an empty workspace. This unconstrained setting forces the LLM to autonomously handle repository structure planning, dependency management, and logic implementation from scratch. To provide realistic, end-to-end validation (Challenge 2), we evaluate the generated tool purely from a user-centric perspective. We execute the tool in an isolated sandbox and compare its terminal outputs and system-level side effects (e.g., file system state changes) against a human-written oracle. To ensure a fair assessment, we propose a multi-tiered equivalence metric—encompassing Execution, Exact, Fuzzy, and Semantic Match—to accurately gauge behavioral correctness without penalizing the agent’s architectural diversity. In summary, the main contributions of this paper are as follows: • We introduce CLI-Tool-Bench, the first benchmark for end-toend software generation. It challenges agents to build functional CLI tools from scratch, granting them complete autonomy without predefined scaffolds. • We propose an automated pipeline for repository-level generation. It utilizes black-box differential testing in isolated Docker environments to rigorously assess execution reliability, behavioral equivalence, and system-level side effects. • We evaluate advanced LLMs and agent frameworks on CLITool-Bench, revealing their great struggles with autonomous systemlevel generation. We also uncover critical behavioral patterns, such as monolithic design preferences and infinite generation loops.
2
Construction and Evaluation Pipeline
To evaluate the 0-to-1 software generation capabilities of LLM agents, we propose an automated benchmark construction and evaluation pipeline. As illustrated in Figure 1, it consists of three core modules: (1) Repository Curation, (2) Schema-Guided Task Synthesis, and (3) Black-Box Differential Evaluation.
2.1
Repository Curation
To construct a high-quality and representative dataset, we target three mainstream languages widely adopted for CLI development, including Python, JavaScript, and Golang [13]. To ensure the reliability of the oracle repositories serving as ground truth in our benchmark, we design a rigorous curation pipeline, with the details as follows: 2.1.1 Static Metadata Filtering. We initially retrieve candidate repositories from GitHub based on specific criteria to ensure benchmark quality and relevance: (1) Stars >10; (2) Primary language purity >60%; (3) Presence of CLI-related keywords in repository descriptions; (4) Presence of an open-source license (e.g., MIT); and (5) Existence of language-specific build configuration files (e.g., “setup.py” or “pyproject.toml” for Python, “package.json” for JavaScript, and “go.mod” for Golang). 2.1.2 Dynamic Entry Identification. To ensure the practical utility of the selected repositories, we introduce a dynamic verification step. We attempt to install each candidate repository globally in an isolated environment using standard package managers (e.g., “pip install .”, “npm install -g .”). After a successful installation, it is crucial to determine the exact command name used to invoke the CLI tool. While this command name is typically identical to the repository name, variations can occasionally occur. To robustly and autonomously identify the correct entry point for subsequent testing, we design a heuristic matching algorithm. Specifically, let 𝑁𝑟𝑒𝑝𝑜 be the repository name and 𝐷𝑏𝑖𝑛 be the target binary directory. During installation, we monitor 𝐷𝑏𝑖𝑛 to capture the set of newly added executables, denoted as 𝐸𝑛𝑒𝑤 . We then select the target executable 𝑒 ∗ ∈ 𝐸𝑛𝑒𝑤 by matching against 𝑁𝑟𝑒𝑝𝑜 using a prioritized sequence: (1) Exact match; (2) Case-insensitive match; (3) Acronym match; and (4) Maximum Levenshtein similarity. Once an executable 𝑒 ∗ is identified, we validate it by executing the command “𝑒 ∗ --help”. A successful execution, indicated by a zero exit code and the output of standard usage documentation, confirms that the identified binary is indeed a functional CLI entry point. 2.1.3 Stratified Manual Validation. Finally, to further ensure the quality of the selected repositories, two software engineering experts, each with over five years of professional programming experience, conduct a rigorous and independent sanity check on every repository in the filtered dataset. This manual review specifically verifies two critical aspects: first, that the heuristically identified command serves as the correct entry point for the repository’s primary CLI tool; and second, that the tool can execute its commands correctly without encountering underlying environmental or dependency errors. To eliminate subjective bias, a repository is retained in the final benchmark if and only if both experts reach a unanimous consensus on its absolute correctness.
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
1. Repository Curation
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
3. Differential Evaluation
2. Task Synthesis
Environment Initialization Raw GitHub Repos
Black-box Test Cases
Static Metadata Filtering
LLM-Directed Fuzzing for Test Generation
Dynamic Entry Identification
Command Schema
Prompt Template
Iterative Schema Extraction
Agent Env
[Exit Code, Stdout, File System]
[Exit Code, Stdout, File System]
Equivalence Evaluation
LLM Stratified Manual Validation
100 Oracle Repositories
Oracle Env
Task Prompts
Standardized Task Prompt Construction
Execution
Behavior
Side-Effect
(𝑀!"#$ )
(𝑀"%& )
(𝑀#'(( )
Figure 1: Overview of the CLI-Tool-Bench framework.
2.2
Schema-Guided Task Synthesis
A critical challenge in benchmark construction is generating comprehensive test cases at scale without heavy manual intervention. We address this challenge by designing an automated task synthesis pipeline that leverages LLMs to extract a structured command schema from the oracle repository and subsequently generate both the evaluation prompts and a massive suite of test cases based on this extracted schema. 2.2.1 Iterative Schema Extraction. Given an oracle repository 𝑅𝑜𝑟𝑎𝑐𝑙𝑒 , we employ an LLM to iteratively parse its README documentation and “--help” outputs. Since modern CLI tools often feature complex, nested command structures, the LLM explores the CLI level by level (e.g., running “tool --help”, then “tool subcommand --help”). It extracts a hierarchical metadata schema 𝑆 for the CLI, defined as a tuple: 𝑆 = ⟨𝐶, 𝑆𝑠𝑢𝑏 , 𝑃, 𝐾⟩ (1) where 𝐶 is the command name, 𝑆𝑠𝑢𝑏 is the set of nested subcommands, 𝑃 represents the accepted parameters and flags along with their data types (e.g., string, path, boolean), and 𝐾 signifies execution constraints (e.g., required arguments). 2.2.2 LLM-Directed Fuzzing for Test Generation. Based on the extracted schema 𝑆, we implement an LLM-directed fuzzing mechanism to generate a diverse set of test intents and corresponding command strings. To systematically cover the CLI’s capabilities, we first unroll the hierarchical schema 𝑆 into a set of distinct Command Classes. A command class represents a unique functional path, defined by a specific command/subcommand and a valid combination of parameters that satisfies the constraints 𝐾 (e.g., required flags or mutually exclusive arguments). For each repository, the LLM utilizes a predefined Python-based fuzzing framework, which provides standardized utilities for command execution, output capture, and assertion checking, to automatically generate customized fuzzing scripts. These scripts generate test cases across various dimensions for the identified classes: (1) Common usage; (2) Boundary conditions; and (3) Error handling (e.g., intentionally omitting required parameters to test exception outputs). To ensure the validity of the generated test cases, we introduce an execution-feedback loop aimed at instantiating every identified
command class. For a given command class 𝐶𝑖 , let 𝑡𝑖 be a generated concrete test command. We execute 𝑡𝑖 against 𝑅𝑜𝑟𝑎𝑐𝑙𝑒 . If the execution fails unexpectedly (e.g., returning a non-zero exit code for a normal usage intent), the error trace (stderr) is fed back to the LLM to refine the command string. This process iterates until we successfully discover at least one valid, working execution instance for 𝐶𝑖 . Once this verified template is established, the LLM further mutates its parameters to generate a broader suite of test cases. Crucially, this automated fuzzing also serves as a strict quality filter for the oracle repositories. If there exists any command class explicitly claimed in the “--help” documentation for which the LLM consistently fails to find a successful execution instance despite repeated refinements (e.g., reaching a maximum retry limit), we deem the oracle implementation incomplete, buggy, or misaligned with its own documentation. In such cases, the entire repository is discarded. Furthermore, while our LLM-directed pipeline is capable of generating an arbitrarily large volume of test cases, executing end-to-end differential tests within isolated Docker sandboxes incurs non-trivial computational overhead. To strike an optimal balance between evaluation comprehensiveness and execution efficiency, we configure the pipeline to synthesize a robust suite of exactly 50 end-to-end test cases for each identified command class. To ensure thorough behavioral verification, this suite deliberately encompasses both positive test cases (where the oracle executes successfully with a zero exit code) and negative test cases (designed to trigger expected errors or exceptions). Ultimately, this automated, schema-guided approach enables the scalable generation of highly diverse, end-to-end test cases, far exceeding the coverage typically achieved through manual crafting. 2.2.3 Standardized Task Prompt Construction. We construct the task prompt using a unified template (detailed in our repository). To strictly mitigate the risk of data contamination, where the evaluated LLM might recognize the target and rely on memorized code from its pre-training corpus rather than generating it from scratch, we apply a rigorous de-identification process. Specifically, we automatically and manually scrub all identifying metadata, including author names, email addresses, GitHub repository links, and specific project branding, from the source texts. The final anonymized prompt comprises three core components to simulate a realistic,
Ruida Hu1 , Xinchen Wang1 , Chao Peng2 , Cuiyun Gao1† , David Lo3
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
greenfield development requirement: (1) The sanitized functional context derived from the README; (2) The complete, iteratively extracted “--help” documentation, which serves as the strict external interface specification; and (3) One concrete, verified successful execution example for each command class (obtained from the fuzzing stage) to unambiguously demonstrate the expected behavior.
2.3
Black-Box Differential Evaluation
To evaluate the LLM-generated software 𝑅𝑡𝑒𝑠𝑡 without relying on predefined repository structures or internal code implementations, we design a black-box Differential Evaluation Engine based on isolated Docker sandboxes. The proposed engine consists of three key phases: 2.3.1 Environment Initialization. For a given task, we instantiate two identical Docker containers based on the latest languagespecific base images (e.g., python:latest). We mount the oracle repository 𝑅𝑜𝑟𝑎𝑐𝑙𝑒 and the generated repository 𝑅𝑡𝑒𝑠𝑡 into their respective containers at a unified workspace path. After executing the standard installation commands, we capture the exact initial state of the workspace as a clean snapshot. This ensures strict stateless isolation and prevents any persistent environmental changes, such as newly created files or modified configurations (i.e., system-level side effects), from polluting subsequent tests. We denote these persistent, restorable base environments as 𝐸𝑜𝑟𝑎𝑐𝑙𝑒 and 𝐸𝑡𝑒𝑠𝑡 for the human-written oracle and the LLM-generated tool, respectively. 2.3.2 Differential Execution. For each test case 𝑡 ∈ 𝑇 , we execute 𝑡 independently in both 𝐸𝑜𝑟𝑎𝑐𝑙𝑒 and 𝐸𝑡𝑒𝑠𝑡 . To comprehensively capture the behavior of the CLI tool, our engine monitors the execution process and records the resulting state. Specifically, the execution yields a state transition tuple: 𝐸𝑥𝑒𝑐 (𝑅, 𝑡, 𝐸) → ⟨𝐶𝑟𝑒𝑡 , 𝑂𝑠𝑡𝑑 , Δ𝑆⟩
(2)
where 𝐶𝑟𝑒𝑡 is the return code, 𝑂𝑠𝑡𝑑 is the standard output, and Δ𝑆 represents the raw system-level side effects (i.e., file system mutations). This tuple serves as the foundational data structure for our subsequent equivalence analysis. 2.3.3 Equivalence Evaluation. Based on the captured execution states, we evaluate the functional equivalence using three rigorous metrics, including Execution Reliability, Behavioral Equivalence, and System-Level Side-Effect Consistency: Execution Reliability (𝑀𝑐𝑜𝑑𝑒 ): We strictly focus on test cases that represent valid functional paths, defined as those where the 𝑜𝑟𝑎𝑐𝑙𝑒 == 0). For oracle executes successfully with a zero exit code (𝐶𝑟𝑒𝑡 these expected-to-work commands, we verify if the LLM-generated 𝑡𝑒𝑠𝑡 == 0. This ensures software also completes successfully, i.e., 𝐶𝑟𝑒𝑡 that the LLM correctly implements the core functionalities explicitly claimed in the documentation, while avoiding the ambiguity of matching diverse non-zero error codes across different internal implementations. Behavioral Equivalence (𝑀𝑜𝑢𝑡 ): We capture the standard output 𝑂𝑠𝑡𝑑 generated by each test case, deliberately ignoring standard error streams to focus purely on the functional payload. Recognizing that LLMs may generate functionally identical CLI tools with
slight formatting variations in their terminal outputs, our differential evaluation engine employs a multi-tiered output comparison mechanism. This mechanism evaluates behavioral equivalence across three progressive relaxation levels: (1) Exact Match that demands strict string equivalence after basic whitespace normalization; (2) Fuzzy Match that utilizes algorithmic similarity metrics (e.g., normalized edit distance) to tolerate minor formatting divergences while preserving core data integrity; and (3) Semantic Match that leverages an LLM-as-a-judge [8] to verify the equivalence of the core informational payload, completely disregarding superficial stylistic differences. The detailed definitions of these metrics are provided in Section 3. System-Level Side-Effect Consistency (𝑀𝑑𝑖 𝑓 𝑓 ): Beyond terminal outputs, real-world CLI tools often interact with the file system (e.g., creating, modifying, or deleting files). To evaluate these external impacts, we track the state of the workspace before and after a command is executed. By comparing these two states, we extract the exact file system changes, which form the raw side-effect Δ𝑆. Furthermore, during execution, tools often generate trivial intermediate artifacts (such as hidden cache folders or temporary logs) that are irrelevant to the core functionality. To prevent these non-essential files from interfering with the evaluation, our engine automatically ignores hidden paths. Let Φ𝑖𝑔𝑛𝑜𝑟𝑒 be the set of these ignored paths. The effective side-effect is calculated by filtering them out: Δ𝑆 ′ = Δ𝑆 \ Φ𝑖𝑔𝑛𝑜𝑟𝑒 . A specific test case passes this metric only if the effective side-effects of the LLM perfectly match those ′ ′ ). of the oracle (Δ𝑆𝑜𝑟𝑎𝑐𝑙𝑒 == Δ𝑆𝑡𝑒𝑠𝑡 To provide a clear overview of the constructed benchmark, Table 1 summarizes the statistical distribution of the curated repositories across three key dimensions: task difficulty, programming language, and application domain. Following the taxonomy established by NL2Repo-Bench, we stratify the task difficulty into three levels based on the oracle’s Lines of Code (Easy ≤ 1500, Medium 1500 − 4000, Hard ≥ 4000), and classify the repositories into nine distinct application domains. For programming languages, we specifically focus on Python, JavaScript (Node.js), and Go. These languages are selected to provide a representative mix of both interpreted and compiled paradigms that are extensively utilized in modern CLI tool development and system automation. This diverse composition ensures that our evaluation comprehensively reflects an LLM’s general-purpose software engineering capabilities. To ensure the high quality and representativeness of our benchmark, we manually curated a final set of 100 repositories from the initial candidate pool. During this selection process, we carefully balanced the dataset across three key dimensions: programming language, difficulty level (measured by Lines of Code, LOC), and application domain. For the domain classification, we adopted the taxonomy introduced by NL2Repo-Bench [7]. The detailed statistical distribution of the curated dataset is presented in Table 1. As illustrated, the final benchmark maintains a relatively even and highly diverse composition, comprehensively covering three mainstream CLI programming languages, various project complexities, and nine distinct real-world application scenarios.
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Table 1: Statistical distribution of our benchmark. Dimension
Sub-category
Count
Avg. LOC
Difficulty
Easy (≤ 1500 LOC) Medium (1500 − 4000 LOC) Hard (≥ 4000 LOC)
42 24 34
623.76 2,597.33 18,445.91
Language
Python JavaScript Go
38 16 46
4,495.34 5,448.56 9,949.89
Domain
Web Development Testing Utility Libraries Machine Learning Data Analysis & Processing Database Interaction Networking Tools Batch File Processing System Tools
8 9 20 12 12 6 7 14 12
5,500.75 1,921.78 8,607.75 3,332.33 3,894.58 3,509.17 6,462.57 20,651.29 3,342.00
3 Experimental Setup 3.1 Selected LLMs and Agent Frameworks To comprehensively evaluate the state-of-the-art in autonomous software generation, we select 7 cutting-edge LLMs, encompassing both leading closed-source models and highly capable opensource models: GPT-5.4, Claude-Sonnet-4.6, DeepSeek-V3.2, Qwen3.5-plus, GLM-5, MiniMax-M2.5, and Kimi-k2.5. To effectively evaluate these models’ capabilities as software engineers, we employ two representative agent frameworks specifically designed for repository-level tasks: • OpenHands (with CodeAct) [23]: A prominent open-source agent framework that utilizes the CodeAct paradigm, allowing the LLM to iteratively execute code, interact with a bash terminal, and observe environmental feedback. • Mini-SWE-Agent [22]: A streamlined adaptation of the popular SWE-agent framework, specifically optimized for iterative repository construction and terminal-based debugging. By evaluating seven models across two frameworks, we conduct a total of 14 distinct agent configurations for each of the 100 repositories in our benchmark.
3.2
Evaluation Metrics and Scoring Mechanism
As established in methodology, evaluating the true functional correctness of generated CLI tools requires a rigorous, multi-layered approach. To capture the agent’s capabilities, we design an evaluation funnel consisting of progressive metrics. 3.2.1 The Evaluation Funnel and Equivalence Metrics. Before comparing any terminal outputs, the generated repository 𝑅𝑡𝑒𝑠𝑡 must pass two fundamental system-level prerequisites: • Build (Global Installation Success Rate): We first verify if 𝑅𝑡𝑒𝑠𝑡 can be successfully installed in the isolated environment using standard package managers. A failure here indicates a fundamentally broken repository. • Exec (Execution Reliability, 𝑀𝑐𝑜𝑑𝑒 ): We strictly focus on the subset of test cases where the oracle executes successfully 𝑜𝑟𝑎𝑐𝑙𝑒 == 0). This metric measures the proportion of these valid (𝐶𝑟𝑒𝑡 test cases where the agent-generated tool also completes without
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY 𝑡𝑒𝑠𝑡 == 0). A generated test case is considered runtime errors (𝐶𝑟𝑒𝑡 valid for further output comparison if and only if it passes the Exec check and perfectly matches the oracle’s system-level side effects ′ ′ ). Building upon these strict prerequisites, we (Δ𝑆𝑜𝑟𝑎𝑐𝑙𝑒 == Δ𝑆𝑡𝑒𝑠𝑡 define three progressive metrics to evaluate the Behavioral Equivalence (𝑀𝑜𝑢𝑡 ) of the standard outputs: • Exact Match (EM): This metric represents the most rigid functional alignment. The standard outputs of the oracle and the agent are first normalized by stripping all whitespace characters (e.g., \n, \t, and spaces). A match is recorded only if the resulting strings are strictly identical. • Fuzzy Match (FM): To accommodate minor formatting divergences (e.g., different table alignments or spacing) while preserving core data integrity, we calculate the normalized Levenshtein edit distance between the two outputs. A match is recorded if the string similarity score meets or exceeds an empirical threshold of 𝜏 = 0.8. • Semantic Match (SM): For cases where the output structure differs significantly but the underlying information is correct, we employ GPT-5.4 as an automated judge. The LLM is prompted to evaluate whether the core semantic content and informational payload of the two outputs are equivalent, completely disregarding superficial stylistic differences. To rigorously validate the reliability of this LLM-as-a-judge approach, we conducted a large-scale human annotation study. Two software engineering experts—each with over five years of professional experience—independently reviewed a random sample of 1,000 pairs of oracle and agent execution outputs, manually labeling them for semantic equivalence. We then compare the human consensus labels against the automated judgments produced by GPT-5.4. This extensive analysis yielded a Cohen’s Kappa coefficient of 𝜅 > 0.9, demonstrating that our Semantic Match metric strongly aligns with human judgment, enabling a more flexible yet accurate evaluation of output consistency.
3.2.2 Multi-Level Scoring Mechanism. To systematically quantify the performance of an agent-generated repository 𝑅𝑡𝑒𝑠𝑡 and populate our final evaluation tables, we compute final scores for the aforementioned metrics. The Build score is simply the binary installation success rate averaged across all repositories. For the remaining metrics (Exec, EM, FM, and SM), we employ a macro-averaging aggregation strategy. To prevent command classes with simpler logic from disproportionately dominating the evaluation, we calculate the scores as follows. Let 𝐶 denote the set of all identified command classes for a given repository 𝑅𝑡𝑒𝑠𝑡 . For each command class 𝑐 ∈ 𝐶, let 𝑇𝑐 be its corresponding suite of 50 test cases. The pass rate of a specific command class under a given evaluation metric (denoted as 𝑚𝑒𝑡𝑟𝑖𝑐 ∈ {Exec, EM, FM, SM}) is calculated as: 1 ∑︁ 𝑃𝑚𝑒𝑡𝑟𝑖𝑐 (𝑐) = I𝑚𝑒𝑡𝑟𝑖𝑐 (𝑡) (3) |𝑇𝑐 | 𝑡 ∈𝑇 𝑐
where I𝑚𝑒𝑡𝑟𝑖𝑐 (𝑡) ∈ {0, 1} is the binary indicator of whether test case ttt successfully passes the specified metric. The final aggregated score for the repository 𝑅𝑡𝑒𝑠𝑡 under the chosen metric is then formally defined as: ( 𝑆𝑐𝑜𝑟𝑒𝑚𝑒𝑡𝑟𝑖𝑐 (𝑅𝑡𝑒𝑠𝑡 ) =
1 Í 𝑐 ∈𝐶 𝑃𝑚𝑒𝑡𝑟𝑖𝑐 (𝑐) |𝐶 |
if installation succeeds
0
if installation fails
(4)
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Ruida Hu1 , Xinchen Wang1 , Chao Peng2 , Cuiyun Gao1† , David Lo3
This formulation ensures that a repository fundamentally failing the initial package manager installation is penalized with a score of 0. For successfully installed repositories, the macro-averaging mechanism guarantees that the agent is evaluated on its comprehensive ability to implement the entire spectrum of the CLI tool’s functionalities evenly. The final values reported in our subsequent experimental results represent the average 𝑆𝑐𝑜𝑟𝑒𝑚𝑒𝑡𝑟𝑖𝑐 across all evaluated repositories.
3.3
Implementation Details
Following the evaluation protocol established by NL2Repo-Bench, we aim to fully assess the repository-level generation capabilities of the evaluated LLMs. Therefore, rather than restricting the agents with default hyperparameters, we remove all artificial constraints—such as maximum iteration limits or token budgets—across all models and frameworks. The generation process is entirely openended, allowing the agent to autonomously determine when the software construction is complete and voluntarily terminate the execution. For all auxiliary tasks within our automated pipeline—including schema extraction, fuzzing test generation, and the Semantic Match judge—we strictly utilize GPT-5.4. To eliminate generation randomness and ensure deterministic evaluation, the temperature for these auxiliary GPT-5.4 API calls is set to zero.
3.4
Research Questions
We investigate the following four research questions (RQs): • RQ1: How do state-of-the-art LLMs perform in ground-up CLI generation, and how do different agent frameworks influence their success rates? • RQ2: How does the software generation capability of these agents scale or degrade across different task difficulty levels? • RQ3: What are the differences in computational overhead and generation efficiency among the models under an unconstrained, open-ended generation setting? • RQ4: Given complete structural freedom, how do the internal structures of agent-generated repositories compare to humanwritten oracles?
4
Experimental Result
In this section, we present the evaluation results of the selected LLMs and agent frameworks to answer our four research questions.
4.1
RQ1: Overall Performance and Framework Impact
To answer RQ1, we evaluate seven state-of-the-art LLMs across two distinct agent frameworks, revealing a clear performance hierarchy (Table 2). Kimi-k2.5 emerges as the absolute frontrunner, achieving the highest average Semantic Match (SM) score of 42.74%, followed closely by MiniMax-M2.5, which excels particularly within the Mini-SWE-Agent framework. A competitive middle tier consists of GPT-5.4, Qwen-3.5-plus, and GLM-5, all hovering around 29% to 30% SM. Conversely, Claude-Sonnet-4.6 exhibits abnormally poor performance (10.48% SM), primarily bottlenecked at the initial Build stage. Furthermore, the choice of framework significantly impacts success rates; Mini-SWE-Agent outperforms OpenHands across
(a) Results on OpenHands.
(b) Results on Mini-SWE-Agent.
Figure 2: Performance comparison of seven models across three programming languages. The radar charts illustrate the varying capabilities of each model, with the radial axes representing the evaluation scores of Semantic Match Score.
almost all models (31.94% vs. 25.75% average SM), suggesting its workspace management and interaction design provide a more conducive environment for the complex reasoning required in CLI tool development. Beyond individual model capabilities, the progressive evaluation funnel exposes fundamental bottlenecks in current agentic workflows. Across all models, there is a steep degradation from the Build stage (average 76.93%) to Execution Reliability (57.90%), followed by a drastic plunge when evaluated by Exact Match (EM, 23.07%). This massive drop highlights that while agents can successfully write syntactically correct and executable code, they struggle to perfectly replicate the exact string outputs of the Oracle. However, the notable recovery in scores observed in the FM score of 37.69% and the SM score of 28.85% validates our methodological design: agents frequently generate functionally correct and semantically equivalent CLI schemas that are unfairly penalized by stringent EM metrics. A deeper analysis of the programming language distribution (Figure 2) further reveals a pronounced language bias inherent in current LLMs. As depicted in the radar charts, almost all evaluated models exhibit larger coverage areas for interpreted languages like Python and JavaScript, while Golang emerges as a severe bottleneck (represented by the constricted inner green polygons). It suggests that models frequently fail to navigate Golang’s strict type matching and rigid compilation constraints, often generating code that fails to build. Overall, while top-tier models demonstrate promising capabilities, the highest average Semantic Match score remains below 43%, indicating that end-to-end CLI tool development remains a highly challenging task with substantial room for future improvement. Finding 1: Despite the promising capabilities of top-tier models, their steep performance degradation across the evaluation funnel and severe struggles with compiled languages cap the highest overall success rate at under 43%, highlighting that end-to-end software generation remains a highly challenging frontier with substantial room for future improvement.
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Table 2: Overall performance of the evaluated LLMs across two agent frameworks. The rightmost columns and bottom row present macro-averages. Best results are highlighted in bold. OpenHands
Model
Build Exec
EM
FM
Mini-SWE-Agent SM
Build Exec
EM
FM
Average (Both Frameworks) SM
Build Exec
EM
FM
SM
GPT-5.4 Claude-Sonnet-4.6 DeepSeek-V3.2 Qwen-3.5-plus GLM-5 MiniMax-M2.5 Kimi-k2.5
80.00 38.00 81.00 74.00 76.00 78.00 89.00
58.63 21.18 37.31 28.14 17.39 5.59 8.80 6.78 59.51 21.18 36.45 26.79 58.00 23.09 37.21 28.13 56.07 20.76 35.27 26.83 63.55 21.60 39.74 28.30 68.05 28.98 45.11 35.31
86.00 53.00 73.00 77.00 80.00 96.00 96.00
60.96 24.25 40.30 31.79 31.43 11.98 20.77 14.18 57.79 20.90 38.83 26.70 60.41 23.53 38.68 29.79 62.08 26.02 42.26 32.21 78.73 31.34 50.84 38.72 78.01 42.51 56.11 50.18
83.00 45.50 77.00 75.50 78.00 87.00 92.50
59.79 22.72 38.81 29.97 24.41 8.79 14.79 10.48 58.65 21.04 37.64 26.74 59.20 23.31 37.94 28.96 59.08 23.39 38.76 29.52 71.14 26.47 45.29 33.51 73.03 35.74 50.61 42.74
Average
73.71
54.46 20.34 34.27 25.75
80.14
61.34 25.79 41.11 31.94
76.93
57.90 23.07 37.69 28.85
(a) Results on OpenHands.
(b) Results on Mini-SWE-Agent.
Figure 3: Correlation between repository complexity and agent performance. Each scatter point represents an individual test case. The color of the points denotes the difficulty level: Green for Easy, Orange for Medium, and Red for Hard. The dashed black line represents the LOESS trend line.
4.2
RQ2: Impact of Task Complexity
To answer this, we analyze the correlation between the Oracle LOC and the Semantic Match Score, as visualized in Figure 3. Intuitively, one might expect a strict negative correlation: as the repository grows larger, the agent’s performance should monotonically degrade due to context window limitations and complex dependency resolutions. However, the trend lines reveal a counterintuitive nonmonotonic (U-shaped) trajectory. In the transition from Easy (< 1500 LOC) to Medium (1500 - 4000 LOC) repositories, we observe an obvious decline in performance. As projects evolve from simple scripts to multi-module structures, the cognitive load on the LLM increases significantly. Agents struggle with cross-file context retrieval and often suffer from the “lost in the middle” phenomenon, leading to a rapid drop in semantic accuracy. Surprisingly, as the LOC scales into the Hard category (> 4000 LOC), the trend line stabilizes and eventually trends upward. We attribute this phenomenon to the structural standardization of enterprise-scale repositories. Extremely large CLI projects rarely rely on ad-hoc argument parsing; instead, they heavily utilize standardized, well-documented CLI frameworks (e.g., “Cobra” in Go, “Click” in Python). LLMs are highly proficient at recognizing these boilerplate structures from their pre-training corpora. Consequently, even if the agent fails to execute the tool perfectly, it can accurately reconstruct the semantic schema (commands, flags, and arguments) based on framework conventions, thereby achieving a higher Semantic Match Score.
(a) Results on OpenHands.
(a) Results on Mini-SWE-Agent.
Figure 4: Cost-effectiveness analysis of LLM agents across different frameworks. The scatter plots illustrate the trade-off between task performance and execution cost. Small translucent dots represent individual task runs, while the large cross markers denote the centroid for each model. The dashed gray lines indicate the overall mean score and mean token consumption, dividing the space into four quadrants. Finding 2: Agent performance exhibits a non-monotonic, Ushaped trend with repository complexity; while performance drops sharply in mid-sized repositories, the observed rebound in massive projects suggests that agents might benefit from the highly standardized structures typical of large-scale frameworks.
4.3
RQ3: Agent Cost and Generation Efficiency
To answer RQ3, we analyze the cost-effectiveness of different LLMs by plotting their Semantic Match Scores against their total token consumption, as shown in Figure 4. The dashed lines represent the average score and token usage, dividing the performance space into four quadrants. Ideally, a highly capable agent should fall into the top-left quadrant, achieving above-average scores with belowaverage token costs. Across both frameworks, GPT-5.4 and GLM5 consistently demonstrate this optimal behavior. They achieve highly competitive semantic scores while maintaining a minimal token footprint. This suggests that these models possess strong zeroshot reasoning capabilities and can generate accurate CLI schemas without relying on extensive, token-heavy trial-and-error loops. Similarly, Kimi-k2.5 emerges as a standout performer, particularly in the Mini-SWE-Agent framework (Figure 4 (b)), where it dominates the top-left quadrant by achieving the highest overall score with remarkably low token consumption.
Ruida Hu1 , Xinchen Wang1 , Chao Peng2 , Cuiyun Gao1† , David Lo3
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Table 3: Generation overhead and estimated API cost per repository, categorized by LLM and agent framework. Steps
Prompt Tokens (K)
Comp. Tokens (K)
Max
Avg
Min
Max
Avg
Min
Max
Avg
Min
Max
26 6
161.86 16.90
43.75 2.81
708.42 56.45
9.32 3.13
1.69 0.30
25.89 8.26
0.54 0.09
0.15 0.01
1.94 0.21
3 2
19 19
194.15 85.88
52.37 8.96
690.85 483.82
7.65 6.86
1.01 0.51
81.78 20.92
0.70 0.36
0.18 0.03
3.30 1.77
60.88 41.79
25 14
132 112
2,057.02 914.87
352.04 43.56
9,851.36 5,626.74
18.87 16.97
4.30 1.82
130.31 51.70
0.58 0.26
0.10 0.01
2.77 1.60
OpenHands Mini-SWE-Agent
35.61 41.27
3 8
94 131
1,161.52 1,076.39
0.51 18.97
5,104.63 6,907.64
14.02 19.73
2.08 1.32
65.33 71.86
0.50 0.48
0.01 0.01
2.10 2.88
GLM-5
OpenHands SWE-Agent
25.70 43.58
3 2
82 116
239.54 264.38
2.87 9.45
1,073.98 1,156.66
9.42 14.33
0.66 0.98
37.56 59.56
0.27 0.31
0.01 0.01
1.19 1.22
MiniMax-M2.5
OpenHands Mini-SWE-Agent
84.09 68.23
20 10
169 153
3,783.14 2,399.24
285.08 21.37
11,736.31 9,503.50
28.06 32.94
3.82 1.04
84.45 105.40
1.17 0.76
0.09 0.01
3.58 2.93
Kimi-k2.5
OpenHands Mini-SWE-Agent
45.70 48.68
4 3
126 193
1,259.57 343.85
24.92 6.04
6,844.40 4,883.85
20.58 35.36
1.54 0.46
75.98 342.21
0.82 0.31
0.03 0.01
4.33 3.96
Model
Framework Avg
Min
GPT-5.4
OpenHands Mini-SWE-Agent
12.38 2.33
6 2
Claude-Sonnet-4.6
OpenHands Mini-SWE-Agent
7.76 6.91
DeepSeek-V3.2
OpenHands Mini-SWE-Agent
Qwen-3.5-plus
Conversely, the right half of the scatter plots reveals a phenomenon of “diminishing returns” in agent trajectories. Models like Minimax-M2.5 and DeepSeek-V3.2 frequently fall into the right-side quadrants, consuming massive amounts of tokens (often exceeding 2 million) but failing to achieve top-tier performance. This high token consumption is typically indicative of “thrashing”, situations where the agent gets trapped in repetitive debugging cycles or generates overly verbose, unhelpful commands without making actual progress toward the task resolution. On the other extreme, Claude-Sonnet-4.6 consistently occupies the bottom-left quadrant. Its extremely low token usage, coupled with the lowest semantic scores, implies a tendency to “fail fast”; the agent likely encounters an early error it cannot resolve and prematurely terminates the trajectory before consuming significant context. Finding 3: Higher token consumption does not equate to better task resolution; while some models get trapped in costly debugging loops with diminishing returns, models like GPT-5.4 and Kimi-k2.5 demonstrate superior cost-effectiveness by achieving top-tier performance with minimal token footprints.
4.4
RQ4: structural Autonomy and Diversity
To answer RQ4, we investigate how different LLMs organize repository structures and manage workspace complexity when granted full structural autonomy. Figure 5 compares the file count distributions of agent-generated repositories against the human Oracle. Human Modularity vs. Agent Monolithic Preference. The evident trend is the divergence in structural design between human developers and autonomous agents. The human Oracle exhibits a broad distribution, reflecting standard software engineering practices where code is modularized into distinct components. In contrast, all evaluated LLMs demonstrate a strong preference for monolithic structures, with their medians tightly clustered between 1 and 3 files. This indicates a shared behavioral strategy among LLMs: centralizing logic into a single or very few files. From an agentic
Cost ($)
(a) File count distribution under OpenHands.
(b) File count distribution under Mini-SWE-Agent. Figure 5: Distribution of total files generated by the human Oracle and various LLMs. The y-axis is truncated at 20 files. Extreme outliers are annotated with red stars and their maximum values at the top of the violins. perspective, this monolithic approach seems to be a practical adaptation to minimize cross-file dependency issues (e.g., ImportError) and to keep the entire system state easily accessible within the model’s limited context window. Workspace Management and Debugging Behaviors. Beyond the median values, the maximum file counts (annotated with red stars) reveal diverse workspace management behaviors during the generation process. Models like GPT-5.4 and Claude-Sonnet4.6 maintain strictly low file counts, indicating a behavior of editing and overwriting existing files in place when fixing bugs. Conversely, several other models (e.g., DeepSeek-V3.2, Qwen-3.5-plus, and MiniMax-M2.5) occasionally generate hundreds of files. Rather
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
than implying lower functional capability, this extreme file sprawl reflects a different debugging pattern: when encountering execution errors, these agents tend to create numerous temporary scripts, duplicate modules, or isolated test files instead of cleaning up or modifying the original code. Furthermore, this behavior is influenced by the environment. For instance, GLM-5 and Kimi-k2.5 maintain compact workspaces under OpenHands (Figure 5 (a)) but exhibit great file sprawl under Mini-SWE-Agent (Figure 5 (b)), suggesting that the underlying framework’s prompt structure and feedback mechanisms impact agents’ file system management.
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
Table 4: Runtime efficiency of the generated CLI tools. Source
5 Discussion 5.1 Beyond Correctness: Runtime Efficiency Robustness to Invalid Inputs. A production-ready CLI tool must not only handle expected inputs but also gracefully manage invalid arguments or edge cases. To evaluate this, we compare the agents’ performance on positive (valid) versus negative (invalid/edge-case) test commands. Runtime Efficiency of Generated Code. To investigate the runtime efficiency of LLM-generated code beyond mere functional correctness, we conduct a controlled performance analysis. Specifically, we isolate a subset of 27 repositories where all models successfully built and executed the CLI tools within the Mini-SWE-Agent framework. As presented in Table 4, we observe a counterintuitive phenomenon: the tools generated by all LLMs consistently outperformed the human-written Oracle in terms of raw execution speed. On average, the human Oracle required 270.37 milliseconds per command. In stark contrast, models like GPT-5.4 and Qwen-3.5-plus achieved great runtime reductions, executing in roughly 126-127 milliseconds—effectively halving the execution time (0.47× relative slowdown). Even the relatively slower models in this subset, such as Kimi-k2.5 (0.88×), still maintained a distinct speed advantage over the human baseline. This unexpected efficiency may contribute to the inherent “minimalism” of LLM-generated solutions compared to human engineering practices. Human developers typically design CLI tools with robust non-functional requirements, incorporating extensive error handling, rich logging mechanisms, and user-friendly formatting (e.g., progress bars or colored terminal outputs). These engineering best practices, while crucial for maintainability and user experience, inevitably introduce runtime overhead. Conversely, LLM agents are highly goal-oriented; they tend to generate the most direct, stripped-down logic necessary to satisfy the immediate functional requirements or pass the provided test cases. They frequently bypass defensive programming and decorative outputs, resulting in leaner, faster-executing binaries. This finding highlights a fascinating trade-off in agentic software engineering: while agents may currently lack the holistic design foresight of human developers, their hyper-focused, minimalist coding style can inadvertently yield superior raw execution performance.
Relative Slowdown
Human Oracle
270.37
1.00×
GPT-5.4 Claude-Sonnet-4.6 DeepSeek-V3.2 Qwen-3.5-plus GLM-5 MiniMax-M2.5 Kimi-k2.5
126.14 167.41 201.44 127.49 138.85 142.78 238.11
0.47× 0.62× 0.75× 0.47× 0.51× 0.53× 0.88×
5.2 Finding 4: While human developers naturally adopt modular structures, LLMs prefer monolithic designs to simplify context management. Besides, agents exhibit diverse debugging behaviors, ranging from in-place editing to extensive file sprawl.
Avg. Tool Runtime (ms)
Case Study
To better understand the bottlenecks in autonomous software generation, we conducted an in-depth qualitative analysis by manually inspecting a diverse set of failed agent trajectories. Aligning with our evaluation funnel (Section 3.2), we categorize the most prevalent and representative failure modes into three distinct phases: Installation, Execution, and Behavioral Equivalence. Figure 6 illustrates these typical failures. To gain deeper insights into the behavioral bottlenecks of LLM agents beyond quantitative metrics, we conducted a qualitative analysis of typical failure modes encountered during CLI tool development, as illustrated in Figure 6. Figure 6 (a) demonstrates a prevalent "build failure" driven by what we term a text-to-file bias. In this Go project scenario, the agent successfully generated the correct source code logic but attempted to resolve dependencies by manually hardcoding them into the “go.mod” file instead of executing standard package management commands like “go mod tidy”. Because LLMs are predominantly trained on static code repositories, they often default to direct text manipulation and lack the “environmental intuition” required to interact with dynamic, command-driven toolchains, ultimately leading to compilation errors due to missing dependencies. Beyond a lack of environmental intuition, agents also exhibit significant deficiencies in maintaining a coherent mental model of the workspace over long trajectories, leading to severe execution failures. Figure 6 (b) illustrates a catastrophic workspace mismanagement scenario where the agent, while attempting to build a Python tool, trapped itself in a recursive directory generation loop, creating deeply nested “build/lib/” structures. This chaotic file system state completely broke the package’s entry points, resulting in a runtime “Command not found” (Exit Code: 127) error during evaluation. This case highlights a critical spatial blindness; when faced with unexpected build errors, the agent blindly retried commands or applied localized patches rather than systematically diagnosing and correcting the corrupted directory structure. Perhaps the most deceptive failure mode is the behavioral mismatch shown in Figure 6 (c), which perfectly illustrates the "illusion of success." In this scenario, the agent-generated “deletor” tool executed flawlessly without any runtime crashes, returning a successful Exit Code 0. However, our file system state evaluation revealed that, unlike the human oracle, which successfully removed the target files, the agent’s tool produced absolutely no system-level side effects. If an evaluation framework relied solely on execution status or standard output, this silent failure would be falsely rewarded as a success. This stark contrast underscores a fundamental principle in evaluating CLI tools: "running without crashing" does
Ruida Hu1 , Xinchen Wang1 , Chao Peng2 , Cuiyun Gao1† , David Lo3
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
go.mod
main.go
module tview go 1.21
package main
// 🟢 Agent manually wrote dependencies require ( github.com/fatih/color v1.15.0 github.com/spf13/cobra v1.7.0 ) ...
import ( "fmt" "os" "strings" "github.com/spf13/cobra" "tview/internal/tview" ) // ... valid logic ...
$ go install . 🔴 main.go:8:2: missing go.sum entry for module providing package github.com/spf13/cobra (imported by tview); to add: go get tview 🔴 internal/tview/tview.go:11:2: missing go.sum entry for module providing package github.com/fatih/color ...
✅[Oracle Execution] $ tree 📁 whattheflag_repo/ ├── 📁 build/ │ └── 📁 lib/ │ └── 📁 build/ │ └── 📁 lib/ │ └── 📁 build/ │ └── 📁 lib/ │ └── 📁 build/ │ └── 📁 lib/ │ └── 📁 whattheflag/ │ ├── data.py │ └── __main__.py ├── 📁 whattheflag/ ├── pyproject.toml └── README.md
$ deletor -cli -d /test_data --older [duration] -skip-confirm Exit Code: 0 [File System Diff] - /test_data/old_file.tmp - /test_data/file_a.txt - /test_data/file_b.log (Files successfully deleted)
❌[Agent Generated]
$ deletor -cli -d /test_data --older [duration] -skip-confirm $ whattheflag tools -1 sh: 1: whattheflag: not found Exit Code: 127
(a) Build Failure.
(b) Execution Failure.
Exit Code: 0 [File System Diff] (No changes detected)
(c) Behavioral Mismatch.
Figure 6: Representative failure modes of LLM agents in CLI tool development. not equate to task completion, firmly validating the necessity of our multi-dimensional evaluation approach that rigorously verifies actual file system state changes.
5.3
Implication
For Researchers: Our findings emphasize the critical need to evolve evaluation methodologies and agent structures. The contrast between exact match and semantic match scores proves that traditional string-based evaluations are insufficient for agentic tasks, necessitating multi-dimensional, state-aware benchmarks like CLITool-Bench. Besides, the prevalent issues of "thrashing" (high token consumption without task progress) and catastrophic workspace mismanagement highlight a structural gap. Future research could focus on equipping agents with better spatial awareness of the file system, long-horizon planning, and self-reflection mechanisms to break out of unproductive debugging loops. For Developers: For software developers and practitioners, our study reveals actionable strategies for optimally collaborating with current LLM agents. Since agents universally prefer monolithic designs and struggle with cross-file dependency resolutions, developers should provide explicit structural scaffolding or mandate the use of highly standardized CLI frameworks (e.g., Click or Cobra) to guide the agent’s generation process. Additionally, while agent-generated tools often exhibit impressive raw execution speed due to their goal-oriented, minimalist coding style, they typically lack defensive programming practices. Therefore, developers must treat these outputs as functional prototypes, actively reviewing and reinforcing them with necessary error handling, robust logging, and security measures before production deployment.
5.4
Threats and Limitations
First, our dataset focuses on three mainstream languages. Future iterations will expand to include heavily compiled languages. Another threat arises from the inherent randomness of LLMs. Since we use LLMs during experiments, results may vary across trials. Hence, we conduct multiple runs and report the average results.
6 Related Work 6.1 Agents for Software Engineering The application of LLMs in software engineering has evolved from static code completion to autonomous, environment-interacting
agents [3, 4, 20, 27]. Early multi-agent frameworks, such as ChatDev [19] and MetaGPT [9], utilized role-playing and Standardized Operating Procedures (SOPs) to orchestrate software design via simulated collaboration. To enable real-world repository interaction, Agent-Computer Interfaces (ACIs) were introduced. Frameworks like SWE-agent [26] and OpenHands [23] allow LLMs to interact directly with command-line terminals, execute tests, and iteratively debug within isolated sandboxes. Recently, minimalist scaffolds like Mini-SWE-agent [22] have demonstrated that state-of-the-art models require remarkably little scaffolding to achieve high success rates, relying primarily on their native reasoning capabilities.
6.2
Software Generation and Agent Evaluation
As agentic capabilities advance, evaluation methodologies have shifted to complex systems. Early benchmarks like HumanEval [6] and MBPP [2] focused on static, function-level synthesis but suffer from data contamination and a lack of repository context. Subsequent efforts addressed this through dynamic problem sourcing (LiveCodeBench [10]) or complex API integration (BigCodeBench [28]). For system-level tasks, SWE-bench [12] established the standard for software maintenance by evaluating patch generation for GitHub issues, while OSWorld [25] and Terminal-Bench [17] assess multi-step command execution in live operating systems. However, evaluating zero-to-one software generation remains a critical challenge. While NL2Repo-Bench [7] explores repository-level generation from natural language, it relies heavily on human-written oracles. This rigid white-box approach severely penalizes functionally correct but structurally diverse solutions. CLI-Tool-Bench bridges these paradigms, extending black-box differential testing to the repository level to evaluate end-to-end software generation without imposing structural constraints.
7
Conclusion
In this paper, we introduce CLI-Tool-Bench, a benchmark for endto-end evaluating LLM agents on software generation. Extensive evaluations reveal that the highest overall success rate remains below 43%, indicating substantial room for future improvement in this challenging domain. Furthermore, our analysis uncovers several intriguing phenomena: agent performance exhibits a counterintuitive U-shaped trend with repository complexity, and higher
Evaluating LLM-Based 0-to-1 Software Generation in End-to-End CLI Tool Scenarios
Conference acronym ’XX, June 03–05, 2018, Woodstock, NY
token consumption does not equate to better task resolution. Finally, we observe that agents universally prefer monolithic designs over modular structures to simplify context management.
Engineering and Methodology 33, 3 (2024), 1–46. [14] Junwei Liu, Kaixin Wang, Yixuan Chen, Xin Peng, Zhenpeng Chen, Lingming Zhang, and Yiling Lou. 2024. Large Language Model-Based Agents for Software Engineering: A Survey. CoRR abs/2409.02977 (2024). arXiv:2409.02977 doi:10. 48550/ARXIV.2409.02977 [15] Stephane H Maes. 2025. The gotchas of ai coding and vibe coding. it’s all about support and maintenance. OSF Preprints (2025). [16] Zhenyy Mao, Jialong Li, Dongming Jin, Munan Li, and Kenji Tei. 2024. Multi-role consensus through llms discussions for vulnerability detection. In 2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C). IEEE, 1318–1319. [17] Mike A Merrill, Alexander G Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E Kelly Buchanan, et al. 2026. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces. arXiv preprint arXiv:2601.11868 (2026). [18] Christian Meske, Tobias Hermanns, Esther Von der Weiden, Kai-Uwe Loser, and Thorsten Berger. 2025. Vibe coding as a reconfiguration of intent mediation in software development: Definition, implications, and research agenda. IEEE Access 13 (2025), 213242–213259. [19] Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative Agents for Software Development. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 15174–15186. doi:10.18653/V1/2024.ACL-LONG.810 [20] Shuofei Qiao, Ningyu Zhang, Runnan Fang, Yujie Luo, Wangchunshu Zhou, Yuchen Eleanor Jiang, Chengfei Lv, and Huajun Chen. 2024. AutoAct: Automatic Agent Learning from Scratch for QA via Self-Planning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 3003–3021. https://aclanthology.org/2024.acl-long.165 [21] Partha Pratim Ray. 2025. A review on vibe coding: Fundamentals, state-of-the-art, challenges and future directions. Authorea Preprints (2025). [22] SWE-Agent. [n. d.]. The 100 line AI agent that solves GitHub issues or helps you in your command line. https://github.com/SWE-agent. [23] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, and et al. 2025. OpenHands: An Open Platform for AI Software Developers as Generalist Agents. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https://openreview.net/forum?id=OJd3ayDDoF [24] Zefan Wang, Zichuan Liu, Yingying Zhang, Aoxiao Zhong, Jihong Wang, Fengbin Yin, Lunting Fan, Lingfei Wu, and Qingsong Wen. 2024. Rcagent: Cloud root cause analysis by autonomous agents with tool-augmented large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management. 4966–4974. [25] Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. 2024. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). [26] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024, Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomczak, and Cheng Zhang (Eds.). [27] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. 2024. AutoCodeRover: Autonomous Program Improvement. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Austria, September 16-20, 2024, Maria Christakis and Michael Pradel (Eds.). ACM, 1592–1604. doi:10.1145/3650212.3680384 [28] Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, James Hoang, Armel Randy Zebaze, Xiaoheng Hong, WenDing Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, and et al. 2025. BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
References [1] Chetan Arora, John Grundy, and Mohamed Abdelrazek. 2024. Advancing requirements engineering through generative ai: Assessing the role of llms. In Generative AI for Effective Software Development. Springer, 129–148. [2] Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. CoRR abs/2108.07732 (2021). arXiv:2108.07732 https://arxiv.org/abs/2108.07732 [3] Islem Bouzenia, Premkumar Devanbu, and Michael Pradel. 2025. Repairagent: An autonomous, llm-based agent for program repair. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE). IEEE, 2188–2200. [4] Dong Chen, Shaoxin Lin, Muhan Zeng, Daoguang Zan, Jian-Gang Wang, Anton Cheshkov, Jun Sun, Hao Yu, Guoliang Dong, Artem Aliev, Jie Wang, Xiao Cheng, Guangtai Liang, Yuchi Ma, Pan Bian, Tao Xie, and Qianxiang Wang. 2024. CodeR: Issue Resolving with Multi-Agent and Task Graphs. CoRR abs/2406.01304 (2024). arXiv:2406.01304 doi:10.48550/ARXIV.2406.01304 [5] Liguo Chen, Qi Guo, Hongrui Jia, Zhengran Zeng, Xin Wang, Yijiang Xu, Jian Wu, Yidong Wang, Qing Gao, Jindong Wang, Wei Ye, and Shikun Zhang. 2024. A Survey on Evaluating Large Language Models in Code Generation Tasks. CoRR abs/2408.16498 (2024). arXiv:2408.16498 doi:10.48550/ARXIV.2408.16498 [6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. 2021. Evaluating Large Language Models Trained on Code. CoRR abs/2107.03374 (2021). arXiv:2107.03374 https://arxiv.org/abs/2107.03374 [7] Jingzhe Ding, Shengda Long, Changxin Pu, Huan Zhou, Hongwan Gao, Xiang Gao, Chao He, Yue Hou, Fei Hu, Zhaojian Li, Weiran Shi, Zaiyuan Wang, Daoguang Zan, Chenchen Zhang, Xiaoxu Zhang, Qizhi Chen, Xianfu Cheng, Bo Deng, Qingshui Gu, Kai Hua, Juntao Lin, Pai Liu, Mingchen Li, Xuanguang Pan, Zifan Peng, Yujia Qin, Yong Shan, Zhewen Tan, Weihao Xie, Zihan Wang, Yishuo Yuan, Jiayu Zhang, Enduo Zhao, Yunfei Zhao, He Zhu, Chenyang Zou, Ming Ding, Jianpeng Jiao, Jiaheng Liu, Minghao Liu, Qian Liu, Chongyao Tao, Jian Yang, Tong Yang, Zhaoxiang Zhang, Xinjie Chen, Wenhao Huang, and Ge Zhang. 2025. NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents. CoRR abs/2512.12730 (2025). arXiv:2512.12730 doi:10.48550/ARXIV.2512.12730 [8] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. 2024. A survey on llm-as-a-judge. The Innovation (2024). [9] Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum? id=VtmBAGCN7o [10] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. https: //openreview.net/forum?id=chfJJYC3iL [11] Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A Survey on Large Language Models for Code Generation. CoRR abs/2406.00515 (2024). arXiv:2406.00515 doi:10.48550/ARXIV.2406.00515 [12] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-world Github Issues?. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https://openreview.net/forum?id=VTF8yNQM66 [13] Wen Li, Austin Marino, Haoran Yang, Na Meng, Li Li, and Haipeng Cai. 2024. How are multilingual systems constructed: Characterizing language use and selection in open-source multilingual software. ACM Transactions on Software