Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software

AI Magazine

Article

Code That Works, Environments That Don’t: Measuring Environment Reproducibility in AI-Generated Software

arXiv:2610.00425v1 [cs.SE] 30 Sep 2026

Bhanu Prakash Vangala | Tanu Malik 1

Department of Electrical Engineering and Computer Science, University of Missouri–Columbia, Missouri, USA |

Correspondence: Bhanu Prakash Vangala (bv3hz@missouri.edu); Tanu Malik (tanu@missouri.edu) Received: Revised: Accepted: Keywords: LLM coding agents | dependency specification | environment gap | code generation | software supply chain

ABSTRACT Code generation has emerged as a central capability of large language models, with coding agents now able to produce functionally correct software projects from natural language prompts. However, functional correctness alone does not capture a critical dimension of generation quality: environment specification, which is defined as the accurate identification of the dependencies required to execute generated code, is equally critical. We develop an agent protocol for environment specification, and introduce a three-layer framework comprising declared , runtime-installed, and necessary and sufficient dependencies to systematically assess coding agents for environment specification. Using this protocol, we evaluate the extent to which coding agents systematically misspecify software environment dependencies, and how does this misspecification vary across three agents, four languages, and fifty programming tasks. Our results show that current coding agents exhibit systematic generalization failures along this dimension, producing dependency specifications that are inconsistent, redundant, or incomplete in ways that functional tests do not detect. Across agents, dependency set agreement is as low as 7% for identical tasks, and newer agents show no meaningful improvement, suggesting the failure is not resolved by scale or recency. The largest divergence occurs between the declared (𝐷 (1) ) and runtime (𝐷 (3) ) dependency layers, implicating environment priors learned from the models’ training distributions as the primary driver. Our findings establish environment specification as a distinct, measurable axis of code generation quality that current benchmarks do not capture, and motivate training objectives and evaluation protocols that jointly optimize for functional correctness and environmental portability.

1

Introduction

Code generation has emerged as one of the most consequential capabilities of large language models. Advanced coding agents [1, 2] can now synthesize complete, multi-file software repositories from natural language descriptions, representing a qualitative leap beyond the line-level autocomplete of earlier tools [3, 4]. However, the benchmarks used to evaluate these agents have not kept pace with their evolving capabilities [5, 6]. Current evaluation frameworks [3, 7, 8] focus almost exclusively on functional code correctness. They fail to measure the correctness of the underlying software environment, such as dependencies, package versions, and system configurations, which is strictly required for execution.

AI Magazine, 2026;v0:1–16 https://doi.org/0000

One might argue that code and its environment are two sides of the same coin, implying that traditional benchmarks implicitly evaluate both. We counter that software environments are fundamentally distinct from source code. While code dictates logic, environments manage external, stateful ecosystems that are prone to silent runtime failures and complex version conflicts [9, 10]. Consequently, evaluating environment generation requires a qualitatively different benchmarking paradigm. Before AI scientists build such benchmarks, we believe it is important to first systematically measure how effectively current agents specify these environments. To address this gap, this paper introduces a framework to evaluate and quantify how reliably coding agents generate software environments. We propose an iterative evaluation protocol. First,

1 of 16

the agent is prompted to generate a complete software artifact containing both source code and its environment specification, establishing a baseline performance metric. Second, we evaluate the agent’s repair capabilities by forcing it to iteratively diagnose and resolve its own environment failures. The protocol ultimately determines the agent’s baseline accuracy, its self-repair efficiency, and the execution bottlenecks common to modern LLMs. Toward this goal, this paper introduces a novel evaluation framework to measure, assess, and quantify the quality of environment specifications generated by coding agents. We propose an iterative evaluation protocol divided into two primary phases. First, an agent is prompted to generate a complete software artifact comprising both source code and its environment specification, establishing a baseline metric for its initial generation capabilities. Second, if the initial environment is under-specified or broken, the agent enters a self-repair phase where it must iteratively diagnose and resolve its own environment failures. Once an agent achieves a successfully executing artifact, the overall quality of the environment specified across all iterations is independently validated using application virtualization. By tracing the necessary and sufficient dependencies loaded into memory during runtime, our framework establishes an absolute ground truth, allowing us to explicitly measure the degree of dependency over-specification. Our framework thus provides a holistic, three-dimensional assessment: it benchmarks initial environment generation; quantifies the repair friction required to resolve under-specification; and exposes over-specification by comparing the agent’s declared dependencies against the lean, optimal runtime ground truth. We instantiate this framework in a controlled empirical study spanning three production coding agents, four language ecosystems, and 1,000 curated project instances. Each instance is executed in a clean, isolated environment with full dependency tracing from manifest through runtime. Using this rich dataset, we systematically characterize the root causes of environment specification failures across different agents, programming languages, and task domains. Our results produce the first systematic evidence that dependency declarations are profoundly inconsistent across agents solving identical tasks. Critically, we find that LLM environment generation is non-deterministic; a single agent rarely reproduces its own dependency choices across independent runs. In fact, the unanimous consensus rate for identical tasks across all languages is exactly zero. Furthermore, the gap between declared and actual runtime dependencies is wide, directionally biased, and heavily ecosystem-dependent. These variations implicate artifacts within the models’ training distributions rather than the inherent difficulty of the tasks themselves. Our earlier study [11] offered first evidence that agent-generated code is not reproducible at the dependency level; here we formalize the underlying evaluation protocol and measurement primitives, and expand the analysis across agents, ecosystems, and stochastic trials. These findings have direct, urgent implications for the reproducibility of agent-generated software and the validity of using agents for autonomous software engineering. Specifically, our study indicates that software environment configuration must be treated as a core target training objective for LLMs. Furthermore, precision, recall, and F1-scores on dependency declarations—measured against traced runtime ground truth—must be

treated as first-class evaluation criteria alongside functional code correctness. We make the following primary contributions: ∙ Three-Layer Dependency Framework: We introduce a formal paradigm for evaluating coding agents by decoupling dependencies into three layers: what the agent declares, what the package manager installs, and what the program actually loads into memory at runtime. ∙ Comprehensive Evaluation Dataset: We construct and open-source a dataset spanning three production coding agents, four language ecosystems, and 1,000 verified project instances, complete with execution traces in isolated environments. ∙ Empirical Analysis of Environment Failures: We deliver a systematic characterization of LLM environment generation, proving that under- and over-specification represent systemic algorithmic gaps in training rather than isolated edge cases. The rest of this paper is organized as follows. Section 2 formalizes the agent evaluation protocol and defines the core agent capabilities required for environment specification. Section 3 details the primitive functions that underpin the three-layer dependency framework. Section 4 outlines our experimental design, followed by the evaluation dimensions in Section 5 and a comprehensive evaluation and metrics analysis in Section 6. Finally, Section 7 reviews related work, and Section 8 concludes the paper.

2

Agent Evaluation Protocol

Given a hardware and operating environment, if a software environment is incorrectly specified, a developer will aim to fix it till it results in executable code. An agent is similar. Our agent evaluation protocol divides this agent behavior into two skills, a generation skill, in which it produces code and a dependency manifest for a given task, and a repair skill, in which it iteratively revises both specified code and dependency manifest in response to runtime failures. Formally, given a coding agent 𝐴 and a programming task 𝑇 with corresponding prompt 𝜋𝑇 , the agent’s generation skill produces the initial code 𝐶 and dependency manifest 𝑀, specified as: (1)

(𝐶, 𝑀) = Gen(𝐴, 𝜋𝑇 ) (0)

(0)

Let the (𝐶, 𝑀) produced by the Gen skill be indexed as (𝐶 , 𝑀 ). Executing (𝐶 (0) , 𝑀 (0) ) in an isolated environment 𝐼 yields an initial signal: 𝑒(0) = Execute(𝐶 (0) , 𝑀 (0) , 𝐼) If 𝑒(0) = ∅ the agent has reliably executed the code given the dependency manifest, and it denotes successful execution. Otherwise it denotes failed execution with 𝑒(0) specifying the error. If 𝑒(0) ≠ ∅, the Repair skill iteratively generates (𝐶 (𝑖) , 𝑀 (𝑖) ) by feeding the previous code, manifest, and error signal to 𝐴:

(𝐶 (𝑖) , 𝑀 (𝑖) ) = Repair(𝐴, 𝐼, [𝐶 (𝑖−1) , 𝑀 (𝑖−1) , 𝑒(𝑖−1) ]),

𝑖 = 1, … , 𝑘

𝑒(𝑖) = Execute(𝐶 (𝑖) , 𝑀 (𝑖) , 𝐼)

2 of 16

AI Magazine, 2026

repair-skill gap ∆: 𝐷 (1) ↔ 𝐷 (2)

𝐷 (2) Installed

𝐷 (1) Declared phantom: 𝐷 (1) ⧵𝐷 (3) hidden: 𝐷 (3) ⧵𝐷 (1)

bloat: 𝐷 (2) ⧵𝐷 (3) hidden: 𝐷 (3) ⧵𝐷 (2)

𝐷 (3) Traced F I G U R E 1 | Relationships among the three dependency layers. 𝐷 (1) is extracted from the initial manifest, 𝐷 (2) from the final successfully installed environment, and 𝐷 (3) from independent runtime tracing via strace. No fixed set-inclusion is assumed between 𝐷 (1) and 𝐷 (2) . Pairwise gaps define phantom dependencies (declared but not needed), hidden dependencies (needed but not declared or installed), and bloat (installed but unused).

The loop halts at the first 𝑖 ≤ 𝑘 for which 𝑒(𝑖) = ∅, or exhausts the budget 𝑘 otherwise. The final artifact is denoted (𝐶 (𝑘) , 𝑀 (𝑘) ), where (𝐶 (𝑘) , 𝑀 (𝑘) ) = (𝐶 (𝑖) , 𝑀 (𝑖) ) for the smallest 𝑖 at which 𝑒(𝑖) = ∅, or the last iterate if no successful execution is reached within budget. From these artifacts, obtained iteratively, we derive three dependency sets: ∙ The declared set 𝐷 (1) ∶= ParseManifest𝑙 (𝑀 (0) ) captures the agent’s initial claims about the environment the code needs given the programming task. ∙ The installed set 𝐷 (2) ∶= Resolve𝑙 (𝑀 (𝑘) , 𝐼) captures the environment installed in 𝐼 at the end of the iterative cycle. In both definitions, 𝑙 denotes the language ecosystem, which is necessary to understand the language code-specific dependencies. ∙ The traced set 𝐷 (3) ∶= Trace(𝐶 (𝑘) , 𝐼) is defined only when 𝑒(𝑘) = ∅, i.e., when the program has been successfully installed and executes without error. Trace is agentindependent and captures the necessary and sufficient environment for the observed execution of 𝐶 (𝑘) in 𝐼. We often use 𝐶 (𝑘) to refer to both the source code and its installed executable in 𝐼 interchangeably, as the distinction is not material to the dependency analysis. These three sets 𝐷 (1) , 𝐷 (2) , and 𝐷 (3) define our three-layer framework. To use these as a basis for measuring the generative and repair skills of coding agents, we do a pairwise comparison of 𝐷 (1) , 𝐷 (2) and 𝐷 (3) . In this comparison, 𝐷 (3) serves as a proxy for ground truth since obtaining 𝐷 (3) is independent of the agent and this truth is obtained via the reference run of the executable corresponding to the code, which identifies dependencies for the specific execution run. Comparing 𝐷 (1) and 𝐷 (3) in both directions reveals two complementary gaps: dependencies present in 𝐷 (1) but absent from 𝐷 (3) are phantom dependencies i.e., declared by the agent’s Gen skill but not actually needed by the code, while dependencies present in 𝐷 (3) but absent from 𝐷 (1) are hidden dependencies i.e., necessary for execution but never declared.

Comparing 𝐷 (2) and 𝐷 (3) exposes the agent’s blindness to deeper dependencies: packages in 𝐷 (2) but not 𝐷 (3) constitute dependency bloat. These are dependencies installed by the package manager but unused at runtime. Alternatively, packages in 𝐷 (3) but not 𝐷 (2) remain hidden even after full resolution. Finally, comparing 𝐷 (1) and 𝐷 (2) quantifies what the Repair skill contributes: packages added to or removed from the manifest across 𝑘 iterations, isolating the effect of iterative error correction from the agent’s initial declaration. Figure 1 summarizes each pairwise comparison and the gap types each exposes.

Protocol Primitives: Dependency Generation 3 and Extraction We now describe the different primitives used in our protocol, namely how Gen and Repair are configured with prompts and tasks, and how ParseManifest, Resolve and Trace lead to 𝐷 (1) , 𝐷 (2) , and 𝐷 (3) . Figure 2 shows the use of these primitives as part of an agentic pipeline that includes Gen and Repair.

3.1

Dependency Generation Primitives

The quality of the results from Gen and Repair depends on two inputs: the complexity of the programming task 𝑇 and the design of the prompt 𝜋𝑇 . We address each through principled design criteria. Tasks are drawn from common real-world programming problems, subject to two constraints: First, each task must be platformindependent i.e., implementable in every language under study so that cross-language comparisons are controlled at the task level rather than confounded by structural differences between tasks. Second, each task must also be self-contained, requiring no external services, APIs, or credentials, so that dependency failures are attributable to the agent and not to environmental inaccessibility. The prompt template instantiated for each task is shown in Figure 3. Every instance shares the same structure of task title, task description, and functional requirements. Further, there are two explicit framing instructions, differing only in the target language runtime and manifest filename, so that cross-agent comparisons are controlled at the prompt level. The two framing instructions specify that the agent must generate a complete project for a clean environment; the clean environment is language-specific and stated explicitly in every instance. Completeness is treated as an experimental variable instantiated in two prompt variations. In the implicit variant, no completeness specification is given, measuring the agent’s default behavior when a developer is succinct. In the explicit variant, the agent is instructed to produce source code, a dependency manifest covering both direct and transitive dependencies pinned to exact versions, and a README — measuring behavior when a developer is meticulous. These two extremes reflect the observation that real-world developer prompts cluster toward one or the other rather than occupying a graded middle ground. A prompt 𝜋𝑇 is considered valid if the agent produces code that passes an accompanying test suite specified within the functional requirements. This grounds prompt adequacy in an objective, reproducible criterion: passing tests confirm the prompt was

3 of 16

F I G U R E 2 | Environment Evaluation Pipeline. Left: the agent 𝐴 exercises its Gen skill to produce an initial project (𝐶 (0) , 𝑀 (0) ), which is executed in an isolated environment 𝐼. On failure, the Repair skill is invoked with the runtime error 𝑒(𝑖) for up to 𝑘 rounds, yielding the final artifact (𝐶 (𝑘) , 𝑀 (𝑘) ). Right: three extraction primitives operate on the pipeline outputs to produce the three dependency layers: 𝐷 (1) via manifest parsing of the initial declaration 𝑀 (0) ; 𝐷 (2) via package-manager resolution of the final manifest 𝑀 (𝑘) ; and 𝐷 (3) via strace-based runtime tracing of 𝐶 (𝑘) in 𝐼, serving as ground truth.

TASK: {task_title}

3.2.1

{task_description}

A dependency manifest is a language-ecosystem-specific configuration file that formally declares the external packages a software project requires to execute, and serves as the input artifact that package managers read to install and resolve dependencies. In our framework, the initial manifest 𝑀 (0) is the agent’s explicit machine-readable claim of what the code needs in response to 𝜋𝑇 , while the final manifest 𝑀 (𝑘) is the manifest produced after iterative repair. ParseManifest𝓁 comprises parsing rules and takes as input 𝑀 (0) to produce 𝐷 (1) , the declared dependency set. The parsing rules are language-specific and currently support four language ecosystems. For Python, it reads each line of requirements.txt and strips version constraints to normalize package names for crossagent comparison. For Java, it extracts every <artifactId> from the <dependencies> section of pom.xml. For JavaScript, it reads the keys of the "dependencies" object in package.json, deliberately excluding "devDependencies" since build-time tools are outside the runtime supply chain; this exclusion is applied consistently across all agents and tasks. For C++, it collects every library name from FetchContent_Declare and find_package calls in CMakeLists.txt. Note that C++ differs architecturally from the other three languages in that CMake does not have a canonical package manager, which has implications for Resolve𝓁 discussed below. Each rule is a deterministic extraction from a single field of a formally specified file format and involves no heuristic inference; the complete parsing implementation is released in our artifact. Because the same manifests are subsequently consumed by each ecosystem’s package manager in the Resolve𝓁 step, any systematic parsing error would surface as a mismatch between 𝐷 (1) and the resolver’s reported dependency set during execution.

FUNCTIONAL REQUIREMENTS: {functional_requirements} Create a complete {language} project for a clean Ubuntu 22.04 machine with only {language_runtime} installed. Include: – Source code – {dependency_file} with all dependencies (direct and transitive) pinned to exact versions – README.md with setup instructions, dependency explanations, build steps, run commands, and expected output F I G U R E 3 | A prompt template. The template is instantiated with language, manifest filename, and runtime; all functional content is identical across languages.

sufficiently specified, ensuring that any dependency misspecification in the output is attributable to the agent rather than to an underspecified prompt.

3.2

Dependency Extraction Primitives

Amongst the three extraction primitives ParseManifest𝑙 , Resolve𝑙 , and Trace, the first two are language-specific although we describe a generalized specification. The Trace primitive is language-independent, but relies on the clean environment.

4 of 16

Parse Primitive

AI Magazine, 2026

3.2.2

Resolve primitive

The primitive Resolve𝓁 determines 𝐷 (2) , the set of installed dependencies, within the isolated environment 𝐼. The Docker instance that instantiates 𝐼 is built from a base OS image with the language runtime, i.e., compiler, interpreter, or virtual machine, and the language-specific package manager. No additional packages are pre-installed beyond this toolchain baseline. Within 𝐼, the primitive Resolve𝓁 executes the final manifest 𝑀 (𝑘) , causing the language-specific package manager to install both the dependencies explicitly declared in 𝑀 (𝑘) and any additional transitive dependencies they require. Resolve𝓁 then queries the package manager for the complete list of packages installed in 𝐼, and reports this as 𝐷 (2) : 𝐷 (2) ∶= pkg_list𝑝𝑜𝑠𝑡 (𝐼, 𝑀 (𝑘) ) The correctness of 𝐷 (2) rests on the package manager itself serving as the measurement instrument: the reported set is the ecosystem’s own authoritative enumeration of installed packages, queried inside 𝐼 immediately after installation, rather than an inference made by our tooling. The clean toolchain baseline of 𝐼 is fixed and identical across all runs, so the enumeration reflects only packages installed from 𝑀 (𝑘) . A failed or partial resolution is not silently absorbed: it surfaces as an installation error in the protocol and is returned to Repair as 𝑒(𝑖) . Given the declared dependency set 𝐷 (1) obtained from ParseManifest𝓁 , Resolve𝓁 also measures the transitive overhead introduced by the package manager: ∆ = 𝐷 (2) ⧵ 𝐷 (1) ;

𝜌=

|𝐷 (2) | |𝐷 (1) |

where ∆ is the set of packages installed that the agent did not explicitly declare, and 𝜌 is the inflation ratio measuring the multiplicative factor by which the package manager expands the agent’s declared dependency set. 𝜌 = 1 indicates no transitive resolution; 𝜌 ≫ 1 indicates aggressive transitive expansion. For example, if 𝑀 (𝑘) declares only pandas and the build succeeds on 𝑀 (0) such that 𝑀 (𝑘) = 𝑀 (0) , then pip additionally installs numpy, python-dateutil, pytz, tzdata, and six as transitive dependencies, giving |𝐷 (1) | = 1, |𝐷 (2) | = 6, |∆| = 5, and 𝜌 = 6.0. Java and JavaScript behave analogously and are described in Section 4. C++ is a special case. Since CMake has no canonical package manager, no transitive dependency resolution occurs at the package manager layer. Consequently, Resolve𝓁 for C++ reduces to 𝐷 (2) ∶= 𝐷 (1) , meaning ∆ = ∅ and 𝜌 = 1 by construction rather than by agent behavior. The 𝐷 (1) -vs-𝐷 (2) comparison is therefore structurally uninformative for C++ and is excluded from cross-language analyses of transitive dependency gaps.

3.2.3

Trace primitive

The Trace primitive operates on the final successfully installed program (𝐶 (𝑘) , 𝑀 (𝑘) ) within the isolated environment 𝐼, determining which packages are actually loaded during a specific execution run. Unlike ParseManifest𝓁 and Resolve𝓁 , the Trace method is independent of both the language runtime and the coding agent: It is independent of the language-runtime due to the choice of the tracing method; It is independent of the coding agent as it does not query any agent-produced artifact but

instead observes program behavior directly for used dependencies. It does however require that Resolve𝓁 has succeeded, since a successfully installed program is a precondition for tracing. Trace produces 𝐷 (3) , the set of dependencies necessary and sufficient for the observed execution of 𝐶 (𝑘) in 𝐼. Comparing 𝐷 (3) against 𝐷 (1) and 𝐷 (2) exposes overclaimed dependencies i.e., packages declared or installed but never loaded at runtime. Formally: 𝐷 (1) ⧵ 𝐷 (3) 𝐷

(2)

⧵𝐷

(3)

(declared but unused);

(2)

(installed but unused)

While modern interpreted language IDEs often warn developers of unused declared dependencies, the problem is equally present in compiled languages such as Java and C++, where a package may be linked but never exercised on a given execution path [12]. Trace uses ptrace to intercept all file- and process-related system calls made by the running program and any child processes it spawns. Each dependency file accessed during execution is recorded in a log. Dependency-specific files are then filtered using a dictionary of language-specific installation paths, such as dist-packages for Python and dependency for Java to map accessed file paths to package names, enabling direct comparison with 𝐷 (1) and 𝐷 (2) . Language-specific path-to-package mapping rules are described in Section 4. Listing 1 Example code showing reference execution

int main ( int argc , char * argv []) { if ( std :: string ( argv [1]) == " -- matrix " ) { // uses Eigen Eigen :: MatrixXd m = Eigen :: MatrixXd :: Random (3 , 3) ; } else if ( std :: string ( argv [1]) == " -- json " ) { // uses nlohmann / json nlohmann :: json j = nlohmann :: json :: parse ( " {} " ) ; } }

Two limitations apply to 𝐷 (3) . First, since tracing captures only dependencies loaded during a specific execution run, conditionally imported packages not exercised on the observed execution path are absent from 𝐷 (3) . See example in Listing 1. We assume such conditional dependencies are rare in practice and do not materially affect aggregate agent behavior across tasks. Second, Trace operates at the file access level and therefore cannot observe header-only libraries, which are included at compile time rather than loaded at runtime. Such libraries may appear in 𝐷 (1) or 𝐷 (2) but will be systematically absent from 𝐷 (3) , which is particularly relevant for C++ where header-only libraries are common. We acknowledge this as a scope limitation of dynamic over static analysis.

4

Experiment Setup

In this section, we first instantiate the protocol of Section 2 with tasks, agents, and language ecosystems. We then describe how the agent protocol is conducted via configured primitives of Section 3. The experimental setup of tasks and extracted dependencies will be made available as an open reproducible artifact.

5 of 16

T A B L E 1 | The 50 benchmark programming tasks span eight application domains. Domain

𝑛

Representative tasks

Data Processing

10

Cryptography

7

Image Processing

6

Networking

6

Text Processing

6

Compression

5

Math/Scientific

5

System Utilities

5

CSV statistical analyzer, JSON schema validator, XML-to-JSON converter, YAML config merger AES-256 file encryption, RSA digital signatures, TOTP generator, bcrypt hasher Format converter, watermarking, histogram analyzer, EXIF metadata extractor HTTP downloader with retry, port scanner, DNS resolver, static file server Markdown-to-HTML, spell checker, TFIDF search engine, template engine ZIP archive manager, GZIP tool, file deduplicator, compression benchmark Matrix operations, linear regression, Dijkstra shortest path, FFT analyzer QR code generator/reader, PDF report builder, Merkle tree verifier, cron parser

4.1

Protocol Instantiation and Execution

Programming Tasks 𝑇. We instantiate the task set 𝑇 with 50 benchmark programming tasks spanning eight application domains, as shown in Table 1. Tasks are selected to satisfy the two criteria established in Section 3.1: platform independence, ensuring each task can be implemented identically across all four language ecosystems in ℒ, and self-containment, requiring no external services, APIs, or credentials so that dependency misspecification is attributable to the agent’s Gen and Repair skills rather than environmental inaccessibility. The eight domains are chosen to provide broad coverage of real-world software categories that developers routinely delegate to coding agents, ranging from data processing and cryptography to system utilities and scientific computing. The distribution across domains, which is weighted toward data processing and cryptography reflects the relative frequency of such tasks in open-source coding benchmarks [3, 7]. The 50-task scale is chosen to ensure statistical power for crossagent and cross-language comparisons while remaining feasible for the fully manual execution protocol described below. Language Ecosystems ℒ. The language ecosystem set consists of ℒ = {Python, Java, JavaScript, C++}. These language ecosystems are used for configurations of ParseManifest𝓁 , Resolve𝓁 , and Trace defined in Section 3. These four ecosystems were selected to represent diversity in runtimes (interpreted vs compiled), package manager design (centralized vs decentralized), and transitive resolution behavior (𝜌 ≫ 1 for npm vs 𝜌 ≈ 1 for CMake), enabling cross-ecosystem analysis of the environment specification gap. Coding agents 𝒜. The agent set is 𝒜 = {𝑎1 , 𝑎2 , 𝑎3 }, instantiated as: ∙ 𝑎1 : Claude Code (Anthropic, Claude Opus with extended thinking). ∙ 𝑎2 : Codex (OpenAI, the latest Codex model via GitHub Copilot). ∙ 𝑎3 : Gemini Code Assist (Google, Gemini with the highest reasoning configuration). These three agents are selected representing the current frontier of coding agent capability, covering the three dominant providers,

6 of 16

and are accessed exclusively through their command-line interfaces at default settings. This mirrors the experience of a developer using these agents in practice. All agents receive the same prompt 𝜋𝑇 for each task, with the iteration budget 𝑘 = 10 repair cycles specified in each agent’s configuration file as described in Section 2, yielding a maximum of 11 agent interactions per instance — one Gen call followed by up to 10 Repair calls. Further, agents preserve full history within the context window during the Repair cycle, enabling the agent to reason about the sequence of previously attempted repairs. Trial Structure and Study Scale. Every agent 𝑎 ∈ 𝒜 receives every prompt 𝜋𝑇 for every task-language pair (𝑇, 𝓁) ∈ 𝒯 × ℒ, yielding |𝒯| × |ℒ| × |𝒜| = 50 × 4 × 3 = 600 primary instances. To assess the stochastic variability of the Gen and Repair skills, 𝑎1 is additionally executed for two further independent trials per tasklanguage pair, each initialized from a fresh conversation with no memory of prior attempts, contributing 50 × 4 × 2 = 400 stochastic trials. 𝑎1 is selected for variance analysis as it is the only agent achieving 100% final execution success (Section 6), guaranteeing complete data across all task-language pairs for a statistically meaningful variance analysis. The total study scale is thus 1000 evaluation runs (Table 2). Section 3.1 defines two prompt variations: implicit, in which no completeness specification is provided, and explicit, in which the agent is instructed to produce source code, a pinned dependency manifest, and a README. All 600 primary instances use the explicit variant, so the reported results characterize agent behavior under maximal prompt specificity; this constitutes the agent’s best case, and the misspecification rates reported in Section 6 should therefore be read as lower bounds on what less specific prompting would produce. A systematic comparison against the implicit variant is left to future work.

T A B L E 2 | Study scale. Dimension

Count

Benchmark programming tasks Target language ecosystems LLM coding agents Primary configurations (50×4×3) Stochastic trials (Claude Code, 2 extra × 200) Maximum agent interactions per instance

50 4 3 600 400 11

Total evaluation runs

1000

Isolated Environment 𝐼. The isolated environment 𝐼 is realized as a per-language Docker container built from the ubuntu:22.04 base image and containing only the language toolchain necessary to execute Resolve𝓁 and Trace𝓁 : ∙ 𝐼𝑃𝑦𝑡ℎ𝑜𝑛 (tmlr-python-eval): Python 3.10, pip ∙ 𝐼𝐽𝑎𝑣𝑎 (tmlr-java-eval): JDK 17, Maven 3 ∙ 𝐼𝐽𝑆 (tmlr-js-eval): Node.js 20 LTS, npm ∙ 𝐼𝐶++ (tmlr-cpp-eval-v2): G++ 12, CMake 3.22, make, git, zlib1g-dev, and standard headers No image contains common system development libraries beyond those listed: packages such as OpenSSL, libcurl, or SQLite3 development headers are absent from every container, so any agent assumption of their presence surfaces as an observable failure rather than silently succeeding. The C++ container

AI Magazine, 2026

necessarily includes zlib1g-dev and standard C/C++ headers as these are required by the CMake toolchain; these are recorded in pkg_list𝑝𝑟𝑒 (𝐼) and subtracted from 𝐷 (2) via the baseline procedure defined in Section 3, ensuring they do not contaminate the installed dependency measurement. Each image additionally contains Sciunit [13], whose ptrace-based engine implements the Trace𝓁 primitive inside the same container; Sciunit is part of the fixed baseline, and its own file accesses are excluded from 𝐷 (3) so that the runtime trace reflects only the agent-generated artifact. Manual execution. All 1000 instances were conducted manually, with each instance comprising a multi-turn interaction: the initial Gen prompt followed by up to ten Repair turns, for a maximum of eleven agent interactions per instance (Table 2). This means the following actions were undertaken by hand: (i) the initial prompt and every subsequent repair turn were submitted to each agent’s CLI, (ii) generated artifacts were transferred into the Docker containers hosted on an AWS EC2 instance, (iii) runtime errors were transcribed verbatim into the ongoing agent conversation, and (iv) outcomes were recorded into structured CSV files. Automation was deliberately avoided since in our experience the diversity of build failures across four ecosystems was too broad for reliable scripting, and an automated harness would either mask real failure modes through generic recovery heuristics or accumulate so many special-case rules that the harness itself would become a confounding factor, especially in the evaluation of the Repair skill. Once the manual execution of a project is completed and the necessary dependencies collected, each container is destroyed via docker run --rm and a fresh container is instantiated for the next programming task.

4.2

Primitive Configuration

ParseManifest𝑙 and Resolve𝑙 . Table 3 shows concrete instantiations of ParseManifest𝑙 and Resolve𝑙 per ecosystem. Trace. We realize Trace(𝐶 (𝑘) , 𝐼) using Sciunit [13], compiled from source and installed in every Docker image. Sciunit attaches to the target process via the Linux ptrace API and intercepts all open(), openat(), read(), and execve() system calls, writing the absolute path of each accessed file to a provenance log. READtagged entries are filtered using ecosystem-specific path rules to map accessed file paths to package names, enabling direct comparison with 𝐷 (1) and 𝐷 (2) . A critical implementation challenge arises for Java. LLMgenerated Java projects frequently use the Maven Shade Plugin to produce a fat JAR bundling all dependency classes into a single archive. Under fat JAR invocation, the JVM issues exactly one openat() call for the archive itself, making the provenance log opaque to dependency analysis. Sciunit cannot determine which packages were loaded since all are bundled inside the same file. We refer to this as the fat JAR opacity problem. To resolve it, we replace fat JAR execution with classpath-based execution before tracing: dependency JARs are unpacked as physically separate files via mvn dependency:copy-dependencies, and the program is traced against the unpacked classpath. With each JAR a distinct file, the JVM issues a separate openat() per loaded JAR, which appears individually in the provenance log. We verified

that classpath-based execution produces functionally identical output to fat JAR execution in all traced cases. For C++, the compiled binary is traced directly. At startup, the dynamic linker issues openat() calls for every shared library the binary was linked against. We retain .so entries and exclude the seven base system libraries loaded by every C++ binary regardless of application code: libc.so, libstdc++.so, libm.so, libgcc_s.so, libpthread.so, libdl.so, and librt.so, leaving only application-level libraries in 𝐷 (3) . The header-only library limitation noted in Section 3 is most acute here: nlohmann_json, the most frequently declared C++ dependency across all agents (65 successful manifests), is compiled directly into the binary and produces no .so file, and therefore no provenance log entry. For projects whose declared dependencies are entirely header-only, 𝐷 (3) = ∅ even when 𝐷 (1) is non-empty. We return to this limitation in Section 6.6.

5

Evaluation Dimensions

We evaluate the agent evaluation protocol and the extraction primitives across five dimensions: ∙ Protocol Convergence. The Gen → Repair cycle produces a successful artifact (𝐶 (𝑘) , 𝑀 (𝑘) ) through iterative resolution. We evaluate whether and how quickly an agent converges, i.e., reaches 𝑒(𝑘) = ∅ within budget, and how convergence rate varies across agents, languages, and task domains. This includes measuring both the binary outcome indicating if the cycle converges, and the number of iterations consumed via repair depth 𝑖 ∗ = min{𝑖 ≤ 𝑘 ∶ 𝑒(𝑖) = ∅}. ∙ Environment Specification Quality. The three dependency sets 𝐷 (1) , 𝐷 (2) , and 𝐷 (3) produced by the extraction primitives form the basis for measuring specification quality. We evaluate two complementary measures. First, pairwise comparison of {𝐷 (1) , 𝐷 (2) , 𝐷 (3) } against 𝐷 (3) as ground truth, quantifying the extent of phantom (𝐷 (1) ⧵ 𝐷 (3) ), hidden (𝐷 (3) ⧵ 𝐷 (1) ), bloated (𝐷 (2) ⧵ 𝐷 (3) ), and transitive (∆ = 𝐷 (2) ⧵ 𝐷 (1) ) dependencies, reported as precision, recall, and F1 scores. Second, the direction and magnitude of misspecification i.e., over- versus under-specification, measured via ∆ and 𝜌. We additionally assess stochastic variability within a single agent using the repeated trials, determining whether misspecification is a deterministic property of the agent’s knowledge or a stochastic artifact of generation. measured via ∆ and 𝜌. We additionally assess stochastic variability within a single agent using the repeated trials , determining whether misspecification is a deterministic property of the agent’s knowledge or a stochastic artifact of generation. ∙ Cross-Agent Comparison. Given per-agent specification quality, we evaluate how agents differ from one another for identical task-language pairs. This covers pairwise dependency set agreement across agents and whether differences are systematic or task-dependent, isolating agent-specific behavior from task and language effects. ∙ Cross-Language Variation. We evaluate whether specification quality varies systematically across the four language ecosystems ℒ = {Python, Java, JavaScript, C++}. If agents perform poorly and that poor performance is languagespecific, this evaluation informs whether agents require

7 of 16

T A B L E 3 | Per-ecosystem realization of ParseManif est𝓁 , Resolve𝓁 , and Trace𝓁 for the three dependency layers. Ecosystem

Layer 1: 𝐷 (1) = ParseManif est𝓁 (𝑀 (0) )

Layer 2: 𝐷 (2) = Resolve𝓁 (𝑀 (𝑘) )

Layer 3: 𝐷 (3) = Trace𝓁 (𝐶 (𝑘) , 𝐼)

Python

Parse requirements.txt line by line; strip version constraints <artifactId> values in pom.xml <dependencies> Keys of "dependencies" in package.json

Diff of pip list before vs. after pip install

Syscall READs under dist-packages/ or site-packages/ Syscall READs of target/dependency/*.jar via classpath execution Syscall READs under node_modules/

Java JavaScript C++

FetchContent_Declare and find_package names in CMakeLists.txt

mvn dependency:list artifact IDs (full transitive closure) npm list --all --json; full node_modules/ tree CMake configure log (𝐷 (2) ≈ 𝐷 (1) ; transitive expansion typically absent)

language-specific training or whether general environment specification ability is the primary deficiency. ∙ Task Complexity Sensitivity. We evaluate whether specification quality degrades with task complexity or varies by programming domain. This determines whether agents fail consistently on particular task types, providing diagnostic signal about which categories of software projects are most vulnerable to environment specification failure.

6

Evaluation Results

We present our results across the five evaluation dimensions of Section 5. The building blocks are already in place: the dependency sets 𝐷 (1) , 𝐷 (2) , 𝐷 (3) come from Section 2, and the inflation ratio 𝜌 from Section 3. Each subsection below only adds the small amount of extra notation it needs. We fix three conventions used throughout. First, subscripts iden(𝑗) tify the setting: 𝐷𝑎,𝑝,𝓁 is the dependency set of agent 𝑎 on task 𝑝 in language 𝓁. Second, a project counts as successful when its final artifact (𝐶 (𝑘) , 𝑀 (𝑘) ) runs without error in the clean environment 𝒟𝓁 within the repair budget; we write Success(𝐶, 𝑀, 𝒟𝓁 ) for this condition and 𝒮𝑎,𝓁 for the set of tasks where agent 𝑎 succeeds in language 𝓁. Third, unless stated otherwise, every reported number is a mean over successful projects.

6.1

Protocol Convergence

Figures 4–6 examine the agents for protocol convergence. We first measure the performance of the Gen skill, i.e., obtaining a successful software artifact for a programming task in the first attempt. We measure this across the 600 instances, ordered first by agent and then by language, so that each cell in Figure 4 represents one (agent, language, task) triple. Dense green regions show that Claude and Codex have a strong Gen skill in Python and JavaScript but a much weaker one in Java and C++, while Gemini inverts the pattern: it is weakest in Python (32% first-attempt success) yet the strongest of the three agents in C++ (80%). No agent is strong in all four languages, and the failure patterns differ across agents, motivating a closer examination of the Repair skill. Figure 5 shows how much of the Repair skill is exercised per agent. For each (agent, language) pair, it reports the fraction of tasks resolved on the first attempt and the fraction requiring 𝑘 repair cycles, with 𝑘 bucketed into the ranges [1, 3], [4, 7], and [8, 10]; the final category marks tasks that failed within the repair budget. The vast majority of repairs complete within 1–3 cycles in every ecosystem. Deep repair chains of four or more cycles are rare and appear almost exclusively in C++. Claude repairs every

8 of 16

Syscall READs of .so files; seven base system libraries excluded

one of its failed first attempts in all four languages, whereas Codex recovers 89% of its failed C++ attempts and Gemini recovers 92% in Java and 90% in C++; outright failures are therefore confined to Codex and Gemini in C++ and to Gemini in Java. Figure 6 aggregates these results by language. To aggregate, we define a success rate at iteration 𝑖 for agent 𝑎 and language 𝓁 as (𝑖)

SR 𝑎,𝓁 =

( )] 1 ∑ [ (𝑖) (𝑖) 𝟏 Success 𝐶𝑎,𝑝,𝓁 , 𝑀𝑎,𝑝,𝓁 , 𝒟𝓁 , |𝒫| 𝑝 ∈ 𝒫

(3)

(0)

so that SR 𝑎,𝓁 is the first-attempt success rate, measuring the Gen (≤𝑘)

(𝑖)

skill alone, and SR 𝑎,𝓁 = max 𝑖≤𝑘 SR 𝑎,𝓁 is the final success rate (≤𝑘)

after up to 𝑘 repair cycles. The self-correction gain SR 𝑎,𝓁 − (0)

SR 𝑎,𝓁 measures the contribution of the Repair skill to eventual convergence. Figure 6 shows solid bars indicating SR (0) and hatched bars indicating the self-correction gain for each agent and language. First-attempt success varies widely: Claude ranges from 96% in JavaScript and 86% in Python down to 36% in C++ and only 18% in Java, Codex spans 44% (C++) to 88% (Python and JavaScript), and Gemini runs from 32% (Python) to 84% (JavaScript). After repair, nearly every configuration converges: Claude reaches 100% in all four ecosystems, Codex falls short only in C++ (94%), and Gemini falls slightly short in Java (96%) and C++ (98%). The Repair skill thus rescues most first-attempt failures, but the size of the hatched region shows how much iterative feedback each configuration consumes to get there. The corresponding analysis by task domain is deferred to Section 6.5. These results carry a direct implication for how coding agents are evaluated. A benchmark that scores only final success would rate all three agents as near-perfect, yet the cost of reaching that success differs by up to a factor of five in repair cycles, and every repair cycle is additional latency, compute, and opportunity for the agent to mutate the dependency set. First-attempt environment specification is therefore a hidden cost driver that functional-correctness benchmarks do not surface, and the first manifest, not the final one, determines how much of that cost a deployment pays.

6.2

Environment Specification Quality

All measures in 4 are aggregated as means per stratum (𝑎, 𝓁) and per ecosystem 𝓁 pooled over agents. Because F1 is computed per project and then averaged, the tabulated F1 need not equal the harmonic mean of the tabulated precision and recall; the discrepancy is largest in C++, where many projects have zero declared–runtime overlap.

AI Magazine, 2026

Gen Skill — First-Attempt Success Across 600 Instances Python

Java

JavaScript

C++

SR0=86%

SR0=18%

SR0=96%

SR0=36%

Python

Java

JavaScript

C++

SR0=88%

SR0=68%

SR0=88%

SR0=44%

Python

Java

JavaScript

C++

Gen success (k = 0) Gen failure (k ≥ 1)

Claude Code

Codex

Gemini 0

SR0=32%

50

100

SR0=50%

SR0=84%

150

200

SR0=80%

Instance index (50 tasks × 4 languages = 200 per agent)

F I G U R E 4 | Gen skill: first-attempt success across all 600 instances (3 agents × 4 languages × 50 tasks). Each cell is one (agent, language, task) triple; green indicates first-attempt success and red indicates failure.

Repair Skill — Iterations k to Convergence Python

Java

JavaScript

C++

100%

100%

100%

100%

100%

100%

100%

89%

Claude Code

Codex

Gemini

0

100%

92% 100% 50 100 150 Instance index (50 tasks × 4 languages = 200 per agent) · % = share of failed-Gen tasks repaired No repair needed (k = 0) k = 1–3 k = 4–7 k = 8–10 No conv. (k > 10)

200

90%

F I G U R E 5 | Repair skill: distribution of repair cycles needed to converge for each (agent, language) pair, bucketed into first-attempt success, 1–3, 4–7, and 8–10 cycles, and failure within the budget.

Protocol Convergence by Language Claude Code Codex Gemini

100%

Manifest Misspecification — Phantom, Hidden, and Bloat Rates

100%

SR0 (first attempt)

Phantom (ϕ)

Repair gain

80%

Hidden (η) Bloat (β)

Mean Rate

Success Rate

80%

60%

40%

40% 20%

20%

0%

60%

0%

Python

Java JavaScript Programming Language

C++

F I G U R E 6 | Protocol convergence by language. Solid bars show the first-attempt success rate SR (0) and hatched extensions show the selfcorrection gain contributed by Repair.

We measure environment specification quality in two complementary ways, building directly on the three-layer framework

Python

Java JavaScript Programming Language

C++

Rates pooled over agents on successful projects; zero-denominator cases counted as zero

F I G U R E 7 | Phantom, hidden, and installed-but-unused bloat rates by ecosystem, averaged over successful projects with zero-denominator cases included as zero.

of Section 2. First, we operationalize the phantom, hidden, and bloat categories of Section 2 as rates, together with the inflation

9 of 16

Environment Growth — Transitive Inflation by Ecosystem

Inflation Ratio ρ

50× 20× med = 8.5

10× 5× med = 1.7

med = 1.5

2×

med = 1

1× Python

Java JavaScript Programming Language

C++

ρ = |D (2)| / |D (1)| per successful project · log scale

F I G U R E 8 | Initial-to-final environment growth by ecosystem. Each box summarizes the distribution of the inflation ratio 𝜌 of Section 3 over all successful projects in that ecosystem (log scale).

T A B L E 4 | Initial-manifest accuracy: treating runtime dependencies 𝐷 (3) as ground truth. Agent

Language

Precision

Recall

F1

Claude Claude Claude Claude Codex Codex Codex Codex Gemini Gemini Gemini Gemini

Python Java JavaScript C++ Python Java JavaScript C++ Python Java JavaScript C++

0.990 0.935 0.929 0.037 0.938 0.939 0.214 0.062 0.899 0.947 0.040 0.059

0.642 0.739 0.406 0.781 0.827 0.684 0.143 0.658 0.721 0.523 0.182 0.708

0.733 0.758 0.493 0.031 0.837 0.735 0.143 0.004 0.756 0.624 0.005 0.004

All All All All

Python Java JavaScript C++

0.944 0.940 0.413 0.049

0.713 0.653 0.258 0.729

0.765 0.708 0.219 0.018

ratio 𝜌 of Section 3. These metrics characterize the nature and magnitude of misspecification independent of the agent, revealing which language ecosystems are structurally harder to specify. Second, retrieval measures of precision, recall, and F1 consider 𝐷 (3) as the ground truth and characterize per-agent accuracy, enabling the direct cross-agent comparison of Section 6.3. (𝑗) For a successful project (𝑎, 𝑝, 𝓁), let 𝑛𝑗 = |𝐷𝑎,𝑝,𝓁 | for 𝑗 ∈ {1, 2, 3}. The three mismatch rates are

𝜙=

|𝐷 (1) ⧵𝐷 (3) | 𝑛1

, 𝜂=

|𝐷 (3) ⧵𝐷 (1) | 𝑛3

, 𝛽=

|𝐷 (2) ⧵𝐷 (3) | 𝑛2

,

(4)

with each rate set to zero when its denominator is zero. The phantom rate 𝜙 measures over-specification: packages declared but never loaded at runtime. The hidden rate 𝜂 measures underspecification: packages loaded at runtime but absent from the initial manifest. The bloat rate 𝛽 measures transitive excess: packages installed by the package manager but never accessed during execution. Treating 𝐷 (1) as a retrieval system predicting 𝐷 (3) , per-project precision, recall, and F1 are |𝐷 (1) ∩ 𝐷 (3) | |𝐷 (1) ∩ 𝐷 (3) | 2PR P= , R= , F1 = , 𝑛1 𝑛3 P+R

10 of 16

(5)

with the conventions P = 1 when 𝑛1 = 0 and 𝑛3 > 0, R = 1 when 𝑛3 = 0 and 𝑛1 > 0, and projects with 𝑛1 = 𝑛3 = 0 excluded entirely. Precision and recall jointly characterize the direction of agent error: high precision but low recall indicates systematic under-specification, while the reverse indicates systematic overspecification. All measures are aggregated as means per stratum (𝑎, 𝓁) and per ecosystem 𝓁 pooled over agents. Figure 7 shows the character of misspecification per ecosystem. Python and Java have relatively low phantom rates (about 5%), but both still show meaningful hidden-dependency rates because transitive packages become active at runtime without being named in the initial manifest. JavaScript is much worse on all three dimensions: about 49% of declared packages are phantom, about 62% of runtime packages are hidden, and about 22% of installed packages are never touched. C++ is dominated by a different pathology: many dependencies are compile-time or header-only and therefore disappear from the runtime trace, inflating phantom and bloat rates under a runtime-ground-truth definition. Figure 8 quantifies how large the eventual installed environment becomes relative to what the agent initially declared, using the inflation ratio 𝜌 defined in Section 3. JavaScript shows by far the largest environment growth, with 𝜌̄ JavaScript = 11.40 and a median of 8.5, meaning that small initial declarations routinely lead to much larger installed environments. Java also shows heavy inflation (𝜌̄ Java = 3.97), while Python is noticeably milder (𝜌̄ Python = 2.06). C++ remains near 𝜌 = 1.0, which matches its non-centralized dependency workflow (Section 3), with only minor deviations in rare fetched-project cases. The figure shows that the size of the initial declared dependency set is a poor proxy for actual footprint in resolver-heavy ecosystems. Table 4 reports the retrieval measures. Python achieves the strongest overall initial-manifest accuracy with macro F1 of 0.765, followed by Java at 0.708. JavaScript drops sharply to 0.219 because both over-declaration and under-declaration are common. C++ falls to 0.018, but that value should be interpreted together with the methodological limitation that many C++ libraries leave little or no runtime footprint after compilation (Section 6.6). The impact of these mismatch rates extends beyond code quality. A dependency manifest is the primary input to software bills of materials, vulnerability scans, and license audits; when nearly half of a JavaScript manifest is phantom and nearly two thirds of what actually runs is hidden, every downstream audit inherits that inaccuracy. Phantom declarations inflate the apparent attack surface and generate false positives in security review, while hidden dependencies are worse: they are runtime-active code that no scan ever examines. At the accuracy levels of Table 4, an agent-produced manifest cannot be treated as a trustworthy audit artifact in JavaScript or C++ without independent runtime verification.

6.3

Cross- and Intra-Agent Comparison

We now measure cross-agent and intra-agent agreement. For both assessments, we define the pairwise declared-dependency agreement between agents 𝑎𝑖 and 𝑎𝑗 on task 𝑝 in language 𝓁 as

𝐽(𝑎𝑖 , 𝑎𝑗 , 𝑝, 𝓁) =

(1)

(1)

(1)

(1)

|𝐷𝑎𝑖 ∩ 𝐷𝑎𝑗 |

,

(6)

|𝐷𝑎𝑖 ∪ 𝐷𝑎𝑗 |

AI Magazine, 2026

Cross-Agent Agreement — Declared Dependencies

Mean Jaccard on D (1)

0.6

Claude–Codex Claude–Gemini half of packages shared

0.5

Codex–Gemini

T A B L E 6 | Environment Gap: C++ system library assumption failures and recovery. “Fail” counts first-attempt failures; “SysLib” counts those whose terminal error is a missing system library; “Rec.” counts SysLib failures eventually repaired.

0.4

Agent

Fail SysLib

SLAR

Rec.

EGAR

Claude Codex Gemini

32 28 10

87.5% 17.9% 0.0%

28 5 0

100.0% 100.0% –

0.3 0.2 0.07

0.1 0.0

Python

Java JavaScript Programming Language

28 5 0

C++

Pairwise Jaccard averaged over tasks where both agents succeed

F I G U R E 9 | Cross-agent agreement on declared dependencies. Each bar reports the mean Jaccard similarity between the declared dependency sets extracted from two agents’ initial manifests for the same task in the same ecosystem; the dotted line marks the level at which half of the declared packages would be shared.

T A B L E 5 | Stochastic variability: Claude’s three independent trials produce near-zero agreement. Language

Mean 𝐽̄intra

Median

UCR

𝑉̄ ∪

𝑉̄ ∩

Python Java JavaScript C++

0.113 0.138 0.068 0.093

0.083 0.111 0.000 0.067

0/50 (0.0%) 0/48 (0.0%) 0/50 (0.0%) 0/35 (0.0%)

5.5 5.6 5.9 5.3

0.2 0.1 0.0 0.1

with 𝐽 = 1 when both sets are empty and 𝐽 = 0 when exactly one is. The mean cross-agent Jaccard 𝐽̄𝓁 and mean intra-agent Jaccard 𝐽̄intra are averages of 𝐽 over all valid agent pairs and, for the intra-agent case, over the three independent Claude Code trials of Section 4. The complete disjointness rate CDR 𝓁 is the fraction of pairs for which 𝐽 = 0, and the unanimous consensus rate UCR 𝓁 is the fraction of tasks for which all three Claude trials produce an identical 𝐷 (1) . Figure 9 shows that agents consistently solve the same programming task with substantially different declared dependency sets. Python is the most convergent ecosystem, yet even there mean pairwise Jaccard similarity remains below 0.5 across all agent pairs, indicating that fewer than half of declared packages are shared on average. JavaScript is the most divergent, with Codex and Gemini agreeing on as little as 𝐽̄ = 0.073, meaning the two agents share fewer than one in thirteen declared packages for the same task. The complete disjointness rate is highest in JavaScript (63.3%), followed by C++ (57.7%) and Java (56.2%), with Python lowest at 36.0%, yielding a macro-average CDR of 53.3% across the four ecosystems. Complete disagreement between agents is therefore not an edge case but the majority outcome. Table 5 asks whether this divergence is vendor-specific or whether it reflects a deeper instability, by repeating Claude Code three independent times on the same tasks. The unanimous consensus rate is 0% in every language, meaning Claude never converged to the same dependency declaration across all three trials for any task in any ecosystem. Intra-agent Jaccard scores are low throughout, with JavaScript again the worst case. Crucially, the mean vocabulary union 𝑉̄ ∪ substantially exceeds the mean unanimous core 𝑉̄ ∩ in every language, confirming that the agent draws from a much larger pool of plausible packages than it consistently commits to. Taken together, the figure and table show

that dependency selection is neither reproducible across vendors nor stable within a single model, a result that implicates the generative process itself rather than any vendor-specific design choice, and suggests that dependency specification is not retrieved from a stable internal knowledge structure but sampled stochastically on each generation. The practical consequence is that agent-generated software is not reproducible at the supply-chain level: two developers prompting the same agent with the same task, or one developer prompting it twice, receive different dependency trees, so every fresh generation effectively produces a different supply chain. Dependency review, pinning decisions, and security approvals performed on one generation do not transfer to the next. Because the instability persists within a single vendor, it also cannot be mitigated by standardizing on one agent; it must be addressed in the generation process itself.

6.4

Cross-Language Variation

The preceding results consistently separate the four ecosystems, and we now draw the language-level picture together. Python is the easiest ecosystem to specify: it combines the highest firstattempt success, the lowest phantom rates, mild environment inflation (𝜌̄ Python = 2.06), and the best initial-manifest accuracy (macro F1 of 0.765). Java sits in the middle: first-attempt success is weaker, but its declaration quality remains high (macro F1 of 0.708) despite heavy transitive inflation. JavaScript is the pathological case on every specification metric: the largest inflation (𝜌̄ = 11.40), the highest phantom and hidden rates, the lowest manifest accuracy (0.219), and the weakest cross- and intra-agent agreement. Poor specification quality is therefore not merely an agent property; it interacts strongly with how aggressively an ecosystem’s package manager expands and obscures the declared set. C++ varies along a different axis: agents fail not because the resolver inflates their declarations, but because they assume system development libraries that the clean container does not provide. For every C++ project that fails at round 𝑖 = 0, we classify the terminal error 𝑒(0) into one of three types: SystemLib, a find_package failure for a system library absent from 𝒟𝓁 (e.g., OpenSSL, libcurl, SQLite3); FetchFail, a FetchContent_Declare network or archive error; and Other, covering compilation, missing-header, linker, or runtime failures unrelated to system library absence. From this taxonomy we report two rates per agent: the system library assumption rate (SLAR), the fraction of an agent’s first-attempt C++ failures whose terminal error is of type SystemLib, and the Environment Gap recovery rate (EGAR), the fraction of those SystemLib failures that the agent eventually repairs within the budget.

11 of 16

Success Rate (pooled across languages)

Protocol Convergence by Task Domain (pooled languages) SR0 (first attempt)

Claude Code Codex Gemini

100%

Repair gain

80%

60%

40%

20%

0%

Data

Pro

c.

y

raph

ptog Cry

ge Ima

c. Pro

Netw

g orkin

Text

c. Pro

ion

press

Com

/Sci

Math

Sys

Utility

Task Domain

F I G U R E 10 | Protocol convergence by task domain, with solid and hatched bars as in Figure 6.

Table 6 reports the clearest C++ manifestation of the environment gap. Claude’s initial C++ failures are overwhelmingly due to incorrect system-library assumptions (SLAR of 87.5%), and every one of those failures is eventually repaired after feedback. Codex shows a lower but still meaningful assumption rate (17.9%), again with perfect recovery once the missing package is revealed. Gemini’s C++ failures arise from other causes rather than missing system libraries. The table shows that the clean Docker image is not just making the benchmark harder; it is exposing a concrete mismatch between the environments agents implicitly assume and the environments they are actually asked to run in. Together, these ecosystem differences imply that mitigation cannot be uniform. JavaScript projects need the strongest specification guardrails, because there the manifest is least trustworthy precisely where the resolver amplifies it most. C++ points to a different remedy: since nearly all system-library assumption failures are repaired once the environment reports the missing package, feedback from the actual deployment target, rather than a richer assumed environment, is the effective corrective. Code that runs on a developer’s fully provisioned machine will still fail in the minimal containers that production systems favor, and the environment gap measures exactly that deployment risk.

6.5

Task Complexity Sensitivity

Finally, we ask whether specification quality depends on what kind of software is being built. Figure 10 repeats the convergence analysis of Section 6.1 per task domain. Final success is at or near 100% in every domain (never below 95% for any agent– domain pair), and no domain is intrinsically hard for all agents: first-attempt success spans 40%–88% across domain–agent pairs, but the weakest domain differs per agent — System Utilities for Claude (40%), Image Processing for Codex (50%), and Data Processing for Gemini (48%), which is simultaneously the strongest domain for the other two agents. Convergence behavior is therefore driven primarily by language ecosystem rather than by task domain: C++ is hard for every agent regardless of what the task computes. A sharper test of task sensitivity isolates the tasks that need no external libraries at all. Following Section 4, a subset of benchmark tasks in each language is identified a priori by the authors as entirely solvable with the standard library (e.g., Python’s csv, hashlib, and json modules; Java’s java.util and java.security packages; Node’s fs, crypto, and path builtins). The unnecessary dependency rate (UDR) is the fraction of

12 of 16

these standard-library-sufficient tasks for which the agent nevertheless declared at least one external package. On these standard-library-sufficient tasks, Claude introduced external packages in 91.8% of these tasks on macro average, reaching 100% in C++ and remaining at or above 84% in every other ecosystem. Gemini is similarly aggressive in JavaScript. Codex is much more conservative in Python and JavaScript, though it still adds unnecessary libraries in more than half of eligible C++ tasks. This result isolates over-dependence from task difficulty: the extra libraries are not required by the benchmark itself. Each unnecessary package is avoidable attack surface, maintenance burden, and audit scope added for zero functional gain, and on standard-library-sufficient tasks that cost is pure waste. Because the behavior persists on tasks where no library is needed, it reflects a generation prior rather than a response to task demands, which makes it a natural target for training objectives that reward minimal, sufficient environment specification alongside functional correctness.

6.6

Limitations

The strongest measurement limitation is in C++. Runtime tracing undercounts header-only and purely compile-time dependencies, which pushes C++ manifest precision and F1 downward relative to ecosystems where runtime loading leaves a clearer file-level footprint. Similarly, although C++ usually has |𝐷 (2) | ≈ |𝐷 (1) |, a small number of fetched projects recursively pull an additional library during their own build process, so exact equality should be interpreted as an ecosystem-level trend rather than as a strict invariant. More broadly, the study uses a controlled benchmark of 50 tasks rather than fully open-ended real-world projects, so the reported rates should be interpreted as evidence about structured code-generation settings rather than as exhaustive estimates for all software development contexts. All results are additionally conditioned on the prompt template of Section 3.1; because the template is held identical across agents and languages, comparative findings are controlled at the prompt level, but absolute rates may vary under different phrasings, and a systematic prompt-robustness analysis remains future work. The manual evaluation protocol and fixed annotations for standardlibrary tasks and C++ failure labels improve control, but they also introduce a degree of judgment that future larger-scale replications should further test. Finally, because commercial coding agents evolve rapidly, the results should be understood as a snapshot of the evaluated agent configurations at evaluation time rather than as immutable model properties.

7

Related Work

LLM Code Generation and Its Evaluation Gaps. The era of LLM code generation began with Codex [3], which solved 28.8% of HumanEval problems. Subsequent models such as AlphaCode [8], StarCoder [14], Code Llama [15], and commercial agents like Claude Code [1], GitHub Copilot, and Gemini Code Assist [2] have expanded from single-function completion to full project synthesis. Surveys by Fan et al. [6] and Jiang et al. [5] cover this landscape comprehensively.

AI Magazine, 2026

The critical gap in this literature is that nearly all benchmarks evaluate functional correctness of isolated snippets [3, 7]. A smaller body of work examines code quality [16], bug patterns [17, 18], and security vulnerabilities [4]. Ouyang et al. [19] studied non-determinism in ChatGPT’s code output but restricted their analysis to functional behavior; they did not examine whether dependency choices are stable across runs. Our stochastic analysis fills this gap directly: we show that a single agent’s dependency declarations are unstable across independent runs of the same task, with unanimous consensus rate zero across all tasks and languages — a form of non-determinism invisible to functional correctness metrics. Recent work has identified package hallucinations, where models recommend packages that do not exist in registries, linked to practical supply-chain attacks [20]. Our work is complementary but distinct: we study the misspecification of existing packages, over-declaration, under-declaration, and transitive blindness, which is a different and broader failure mode that persists even when every named package exists. None of this work examines dependency management as a first-class evaluation concern across the full lifecycle from declaration through resolution to runtime execution. Our own earlier study [11] provided initial evidence of dependency gaps in agent-generated code; the present article formalizes the evaluation protocol underlying that study and extends the analysis. Dependency Ecosystems and Their Distinct Challenges. Software dependency management is not a uniform problem: each ecosystem has its own resolution semantics, transitive behavior, and failure modes, and these differences matter profoundly for LLM-generated code. A key contribution of this paper is measuring these ecosystem-specific failure modes under a unified framework, rather than studying any single ecosystem in isolation. We breifly review relevant works within each ecosystem: JavaScript/npm.. Zimmermann et al. [21] found that installing a median npm package transitively imports 79 packages maintained by 39 individuals. The npm ecosystem’s deeply nested dependency graphs mean that a single declared package can pull in dozens or even hundreds of transitive dependencies, creating a vast and largely invisible attack surface. Decan et al. [9] confirmed that npm exhibits the most aggressive growth and fragility among seven ecosystems studied. Our results quantify this aggression in the context of agent-generated code specifically: we observe a mean inflation ratio 𝜌̄ = 11.40 for JavaScript, with individual projects expanding a three-package declaration into over 140 installed packages, a transitive footprint no agent explicitly reasoned about. Java/Maven.. Soto-Valero et al. [22] found that 75.1% of Maven Central artifacts contain at least one bloated dependency — a library declared in the build file but never used at runtime. Maven’s transitive resolution is opaque: a developer declares a handful of dependencies in pom.xml, and Maven silently resolves an entire tree of transitive artifacts. Wang et al. [10] further documented the risks of third-party library management in Java projects. Whereas these studies measure bloat in human-authored artifacts, we measure it in agent-generated code where the selection decisions are made by a model rather than a developer, finding that bloat is pervasive and that agents show no awareness of the transitive footprint their declarations induce.

C++/CMake.. C++ dependency management is fundamentally different from the other three ecosystems. There is no single dominant package manager; dependencies are handled through CMake’s FetchContent, find_package, or manual vendoring. The key issue for LLM-generated code is the system library assumption gap: agents trained on code from rich development environments assume that system libraries such as libssl-dev or libopencv-dev are already installed. We provide the first systematic measurement of this assumption gap across multiple agents, showing that it accounts for the majority of C++ build failures and that its prevalence differs substantially across agents — implicating training distribution rather than task difficulty as the primary driver. Python/pip.. Python’s ecosystem is comparatively wellbehaved for dependency management. The pip freeze convention produces exact version pins, and transitive resolution is relatively flat. However, system library issues still arise for packages with native extensions — pyzbar requires libzbar0 and opencv-python requires libGL.so — and our results show that even in this most tractable ecosystem, agents achieve macro F1 of only 0.765 against runtime ground truth, leaving meaningful specification gaps.

Cross-ecosystem analysis. Kikas et al. [23] studied the structural evolution of dependency networks in JavaScript, Ruby, and Rust. Cox [24] articulated the fundamental tension between code reuse and dependency risk. Supply-chain attacks exploiting dependencies are a major concern [25, 26], prompting government mandates for Software Bills of Materials [27, 28]. Our paper provides the first cross-ecosystem comparison of dependency specification quality in agent-generated code, using a unified three-layer measurement framework that enables direct comparison across Python, Java, JavaScript, and C++ rather than treating each ecosystem as an isolated problem. Software Bloat and Unused Dependencies. Bruce et al. [29] found 14.8% of compiled Java methods unreachable. Quach et al. [12] proposed piece-wise compilation for C/C++ attack surface reduction. Soto-Valero et al. [30] showed that most transitive Maven dependencies have very low runtime utilization. These studies measure bloat in human-authored artifacts using static or coverage-based analysis. Our study is the first to measure dependency bloat in agent-generated code, where the selection decisions are made by a model rather than a developer, and to use dynamic runtime tracing as the ground truth rather than static reachability, capturing the actual runtime footprint rather than an approximation of it. Static vs Dynamic Dependency Analysis. Tools for automatically inferring dependencies from source code, such as pipreqs for Python, jdeps for Java, and import-graph analyzers more broadly, represent the non-agent baseline for dependency specification. These tools use static analysis, which has known limitations: they cannot capture dynamically imported packages, conditionally loaded modules, or runtime-only dependencies exercised on specific execution paths. Dynamic tracing via system call interposition addresses these limitations at the cost of execution-path specificity — capturing only dependencies exercised during a particular run. We use dynamic tracing via ptrace-based provenance capture [31, 32] as our runtime ground truth precisely because it captures what the program actually loads.

13 of 16

Software Bills of Materials and Manifest Accuracy. Regulatory mandates for Software Bills of Materials [27, 28] have prompted a growing literature on SBOM generation accuracy and completeness [33, 34]. Existing SBOM generation tools operate on human-authored codebases; the accuracy of agent-generated manifests as SBOMs has not been studied. Our three-layer framework is directly applicable to this question: the declared dependency set 𝐷 (1) is precisely what an agent-generated SBOM would contain, and our precision, recall, and F1 measurements against runtime ground truth 𝐷 (3) provide the first empirical characterization of agent-generated SBOM accuracy. The finding that macro F1 ranges from 0.765 for Python to 0.018 for C++ has direct implications for the trustworthiness of agent-generated SBOMs in security-critical contexts. Computational Reproducibility and Provenance. Reproducibility remains a persistent challenge in computational science [35, 36]. Docker containers [37] capture static environment snapshots but cannot reveal what a program actually loads at runtime — a limitation that makes them insufficient as a ground truth for dependency analysis. Provenance-based systems go further. ReproZip [31] traces system calls to build reproducible archives. Sciunit [13] uses its PTU engine building on CDE’s system call interposition approach [38] to intercept open(), openat(), read(), and execve() via ptrace, recording every file accessed during execution. We use Sciunit as our runtime dependency oracle, extending its application from reproducibility packaging to dependency ground-truth extraction in the context of agent-generated code evaluation. LLM Evaluation Methodology. Beyond code generation specifically, our work contributes to the broader literature on LLM evaluation methodology. Holistic evaluation frameworks such as HELM [39] argue that model capabilities should be measured across multiple dimensions simultaneously rather than through single-metric proxies. Our three-layer dependency framework instantiates this principle for the code generation setting: rather than collapsing environment specification quality into a single pass/fail execution outcome, we decompose it into declaration accuracy, resolution behavior, and runtime footprint, each of which implicates a different failure mode and a different remediation. We argue that environment specification should be incorporated as a standard evaluation dimension in code generation benchmarks alongside functional correctness, and that the metrics introduced here - phantom rate, hidden rate, bloat rate, and manifest F1 - provide a reusable evaluation vocabulary for this dimension.

8

Conclusion

This study shows that dependency generation by current LLM coding agents is highly inconsistent and often misaligned with the environments their code actually requires. Although these agents can frequently repair projects to functional success, they do not reliably produce correct or minimal dependency specifications. Our results directly answer the central research question: current LLMs systematically misspecify software environment dependencies, and this misspecification varies substantially across ecosystems and across agents. The problem appears both as under-specification, where required dependencies are missing

14 of 16

from the manifest, and as over-specification, where unnecessary libraries inflate the software environment. These findings have immediate implications for both research and practice. Successful code generation should not be treated as evidence of environment correctness without additional validation. More reliable and portable code generation will require evaluation frameworks, validation pipelines, and training objectives that explicitly target accurate and minimal dependency specification alongside task completion. One concrete direction is to treat dependency specification as a first-class learning target: future systems could incorporate environment-grounded fine-tuning, dependency-aware reward signals, or manifest-verification steps in iterative generation and repair loops so that dependency correctness is optimized alongside functional task completion.

References 1. Anthropic 2024. “Claude Code: Agentic Coding Tool” https:// www.anthropic.com/claude-code, Accessed: 2024. 2. Gemini Team, Google. “Gemini: A Family of Highly Capable Multimodal Models”. arXiv preprint arXiv:2312.11805 (2023). 3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. “Evaluating Large Language Models Trained on Code”. arXiv preprint arXiv:2107.03374 (2021). 4. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. “Asleep at the Keyboard? Assessing the Security of GitHub Copilot’s Code Contributions” In Proceedings of the 2022 IEEE Symposium on Security and Privacy (SP), 754–768. IEEE. 5. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. “A Survey on Large Language Models for Code Generation”. arXiv preprint arXiv:2406.00515 (2024). Accepted to ACM TOSEM 2025; doi:10.1145/3747588. 6. Angela Fan, Beliz Gokkaya, Mark Harman, Mitya Lyubarskiy, Shubho Sengupta, Shin Yoo, and Jie M. Zhang. “Large Language Models for Software Engineering: Survey and Open Problems”. arXiv preprint arXiv:2310.03533 (2023). 7. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. “Program Synthesis with Large Language Models”. arXiv preprint arXiv:2108.07732 (2021). 8. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, PoSen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel J. Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. “Competition-Level Code Generation with AlphaCode”. Science 378, no. 6624 (2022): 1092–1097. 9. Alexandre Decan, Tom Mens, and Philippe Grosjean. “An Empirical Comparison of Dependency Network Evolution in Seven

AI Magazine, 2026

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

Software Packaging Ecosystems”. Empirical Software Engineering 24, no. 1 (2019): 381–416. Ying Wang, Bihuan Chen, Kaifeng Huang, Bowen Shi, Congying Xu, Xin Peng, Yang Liu, and Yijian Wu. 2020. “An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projects” In Proceedings of the 2020 IEEE International Conference on Software Maintenance and Evolution (ICSME), 35–45. IEEE. Bhanu Prakash Vangala, Ali Adibifar, Ashish Gehani, and Tanu Malik. 2026. “AI-Generated Code Is Not Reproducible (Yet): An Empirical Study of Dependency Gaps in LLM-Based Coding Agents” , . Anh Quach, Aravind Prakash, and Lok Yan. 2018. “Debloating Software through Piece-Wise Compilation and Loading” In Proceedings of the 27th USENIX Security Symposium (USENIX Security ’18), 869–886. USENIX Association. Quan Pham, Tanu Malik, and Ian Foster. 2013. “Using Provenance for Repeatability” In Proceedings of the 5th USENIX Workshop on the Theory and Practice of Provenance (TaPP ’13), Lombard, IL: USENIX Association. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, Oleh Shliazhko, Nicolas Gontier, Nicholas Meade, Armel Zebaze, Ming-Ho Yee, Logesh Kumar Umapathi, Jian Zhu, Benjamin Lipkin, Muhtasham Oblokulov, Zhiruo Wang, Rudra Murthy, Jason Stillerman, Siva Sankalp Patel, Dmitry Abulkhanov, Marco Zocca, Manan Dey, Zhihan Zhang, Nour Fahmy, Urvashi Bhattacharyya, Wenhao Yu, Swayam Singh, Sasha Luccioni, Paulo Villegas, Maxim Kunakov, Fedor Zhdanov, Manuel Romero, Tony Lee, Nadav Timor, Jennifer Ding, Claire Schlesinger, Hailey Schoelkopf, Jan Ebert, Tri Dao, Mayank Mishra, Alex Gu, Jennifer Robinson, Carolyn Jane Anderson, Brendan Dolan-Gavitt, Danish Contractor, Siva Reddy, Daniel Fried, Dzmitry Bahdanau, Yacine Jernite, Carlos Muñoz Ferrandis, Sean Hughes, Thomas Wolf, Arjun Guha, Leandro von Werra, and Harm de Vries. “StarCoder: may the source be with you!”. arXiv preprint arXiv:2305.06161 (2023). Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. “Code Llama: Open Foundation Models for Code”. arXiv preprint arXiv:2308.12950 (2024). Yue Liu, Thanh Le-Cong, Ratnadira Widyasari, Chakkrit Tantithamthavorn, Li Li, Xuan-Bach D. Le, and David Lo. “Refining ChatGPT-Generated Code: Characterizing and Mitigating Code Quality Issues”. ACM Transactions on Software Engineering and Methodology 33, no. 5 (2024): 116:1–116:26. Kevin Jesse, Toufique Ahmed, Premkumar T. Devanbu, and Emily Morgan. 2023. “Large Language Models and Simple, Stupid Bugs” In Proceedings of the 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR), 563–575. IEEE. Florian Tambon, Arghavan Moradi Dakhel, Amin Nikanjam, Foutse Khomh, Michel C. Desmarais, and Giuliano Antoniol. “Bugs in Large Language Models Generated Code: An Empirical Study”. Empirical Software Engineering 30, no. 3 (2025). Preprint: arXiv:2403.08937, 2024. Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. “LLM is Like a Box of Chocolates: the Non-determinism of ChatGPT in Code Generation”. arXiv preprint arXiv:2308.02828 (2023).

20. Bar Lanyado, Ortal Keizman, and Yair Divinsky. 2023. “Can You Trust ChatGPT’s Package Recommendations?” Vulcan Cyber Voyager18 Research Team blog post, https://vulcan.io/blog/ ai-hallucinations-package-risk, Accessed: 2024. 21. Markus Zimmermann, Cristian-Alexandru Staicu, Cam Tenny, and Michael Pradel. 2019. “Small World with High Risks: A Study of Security Threats in the npm Ecosystem” In Proceedings of the 28th USENIX Security Symposium (USENIX Security ’19), 995–1010. USENIX Association. 22. César Soto-Valero, Nicolas Harrand, Martin Monperrus, and Benoit Baudry. “A Comprehensive Study of Bloated Dependencies in the Maven Ecosystem”. Empirical Software Engineering 26, no. 3 (2021): Article 45, 1–44. 23. Riivo Kikas, Georgios Gousios, Marlon Dumas, and Dietmar Pfahl. 2017. “Structure and Evolution of Package Dependency Networks” In Proceedings of the 14th International Conference on Mining Software Repositories (MSR), 102–112. IEEE. 24. Russ Cox. “Surviving Software Dependencies”. Communications of the ACM 62, no. 9 (2019): 36–43. 25. Piergiorgio Ladisa, Henrik Plate, Matias Martinez, and Olivier Barais. 2023. “SoK: Taxonomy of Attacks on Open-Source Software Supply Chains” In Proceedings of the 2023 IEEE Symposium on Security and Privacy (SP), 1509–1526. IEEE. 26. Chengwei Liu, Sen Chen, Lingling Fan, Bihuan Chen, Yang Liu, and Xin Peng. 2022. “Demystifying the Vulnerability Propagation and Its Evolution via Dependency Trees in the NPM Ecosystem” In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22), 672–684. ACM. 27. National Telecommunications and Information Administration 2021. “The Minimum Elements for a Software Bill of Materials (SBOM)” U.S. Department of Commerce, https://www.ntia.doc.gov/files/ntia/publications/sbom_ minimum_elements_report.pdf, Pursuant to Executive Order 14028 §4(f). 28. The White House 2021. “Executive Order on Improving the Nation’s Cybersecurity” Executive Order 14028 of May 12, 2021, https://www.whitehouse. gov/briefing-room/presidential-actions/2021/05/12/ executive-order-on-improving-the-nations-cybersecurity/, . 29. Bobby R. Bruce, Tianyi Zhang, Jaspreet Arora, Guoqing Harry Xu, and Miryung Kim. 2020. “JShrink: In-Depth Investigation into Debloating Modern Java Applications” In Proceedings of the 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2020), 135–146. ACM. 30. César Soto-Valero, Thomas Durieux, Nicolas Harrand, and Benoit Baudry. “Coverage-Based Debloating for Java Bytecode”. ACM Transactions on Software Engineering and Methodology 32, no. 2 (2023): Article 38, 1–34. 31. Fernando Chirigati, Rémi Rampin, Dennis Shasha, and Juliana Freire. 2016. “ReproZip: Computational Reproducibility With Ease” In Proceedings of the 2016 ACM SIGMOD International Conference on Management of Data (SIGMOD ’16), 2085–2088. ACM. 32. Dai Hai Ton That, Gabriel Fils, Zhihao Yuan, and Tanu Malik. 2017. “Sciunits: Reusable Research Objects” In Proceedings of the 2017 IEEE 13th International Conference on e-Science (eScience ’17), 158–167. IEEE. 33. Nusrat Zahan, Elizabeth Lin, Mahzabin Tamanna, William Enck, and Laurie Williams. “Software Bills of Materials Are Required. Are We There Yet?”. IEEE Security & Privacy 21, no. 2 (2023): 82–88. 34. Boming Xia, Tingting Bi, Zhenchang Xing, Qinghua Lu, and Liming Zhu. 2023. “An Empirical Study on Software Bill of Materials: Where We Stand and the Road Ahead” In Proceedings of the 45th IEEE/ACM International Conference on Software Engineering (ICSE), 2630–2642. 35. Monya Baker. “1,500 Scientists Lift the Lid on Reproducibility”. Nature 533, no. 7604 (2016): 452–454.

15 of 16

36. Christian Collberg and Todd A. Proebsting. “Repeatability in Computer Systems Research”. Communications of the ACM 59, no. 3 (2016): 62–69. 37. Carl Boettiger. “An Introduction to Docker for Reproducible Research”. ACM SIGOPS Operating Systems Review 49, no. 1 (2015): 71–79. 38. Philip J. Guo and Dawson Engler. 2011. “CDE: Using System Call Interposition to Automatically Create Portable Software Packages” In Proceedings of the 2011 USENIX Annual Technical Conference (USENIX ATC ’11), Portland, OR: USENIX Association. 39. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. “Holistic Evaluation of Language Models”. Transactions on Machine Learning Research (2023). arXiv:2211.09110.

T A B L E A2 | Metric summary: definitions, key quantities, and result section.

APPENDIX

jackson-core, and jackson-annotations reveal systematic omissions in dependency reasoning.

A

ID

Metric

M1 Success rate under self-correction M2 Environment growth M3 Phantom / hidden / bloat rates M4 Manifest accuracy M5 Cross-agent consistency M6 Stochastic variability M7 Unnecessary external libraries M8 Environment gap failures

Key quantity Results SR (0) , SR (≤𝑘)

§6.1

𝜌 𝜙, 𝜂, 𝛽 P, R, F1 𝐽̄𝓁 , CDR 𝐽̄intra , UCR UDR SLAR, EGAR

§6.2 §6.2 §6.2 §6.3 §6.3 §6.5 §6.4

Supplementary Structural Metrics

This appendix reports supplementary results that support interpretation of the primary findings. Manifest size vs. first-attempt success.. We test whether projects with larger declared dependency sets fail more frequently on their first execution attempt. For each ecosystem 𝓁, we com(1) pute the Spearman rank correlation between |𝐷𝑎,𝑝,𝓁 | and the round-zero success indicator 𝟏[Success(𝐶 (0) , 𝑀 (0) , 𝒟𝓁 )] over all (𝑎, 𝑝) pairs. A negative correlation would indicate that manifest bloat is itself a source of build fragility.

T A B L E A1 | Spearman correlation between manifest size and firstattempt success. Ecosystem

𝜌𝑆

𝑝-value

Python Java JavaScript C++

−0.276 −0.260 −0.093 −0.410

0.0006 0.0013 0.2564 0.0000

Table A1 shows that larger declared dependency sets are associated with lower first-attempt success in Python, Java, and C++. The strongest negative association appears in C++, while JavaScript shows the same direction but not a statistically meaningful effect in the current benchmark. Summary.. Table A2 cross-references the eight primary metrics against the results subsections where they are reported.

B

Complete Problem List

Most frequent phantom and hidden packages.. We compile ranked frequency tables of the packages that appear most often in 𝐷 (1) ⧵ 𝐷 (3) (phantom) and 𝐷 (3) ⧵ 𝐷 (1) (hidden) across all projects in each ecosystem. These tables serve as qualitative diagnostics: recurring phantom packages such as pycparser, yargs, and nlohmann_json reveal systematic over-declaration patterns, while recurring hidden packages such as six, python-dateutil,

16 of 16

AI Magazine, 2026

T A B L E B3 | All 50 programming problems. ID

Problem

Domain

p_01 p_02 p_03 p_04 p_05 p_06 p_07 p_08 p_09 p_10 p_11 p_12 p_13 p_14 p_15 p_16 p_17 p_18 p_19 p_20 p_21 p_22 p_23 p_24 p_25

CSV Statistical Analyzer Data Processing JSON Schema Validator Data Processing XML to JSON Converter Data Processing YAML Config Merger Data Processing Log File Pattern Analyzer Data Processing SQLite Database Manager Data Processing Data Profiling Tool Data Processing Multi-Format Data Converter Data Processing Time Series Trend Detector Data Processing Duplicate Record Finder Data Processing AES-256 File Encryption Cryptography RSA Digital Signatures Cryptography HMAC Integrity Checker Cryptography TOTP Generator Cryptography Password Strength Analyzer Cryptography X.509 Certificate Parser Cryptography Bcrypt Password Hasher Cryptography Image Format Converter Image Processing Image Resizer / Thumbnails Image Processing Image Watermarking Tool Image Processing Image Histogram Analyzer Image Processing EXIF Metadata Extractor Image Processing Color Palette Extractor Image Processing HTTP File Downloader Networking TCP Port Scanner Networking

Cx.

ID

Problem

Domain

Cx.

M M M M M H H H H M H H M M M H M M M M M M H M M

p_26 p_27 p_28 p_29 p_30 p_31 p_32 p_33 p_34 p_35 p_36 p_37 p_38 p_39 p_40 p_41 p_42 p_43 p_44 p_45 p_46 p_47 p_48 p_49 p_50

DNS Resolver HTTP REST Client w/ Retry Static HTTP File Server Network Latency Monitor Markdown to HTML Converter Spell Checker TF-IDF Search Engine Text Diff Tool HTML to Plain Text Template Engine ZIP Archive Manager GZIP Compression Tool Tar Archive Handler File Deduplicator Compression Benchmark Matrix Operations Calculator Linear Regression Fitter Dijkstra Shortest Path FFT Spectrum Analyzer Numerical Integration QR Code Generator / Reader PDF Report Generator Merkle Tree Verifier Cron Expression Parser Structured Log Processor

Networking Networking Networking Networking Text Processing Text Processing Text Processing Text Processing Text Processing Text Processing Compression Compression Compression Compression Compression Math/Scientific Math/Scientific Math/Scientific Math/Scientific Math/Scientific System Utilities System Utilities System Utilities System Utilities System Utilities

M H H M M M H M M H M M M M H H H M H H M H M M M

Cx. = complexity: M = Medium, H = High.

T A B L E B4 | Most frequent phantom packages 𝐷 (1) ⧵ 𝐷 (3) by ecosystem.

Ecosystem Most frequent phantom packages Python Java JavaScript C++

pycparser (6), pandas (5), tabulate (2), tzdata (1), python-dateutil (1) gson (2), zxing-javase (2), zxing-core (2), slf4j-api (1), javalin (1) yargs (46), sharp (8), fast-csv (4), lodash (4), js-yaml (3) nlohmann_json (65), openssl (16), opencv (11), yaml-cpp (5), mbedtls (5)

T A B L E B5 | Most frequent hidden packages 𝐷 (3) ⧵ 𝐷 (1) by ecosystem.

Ecosystem Most frequent hidden packages Python Java

JavaScript

C++

six (31), python-dateutil (23), pytz (20), packaging (14), kiwisolver (11) jackson-annotations (21), jackson-core (20), slf4j-api (14), commons-logging (12), error_prone_annotations (12) ansi-styles (15), supports-color (13), has-flag (13), emoji-regex (9), is-fullwidthcode-point (9) crypto (16), z (14), zstd (6), nettle (6), krb5 (5)

17 of 16

Related documents

• Code That Works, Environments That Don't: Measuring Environment Reproducibility in AI-Generated Software#1155360
• How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method#972210
• CodeAssay: A Multi-Metric Benchmark with Audited Ground Truth for LLM Code Generation#1014787
• Do Code Language Models Use Tests? A Behavioral and Representational Study of Test-Driven Code Generation#411115
• Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality#373456