DeployBench : Benchmarking LLM Agents for Research Artifact Deployment
arXiv:2606.05238v1 [cs.SE] 3 Jun 2026
Yuanli Wang 1 , Yaoyao Qian2 , Yue Zhang3 , Hanhan Zhou4 , Jindan Huang5 , Tianfu Fu6 , Qiuyang Mang7 , Huanzhi Mao7 , Wenhao Chai8 , Wendong Fan9 and Liqiang Jing3 1 Boston University, 2 Northeastern University, 3 University of Texas at Dallas, 4 George Washington University, 5 Tufts University, 6 xAI, 7 University of California, Berkeley, 8 Princeton University, 9 Eigent.Ai Correspondence: [email protected]
Abstract: LLM agents have made rapid progress on software engineering and ML research tasks, but these advances often assume access to a working runnable environment. For research artifacts released alongside published papers, setting up such an environment from a fresh machine remains a major bottleneck. Existing environment setup benchmarks do not cover the full scope of researchartifact deployment, which involves multi-language toolchains, system-level dependencies beyond containers (e.g., GPU/CUDA and kernel configurations), and legacy artifact compatibility. We introduce DeployBench , a multi-domain benchmark of 51 research-artifact deployment tasks spanning AI/ML, computer systems, and scientific computing, covering all these dimensions. Each task is verified by a hidden pipeline that executes the paper’s designated experiment and checks its outputs. Evaluating four state-of-the-art LLMs with OpenHands yields pass-rates from 7.8%–51.0%. Failures are dominated by a completion-judgment problem: 97 of 154 are agent-terminated self-stops, where the agent’s pre-finish checks validate a different or weaker target than the paper-specific task requires. DeployBench highlights the gap between current agents and autonomous deployment, and offers a realistic testbed for scientific research agents. Repository: https://github.com/pentium3/DeployBench
1.
Introduction
Large language model (LLM) agents have expanded beyond text generation into autonomous software engineering and scientific research, tackling repository-level software engineering tasks [1, 2], reproducing open-source research code [3, 4], and performing ML engineering, data science tasks [5, 6] and frontier research [7, 8]. However, these evaluations often assume access to a runnable environment where dependencies, build tools, and runtime stacks are already configured. In practice, research artifacts are not always released in a runnable state [9], and the effort required to reproduce them can be substantial [10, 11]. Setting up an artifact from scratch often requires resolving version-drifted dependencies, aligning GPU/CUDA stacks, patching legacy code against modern toolchains, and configuring system-level components such as custom Linux kernels that containers cannot host. This setup phase is often the primary barrier preventing researchers from reproducing, validating, and building upon published work. Existing environment-setup benchmarks do not fully test this capability [12–14]. Prior work primarily evaluates general-purpose repository setup using container builds, static checks, or unit tests as
success signals, or studies research-code deployment within narrower Python-based AI research or containerized settings. As a result, they do not measure whether agents can deploy diverse research artifacts from raw infrastructure and run a paper-specific target. To address this gap, we introduce DeployBench , a benchmark designed to evaluate whether LLM agents can deploy published research artifacts from raw infrastructure to runnable environments. DeployBench includes 51 deployment tasks drawn from top-tier venues between 2008 and 2025, spanning AI/ML (25), computer systems (19), and scientific computing (7). The tasks cover 11 programming-language ecosystems and include 22 GPU-dependent workloads, 5 tasks that compile and boot custom Linux kernels in virtual machines, and 10 legacy artifacts (2011–2018) requiring compatibility repair against modern toolchains. Each task provides the agent with the paper, its code repository, and a fresh cloud VM with no pre-installed drivers. Task success is determined by a hidden 2-layer verifier (a global rule-based parser plus a task-specific check) that runs a lightweight version of the paper’s designated experiment covering the main pipeline and key dependencies (e.g., training the model for one epoch), and checks expected outputs and runtime evidence. We also confirm that all included tasks can be deployed by running the reference setup scripts. We evaluate 4 state-of-the-art LLMs (GPT-5.3-Codex, Gemini-3.1-Pro, Grok-4.20, GPT-5.4-Mini) on DeployBench with OpenHands as agent scaffold [15]. The pass rates of these 4 LLMs range from 7.8% to 51.0%. Even the strongest model solves only about half the tasks. Failure analysis reveals five recurring root-cause patterns, with completion judgment as a dominant problem. Among 154 failures, 97 are agent-terminated self-stops. In these cases, the agent’s own pre-finish checks often run against a different or weaker target than the paper-specific task requires. This suggests that agents still fall short both in deploying research artifacts and in deciding whether the resulting setup satisfies the paper-specific task.
2.
Related Work
LLM agents for software engineering. LLM agents have been applied to a wide range of repositorylevel software engineering tasks [5, 6, 16–20]. To perform such tasks, agent scaffolds such as SWEagent [2] and OpenHands [15] have emerged, with progress measured on benchmarks such as SWE-bench [1] and Terminal-Bench [21]. However, before such downstream tasks can be performed, the repository must first be set up in a runnable environment, which remains a bottleneck due to version drift and dependencies that span Python packages, system libraries, GPU drivers, etc. Benchmarks for environment setup. Motivated by this gap, recent benchmarks have begun to study environment setup and repository deployment directly. Installamatic [22] introduces a benchmark of repository installation tasks from open-source Python projects, together with an agent that searches project documentation to install and verify target repositories. ExecutionAgent [23] generates project-specific scripts for building arbitrary projects and running their tests. EnvBench [12] evaluates automated environment setup across Python and JVM repositories, and SetupBench [11] evaluates environment bootstrapping from Linux sandboxes with deterministic success commands. Repo2Run [24] introduces an agentic framework and benchmark for building Docker-based executable test environments for Python repositories. EnConda-Bench [25] is complementary to these end-to-end setup benchmarks: it injects realistic README errors and provides process-level trajectory evaluation of 2
25 20
Paper Artifact paper + code repo AI/ML
Computer System
DeployBench Task Task instruction + Hidden verifier
Terminal agent
Set up environment Output + VM state
Hidden verifier
Scientific Computing
Figure 1: DeployBench overview.
Verifier script Check on final VM
Count
Clean Linux VM per task Task instruction
15
Domain AI/ML Computer Systems Sci. Computing
10 5 0 l t t hon ++ us tran kel ava rip Lua Julia y P t C/C R For Has J avaSc J
R cket Ra
Figure 2: Distribution of tasks across domains and main languages.
planning, error diagnosis, repair, and final execution. Closer to research deployment, CSR-Bench [13] evaluates LLM agents on deploying computer science research repositories, with tasks concentrated in NLP/CV/AI/ML/DM topics, through setup-command generation, data preparation, and experiment execution. ResearchEnvBench [14], the closest to our setting, benchmarks environment synthesis for research code, but operates inside Docker containers with CUDA drivers pre-installed, and covers only recent AI repositories in Python without older codebases or other domains. Beyond benchmarks, agent systems such as HerAgent [26] and DockSmith [27] treat environment construction as a learning target, focusing on agentic strategy improvement rather than benchmark scope. In contrast, DeployBench targets autonomous deployment in more realistic settings: it starts from a bare metal Linux VM rather than a container, spans multiple research domains beyond AI, and involves legacy codebases that were written years ago and must now be made to build and run on modern systems. A task is solved only when the final environment executes a designated paper experiment.
3.
DeployBench
We introduce DeployBench , a benchmark for evaluating whether LLM agents can construct environments that execute designated experiments from published research artifacts, starting from raw infrastructure. Beyond installing packages, this requires the agent to infer runtime requirements from the paper and repository, construct the needed software and system stack, repair compatibility issues, and determine whether the resulting environment is ready for the target experiment.
3.1.
Task Formulation
Given a research paper and its code repository, a DeployBench task asks an agent to turn a clean cloud VM into an environment in which a designated experiment from the paper executes successfully (Figure 1). Each task provides the agent with three inputs: the paper, the code repository, and a minimal Linux VM with no task-specific dependencies pre-installed. We disallow Docker so that success reflects native dependency resolution and system setup rather than reuse of pre-packaged containers. This is also necessary for system-level artifacts such as the QEMU/kernel tasks. Task success is determined by a hidden, task-specific verifier described in Section 3.3. 3
3.2.
Benchmark Construction
Paper Selection. DeployBench comprises 51 tasks drawn from research artifacts published between 2008 and 2025, spanning AI/ML (25 tasks from venues such as NeurIPS, ICML, ICLR, CVPR, ECCV, RSS, CoRL), computer systems (19 tasks from OSDI, SOSP, EuroSys, ATC, USENIX Security), and scientific computing (7 tasks from non-CS scientific domains, including JOSS and Cell artifacts). We select papers from top venues in each domain and prioritize artifacts with widely used public repositories. For systems and security tasks, we also prioritize papers that have passed artifact evaluation [28] when such badges or reports are available. We manually inspect each candidate repository and attempt to run the artifact ourselves, excluding clearly broken, incomplete, or unreproducible artifacts. Each DeployBench task is validated by executing a verification target derived from a designated paper experiment on a fresh VM, rather than by import checks, command success, or repository-provided tests alone. Verifier Construction and Reproducibility Check. For each task, we first construct a working reference deployment on a clean VM to confirm that the artifact can be reproduced and to understand its dependencies and execution workflow. This step is used to validate task feasibility and verifier design. We then select a verification target from the paper’s original evaluation that is lightweight enough to run within a practical time budget but substantial enough to cover the main pipeline and its key dependencies (for example, training the model for one epoch). The task-specific verifier and reference setup script are manually constructed for each task. The verifier re-runs the target in the final agent-produced environment and checks the output against task-specific success conditions. As a reliability check, the reference setup script reproduces the environment from scratch and passes the verifier on all 51 tasks. Per-task agent time budgets are set individually based on setup complexity and command execution time, ranging from 10 to 60 minutes. We exclude candidates that cannot be reproduced during reference construction, such as those with unavailable datasets or model weights. Task Composition and Complexity. Across the 51 tasks, implementation languages span 11 ecosystems (Python, C/C++, Rust, Fortran, Julia, R, Haskell, Lua, Racket, JavaScript, Java), with several tasks combining multiple languages in a single build pipeline. The final set also spans a wide range of setup difficulty: 12 tasks are labeled easy, 20 medium, and 19 hard. These labels are assigned by three authors with graduate-level CS research experience who constructed the reference deployments, based on dependency layers, GPU/CUDA or QEMU/kernel setup, legacy compatibility repair, and debugging required during reference construction. Deployment requirements vary along multiple axes: 22 tasks require GPU execution while 29 are CPU-only, 5 require building a custom Linux kernel and booting it inside a QEMU virtual machine, and 10 are legacy artifacts released between 2011 and 2018 (not updated since 2020) that require compatibility repair against modern OS versions, compilers, and library APIs. Many tasks also include runtime-consuming setup steps that affect the deployment budget, such as compiling large codebases from source, downloading required weights or datasets to expected paths, or patching legacy APIs. Figure 2 summarizes the distribution across domains and main languages.
3.3.
Evaluation
After the agent finishes, reaches the step limit, or times out, we evaluate the final VM state using a hidden task-specific verifier. The verifier has two layers (Figure 3) that check different failure cases. 4
Generic failure found?
Hidden Verifier
Run target experiments on final VM
Identify task-agnostic failure patterns
• • • •
Dependency errors Runtime exceptions Crash / out of memory …
Success
Failed
no
Layer1: Global parser verifier logs
yes
Layer2: Task-specific check Validate target completion status • • •
yes
Diagnostic judge
Expected output runtime evidence ..
Check passed?
Diagnose & Summarize root causes
no
Failed
Figure 3: Evaluation pipeline.
Layer 1 is a global rule-based log parser that catches common execution failures shared across tasks, such as missing dependencies, compilation errors, runtime crashes, and out-of-memory errors. If no Layer-1 failure is detected, Layer 2 runs task-specific checks for expected outputs, runtime evidence, generated artifacts, or service availability. Beyond Layer 1’s generic failure checks, this second layer verifies whether the paper-specific target actually produces the required outputs or runtime evidence, rather than merely reaching a superficially runnable state. A task is solved only if it passes both layers. For failed tasks, we additionally run a separate LLM-based diagnostic agent over the final VM state and verifier outputs to attribute the root cause (full prompt in Appendix A3). This diagnostic step is used only for failure attribution. All success rates are computed solely from deterministic verifier outcomes.
3.4.
Comparison with Existing Benchmarks
Table 1 positions DeployBench against recent benchmarks for repository setup and environment synthesis along four axes: (1) multiple language ecosystems per task (Multi-Lang.), (2) per-task specific validation beyond generic success signals (imports, command success, builds, or repository-provided unit tests) (Per-Task Verifier), (3) host/VM-level setup beyond containers such as GPU/CUDA driver install, kernel builds, or QEMU execution (System-Level Setup), and (4) older artifacts requiring compatibility repair (Legacy Compat.). Among benchmarks with task-specific validation, DeployBench uses a verifier that executes a lightweight target derived from a designated paper experiment on the final fresh-VM environment. ResearchEnvBench [14] shares per-task verification and tests CUDA-stack alignment within Docker containers, but is restricted to AI repositories in Python with drivers pre-installed and excludes legacy artifacts. EnvBench [12] covers only two language ecosystems (Python and Java). EnConda-Bench [25] and CSR-Bench [13] verify only via generic signals (Docker build , repository-provided unit tests , or generic command-execution signals) rather than task-specific validation. In contrast, DeployBench is the only benchmark combining all four axes.
5
Task Source
Benchmark
Multi-Lang.
Per-Task System-Level Legacy Verifier Setup Compat.
EnvBench [12] SetupBench [11] Installamatic [22] ExecutionAgent [23] Repo2Run [24] EnConda-Bench [25] ResearchEnvBench [14] CSR-Bench [13]
Python + Java OSS repos OSS repos Python OSS repos OSS repos Python OSS repos Python OSS repos + README errors AI research code repos AI research code repos
✓ ✓ ✗ ✓ ✗ ✗ ✗ ✗
✗ ✓ ✗ ✗ ✗ ✗ ✓ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✓ ✗
✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗
DeployBench (Ours)
Multi-domain papers + code repos
✓
✓
✓
✓
Table 1: Comparison of DeployBench with existing environment-setup benchmarks. ✓ , ✓ , and ✗ denote full, partial, and no support along each axis.
4.
Experiment
4.1.
Experimental Setup
Models. We evaluate four frontier language models under OpenHands agent scaffold: OpenAI GPT-5.3-Codex, OpenAI GPT-5.4-Mini, Google Gemini-3.1-Pro, and xAI Grok-4.20. For the diagnostic agent on failed runs, we use the Codex CLI tool (an OpenAI command-line agent based on GPT-5.3, high-reasoning mode) with SSH access to the final VM state and verifier logs. Prompting. The same system prompt instructs the agent to set up the artifact natively on a fresh Linux VM (Docker disallowed), run a simple test, and write a RUNBOOK.MD file summarizing the setup process. The full prompt is in Appendix A2. Execution environment. All runs use OpenHands [15] on a fresh Google Cloud VM running Ubuntu 22.04 (16 vCPUs, 64 GB RAM). GPU tasks use a VM with one NVIDIA L4 GPU (24 GB memory) and no pre-installed CUDA drivers. Each task uses a fresh VM, a task-specific time budget, and a fixed verification budget. The agent never sees the hidden verifier scripts. Metric. Our primary metric is task success rate, the fraction of tasks whose verification pipeline returns a passing verdict.
4.2.
Model Performance
4.2.1.
Overall Results
Table 2 reports the aggregate task success rates of all evaluated models on DeployBench : GPT-5.3Codex leads at 51.0%, followed by Gemini-3.1-Pro (27.5%), Grok-4.20 (11.8%), and GPT-5.4-Mini (7.8%). Even the strongest model solves only about half of the tasks, showing that end-to-end research artifact deployment remains a challenging task. The gap between models indicates that DeployBench can distinguish the capabilities of the model under the same agent scaffold.
6
Breakdown
Group
# Tasks
GPT-5.3-Codex
Gemini-3.1-Pro
Grok-4.20
GPT-5.4-Mini
Overall
All tasks
51
51.0%
27.5%
11.8%
7.8%
Category
AI/ML Sci. Computing Systems
25 7 19
52.0% 42.9% 52.6%
16.0% 57.1% 31.6%
8.0% 14.3% 15.8%
0.0% 0.0% 21.1%
Difficulty
Easy Medium Hard
12 20 19
66.7% 45.0% 47.4%
33.3% 40.0% 10.5%
25.0% 5.0% 10.5%
8.3% 10.0% 5.3%
Repository age
Legacy (<2020) Recent (≥2020)
10 41
40.0% 53.7%
20.0% 29.3%
10.0% 12.2%
0.0% 9.8%
Table 2: Task success rates by overall performance, task category, artifact difficulty, and repository age.
4.2.2.
Results by Task Category
Table 2 also breaks performance down by task category, difficulty, and repository age. The differences between categories are mainly due to their deployment requirements. In AI/ML, 88.0% of tasks require GPU-capable environments, making CUDA/GPU setup a major source of category-specific difficulty. In systems, the five QEMU/kernel tasks have an average success rate of only 20.0%, showing that VM- and kernel-level execution adds another source of deployment difficulty. Difficulty labels match the overall success pattern: at least one model solves 83.3% of easy tasks, compared with 55.0% of medium tasks and 52.6% of hard tasks. Comparing legacy and recent tasks, since they run on the same modern OS image and hardware, this gap reflects compatibility drift rather than paper age itself: compatibility repair is needed in 5 of 10 legacy tasks versus 3 of 41 recent tasks, often because older code must be adapted to modern compilers and library APIs. Section 5.4 gives a concrete example. 4.2.3.
Token Usage and Runtime Analysis Tokens (avg per task)
Model GPT-5.3-Codex Gemini-3.1-Pro Grok-4.20 GPT-5.4-Mini
Runtime (minutes, avg per task)
Success rate Steps
Total
Input Output Solved Failed Total Solved
51.0% 27.5% 11.8% 7.8%
2.15M 2.55M 0.75M 0.65M
2.12M 2.54M 0.75M 0.64M
44.5 57.8 15.2 21.7
30.7K 4.3K 4.1K 3.6K
1.63M 1.49M 0.42M 0.45M
2.69M 2.94M 0.80M 0.66M
18.1 15.9 14.0 6.1
14.6 10.1 11.5 10.2
Failed 21.8 18.1 14.3 5.8
Table 3: Per-task token usage and runtime split by model and verifier outcome (Solved / Failed).
Table 3 reports per-model token usage and runtime. Token usage does not align with success ranking: Gemini-3.1-Pro consumes the most tokens (2.55M on average) yet ranks second in success, while GPT-5.4-Mini uses the fewest tokens (0.65M) and has the lowest success rate. Within every model, failed runs use more tokens than solved runs (Codex 2.69M vs 1.63M, Gemini 2.94M vs 1.49M, Grok 0.80M vs 0.42M, Mini 0.66M vs 0.45M), consistent with the higher Loop-style retry rate on failed runs (60.3% vs 44.3%, Section 5.2). Runtime follows the same per-model pattern except for GPT-5.4-Mini, which fails faster than it succeeds (5.8 min vs 10.2 min) because most of its failures are early self-stops. 7
5.
Further Analysis
To inform future agent design, we analyze the deployment-specific failure modes and trajectory patterns of the four evaluated models, drawing on verifier outputs, global-parser error types, and the diagnostic-agent reports described in Section 3.3.
5.1.
Failure Analysis
Table 4 categorizes the 154 failed runs by where they are caught. The global check (Layer 1) detects 46.1% of runs. The remaining 53.9% pass the global check but still fail the task-specific verifier (Layer 2), meaning the environment reaches a superficially runnable state but does not produce the artifacts, services, outputs, or runtime evidence required by the designated experiment. Failure location
Count
Percentage
Layer 1 – Runtime crash Layer 1 – Dependency error Layer 1 – Compilation error
39 27 5
25.3% 17.5% 3.2%
Layer 1: global check Layer 2: task-specific check
71 83
46.1% 53.9%
Table 4: Failure locations in the evaluation pipeline.
From the diagnostic-agent reports, failures cluster into 5 recurring root-cause patterns: dependency and package resolution (incomplete or version-incompatible installs), GPU/CUDA setup (CPU-only frameworks, missing drivers, mismatched CUDA-dependent libraries), native-build and toolchain (failed compilation of non-Python components), artifact and output mismatch (expected binaries, checkpoints, datasets, or outputs missing or misplaced), and system- and VM-level execution (QEMU, kernel boot, service startup, or non-interactive configuration failures). The 20 all-fail tasks (failed by all four models) typically combine multiple of these patterns: all 20 hit at least two and 14 hit three or more (3.4 patterns per task on average). The 30 mixed-outcome tasks (at least one model succeeded and at least one failed) average only 2.5 patterns per task, indicating that all-fail tasks compound more failure modes simultaneously. For example, gan neurips14 gpu combines a wrong Theano version, an absent CUDA toolkit (nvcc and cuDNN missing), and a stale pylearn2 commit. kflex sosp24 combines an incomplete QEMU rootfs (missing libbenchmark, libjemalloc, and libabsl shared objects) with a wrong system base. Appendix A4 reports the full counts for each failure pattern. These results show that per-task verification is necessary: generic Layer 1 checks would miss the 53.9% of failures caught only by the task-specific verifier.
5.2.
Trajectory Analysis
We use trajectory analysis to identify which process signals are useful for agent design. Overall, trajectory length is a weak success signal, while how models use the agent scaffold and how they recover from errors provide more useful signals. 8
Trajectory length and phase coverage. Table 5 reports per-task average trajectory length under three units: agent steps, shell commands, and sub-commands (segments of a shell command split on top-level &&, ||, or ; which expose work hidden inside chained commands. see Appendix A5). Longer trajectories do not imply better deployment. Gemini-3.1-Pro produces the longest trajectories (57.8 steps, 55.3 commands) but underperforms GPT-5.3-Codex, which uses fewer steps (44.5). Grok-4.20 issues only 10.4 commands on average, but these expand to 115.3 sub-command segments because it chains many operations inside each shell command. Thus, trajectory length is useful for understanding how an agent works, but it does not predict whether the task will succeed. We also classify sub-commands into deployment phases using heuristic keyword parsing (definitions in Appendix A5). Failed runs do not skip entire stages. Instead, compared with successful runs, they spend less time on environment inspection and more time in repeated install/build retries, suggesting that phase coverage alone is not enough to explain deployment success. The full phase breakdown is in Appendix A5. Model
Steps
Cmds.
Subcmds.
Success Rate
GPT-5.3-Codex GPT-5.4-Mini Gemini-3.1-Pro Grok-4.20
44.5 21.7 57.8 15.2
37.2 19.9 55.3 10.4
178.7 75.3 84.5 115.3
51.0% 7.8% 27.5% 11.8%
Table 5: Per-task average trajectory length by model.
Model-specific workflow and tool-use patterns. The models follow different workflow styles. GPT5.3-Codex follows a more plan-driven workflow, while Gemini-3.1-Pro produces longer Bash-heavy trajectories. GPT-5.4-Mini often self-stops with partial runbooks, and Grok-4.20 more often aborts after failed commands. In terms of tool-use patterns, one concrete difference is command-timeout use: GPT-5.3-Codex sets the OpenHands bash tool’s optional timeout parameter on 100% of bash calls, while Gemini-3.1-Pro sets it on only 2%. This difference explains why Gemini-3.1-Pro has many more soft-timeout events, where the shell tool stops waiting before the command completes. These events make its trajectories look longer without reflecting additional deployment work. Therefore, part of the trajectory-length gap reflects how models use the agent scaffold, not only how much deployment work they perform. We do not claim that setting timeouts would directly improve success rates, but future agents should make agent tool-use policies explicit. Details are discussed in Appendices A6. Recovery after failed commands. Failed shell commands (non-zero exit or OpenHands soft-timeout) appear in both successful and failed runs. However, what differs is how the agent responds. Using keyword and regex heuristics, we assign each failed command one of five recovery categories: Inspect (read logs/files), Adapt (change the command before retrying), Loop, Abort/Finish, or Other (full definitions in Appendix A7). Successful runs more often inspect logs or adapt the command (Inspect+Adapt: 48.5% vs. 32.7% in failed runs), while failed runs more often repeat waiting or retry loops (60.3% vs. 44.3% in successful runs; Table 6). Because OpenHands soft-timeouts are also counted as Loop, especially for Gemini-3.1-Pro, this category captures unresolved waiting or retry behavior rather than only explicit repeated retries. This suggests a concrete target for future agents: better log reading, dependency tracking, and retry decisions, rather than simply larger time or step budgets. 9
Outcome
Failed Cmds.
Inspect
Adapt
Loop
Abort/Finish
Other
332 2,102
26.2% 19.0%
22.3% 13.7%
44.3% 60.3%
4.8% 6.0%
2.4% 1.0%
Success Failed
Table 6: Recovery behavior after failed commands.
5.3.
Completion Judgment
Model
Self-stop failures
RUNBOOK-success failures
RUNBOOK-partial failures
Meaningful check
Shallow-only check
12/25 42/47 15/37 28/45
12/12 17/42 15/15 11/28
0/12 20/42 0/15 12/28
12/12 33/42 15/15 21/28
0/12 9/42 0/15 7/28
GPT-5.3-Codex GPT-5.4-Mini Gemini-3.1-Pro Grok-4.20
Table 7: Self-stop incidence over all failed runs, with RUNBOOK status and pre-finish check types over selfstops.
Self-stop is the most common termination mode among failed runs: 97 of 154 failures are agentterminated rather than reaching timeout or max-step. GPT-5.4-Mini (42/47) and Grok-4.20 (28/45) show the highest self-stop rates among the four models (Table 7). Within these 97 self-stopped failures, we find two types of completion-judgment errors. First, at the RUNBOOK self-report level, 87 of 97 self-stops show RUNBOOK/verifier mismatch: 55 report SUCCESS while failing verification (RUNBOOK-success), and 32 report PARTIAL while still failing verification (RUNBOOK-partial). Only 1 of 33 self-stops with PARTIAL reports ultimately passes verification. Second, at the pre-finish check level, the check that the agent runs before finishing is often inadequate. All 12 of GPT-5.3-Codex’s and all 15 of Gemini-3.1-Pro’s self-stopped failures do run a meaningful check (a target script, binary, or training/inference command on the artifact), but still fail verification because the agent validates the wrong target, runs a check that is weaker than the verifier, or misses task-specific requirements. We call this self-validation drift. GPT-5.4-Mini (9/42) and Grok-4.20 (7/28) additionally show a subset of cases where the pre-finish check is shallow only (imports, --help, toy examples, or RUNBOOK-only writes), never exercising the real artifact. Future deployment agents need to derive a verifier-like completion check from the paper and repository: a lightweight run that covers the paper-specific target and key dependencies. Self-designed smoke tests and RUNBOOK self-assessment are not reliable substitutes.
5.4.
Case Studies
We present two representative cases: adalora iclr23 shows completion-judgment error. silt sosp11 shows legacy compatibility repair requiring writing new compatibility code.
10
adalora iclr23 (with GPT-5.3-Codex) Agent behavior. The agent inspects the AdaLoRA repository and paper, creates a Python environment, installs a CUDA PyTorch stack and the local loralib package, and writes a custom smoke test around a toy SVDLinear model. The smoke test passes, after which the agent writes RUNBOOK = SUCCESS and self-stops. However, this smoke test exercises only loralib and does not require the HuggingFace evaluation stack used by the paper’s experiment. Root cause. The hidden verifier instead runs the paper’s actual experiment through run glue.py: fine-tuning DeBERTa-v3-base on GLUE CoLA for a few steps with AdaLoRA’s SVD-based LoRA modules. This requires the HuggingFace stack to be present in the environment, which the agent’s toy-model smoke test never installed. The Layer 2 check aborts with error ModuleNotFoundError: huggingface hub. Takeaway. The agent’s smoke test checked only loralib rather than the lightweight paper-specific target used by the verifier. This illustrates self-validation drift. silt sosp11 (with GPT-5.4-Mini) Agent behavior. The agent inspects the SILT repository (an in-memory key-value store system built by the paper), installs build dependencies, and starts the build. Compilation fails on several legacy APIs, including old OpenSSL calls, removed TBB atomic types, and a private libc header. The agent patches the easy parts, such as include paths and simple type substitutions, but cannot solve the deeper TBB API gap. Finally, the agent self-stops with RUNBOOK = PARTIAL. Root cause. A working deployment requires writing new compatibility code that wraps SILT’s legacy TBB atomic API onto modern std::atomic, not just substituting types or includes. GPT-5.3-Codex succeeded by writing a 52-line fawnds/tbb/atomic.h shim and modifying eight other source files, and the resulting binary built and passed verification. GPT-5.4-Mini’s text-level substitution did not provide this wrapper, and the agent stopped and honestly reported PARTIAL, rather than overclaiming success. Takeaway. Legacy artifact deployment requires not just recognizing compatibility breakage but repairing it across the full dependency stack, sometimes by writing new code (the TBB compatibility shim) rather than only editing existing source. Benchmarks restricted to actively-maintained repositories cannot cover this category (Table 1).
6.
Conclusion
We present DeployBench , a benchmark that evaluates whether LLM agents can deploy published research artifacts from raw infrastructure to a runnable state. Across four frontier models, pass rates range from 7.8% to 51.0%. Failure analysis reveals a dominant completion-judgment problem: 97 of 154 failures are agent-terminated self-stops. These results point to three missing capabilities for agents: dependency repair, system-level setup, and strong completion checks.
11
References [1] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations, 2024. [2] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. ArXiv, abs/2405.15793, 2024. [3] Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai’s ability to replicate ai research. ArXiv, abs/2504.01848, 2025. [4] Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, George Michalopoulos, Yue Zhang, Ziyang Zhang, Mian Zhang, Zhiyu Chen, and Xinya Du. Lmr-bench: Evaluating llm agent’s ability on reproducing language modeling research. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2025. [5] Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Aleksander Madry, and Lilian Weng. MLEbench: Evaluating machine learning agents on machine learning engineering. In International Conference on Learning Representations, 2025. [6] Liqiang Jing, Zhehui Huang, Xiaoyang Wang, Wenlin Yao, Wenhao Yu, Kaixin Ma, Hongming Zhang, Xinya Du, and Dong Yu. DSBench: How far are data science agents from becoming data science experts? In International Conference on Learning Representations, 2025. [7] Bohan Lyu, Yucheng Yang, Siqiao Huang, Jiaru Zhang, Qixin Xu, Xinghan Li, Xinyang Han, Yicheng Zhang, Huaqing Zhang, Runhan Huang, Kaicheng Yang, Zitao Chen, Wentao Guo, Junlin Yang, Xinyue Ai, Wenhao Chai, Yadi Cao, Ziran Yang, Kun Wang, Dapeng Jiang, Huan ang Gao, Shange Tang, Chengshuai Shi, Simon S. Du, Max Simchowitz, Jiantao Jiao, Dawn Song, and Chi Jin. Mls-bench: A holistic and rigorous assessment of ai systems on building better ai, 2026. [8] Qiuyang Mang, Wenhao Chai, Zhifei Li, Huanzhi Mao, Shang Zhou, Alexander Du, Hanchen Li, Shu Liu, Edwin Chen, Yichuan Wang, Xieting Chu, Zerui Cheng, Yuan Xu, Tian Xia, Zirui Wang, Tianneng Shi, Jianzhu Yao, Yilong Zhao, Qizheng Zhang, Charlie Ruan, Zeyu Shen, Kaiyuan Liu, Runyuan He, Dong Xing, Zerui Li, Zirong Zeng, Yige Jiang, Lufeng Cheng, Ziyi Zhao, Youran Sun, Wesley Zheng, Meiyuwang Zhang, Ruyi Ji, Xuechang Tu, Zihan Zheng, Zexing Chen, Kangyang Zhou, Zhaozi Wang, Jingbang Chen, Aleksandra Korolova, Peter Henderson, Pramod Viswanath, Vijay Ganesh, Saining Xie, Zhuang Liu, Dawn Song, Sewon Min, Ion Stoica, Joseph E. Gonzalez, Jingbo Shang, and Alvin Cheung. Frontiercs: Evolving challenges for evolving intelligence, 2025. [9] Marcos Nazario, Rodrigo Bonifacio, and Gustavo Pinto. Mitigating configuration differences between development and production environments: A catalog of strategies, 2025.
12
[10] Yanzhou Mu, Rong Wang, Juan Zhai, Chunrong Fang, Xiang Chen, Jiacong Wu, An Guo, Jiawei Shen, Bingzhuo Li, and Zhenyu Chen. Understanding llm-centric challenges for deep learning frameworks: An empirical analysis, 2025. [11] Avi Arora, Jinu Jang, and Roshanak Zilouchian Moghaddam. SetupBench: Assessing software engineering agents’ ability to bootstrap development environments. ArXiv, abs/2507.09063, 2025. [12] Aleksandra Eliseeva, Alexander Kovrigin, Ilia Kholkin, Egor Bogomolov, and Yaroslav Zharov. EnvBench: A benchmark for automated environment setup. ArXiv, abs/2503.14443, 2025. [13] Yijia Xiao, Runhui Wang, Luyang Kong, Davor Golac, and Wei Wang. CSR-Bench: Benchmarking LLM agents in deployment of computer science research repositories. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 12705–12723, 2025. [14] Yubang Wang, Chenxi Zhang, Bowen Chen, Zezheng Huai, Zihao Dai, Xinchi Chen, Yuxin Wang, Yining Zheng, Jingjing Gong, and Xipeng Qiu. ResearchEnvBench: Benchmarking agents on environment synthesis for research code execution. ArXiv, abs/2603.06739, 2026. [15] Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Daniel Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. OpenHands: An open platform for AI software developers as generalist agents. In International Conference on Learning Representations, 2025. [16] Qinyu Luo, Yining Ye, Shihao Liang, Zhong Zhang, Yujia Qin, Yaxi Lu, Yesai Wu, Xin Cong, Yankai Lin, Yingli Zhang, Xiaoyin Che, Zhiyuan Liu, and Maosong Sun. RepoAgent: An LLM-powered open-source framework for repository-level code documentation generation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 436–464, 2024. [17] Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, Pengfei Gao, Ping Yang, and Shuiguang Deng. CoReQA: Uncovering potentials of language models in code repository question answering. ArXiv, abs/2501.03447, 2025. [18] Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen, and Xiaodong Gu. SWE-QA: Can language models answer repository-level code questions? ArXiv, abs/2509.14635, 2025. [19] Zeyu Leo Liu, Shrey Pandit, Xi Ye, Eunsol Choi, and Greg Durrett. CodeUpdateArena: Benchmarking knowledge editing on API updates. ArXiv, abs/2407.06249, 2024. [20] Alexander Zhang, Marcus Dong, Jiaheng Liu, Wei Zhang, Yejie Wang, Jian Yang, Ge Zhang, Tianyu Liu, Zhongyuan Peng, Yingshui Tan, Yuanxing Zhang, Zhexu Wang, Weixun Wang, Yancheng He, Ken Deng, Wangchunshu Zhou, Wenhao Huang, and Zhaoxiang Zhang. CodeCriticBench: A holistic code critique benchmark for large language models. ArXiv, abs/2502.16614, 2025. [21] Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, 13
Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, Anurag Kashyap, Jan-Lucas Uslu, Jeffrey Li, Jianbo Wu, Minghao Yan, Song Bian, Vedang Sharma, Ke Sun, Steven Dillmann, Akshay Anand, Andrew Lanpouthakoun, Bardia Koopah, Changran Hu, Etash Guha, Gabriel H. S. Dreiman, Jiacheng Zhu, Karl Krauth, Li Zhong, Niklas Muennighoff, Robert Amanfu, Shangyin Tan, Shreyas Pimpalgaonkar, Tushar Aggarwal, Xiangning Lin, Xin Lan, Xuandong Zhao, Yiqing Liang, Yuanli Wang, Zilong Wang, Changzhi Zhou, David Heineman, Hange Liu, Harsh Trivedi, John Yang, Junhong Lin, Manish Shetty, Michael Yang, Nabil Omi, Negin Raoof, Shanda Li, Terry Yue Zhuo, Wuwei Lin, Yiwei Dai, Yuxin Wang, Wenhao Chai, Shang Zhou, Dariush Wahdany, Ziyu She, Jiaming Hu, Zhikang Dong, Yuxuan Zhu, Sasha Cui, Ahson Saiyed, Arinbjörn Kolbeinsson, Jesse Hu, Christopher Michael Rytting, Ryan Marten, Yixin Wang, Alex Dimakis, Andy Konwinski, and Ludwig Schmidt. Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces, 2026. [22] Louis Milliken, Sungmin Kang, and Shin Yoo. Beyond pip install: Evaluating LLM agents for the automated installation of Python projects. In 2025 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), pages 1–11, 2025. [23] Islem Bouzenia and Michael Pradel. You name it, i run it: An LLM agent to execute tests of arbitrary projects. Proceedings of the ACM on Software Engineering, 2(ISSTA):1054–1076, 2025. [24] Ruida Hu, Chao Peng, Xinchen Wang, Junjielong Xu, and Cuiyun Gao. Repo2Run: Automated building executable environment for code repository at scale. In Advances in Neural Information Processing Systems, 2025. [25] Jiayi Kuang, Yinghui Li, Xin Zhang, Yangning Li, Di Yin, Xing Sun, Ying Shen, and Philip S. Yu. Process-level trajectory evaluation for environment configuration in software engineering agents. In International Conference on Learning Representations, 2026. [26] Xiang Li, Siyu Lu, Federica Sarro, Claire Le Goues, and He Ye. HerAgent: Rethinking the automated environment deployment via hierarchical test pyramid. ArXiv, abs/2602.07871, 2026. [27] Jiaran Zhang, Luck Ma, Fanqi Wan, Di Qi, Xu Zhao, Jieyi Hou, Zhe Xie, Mengqiang Ren, Xin Wu, Zhewei Huang, Liangyu Chen, Qi Han, and Xiangyu Zhang. DockSmith: Scaling reliable coding environments via an agentic docker builder. ArXiv, abs/2602.00592, 2026. [28] Systems Research Artifacts. Systems Research Artifacts. https://sysartifacts.github.io/, 2026. Last accessed: May 2026.
14
Appendix The appendix is structured as follows: • Full list of benchmark source artifacts (Section A1). • The full agent system prompt used in all runs (Section A2). • The full diagnostic-agent prompt used to diagnose failed runs (Section A3). • Failure pattern counts across all-fail and mixed-outcome tasks (Section A4). • Additional trajectory phase definitions and outcome-level phase distributions (Section A5). • Additional model-specific trajectory signal definitions and metrics (Section A6). • Additional failed-command recovery definitions and distributions (Section A7). • Additional stopping behavior statistics by stop type (Section A8).
15
A1.
List of Benchmark Source Artifacts
Table A1 lists every task in DeployBench by source artifact , venue and source year, grouped by the three domains used in Section 3.2. Two source artifacts (Generative Adversarial Nets and ModelAgnostic Meta-Learning) each contribute two task variants, one CPU-only and one GPU-enabled, so the 51 tasks are drawn from 49 unique source artifacts. DeployBench uses these public source artifacts for research evaluation of deployment agents by running lightweight targets derived from the corresponding paper experiments. The benchmark release provides task metadata and our setup/verifier scripts, and does not redistribute third-party repositories, datasets, or model weights. Table A1: Full list of source artifacts in DeployBench (51 tasks across 49 unique artifacts). Source Artifact / Paper Title
Venue / Source Year
AI/ML (25 tasks) AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning ImageNet Classification with Deep Convolutional Neural Networks BERTScore: Evaluating Text Generation with BERT Counterfactual Reasoning for Out-of-distribution Multimodal Sentiment Analysis Deep Mutual Learning ERGO: Event Relational Graph Transformer for Document-level Event Causality Identification Generative Adversarial Nets (CPU variant) Generative Adversarial Nets (GPU variant) Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection Hyena Hierarchy: Towards Larger Convolutional Language Models High-Resolution Image Synthesis with Latent Diffusion Models LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (CPU variant) Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (GPU variant) MixText: Linguistically-Informed Interpolation of Hidden Space for SemiSupervised Text Classification MMSegmentation: OpenMMLab Semantic Segmentation Toolbox and Benchmark A Neural Algorithm of Artistic Style Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space QLoRA: Efficient Finetuning of Quantized LLMs Sample Efficient Grasp Learning Using Equivariant Models One-shot Imitation Learning via Interaction Warping A Simple Framework for Contrastive Learning of Visual Representations TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting YOLOv7: Trainable Bag-of-Freebies Sets New State-of-the-Art for Real-Time Object Detectors
ICLR 2023 NeurIPS 2012 ICLR 2020 ACM MM 2022 CVPR 2018 COLING 2022 NeurIPS 2014 NeurIPS 2014 ECCV 2024 ICML 2023 CVPR 2022 SIGIR 2020 ICML 2017 ICML 2017 ACL 2020 GitHub release, 2024 CVPR 2016 NeurIPS 2025 NeurIPS 2017 NeurIPS 2023 RSS 2022 CoRL 2023 ICML 2020 ICLR 2024 CVPR 2023 (continued on next page)
16
(Table A1 continued from previous page) Source Artifact / Paper Title
Venue / Source Year
Computer Systems (19 tasks) Detecting Logic Bugs in Database Engines via Equivalent Expression Transformation Efficient Auditing of Event-driven Web Applications Specification and Verification in the Field: Applying Formal Methods to BPF Just-inTime Compilers in the Linux Kernel K9db: Privacy-Compliant Storage for Web Applications by Construction Fast, Flexible, and Practical Kernel Extensions NrOS: Effective Replication and Sharing in an Operating System Operating System Support for Safe and Efficient Auxiliary Execution PaSh: Light-touch Data-Parallel Shell Processing Practically Correct, Just-in-Time Shell Script Parallelization Polyjuice: High-Performance Transactions via Learned Concurrency Control QSYM: A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing RUDRA: Finding Memory Safety Bugs in Rust at the Ecosystem Scale Home, SafeHome: Smart Home Reliability with Visibility and Atomicity Speedy Transactions in Multicore In-Memory Databases SILT: A Memory-Efficient, High-Performance Key-Value Store Storm: Refinement Types for Secure Web Applications Theseus: An Experiment in Operating System Structure and State Management µSlope: High Compression and Fast Search on Semi-Structured Logs Efficiently Mitigating Transient Execution Attacks using the Unmapped Speculation Contract
OSDI 2024 EuroSys 2024 OSDI 2020 OSDI 2023 SOSP 2024 OSDI 2021 OSDI 2022 EuroSys 2021 OSDI 2022 OSDI 2021 USENIX Security 2018 SOSP 2021 EuroSys 2021 SOSP 2013 SOSP 2011 OSDI 2021 OSDI 2020 OSDI 2024 OSDI 2020
Scientific Computing (7 tasks) GROMACS: High Performance Molecular Simulations through Multi-level Parallelism from Laptops to Supercomputers The Kaldi Speech Recognition Toolkit High-level, High-resolution Ocean Modeling at All Scales with Oceananigans OpenFOAM: Open Source CFD Toolbox QUANTUM ESPRESSO: A Modular and Open-Source Software Project for Quantum Simulations of Materials Integrated Analysis of Multimodal Single-Cell Data wannier90: A Tool for Obtaining Maximally-Localised Wannier Functions
SoftwareX 2015 ASRU 2011 JOSS 2020 GitHub release, 2025 J. Phys. Condens. Matter, 2009 Cell 2021 Comput. Phys. Commun., 2008
17
A2.
Agent System Prompt
For every run we provide the agent with the same system prompt, reproduced verbatim in Figure A1. The placeholder <WORKDIR> is substituted with the per-task absolute workspace path on the VM (e.g., /home/vm/Desktop/<task>). The agent is also given the per-task deployment instructions (paper PDF and code repository under <WORKDIR>), described separately in Section 3.2.
A3.
Diagnostic-Agent Prompt for Failed Runs
For each failed run, we run an LLM-based diagnostic agent. The agent is given SSH access to the final VM state, the path to the reference setup script for the task, and the verifier outputs, and is asked to identify the root cause of the failure. Diagnostic reports are used only for failure attribution and never override the verifier’s pass/fail verdict (Section 3.3). The full system prompt is reproduced verbatim in Figure A2. Placeholders <SANDBOX IP>, <SANDBOX KEY>, <WORKDIR>, and <TASK DEF PATH> are substituted at run time with the target VM’s IP, the SSH key path, the per-task workspace, and the per-task definition directory respectively.
A4.
Failure Pattern Counts for Tasks with Failed Runs
This section provides the failure-pattern counts supporting the failure analysis in Section 5.1. For each task, we map the diagnostic agent’s Root Cause report from failed runs to the five recurring failure patterns using keyword and regex heuristics. A task is counted under a pattern if any of its failed runs is attributed to that pattern. Table A2 reports the number of tasks in each group assigned to each pattern: the 20 all-fail tasks and the 30 mixed-outcome tasks (at least one model succeeded and at least one failed). Every all-fail task combines at least two patterns and 70% combine three or more, versus 40% on the mixed-outcome side. Dependency/package issues and native-build failures are the most frequent patterns in both groups, while GPU/CUDA setup, artifact/output mismatch, and system/VM-level execution are more concentrated in the all-fail set.
A5.
Additional Trajectory Analysis
We segment each shell command into sub-commands by splitting on top-level &&, ||, or ;, which exposes work hidden inside large chained commands. This section provides the phase taxonomy used in Section 5.2. Table A3 defines each sub-command phase, and Table A4 reports the phase distribution split by verifier outcome (success vs. failed).
18
You are an autonomous LLM agent whose job is to set up a research paper's artifact on a provided Ubuntu machine and verify it with a simple test. You do NOT run the full paper experiments. Hard constraints (must follow) - You have terminal access to a real Ubuntu host with root privileges. - Username: vm - passwordless sudo inside the sandbox - ALL project-related files (source tree, extracted archives, build outputs, virtualenvs/conda envs, downloaded datasets, logs, VM disk images, checkpoints) MUST live under: <WORKDIR> - I will later run experiments from within <WORKDIR>, so your setup must be reproducible from there. - Do NOT use Docker. If the authors provide a Dockerfile or "run via docker" instructions, ignore them and reproduce the same environment directly on the host. - For system paths (e.g. /etc, /usr, /var): do NOT use FileEditor to view/edit files directly (it won't use sudo). Use execute_bash + sudo (or copy to a writable path, edit, then sudo-copy back). - Only if the artifact is inherently system-level and cannot reasonably run directly (e.g., modified Linux kernel / kernel module requiring reboot, special kernel configs), you MAY create and use a VM inside the host and run it there (e.g., QEMU/KVM). If you do, keep all VM images/config under <WORKDIR> and provide exact VM run commands. Goal - Install dependencies, build/configure the artifact, and complete any required downloads (e.g. models, datasets) so that the environment is ready. - Run a simple smoke test to confirm the setup works (e.g. a small demo). Do NOT run the full paper experiments or long-running benchmarks. - Prefer the simplest native setup: system packages via apt, plus Python venv as needed. - If the host machine has a GPU and the artifact can use it, you must make the GPU usable and use it; CPU-only fallback does not count as success. Operating procedure (follow in order) 1) Initial inspection - List the contents of <WORKDIR>. - Identify: the code folder and the paper PDF. - Create vm in a new directory vm under <WORKDIR>, ONLY if you must use a VM 2) Read instructions and infer requirements - Read the paper PDF to understand: required OS/kernel assumptions, hardware assumptions, and what a minimal smoke test would be (not the full benchmarks). - Read README / INSTALL / scripts in the code. - If instructions assume Docker, translate them into native host steps. Agent skills (optional -- use when helpful) You have access to the following helper functions. Use them when they are useful; the file editor cannot read binary files (e.g. PDF). Run them via bash so output appears in the terminal. Use absolute paths (e.g. under <WORKDIR>). - parse_pdf(file_path): Extract and print text from a PDF (e.g. the paper). Does not include figures. - parse_docx(file_path): Extract and print text from a DOCX file. - parse_pptx(file_path): Extract and print text from a PowerPoint file. - parse_latex(file_path): Convert LaTeX to plain text and print. Invoke from the OpenHands Python environment (required for imports): cd $HOME/OpenHands/openhands-cli && uv run python -c 'from openhands.runtime.plugins.agent_skills.file_reader.file_readers import parse_pdf; parse_pdf("ABS_PATH")' Replace parse_pdf with parse_docx, parse_pptx, or parse_latex as needed, and ABS_PATH with the absolute file path. Output will appear in the command observation. 3) Dependency resolution - Determine all build/runtime dependencies (compilers, libraries, Python/Rust/Go/Java, CUDA, etc.). - Install with apt when appropriate. - For language-specific deps: - Python: Keep the paper's environment isolated from the agent's own runtime; Create and use a project-specific venv under <WORKDIR>/env/. Do not use uv, conda, or any other environment. - Rust/Go/Node: install toolchains if required - User consistency: Use sudo only for system packages (apt install) and global binaries (/usr/local/bin). Everything else -- language toolchains (Rust, Python venv), package managers, and build commands -- must be installed and run as the same non-root user I provided. - Record exact versions where possible (package versions, git commit hashes, pip freeze, compiler versions). 4) Build and configure - Build the artifact as required (e.g., make/cmake/bazel/meson). - Fix path issues so everything runs when invoked from within <WORKDIR>. - If the artifact or smoke test requires downloaded models, datasets, or weights: download them and ensure the smoke test can use them. Do not skip downloads needed for a minimal run. 5) Run a simple smoke test - Execute a minimal check that the setup works (e.g. small demo, or a short run with minimal data). Do NOT run the full paper experiments or long benchmarks. 6) If a VM is needed (only as last resort) - Explain why native host execution is infeasible. - Use QEMU/KVM if available; create VM disk under <WORKDIR>/vm/. - Provide: - VM OS image source and checksums if applicable - VM config (CPU/RAM/disk) and exact launch command(s) - How files are shared between host and VM (e.g., virtiofs/9p/scp) while keeping project files in <WORKDIR> - The exact commands inside the VM to build and run the smoke test Communication requirements (what you must report at the end) - As soon as you have finished the setup (success or blocked), you MUST call the finish tool with a final message. Do not stop after a bash or file action without concluding. - You MUST write a RUNBOOK file on disk so it can be collected: create exactly one file named RUNBOOK.MD under <WORKDIR> (i.e. <WORKDIR>/RUNBOOK.MD) with the runbook content. The runbook must include: - Setup status: SUCCESS / PARTIAL / FAILED (and why) - Artifact root directory (exact path under <WORKDIR>) - Exact commands to: 1) activate the environment, 2) run the smoke test you used to verify the setup - Optionally: how one would run the main paper experiment(s) (you did not run them; document for reference only) - Expected output of the smoke test (and, if documented, of the main experiment) - Versions: OS, kernel, gcc/clang, python, pip freeze (or equivalent) - If VM used: VM location, launch command, login credentials, and how to run the smoke test inside it - Your final (finish) message to the user should summarize the same information; the RUNBOOK.MD file is the canonical record that will be collected from the machine. - If you are blocked (e.g. missing dependency, build error), still write RUNBOOK.MD with what was done and what failed, then call finish. Safety / guardrails - Do not exfiltrate data. - Do not delete or move the user-provided archives and paper. - Do not modify files outside <WORKDIR> except for system dependency installation (apt) that is necessary.
Figure A1: Agent system prompt. The task-specific paths are replaced by placeholders.
19
You are an expert systems engineer tasked with diagnosing why a research software environment failed to set up correctly on a bare-metal Linux machine. You have SSH access to the target machine. Perform a thorough, systematic diagnosis and produce a structured report. ## Hard constraints (must follow) - You have terminal access to a bare-metal Ubuntu sandbox with root privileges. - sandbox IP: <SANDBOX_IP> - Username: vm - ssh key file: <SANDBOX_KEY> - passwordless sudo inside the sandbox - ALL project-related files (source tree, extracted archives, build outputs, virtualenvs/conda envs, downloaded datasets, logs, VM disk images, checkpoints) are live under: <WORKDIR> ## Reference Setup Procedure The intended environment setup steps for this task are documented in: <TASK_DEF_PATH>/setup_reference.sh Read this file first (`ssh -i <SANDBOX_KEY> -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vm@<SANDBOX_IP> cat <TASK_DEF_PATH>/setup_reference.sh`) to understand what the correct setup looks like, then use it as the ground truth when diagnosing what went wrong. ## Your Goal Identify the root cause(s) of environment setup failure. Be specific: name the exact package, version conflict, missing driver, failed download, or compilation error. Do not speculate -- base every conclusion on evidence you collect from the machine. ## Diagnosis Checklist Work through each section. Skip sections that are clearly irrelevant to the task. ### 1. GPU & CUDA (if verify script needs gpu) - GPU model and driver version: `nvidia-smi` - CUDA toolkit version: `nvcc --version`, `ls /usr/local/cuda*` - cuDNN presence: `find /usr -name 'cudnn*.h' 2>/dev/null` - Check if installed packages match the CUDA version (e.g. torch+cu121 vs CUDA 12.4) ### 2. System Dependencies - OS and kernel: `uname -a`, `lsb_release -a` - Key system libraries: `dpkg -l | grep -E 'libssl|libffi|zlib|libc6|build-essential|cmake|ninja|openmpi|libopenblas'` - Missing shared libraries for any binary: `ldd <binary> 2>&1 | grep 'not found'` - Check for version mismatches between system libs and what packages expect ### 3. Python Environment - Python version: `python3 --version` - Active venv / conda env: `which python`, `pip --version` - Installed packages: `pip freeze` - Failed installs: check `pip install` logs if available - Conflicting dependencies: look for version pins that are mutually incompatible - Missing packages that the code imports but are not installed ### 4. Dataset & Model Downloads - Check if expected data files exist in the workspace - Check download logs for HTTP errors, timeouts, or authentication failures - Verify file integrity if checksums are available (truncated downloads) - Check available disk space: `df -h` ### 5. Compilation & Build Errors - Check for `.log` files in the build directory - Look at `make` output if captured - Check for missing headers or libraries that caused compilation to fail - Verify compiler version: `gcc --version`, `g++ --version`, `nvcc --version` - Check CMake cache for misconfigured paths ### 6. Version Compatibility - Identify any API breakage between installed package versions and the code - Check if the code was written for an older version of a framework (e.g. old PyTorch API, old TF 1.x style code) - Look for deprecation errors or `AttributeError`/`ImportError` in logs ### 7. Verify Script Failure - Show the exact output of the verify script: `cat verify.sh`, then run it and capture output - Identify which specific check failed and why - Show the content of any missing output files that were expected ### 8. Runtime Errors - Check workspace for any `.log`, `stderr`, `stdout`, or `error` files - Look at the last N lines of any training/evaluation logs - Identify OOM (out-of-memory) errors, NCCL errors, or timeout issues ## Output Format Produce a report with the following structure: **Root Cause**: [One sentence summary of the primary failure reason] **Evidence**: - [Specific finding 1 with exact output or file path] - [Specific finding 2] - ... **Secondary Issues** (if any): - [Other problems found that may cause failure even after fixing root cause] **Recommended Fix**: - [Concrete, actionable steps to fix each issue] Be concise. Paste only the relevant portions of command output, not full dumps. If a section shows no issues, say "OK" and move on.
Figure A2: Diagnostic-agent prompt for failed-run diagnosis. The runtime-specific values are replaced by placeholders.
20
Pattern
All-fail (20)
Dependency / package / version Native build / toolchain GPU/CUDA setup Artifact / output mismatch System / VM-level execution
17 (85%) 17 (85%) 12 (60%) 11 (55%) 11 (55%)
23 (77%) 22 (73%) 11 (37%) 10 (33%) 8 (27%)
3.40 20/20 (100%) 14/20 (70%)
2.47 25/30 (83%) 12/30 (40%)
Avg. patterns per task Tasks with ≥ 2 patterns Tasks with ≥ 3 patterns
Mixed-outcome (30)
Table A2: Failure-pattern counts across the 20 all-fail tasks and the 30 mixed-outcome tasks. Per-task pattern union is computed across failed runs: a task is counted under a pattern if any failed run for that task is attributed to that pattern.
Phase
Description
Explore Env. Inspect FileRead Install Build FileEdit Config Assets Run/Test Service Finish
Browse repository structure or metadata, such as listing files, searching paths, or checking git state. Inspect the execution environment, such as OS, Python, GPU, driver, disk, or installed packages. Read documentation, source files, logs, or paper text through shell commands. Install system-level or language-level dependencies, such as apt, pip, npm, or cargo packages. Compile or build native components, binaries, kernels, or other generated artifacts. Modify source files, scripts, patches, or configuration files through shell commands. Configure paths, environment variables, permissions, directories, symlinks, or runtime settings. Download or clone external datasets, models, checkpoints, repositories, or other required assets. Run artifact code, demos, tests, experiments, training, inference, binaries, or service checks. Start or manage services, background processes, emulators, simulators, or QEMU-based execution. Write the required runbook or perform final completion-related actions.
Table A3: Deployment phase taxonomy used for sub-command-level trajectory analysis.
Outcome
Explore
Env. Inspect
FileRead
Install
Build
FileEdit
Config
Assets
Run/Test
Service
Finish
Success Failed
52.6% 54.3%
10.7% 6.8%
13.1% 12.5%
3.8% 5.0%
3.0% 4.0%
1.5% 1.6%
5.6% 6.7%
0.9% 0.8%
5.2% 5.5%
1.9% 1.1%
1.6% 1.7%
Table A4: Sub-command-level deployment phase distribution, split by verifier outcome (successful vs. failed runs). Each cell reports the share of sub-commands labeled with that phase.
21
A6.
Additional Trajectory Pattern Analysis
This section defines the trajectory signals used for the model-specific pattern analysis in Section 5.2. Table A5 explains each signal, and Table A6 reports the corresponding model-level counts. Here Soft timeout is an observation produced when the shell tool stops waiting , not the bash tool’s optional input parameter timeout chosen by the agent . Signal
Source
Definition
Task tracker
OpenHands action
An OpenHands planning tool that lets the agent create or update an explicit TODO list during the run. We count whether each run uses this tool at least once.
Structured File Actions
OpenHands action
Native OpenHands file-reading or file-editing actions. These are distinct from reading or editing files through Bash commands such as cat, sed, tee, or heredocs.
MessageAction
OpenHands action
A natural-language message emitted by the agent. We use this as a signal of narration or status reporting, not as a deployment action by itself.
Soft timeout
OpenHands observation
shell
A tool-level observation marked with exit code=-1. It indicates that the shell tool stopped waiting before the command completed, rather than that the command necessarily terminated with a normal shell error.
Max-step termination
OpenHands time status
run-
A run stopped because OpenHands reached its configured maximum number of agent iterations.
Heredoc use
Derived commandlevel metric
A run contains shell heredoc patterns, often used to write scripts, configuration files, or RUNBOOK content through Bash.
Table A5: Trajectory signals used in the model-specific pattern analysis. Some signals are native OpenHands actions or runtime observations, while others are derived during our post-processing.
Model GPT-5.3-Codex GPT-5.4-Mini Gemini-3.1-Pro Grok-4.20
Task tracker
Structured File Actions
Heredoc
Soft Timeout
Max-step
Message Share
50/51 2/51 0/51 4/51
17/51 0/51 12/51 24/51
40/51 46/51 0/51 0/51
10/1898 47/1015 1299/2820 91/528
4/51 1/51 14/51 1/51
0.1% 2.9% 0.0% 13.3%
Table A6: Detailed model-specific trajectory metrics. Task tracker, Structured File Actions, and Heredoc report the number of runs in which the behavior appears. Soft Timeout is reported over paired shell-command observations. Max-step is the number of runs that hit max-step termination. Message Share is the fraction of OpenHands agent actions that are MessageActions.
22
A7.
Additional Recovery Analysis
This section expands the recovery analysis in Section 5.2. Table A7 defines the five recovery labels, Table A8 shows how one label is selected when multiple behaviors appear, and Table A9 reports the per-model distribution. The per-outcome distribution is reported in the main text (Table 6). Primary recovery label selection. When multiple recovery types apply, we pick the highest-priority match. Table A8 lists the nine ordered trigger conditions , each resolving to one of the five final labels. Per-model distribution. GPT-5.3-Codex actively diagnoses failures by inspecting logs (136/189), Gemini-3.1-Pro mostly spins in Loop (much of it soft-timeout polling rather than real recovery), and Grok-4.20 tends to give up early (Table A9). This supports the outcome-level finding (Section 5.2) that successful runs are more often inspect or adapt , while failed runs are more often repeat waiting or retry loops . Recovery label
Description
Inspect
The agent reads logs, error output, source files, documentation, or configuration files after a failed command.
Adapt
The agent changes the command, script, arguments, configuration, or environment before retrying. This includes generic modify-and-retry, switching to a different package, dependency version, toolchain, or installation path, and clean-and-rebuild actions.
Loop
The agent enters repeated or unresolved recovery behavior, including softtimeout polling and repeated failure without meaningful change.
Abort/Finish
The agent stops, writes a final report, or otherwise gives up shortly after the failed command.
Other
The agent’s next actions are an exact retry of the failed command, or do not fit any of the above categories.
Table A7: Recovery taxonomy used to classify responses after failed shell-command observations.
A8.
Additional Stopping Analysis
This section expands the stopping behavior analysis in Section 5.3. We use three stop types in the trajectory analysis: self-stop means that the agent explicitly ends the run, timeout means that the run reaches its wall-clock time budget, and max-step means that the run reaches the maximum number of allowed agent interaction steps. Table A10 reports verifier outcomes under each stop type.
23
Priority
Final label
Trigger condition (first match wins)
1
Abort/Finish
2
Loop
3
Other
4
Inspect
5
Adapt
6
Adapt
7
Adapt
8
Loop
9
Other
No subsequent shell action, or the next action writes a runbook / matches a Finish signal. Soft-timeout polling: failed command has exit code=-1 and the next action is empty input, Ctrl-C, or another soft timeout. Next command is byte-equal (after whitespace normalization) to the failed command (exact retry without meaningful change). Any of the next 1–3 actions reads logs, source, or documentation (tail, cat, sed -n, grep, rg, less, find, ls, dmesg, pip show). Window text contains an install command together with version pins or driver/CUDA markers (==, <=, >=, cuda, cu11, cu12, driver, headless, python[0-9]). Window text contains a cleanup command (make clean, rm -rf, git clean, cargo clean, rebuild). Next command is different from the failed command and lands in an action phase (Install, Build, FileEdit-via-Bash, Config, Run/Test, Service). Repeated failure: subsequent window has further non-zero exits and contains at most two distinct normalized command forms. None of the above conditions match.
Table A8: Priority order used to assign one final recovery label per failed command (higher priority wins when multiple conditions match). The same final label can arise from different trigger conditions: Loop from priorities 2 and 8, Adapt from priorities 5, 6, and 7, and Other from priorities 3 and 9.
Model GPT-5.3-Codex GPT-5.4-Mini Gemini-3.1-Pro Grok-4.20
Failed Cmds.
Inspect
Adapt
Loop
Abort/Finish
Other
189 305 1748 192
136 141 181 28
34 83 229 15
7 37 1294 76
7 41 25 70
5 3 19 3
Table A9: Per-model recovery distribution after failed shell-command observations.
24
Model
Stop type
Failed
Success
GPT-5.3-Codex GPT-5.3-Codex GPT-5.3-Codex GPT-5.4-Mini GPT-5.4-Mini GPT-5.4-Mini Gemini-3.1-Pro Gemini-3.1-Pro Gemini-3.1-Pro Grok-4.20 Grok-4.20 Grok-4.20
Self-stop Timeout Max-step Self-stop Timeout Max-step Self-stop Timeout Max-step Self-stop Timeout Max-step
12/31 9/16 4/4 42/45 4/5 1/1 15/28 8/9 14/14 28/32 16/18 1/1
19/31 7/16 0/4 3/45 1/5 0/1 13/28 1/9 0/14 4/32 2/18 0/1
Table A10: Verifier outcome by stop type. Each row uses runs with the corresponding stop type as the denominator.
25