arXiv CSopen access

D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

D3-Gym: Constructing Real-World Verifiable Environments for Data-Driven Discovery Hanane Nour Moussa1 ,∗ Yifei Li1 ,∗ Zhuoyang Li1 , Yankai Yang1 , Cheng Tang1 , Tianshu Zhang1 , Nesreen K. Ahmed2 , Ali Payani2 , Ziru Chen1 , Huan Sun1 1 The Ohio State University 2 Cisco Research {moussa.45,li.14042,sun.397}@osu.edu

arXiv:2604.27977v1 [cs.AI] 30 Apr 2026

Abstract Despite recent progress in language models and agents for scientific datadriven discovery, further advancing their capabilities is held back by the absence of verifiable environments representing real-world scientific tasks. To fill this gap, we introduce D3-Gym, the first automatically constructed dataset with verifiable environments for scientific Data-Driven Discovery. D3-Gym comprises (1) 565 tasks sourced from 239 real scientific repositories across four disciplines where (2) each task is equipped with a natural language instruction, an executable environment with pre-installed dependencies, input dataset and artifact previews, a reference code solution, and an automatically synthesized evaluation script. Rigorous evaluation of the quality of the verification signal in D3-Gym confirms that our evaluation scripts achieve 87.5% agreement with human-annotated gold standards and strong alignment in domain-specific evaluation logic, showing their scientific soundness. Further, training on trajectories sampled from D3-Gym yields consistent and substantial gains across Qwen3 models of varying sizes on ScienceAgentBench, boosting Qwen3-32B by 7.8 absolute points and substantially shrinking the gap with strong proprietary models. All D3Gym artifacts (environments, creation workflow, trajectories, and models) can be found at https://github.com/OSU-NLP-Group/D3-Gym. Real-world scientific tasks

Diverse disciplines and datasets

Use the RWR algoritm to predict circRNA disease FASTA files

cross-validation and report accuracy, recall, precision, F1, AUC, and AUPR. Save metrics to RWR.csv and the ROC curve as RWR_ROC.png.

Executable Environments

Molecular structures

D3-Gym

GIS maps

Tabular

EEG time series

Task-specific verification Eval Script Verification Signals

rdkit ase pysam geopandas

Solution

SR@3 on SAB (%)

associations from provided data. Perform 5-fold

30 20 10

LLM

File dependencies and scientific libraries

Downstream uses

(a)

Ours Open-weight Proprietary D3-Gym 32B 27.4 Qwen3 Qwen3 80B 235B D3-Gym 21.4 22.6 14B 19.6

Qwen3 14B 13.7

14B

Claude Sonnet 4.5 35.2 o1-preview 34.3

Qwen3 32B 19.6 N/A

32B

80B

235B

Model size (b)

Figure 1: (a) Overview of D3-Gym. (b) Success Rate (SR@3) on ScienceAgentBench (SAB) (Chen et al., 2025) of D3-Gym trained models based on Qwen3-14B and Qwen3-32B and other larger open-weight and proprietary reasoning models.

Introduction

Language agents (Su et al., 2024; Sumers et al., 2024) are beginning to reshape scientific research. Powered by large language models (LLMs), they can now meaningfully assist ∗ Equal contribution.

researchers in their workflows, from literature synthesis (Skarlinski et al., 2024; Asai et al., 2026) to domain-specific scientific reasoning (Baker et al., 2025; Narayanan et al., 2025). In particular, there has been significant research interest in developing language agents for data-driven discovery (Hey et al., 2009; Majumder et al., 2024), where agents can programmatically test new hypotheses (Majumder et al., 2025; Mitchener et al., 2025) and derive scientific insights from data (Chen et al., 2025; Shojaee et al., 2025a;b). Despite this progress, further advancing these capabilities, particularly for open-weight LLMs (Zhu et al., 2025), is held back by the absence of suitable infrastructure for scientific tasks. A critical component in this infrastructure is real-world verifiable environments. Recent work such as Autoresearch (Karpathy, 2026) highlights their value: given a manually prepared task instruction, a dataset, an evaluation script, and a repository with pre-installed dependencies, coding agents can use the resulting executable feedback loop to improve the task performance and run dozens of experiments in a few hours for faster iteration. This result highlights how powerful task-specific verification signals can be for developing stronger models and agents. However, it also reveals a key bottleneck: constructing such environments currently requires substantial manual effort, making them difficult to scale beyond a small number of tasks. In this paper, we fill this gap by presenting an automated pipeline that constructs real-world, verifiable coding environments for data-driven discovery. Unlike software engineering, where unit tests already present in repositories can be directly repurposed as verification signals (Yang et al., 2025b; Jain et al., 2025; Pan et al., 2025), automatically constructing environments for scientific tasks is particularly challenging because related repositories rarely contain pre-existing evaluation scripts, and programs produce domain-specific outputs whose correctness cannot be assessed by any universal criterion. To address this challenge, we start from a large pool of candidate tasks collected using AutoSDT (Li et al., 2025) and design a series of rigorous filtering steps, followed by task execution and output validation, to retain only the highest-quality and most ecologically valid task instances. Moreover, we address a key challenge, evaluation script generation, to create verifiable environments around these task instances. Specifically, we use LLMs to synthesize evaluation scripts in two stages: first writing a detailed, task-specific evaluation plan, and then translating that plan into an executable script to complete environment construction. The resulting dataset, D3-Gym, comprises 565 verifiable environments sourced from 239 unique scientific repositories spanning bioinformatics, computational chemistry, geographic information science, and psychology and cognitive neuroscience (Figure 1a). Each instance includes an executable environment with pre-installed dependencies, input data files and their previews, a natural language task instruction, a reference solution adapted from the original repository, and a task-specific evaluation script with appropriate metrics and acceptance criteria. To the best of our knowledge, D3-Gym is the first automatically constructed dataset with verifiable environments for scientific data-driven discovery. Through comprehensive experiments, we demonstrate the quality of synthesized evaluation scripts and the utility of D3-Gym in improving open-weight LLMs. On a set of 50 tasks with human-annotated gold evaluation scripts, our synthesized evaluation scripts achieve 87.5% agreement on pass/fail verdicts and closely match the underlying evaluation logic, confirming that they are both accurate and methodologically sound. Furthermore, rejection sampling fine-tuning (Yuan et al., 2023) on D3-Gym substantially improves the Qwen3 family models (Team, 2025) on ScienceAgentBench (Chen et al., 2025). Notably, our 32B model improves its success rate over three trials by 7.8 absolute points, surpassing Qwen3235B-A22B and approaching strong proprietary models such as OpenAI o1 (Figure 1b). Overall, D3-Gym provides a scalable foundation for training and evaluation for data-driven discovery in real-world, verifiable environments.

Constructing Verifiable Environments

As shown in Figure 2, we construct D3-Gym through a four-stage workflow. The first three stages (§2.1) collect candidate tasks from scientific repositories, filter them for data integrity, and execute them to obtain reference outputs. These stages serve the purposes of 2

Tasks Collected using AutoSDT Task instruction

Code solution

Dependencies folder

Filtering and preview creation Coding agent Keep highest quality tasks

Create dataset previews

Execution and verification

Evaluation script generation

Data-driven discovery training environments Execution Environment

Install dependencies and execute code

Generate domain-specific evaluation plan

Task instruction Dataset previews

Judge output validity

Generate evaluation script

Code solution Evaluation script

Figure 2: Overview of the D3-Gym construction workflow. Candidate tasks from scientific repositories (Li et al., 2025) undergo quality filtering and dataset preview creation, then execution and validation to obtain reference outputs. These provide the context needed to generate high-quality, task-specific evaluation scripts via a planning-then-coding approach. ensuring task quality and produce the essential context that enables the core contribution of our workflow: automated generation of task-specific evaluation scripts with scientifically grounded metrics and acceptance criteria (§2.2). We validate the reliability of these scripts against human-annotated gold standards in §2.3. Further details are in Appendix C. 2.1

Task Pre-processing

Candidate Task Collection. We collect an initial pool of candidate tasks using AutoSDT (Li et al., 2025), which crawls scientific research repositories from GitHub, applies multistep filtering to identify files implementing data-driven scientific workflows, adapts them into reference code solutions, and pairs each with a natural language task instruction. The pipeline also extracts a dependency folder for each task containing all file dependencies from the original repository. Expert evaluation confirms that 93% of collected tasks are scientifically meaningful (Li et al., 2025). We ensure that all repositories used in ScienceAgentBench (Chen et al., 2025) are excluded, as it serves as our evaluation set. Filtering and Dataset Preview Creation. A limitation of AutoSDT is that its code adaptation stage may substitute synthetic data when an input file is absent from the original repository (Li et al., 2025). Since our goal is to construct high-fidelity environments grounded in real scientific workflows, we discard such tasks with synthetic data and retain only those whose reference solutions operate exclusively on real data files present in the original repository. We implement this filtering using Claude Code (Anthropic, 2025b), which verifies file dependencies and, for passing tasks, produces a dataset preview file for each input data file: a structured excerpt exposing the data schema. These previews provide concrete data context and later inform evaluation script generation. Execution and Output Verification. We execute each remaining reference solution in an isolated environment, using pipreqs1 to install required packages. While the previous step ensures data integrity, execution can still fail or produce degenerate outputs due to environment mismatches or subtle errors introduced during code adaptation. We employ a multimodal LLM-as-judge (GPT-5.2, 92.31% agreement with human; see Appendix C.3) to verify that (1) all requested outputs were produced and (2) each output is meaningful and non-degenerate. Tasks failing either criterion are discarded. The verified outputs, together with the dataset previews, form the context for evaluation script generation. 2.2

Evaluation Script Generation

A central challenge addressed in this work is generating discriminative evaluation scripts that correctly capture what constitutes a correct solution for a given task while rejecting incorrect ones. Inspired by Saha et al. (2025) and Yang et al. (2025a), we decompose evaluation script generation into two sequential phases: planning and coding. This separation encourages the planning LM to reason carefully about scientific validity without being distracted by implementation details, while allowing the coding LM to focus on producing 1 https://pypi.org/project/pipreqs/

Pass/Fail Agreement (%)

Evaluation Logic (1–5, ↑)

Method

Acc.

Recall

Spec.

Metric

Threshold

Artifact

Direct Prompting

85.1

0.0

100.0

2.07

2.16

2.66

Silver Eval. Scripts (Ours) – Planning – Dataset Preview – Code Output

87.5 74.4 62.9 84.6

66.1 3.4 18.6 0.0

91.0 86.3 70.4 98.9

4.00 2.00 1.96 2.89

3.82 2.04 1.74 2.64

4.20 2.68 2.64 2.62

Table 1: Quality validation of silver evaluation scripts against human-annotated gold scripts. Left: pass/fail agreement. High specificity with near-zero recall (e.g., Direct Prompting) indicates scripts that reject nearly all solutions indiscriminately. Right: evaluation logic alignment on metric choice, threshold & tolerance, and target artifact. correct, executable code faithful to the plan. This design choice is validated by our ablation in §2.3, where removing the planning phase significantly reduces performance. Specifically, in the planning phase, we provide Claude Sonnet 4.5 with the task instruction, dataset previews, and verified reference outputs and leverage its parametric scientific knowledge to produce a detailed, task-specific evaluation plan. The plan specifies three components: (1) which output artifacts to inspect, (2) which metrics are scientifically appropriate for each artifact, and (3) the acceptance criteria to apply, including tolerance thresholds, exact versus approximate matching, and domain-justified performance bounds. The evaluation plan is then passed to Claude Sonnet 4.5 in a separate call, which generates the final executable evaluation script in a single pass. We refer to these automatically generated scripts as silver evaluation scripts, to distinguish them from human-annotated gold evaluation scripts used to assess their reliability as follows. 2.3

Validating Evaluation Script Quality

A critical property of any evaluation script is the reliability of its verification signal. We therefore conduct a rigorous evaluation of our silver evaluation scripts along two complementary dimensions: (1) execution-based agreement, which measures whether silver scripts produce the same pass/fail verdicts as human-annotated gold scripts, and (2) evaluation logic agreement, which assesses whether the underlying evaluation methodology aligns with that of human-annotated gold scripts. 2.3.1

Setup

Validation set. We curate a held-out set of 50 task instances with gold evaluation scripts manually written and validated by Ph.D. students over a total of 175 person-hours. For each instance, we generate a corresponding silver evaluation script using the methodology described in §2.2 To obtain candidate solutions for computing execution-based metrics, we sample 10 solutions per task from Claude Opus 4.6 and GPT-5.4. These models were chosen because of their higher execution success rate yields the largest pool of solutions that pass execution and proceed to evaluation. After filtering for successful execution, this produces 424 candidate solutions, each scored independently by both the silver and gold scripts. Metrics. (1) Accuracy measures the proportion of candidate solutions for which the silver and gold scripts return the same verdict. (2) Recall measures the fraction of gold-passing solutions that also pass the silver script, capturing whether silver scripts wrongly penalize correct solutions. (3) Specificity measures the fraction of gold-failing solutions that also fail the silver script, capturing whether silver scripts wrongly accept incorrect ones. Finally, beyond these execution-based metrics, we assess the scientific validity of silver scripts directly via an LLM-as-judge (85% exact, 98% within-1 agreement with human annotators; see Appendix C.3) that scores each silver script against its gold counterpart on three essential aspects using a 1–5 Likert scale: (4) Metric Choice (whether the silver script uses the same metrics), (5) Threshold & Tolerance (whether it applies similar acceptance criteria, such 4

as comparable thresholds and error tolerance ranges), and (6) Target Artifact (whether it evaluates the same output artifacts). Further setup details are in Appendix D. Baselines. We compare our full evaluation script generation method (§2.2) against a direct prompting baseline that provides the task instruction and code solution to an LLM and produces the evaluation script in a single step. We also ablate three components of our pipeline: evaluation planning, access to dataset preview, and access to the code’s output. All evaluation script generation methods use Claude Sonnet 4.5 as the backbone LLM. 2.3.2

Results

As shown in Table 1, our method achieves the highest accuracy. A consistent pattern across the baseline and all ablations is the imbalance between low recall and high specificity, where the generated scripts are overly strict and reject solutions that the gold scripts would accept. Our full pipeline significantly mitigates this tendency, achieving 66.1% recall while maintaining 91.0% specificity. The ablations further reveal the contribution of each component. Removing evaluation planning substantially harms performance, reducing recall to just 3.4%, confirming that separating high-level planning from code generation is critical for such domain-specific tasks. Removing the dataset preview introduces schema expectation errors, substantially degrading both accuracy and specificity. Finally, removing access to the code solution’s output reduces recall to 0%, collapsing behavior to that of the direct prompting baseline. Meanwhile, the evaluation logic agreement scores in Table 1 confirm the methodological soundness of the silver scripts. Our method substantially outperforms the baseline and all ablations across all three aspects. The somewhat lower threshold score reflects a tendency of silver scripts to adopt slightly stricter acceptance criteria than their gold counterparts. Our manual analysis confirms that these remain within scientifically reasonable bounds: silver scripts target the same underlying quantities but may select among equally defensible conventions, such as requiring multiple complementary metrics jointly rather than a single one, or adopting a 1.96σ confidence interval instead of a 2σ envelope. The net effect is that borderline-correct outputs may fail the silver script while passing the gold one (examples in Appendix D.4). This mild strictness explains the 66.1% recall: solutions that narrowly pass gold thresholds are rejected by tighter silver criteria. Still, this does not compromise quality, since the high specificity ensures that accepted solutions are reliably correct.

D3-Gym

Using our verifiable environment creation workflow (§2), we present D3-Gym. Each task instance in D3-Gym comprises: an executable environment with pre-installed dependencies, dataset preview files, a natural language task instruction, a scientist-authored reference solution, and a task-specific evaluation script generated automatically. D3-Gym comprises a diverse set of 565 tasks drawn from 239 unique scientific repositories. Discipline and task type distribution. As shown in Figure 3, D3-Gym spans four scientific disciplines: Computational Chemistry, Bioinformatics, Geographic Information Science, and Psychology & Cognitive Neuroscience. Task types reflect the full arc of real-world scientific workflows, ranging from common operations such as statistical analysis, numerical computation, and visualization to more specialized tasks such as feature engineering, machine learning, and signal and image processing. Tasks frequently span multiple types, underscoring the multi-faceted nature of real scientific workflows. We provide examples of tasks in D3-Gym in Appendix D.1 Input and output modalities. As illustrated in Figure 3, D3-Gym encompasses a wide spectrum of scientific data formats. On the input side, in addition to conventional tabular, textual, and image data, a substantial portion of tasks operate on domain-specific representations such as molecular structure encodings (e.g. SMILES, FASTA), molecular simulation trajectories (e.g. TRAJ), and geospatial records (e.g. GPS logs). Outputs are similarly varied, 5

(a)

(b) 381

Statistical & numerical Computational chemistry (217)

565

Psychology & cog. neuroscience (173)

tasks

352

Data wrangling

257

Visualization & reporting

241

Feature eng. & ML prep

Bioinformatics (111) Geographic information science (64)

Signal & image proc.

Machine learning

194

Output

120

364

226

204

Tabular data

Text

Structured data

Molecular & chemical

Binary & serialized

Biological sequences

Images

Audio & signal

Geospatial

Time series

Visualizations

Figure 3: D3-Gym statistics. (a) Distribution of tasks by scientific discipline. (b) Distribution of tasks by type. Tasks carry multiple type labels (mean 2.4 labels per task). (c) Distribution of tasks by input and output modality. Tasks may carry multiple modalities (input: mean 1.11, max 3; output: mean 1.53, max 4). including visualizations and serialized artifacts such as trained models, Numpy arrays, time series, and domain-specific structures. This variety in modalities reflects the richness of end-to-end scientific workflows represented in D3-Gym. Scientific package coverage. Our analysis also reveals that tasks draw on a substantial number of third-party packages, spanning both the general-purpose scientific stack (numpy, pandas, matplotlib, scipy) and a rich set of domain-specific libraries: rdkit (molecular property computation) and ase (atomistic simulations) for computational chemistry, Bio and pysam (sequence alignment and genomic data parsing) for bioinformatics, LFPy (neural signal modeling) and dyconnmap (brain connectivity analysis) for neuroscience, and geopandas and folium (geospatial data manipulation and mapping) for geographic information science. This breadth of domain-specific tooling is a direct reflection of the genuine scientific diversity of D3-Gym, and a key source of its difficulty. Task difficulty. To gauge the difficulty of the tasks in D3-Gym, we randomly sample 75 tasks and evaluate GPT-5.4, Claude Opus 4.6, Qwen3-235B-A22B, and Qwen3-32B on their ability to generate executable code and to pass the evaluation script (full results in Table D.4, Appendix D.5). Even the most capable frontier models solve only around a third of the tasks (e.g., Claude Opus 4.6 solves merely 36%), and none consistently produce fully executable code, with GPT-5.4 and Claude Opus 4.6 achieving only 82.66% and 78.66% valid execution rates, respectively, likely due to limited pretraining exposure to the highly specialized scientific packages these tasks require. Overall, these results confirm that our tasks are solvable yet remain non-trivial.

Training using D3-Gym

D3-Gym provides a versatile training environment that can support a variety of approaches for improving model capabilities. Here, we demonstrate its utility by training open-weight models for scientific data-driven discovery — a setting where reliable, transparent, and reproducible systems are especially important (Spirling, 2023). Given the current performance gap of open-weight models on these tasks, we opt for training approaches that offer dense supervision and are effective at expanding model capabilities (Yue et al., 2025). We therefore adopt rejection-sampling fine-tuning (RFT) (Yuan et al., 2023), in which a model generates trajectories comprising full reasoning traces and final solutions for tasks in a training environment, and only successful trajectories are retained for fine-tuning. We experiment with two settings: RFT-Distill, where a smaller student model learns from trajectories generated by a stronger teacher, and RFT-Self, where a model learns from its own successful trajectories, a form of self-improvement. 6

ScienceAgentBench

Method

ScienceAgentBench-Verified

VER

SR@3

VER@3

VER

SR@3

VER@3

Qwen3-4B-Instruct Base RFT-Distill

4.2 7.2 (+3.0)

25.8 30.4 (+4.6)

5.9 8.8 (+2.9)

35.3 47.1 (+11.8)

4.2 7.2 (+3.0)

25.8 30.4 (+4.6)

5.9 8.8 (+2.9)

35.3 47.1 (+11.8)

Qwen3-4B Base RFT-Distill RFT-Self

4.9 7.2 6.9 (+2.0)

19.6 6.9 27.5 23.5 8.8 37.3 23.2 (+3.6) 11.8 (+4.9) 40.2 (+12.7)

5.2 7.2 6.9 (+1.7)

20.1 7.3 23.2 8.8 23.2 (+3.1) 11.8 (+4.5)

29.2 36.3 38.2 (+9.0)

Qwen3-8B Base RFT-Distill RFT-Self

6.5 29.7 11.8 8.5 31.7 14.7 10.1 (+3.6) 36.9 (+7.2) 12.8 (+1.0)

43.1 48.0 43.1 (0.0)

8.2 31.2 13.8 8.2 29.1 13.7 10.8 (+2.6) 33.7 (+2.5) 13.7 (-0.1)

44.7 43.1 41.2 (-3.5)

Qwen3-14B Base RFT-Distill RFT-Self

9.1 35.3 13.7 10.5 35.0 13.7 13.1 (+4.0) 37.6 (+2.3) 19.6 (+5.9)

49.0 48.1 54.9 (+5.9)

10.3 36.3 15.0 10.5 33.0 13.7 14.1 (+3.8) 37.3 (+1.0) 20.6 (+5.6)

50.0 47.1 54.9 (+4.9)

Qwen3-32B Base RFT-Self

14.7 36.3 19.6 18.9 (+4.2) 43.8 (+7.5) 27.4 (+7.8)

50.9 58.8 (+7.9)

15.0 37.0 20.0 19.3 (+4.3) 41.8 (+4.8) 28.4 (+8.4)

51.0 54.9 (+3.9)

Table 2: Main results on ScienceAgentBench and ScienceAgentBench-Verified (%, ↑). We report average Success Rate (SR) and Valid Execution Rate (VER) across three independent runs, as well as best-of-three performance (SR@3 and VER@3). For RFT-Self, numbers in parentheses indicate absolute change from the corresponding base model. Best results within each model block are bolded.

4.1

Experimental Setup

Models. We experiment with four student models from the Qwen3 family (Team, 2025): Qwen3-4B, Qwen3-8B, and Qwen3-14B in thinking mode, and Qwen3-4B-Instruct as a non-thinking variant. For expert distillation, we use Qwen3-32B as the teacher. Training. For each model, we sample 16 trajectories per task in D3-Gym, yielding 6,780 trajectories per model, and retain only those that pass the silver evaluation scripts. The number of successful trajectories reflects growing capability with model size: Qwen3-4B (1,126), Qwen3-8B (1,306), Qwen3-14B (1,788), and Qwen3-32B (2,153). In RFT-Distill, the student models are fine-tuned on successful trajectories from the Qwen3-32B teacher, and in RFT-Self, the models are fine-tuned on their own successful trajectories. For Qwen3-4BInstruct, we apply only RFT-Distill using the final solutions without reasoning traces. Evaluation. We evaluate on ScienceAgentBench (Chen et al., 2025), where a model is given a task instruction and dataset information and must generate a complete Python program that processes the input data, implements the required analysis, and saves results to the correct output path. We also evaluate on ScienceAgentBench-Verified, a manually verified version of ScienceAgentBench introduced in this work. Specifically, the authors inspected tasks in the original benchmark that all LLMs fail to solve. Among these tasks, we identified and fixed nine unclear instructions, two erroneous gold programs, and one redundant gold execution result (more details in Appendix F). We report two metrics on both ScienceAgentBench variants: Success Rate (SR), whether program outputs satisfy human-annotated success criteria, and Valid Execution Rate (VER), whether the program executes without errors and produces output in the correct location. For each model we conduct 3 independent inference runs and report the average across runs, reflecting typical performance, and the best of 3, reflecting broader model capability. Further training details are provided in Appendix E. 7

D3-Gym-4B-Thinking (11.8) D3-Gym-4B-Instruct (8.8) Qwen3-4B-Thinking AutoSDT-4B-Instruct (6.9) (6.8) Qwen3-4B-Instruct (5.9)

9 6 3 0

Qwen3-4B Qwen3-8B Qwen3-14B Qwen3-32B

25 SR@3 on SAB (%)

SR@3 on SAB (%)

27.4%

19.6%

12.8% 11.8%

10 5

Training samples

(a)

200

500

Training samples

(b)

Figure 4: (a) Comparison between training on AutoSDT-5K (Li et al., 2025) and our D3-Gym. (b) Scaling analysis across different model sizes. 4.2

Main Results

Training on D3-Gym consistently improves performance across model sizes. As shown in Table 2, training on trajectories sampled from D3-Gym improves both SR and VER across all model sizes, indicating that models learn to produce programs that are both executable and scientifically correct. Gains are evident not only in average performance but also in best of 3 performance (SR@3), where fine-tuned models solve substantially more tasks across three independent runs, reflecting broader capability gains rather than just improved typical-case performance. Generally, these gains also scale with model size: SR@3 improves by +1.0, +5.9, and +7.8 for the 8B, 14B, and 32B models respectively. Models learn effectively from their own trajectories. Across thinking models, RFT-Self generally matches or outperforms RFT-Distill, with the gap widening at larger scales. For Qwen3-4B and Qwen3-8B, neither approach is consistently better: RFT-Distill leads in some settings (e.g., average SR) while RFT-Self leads in others (e.g., SR@3), suggesting that at smaller scales the two strategies are roughly comparable. At 14B, however, RFTSelf becomes consistently superior, improving SR@3 by 5.9 points while RFT-Distill yields no improvement and even degrades VER@3. A plausible explanation is that on-policy trajectories are better aligned with the model’s own distribution (Zhao et al., 2026). To verify this, we score self-generated and 32B teacher-generated responses under each RFT-Self model on 64 randomly sampled shared prompts, using negative log-likelihood (NLL) and perplexity (PPL). Both Qwen3-8B (NLL: 0.252 vs. 0.652; PPL: 1.29 vs. 1.93) and Qwen3-14B (NLL: 0.261 vs. 0.543; PPL: 1.30 vs. 1.73) assign substantially lower NLL and PPL to selfgenerated responses, with self-generated responses preferred on all 64 prompts in both cases. This confirms that self-generated trajectories are closer to the model’s learned distribution, enabling more effective learning. D3-Gym-32B approaches the performance of larger and proprietary models. As shown in Figure 1b, training on D3-Gym enables smaller models to match or even surpass significantly larger ones. Notably, D3-Gym-32B outperforms both Qwen3-80B-A3B and Qwen3-235BA22B, two larger mixture-of-experts reasoning models, and narrows the gap with the strong proprietary o1-preview (OpenAI, 2024) and Claude Sonnet 4.5 (Anthropic, 2025a). Similarly, D3-Gym-14B sees substantial gains, reaching the performance of Qwen3-32B despite being less than half its size. 4.3

Analysis

Training on D3-Gym outperforms static SFT data. Figure 4a compares training on data from D3-Gym against AutoSDT-5K (Li et al., 2025), a static dataset of 5K instruction–solution 8

D3-Gym-32B Qwen3-32B 50

50 38.9

20 10 0

33.3

28.9

18.4 17.8 10.5

33.3 22.2

22.2

14.3 7.1

8.3

a. Vis. arn. ng. atial E An p. Data ep Le eat. eosp m F G Co De

D3-Gym-32B Qwen3-32B

Error count

SR@3 on SAB (%)

55 53 47

40 30

a. is. is. p V l. V . An Ma Mo Stat

(a) Per-category SR@3 (%)

9 7

ta Daema h Sc

I/Li

ta Da ape h S pe/

. h lgo Pat c/A i File g Lo

ua Vis

(b) Error types analysis

Figure 5: Detailed breakdown of performance and error types on ScienceAgentBench for Qwen3-32B vs. D3-Gym-32B pairs that lacks executable environments or verification signals. When used for standard SFT, D3-Gym is more sample-efficient: Qwen3-4B-Instruct trained on solutions sampled from D3-Gym achieves roughly 30% relative improvement over its AutoSDT-5K baseline using less than half the training samples, likely due to the rigorous filtering that guarantees task quality in D3-Gym. Beyond sample efficiency, D3-Gym’s executable environment and verification signals enable sampling full reasoning trajectories, unlocking training of thinking models — a capability that static instruction–solution datasets do not support. For example, Qwen3-4B improves from 6.9% to 11.8% SR@3 when trained on its own verified reasoning trajectories. Larger models show more consistent improvement with training scale. Figure 4b shows SR@3 under RFT-Self as the number of training trajectories increases up to each model’s full set of successful trajectories. The 14B and 32B models improve steadily with additional data, whereas the 8B model gains marginally before plateauing and the 4B model initially degrades with a small number of training samples before recovering. This suggests that larger models are better able to extract useful signal from increasing amounts of training data, consistent with previous studies (Li et al., 2025; Jain et al., 2025). Performance improves across diverse task categories. Figure 5a breaks down SR@3 by task category for the Qwen3-32B base model and D3-Gym-32B. Training yields gains across a broad range of categories including computational analysis, data and map visualization, geospatial analysis, and statistical analysis, with deep learning as the only category showing a regression, a category with only two tasks. This breadth of improvement suggests that the tasks in D3-Gym are sufficiently diverse to expand model capabilities across scientific domains rather than overfitting to a narrow subset of skills. Failure modes are greatly mitigated, but some prevail. We manually analyze errors from both models across six types: Data schema (incorrect assumptions about data format, e.g., column names), API & library misuse (hallucinated or deprecated methods), Data type & shape (operations on incompatible shapes or types), File path (incorrect load/save paths), Logical & algorithmic (wrong computations), and Visualization (outputs not adhering to task specifications). As shown in Figure 5b, D3-Gym-32B substantially reduces data schema errors by 44.7%, data type/shape errors by 60.4%, and visualization errors by 15.0%, greatly improving execution validity. However, logical and algorithmic errors increase: with more programs now passing execution, the model more frequently produces runnable but semantically incorrect code, often due to wrong output schemas and intermediate computations. API & library errors also remain prevalent despite a modest decrease, suggesting a deep limitation in the parametric knowledge of current language models. 9

Conclusion

We introduce D3-Gym, the first automatically constructed dataset of verifiable environments for scientific data-driven discovery. Training on trajectories sampled from D3-Gym yield consistent and substantial improvements for open-weight models across scales, narrowing the gap with strong proprietary systems. Overall, D3-Gym represents a versatile resource for advancing language models in scientific data-driven discovery.

Acknowledgments The authors would like to thank colleagues from the OSU NLP group for constructive feedback. The authors thank Xinming Tu for reaching out and helping to construct ScienceAgentBench-Verified. This research was sponsored in part by NSF OAC 2112606, Cisco, and Ohio Supercomputer Center (Center, 1987). The views and conclusions contained herein are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notice herein.

Author Contributions HNM led the project, implemented the D3-Gym creation workflow, conducted quality evaluation, model training, and analysis, and wrote the manuscript. YL co-led the project, managed the curation of the validation set for quality evaluation, conducted training and evaluation experiments, contributed to the development of SAB-verified, and assisted with manuscript writing. ZL assisted with initial candidate task collection and co-managed the annotation of the validation set. YY, CT, and TZ contributed to the annotation of the validation set. NKA and AP provided feedback during biweekly project discussions. ZC provided guidance on project direction and ideas, contributed to the development of SAB-verified, and revised the manuscript. HS advised the project and provided guidance, contributed to core ideas, and revised the manuscript.

References Anthropic. Introducing Claude Sonnet 4.5, September 2025a. URL https://www.anthropic. com/news/claude-sonnet-4-5. Anthropic. Claude code, 2025b. URL https://docs.anthropic.com/en/docs/claude-code. Command-line tool for agentic coding. Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D’Arcy, David Wadden, Matt Latzke, Jenna Sparks, Jena D. Hwang, Varsha Kishore, Minyang Tian, Pan Ji, Shengyan Liu, Hao Tong, Bohao Wu, Yanyu Xiong, Luke Zettlemoyer, Graham Neubig, Daniel S. Weld, Doug Downey, Wen-tau Yih, Pang Wei Koh, and Hannaneh Hajishirzi. Synthesizing scientific literature with retrieval-augmented language models. Nature, 650:857–863, 2026. doi: 10.1038/s41586-025-10072-4. Frazier N. Baker, Daniel Adu-Ampratwum, Reza Averly, Botao Yu, Huan Sun, and Xia Ning. Larc: Towards human-level constrained retrosynthesis planning through an agentic framework, 2025. URL https://arxiv.org/abs/2508.11860. Ohio Supercomputer Center. Ohio supercomputer center, 1987. URL http://osc.edu/ark: /19495/f5s1ph73. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan 10

Sun. Scienceagentbench: Toward rigorous assessment of language agents for data-driven scientific discovery. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=6z4YKr0GK6. Dayuan Fu, Shenyu Wu, Yunze Wu, Zerui Peng, Yaxing Huang, Jie Sun, Ji Zeng, Mohan Jiang, Lin Zhang, Yukun Li, Jiarui Hu, Liming Liu, Jinlong Hou, and Pengfei Liu. davincienv: Open swe environment synthesis at scale, 2026. URL https://arxiv.org/abs/2603. 13023. Zhongmou He, Yee Man Choi, Kexun Zhang, Jiabao Ji, Junting Zhou, Dejia Xu, Ivan Bercovich, Aidan Zhang, and Lei Li. Hardtests: Synthesizing high-quality test cases for llm coding, 2025. URL https://arxiv.org/abs/2505.24098. Tony Hey, Stewart Tansley, Kristin Tolle, and Jim Gray. The Fourth Paradigm: Data-Intensive Scientific Discovery. Microsoft Research, October 2009. ISBN 978-0-9825442-0-4. URL https://www.microsoft.com/en-us/research/publication/ fourth-paradigm-data-intensive-scientific-discovery/. Naman Jain, Jaskirat Singh, Manish Shetty, Tianjun Zhang, Liang Zheng, Koushik Sen, and Ion Stoica. R2e-gym: Procedural environment generation and hybrid verifiers for scaling open-weights SWE agents. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=7evvwwdo3z. Andrej Karpathy. autoresearch. https://github.com/karpathy/autoresearch, 2026. GitHub repository, accessed March 31, 2026. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023. Yifei Li, Hanane Nour Moussa, Ziru Chen, Shijie Chen, Botao Yu, Mingyi Xue, Benjamin Burns, Tzu-Yao Chiu, Vishal Dey, Zitong Lu, Chen Wei, Qianheng Zhang, Tianyu Zhang, Song Gao, Xuhui Huang, Xia Ning, Nesreen K. Ahmed, Ali Payani, and Huan Sun. AutoSDT: Scaling data-driven discovery tasks toward open co-scientists. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 30396–30418, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.1546. URL https://aclanthology.org/2025.emnlp-main.1546/. Yifei Liu, Li Lyna Zhang, Yi Zhu, bingcheng dong, Xudong Zhou, Ning Shang, Fan Yang, Cheng Li, and Mao Yang. rstar-coder: Scaling competitive code reasoning with a largescale verified dataset. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=NzPwDutzz8. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Sanchaita Hazra, Ashish Sabharwal, and Peter Clark. Position: Data-driven discovery with large generative models. In Forty-first International Conference on Machine Learning, 2024. URL https: //openreview.net/forum?id=5SpjhZNXtt. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. Discoverybench: Towards data-driven discovery with large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https: //openreview.net/forum?id=vyflgpwfJW. Ludovico Mitchener, Jon M Laurent, Benjamin Tenmann, Siddharth Narayanan, Geemi P Wellawatte, Andrew White, Lorenzo Sani, and Samuel G Rodriques. Bixbench: a comprehensive benchmark for llm-based agents in computational biology, 2025. URL https://arxiv.org/abs/2503.00096. 11

Siddharth M. Narayanan, James D. Braza, Ryan-Rhys Griffiths, Albert Bou, Geemi Wellawatte, Mayk Caldas Ramos, Ludovico Mitchener, Samuel G. Rodriques, and Andrew D. White. Training a scientific reasoning model for chemistry, 2025. URL https://arxiv.org/abs/2506.17238. Deepak Nathani, Lovish Madaan, Nicholas Roberts, Nikolay Bashlykov, Ajay Menon, Vincent Moens, Mikhail Plekhanov, Amar Budhiraja, Despoina Magka, Vladislav Vorotilov, Gaurav Chaurasia, Dieuwke Hupkes, Ricardo Silveira Cabral, Tatiana Shavrina, Jakob Nicolaus Foerster, Yoram Bachrach, William Yang Wang, and Roberta Raileanu. MLGym: A new framework and benchmark for advancing AI research agents. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=ryTr83DxRq. OpenAI. Learning to reason with LLMs, 2024. learning-to-reason-with-llms/.

URL https://openai.com/index/

Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with SWE-gym. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum? id=Cq1BNvHx74. Archiki Prasad, Elias Stengel-Eskin, Justin Chen, Zaid Khan, and Mohit Bansal. Learning to generate unit tests for automated debugging. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id=yeVBHPLXxi. Rushi Qiang, Yuchen Zhuang, Yinghao Li, Dingu Sagar V K, Rongzhi Zhang, ChangHao Li, Ian Shu-Hei Wong, Sherry Yang, Percy Liang, Chao Zhang, and Bo Dai. MLE-dojo: Interactive environments for empowering LLM agents in machine learning engineering. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025. URL https://openreview.net/forum?id=5W5mFU4oMO. Rushi Qiang, Yuchen Zhuang, Anikait Singh, Percy Liang, Chao Zhang, Sherry Yang, and Bo Dai. MLE-smith: Scaling MLE tasks with automated multi-agent pipeline. In The Fourteenth International Conference on Learning Representations, 2026. URL https: //openreview.net/forum?id=mXQslpfSU5. Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason E Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking-LLM-as-a-judge. In Forty-second International Conference on Machine Learning, 2025. URL https://openreview.net/forum? id=PNRznmmWP7. Parshin Shojaee, Kazem Meidani, Shashank Gupta, Amir Barati Farimani, and Chandan K. Reddy. LLM-SR: Scientific equation discovery via programming with large language models. In The Thirteenth International Conference on Learning Representations, 2025a. URL https://openreview.net/forum?id=m2nmp8P5in. Parshin Shojaee, Ngoc-Hieu Nguyen, Kazem Meidani, Amir Barati Farimani, Khoa D Doan, and Chandan K. Reddy. LLM-SRBench: A new benchmark for scientific equation discovery with large language models. In Forty-second International Conference on Machine Learning, 2025b. URL https://openreview.net/forum?id=SyQPiZJVWY. Michael D. Skarlinski, Sam Cox, Jon M. Laurent, James D. Braza, Michaela Hinks, Michael J. Hammerling, Manvitha Ponnapati, Samuel G. Rodriques, and Andrew D. White. Language agents achieve superhuman synthesis of scientific knowledge, 2024. URL https: //arxiv.org/abs/2409.13740. Arthur Spirling. Why open-source generative AI models are an ethical way forward for science. Nature, 616:413, 2023. doi: 10.1038/d41586-023-01295-4. Yu Su, Diyi Yang, Shunyu Yao, and Tao Yu. Language agents: Foundations, prospects, and risks. In Jessy Li and Fei Liu (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pp. 17–24, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.emnlp-tutorials.3. URL https://aclanthology.org/2024.emnlp-tutorials.3/. 12

Theodore Sumers, Shunyu Yao, Karthik R Narasimhan, and Thomas L. Griffiths. Cognitive architectures for language agents. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https://openreview.net/forum?id=1i6ZCvflQJ. Survey Certification, Featured Certification. Qwen Team. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Zihan Wang, Siyao Liu, Yang Sun, Ming Ding, and Hongyan Li. CodeContests+: Highquality test case generation for competitive programming. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 5576–5600, Suzhou, China, November 2025. Association for Computational Linguistics. ISBN 979-8-89176-335-7. doi: 10.18653/v1/ 2025.findings-emnlp.299. URL https://aclanthology.org/2025.findings-emnlp.299/. Yiqing Xie, Alex Xie, Divyanshu Sheth, Pengfei Liu, Daniel Fried, and Carolyn Rose. RepoST: Scalable repository-level coding environment construction with sandbox testing. In Second Conference on Language Modeling, 2025. URL https://openreview.net/forum?id= 2txrMBpw3q. Dayu Yang, Tianyang Liu, Daoan Zhang, Antoine Simoulin, Xiaoyi Liu, Yuwei Cao, Zhaopu Teng, Xin Qian, Grey Yang, Jiebo Luo, and Julian McAuley. Code to think, think to code: A survey on code-enhanced reasoning and reasoning-driven code intelligence in LLMs. In Christos Christodoulopoulos, Tanmoy Chakraborty, Carolyn Rose, and Violet Peng (eds.), Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pp. 2586–2616, Suzhou, China, November 2025a. Association for Computational Linguistics. ISBN 979-8-89176-332-6. doi: 10.18653/v1/2025.emnlp-main.130. URL https://aclanthology.org/2025.emnlp-main.130/. John Yang, Kilian Lieret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. SWE-smith: Scaling data for software engineering agents. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025b. URL https://openreview.net/ forum?id=63iVrXc8cC. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models. arXiv preprint arXiv:2308.01825, 2023. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=4OsgYD7em5. Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. ACECODER: Acing coder RL via automated test-case synthesis. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12023–12040, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long.587. URL https: //aclanthology.org/2025.acl-long.587/. Siyan Zhao, Zhihui Xie, Mengchen Liu, Jing Huang, Guan Pang, Feiyu Chen, and Aditya Grover. Self-distilled reasoner: On-policy self-distillation for large language models. arXiv preprint arXiv:2601.18734, 2026. Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan YeYanhan, and Zheyan Luo. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 400–410, 2024. Yuqi Zhu, Yi Zhong, Jintian Zhang, Ziheng Zhang, Shuofei Qiao, Yujie Luo, Lun Du, Da Zheng, Ningyu Zhang, and Huajun Chen. Why do open-source llms struggle with data analysis? a systematic empirical study, 2025. URL https://arxiv.org/abs/2506.19794. 13

Appendix We provide more details omitted from the main text in the Appendix as follows: • Appendix A: Limitations • Appendix B: Related Work • Appendix C: Details of D3-Gym construction workflow – Appendix C.1: Pipeline Statistics – Appendix C.2: Prompts – Appendix C.3: LLM judge agreement with human • Appendix D: D3-Gym details – Appendix D.1: Example tasks – Appendix D.2: Cost breakdown – Appendix D.3: Quality validation of D3-Gym verification signal – Appendix D.4: Examples of Gold and Silver Evaluation Scripts – Appendix D.5: Task difficulty • Appendix E: Training Details • Appendix F: ScienceAgentBench-Verified • Appendix G: Repository Licenses

Limitations

We recognize the following limitations and future work directions: D3-Gym Environment Scale. D3-Gym currently comprises 565 tasks, which remains relatively limited compared to training environments in other domains such as software engineering. However, this reflects the inherent difficulty of sourcing high-quality, scientifically meaningful tasks rather than a constraint of our workflow, which can readily collect more tasks provided that sufficient suitable repositories are available. Moreover, the executable and verifiable nature of D3-Gym enables sampling thousands of training trajectories, which our results confirm is sufficient to yield substantial improvements across model sizes. Training method. We focus on RFT as our training method, as it provides dense supervision that is particularly effective when base model performance is weak, as is the case for the open-weight models we train. Our results confirm the effectiveness of this approach in expanding model capabilities across all scales. While our verifiable training environment would also support other techniques including reinforcement learning, the limited baseline capabilities of the models, the complexity of data-driven discovery tasks, and the sparsity of the reward signal make this a challenging direction that we leave for future work.

Related Work

Environment creation for general coding tasks. Recent work has explored methods for generating executable training environments for different coding tasks. In software engineering, SWE-Gym (Pan et al., 2025), RepoST (Xie et al., 2025), R2E-Gym (Jain et al., 2025), and SWE-smith (Yang et al., 2025b) construct environments from real GitHub repositories, leveraging existing commits and unit tests for task synthesis and verification. Beyond software engineering, MLE-Dojo (Qiang et al., 2025), MLGym (Nathani et al., 2025), and MLE-Smith (Qiang et al., 2026) target machine learning tasks where correctness is assessed via standard metrics such as F1 and RMSE. In all these settings, verification relies on preexisting signals such as unit tests already in the repository or well-known metrics shared across tasks. Scientific repositories offer neither, as the codebases rarely include test suites, 14

and correctness is inseparable from the scientific domain, varying with the data modality and analytical goal of each task. Evaluation logic must therefore be constructed from scratch for every task instance, a challenge absent from all of the above works. Training data scaling for data-driven discovery. The most closely related work to ours is AutoSDT (Li et al., 2025), which introduces a pipeline for automatically collecting scientific coding tasks at scale to construct SFT datasets. However, AutoSDT is inherently limited by the absence of execution environments and evaluation scripts alongside its task instances, which precludes generating trajectories or validating alternative solutions. Our work builds on their pipeline and directly addresses this limitation by constructing executable and verifiable training environments, enabling execution and correctness feedback that unlocks training settings beyond standard SFT, including self-improvement by training using onpolicy trajectories. Evaluation Script Generation. Several recent works have tackled automated test and evaluation generation for general coding tasks, both to improve test quality (He et al., 2025; Liu et al., 2025) and to enable downstream training (Zeng et al., 2025; Wang et al., 2025; Prasad et al., 2025). Across all these works, correctness reduces to a domain-agnostic criterion: whether candidate output matches a reference under string or near-string comparison. Our setting is fundamentally different, as evaluation logic for scientific data-driven discovery tasks is tightly coupled to the domain, analytical goal, and data modality of the task, requiring reasoning about which metrics, thresholds, and output artifacts are scientifically appropriate. This challenge is not addressed by any of the above works.

Details of D3-Gym construction workflow

C.1

Pipeline Statistics

The creation workflow of D3-Gym follows the steps provided in section 2. Concretly, we begin by collecting 5,111 candidate tasks using the AutoSDT pipeline (Li et al., 2025). After filtering to retain only tasks whose reference solutions operate exclusively on dependencies present in the original repository and generating dataset previews, 1,586 tasks remain. Of these, 1,263 execute successfully in isolated conda environments and produce output artifacts. Finally, 565 pass the multimodal LLM-as-judge output verification and receive silver evaluation scripts, constituting the final D3-Gym environment — approximately 11% of the initial pool, reflecting the stringent filtering needed to obtain the highest quality tasks. C.2

Prompts

In this section we present all the agent and LLM instructions used in our workflow, namely for task filtering and preview creation, output validity judgment, and evaluation script planning and generation. Task Filtering Agent Prompt You are a coding agent with access to the local file system. You are given a candidate task consisting of a Python program (program.py) and its associated data files. Your objective is twofold: (1) determine whether this task uses only real data and real library dependencies, with no mock, stub, or simulated logic; and (2) if the task is valid, generate dataset preview files for downstream use. Phase 1: Task Validation. Carry out the following steps to determine task validity: 1. Identify Data Files. Inspect the Python program and extract every file path that it attempts to open, read, load, or process. For each path: • Verify whether the file exists on the current file system. • Read a small portion of the file to confirm it contains real, meaningful data (not empty, not placeholder, not all zeros, not constant meaningless values, not randomly generated toy content). If all referenced data files exist and contain valid data, set dummy data = 0; otherwise set dummy data = 1. 2. Check for Mock or Simulated Logic. Analyze the program for any of the following: • Mock objects or unittest.mock usage (e.g., Mock(), MagicMock, patch). • Custom stub or fake class implementations replacing real libraries.

• Synthetic data generation used as primary input instead of loading from files. • Hardcoded dummy values pretending to be real data. • try-except blocks catching ImportError and defining mock replacements. • Any workaround for missing dependencies that bypasses the real library. If the program contains any such logic, set has mock = 1; otherwise set has mock = 0. 3. Render Verdict. If dummy data == 0 and has mock == 0, set valid = 1; otherwise set valid = 0. If valid = 0, skip Phase 2 and proceed directly to the output. Phase 2: Dataset Preview Generation (only if valid = 1). For each data file referenced by the program, generate a preview file showing only the raw data schema: • For CSV or tabular data: the header row plus 3–5 example rows. • For JSON: a snippet showing the structure with 1–2 entries. • For text files: the first few lines of raw content. • For binary or image files: a brief factual description (e.g., “PNG images, 224×224, RGB”). Each preview must follow the format: [START Preview of <file_path>] <raw data snippet> [END Preview of <file_path>]

Multimodal Gold-Result Judge Prompt You are a meticulous evaluator for data-driven discovery workflows. Each task represents an analysis workflow on a dataset—such as statistical analysis, machine learning model training, data visualization, or computational simulation. Inputs Provided. 1. A task instruction describing the analysis to perform. 2. Dataset preview files showing the structure and content of the input data. 3. A list of output files created in pred results/ and their content (a.k.a gold results). The gold results were generated by synthesized programs. Complete correctness is not guaranteed due to possible errors in code synthesis, data processing, or the execution environment. Evaluation Criteria. For a task to PASS, all of the following must hold: 1. All requested output files must exist. Read the task instruction to identify every output file that should be produced. Verify each one appears in the output files listing. If any requested file is missing, the task FAILS. 2. Each output file must be valid. For every file, verify that: • • • •

It is non-empty and contains substantive content (no placeholders, no “TODO”, no all-zeros). It contains no error messages or stack traces. Its content aligns with what the task instruction requested. Its format and values are reasonable for the analysis type.

Use your domain knowledge to assess whether the results are plausible (e.g., metric values within expected ranges, predictions that are sensible given the data). Evaluation Procedure. 1. Identify required output files from the task instruction. 2. Check file existence against the list of required outputs. 3. Validate each file for correctness, completeness, and validity. 4. Render a final decision: PASS only if all files exist and all are valid. Output Format. Return a JSON object with exactly three fields: {"valid": true|false, "reason": "<explanation>"}

Evaluation Script Planner Prompt You are an expert in scientific data analysis and evaluation methodology. Your task is to analyze a data-driven discovery task and produce a concrete evaluation plan that a coding agent will implement. Inputs Provided. • Task Description: the natural-language instruction defining the analysis to perform. • Dataset Information: the dataset path, folder structure, and schema previews. • Expected Output Files: the list of files the gold program produces. • Reference Result Files: the actual content of the reference outputs. Instructions. Produce a structured evaluation plan covering the following:

1. Task Type. Classify the task (e.g., classification, regression, clustering, visualization, statistical analysis, simulation). 2. Evaluation Metrics. Specify which metric(s) are appropriate and justify why: • Classification: if the dataset is imbalanced, prefer F1-score, balanced accuracy, AUROC, or AUPRC over accuracy. • Regression: consider MAE, RMSE, R2 , or domain-specific error metrics. • Statistical analysis: consider p-values, effect sizes, confidence intervals. • Clustering: consider silhouette score, Davies–Bouldin index, or domain-specific quality measures. • Other: specify domain-appropriate criteria. 3. Success Thresholds. Define specific threshold values for each metric, grounded in domain standards and task complexity. 4. Special Considerations. Note any domain-specific requirements such as handling of missing data, biological versus statistical significance, cross-validation needs, or output format constraints. 5. Evaluation Steps. Provide a clear 3–5 step evaluation procedure that the coding agent should follow. Output Requirements. The plan must be concise, unambiguous, and specify exactly one evaluation strategy. Avoid presenting multiple alternatives or optional branches—the coding agent will implement the plan directly as written.

Evaluation Script Coder Prompt You are a Python coding assistant. Your task is to generate an evaluation script for a scientific computing benchmark, following the evaluation plan provided below. Inputs Provided. • Evaluation Plan: the structured plan produced by the evaluation planner. • Task Instruction: the natural-language description of the analysis task. • Dataset Information: the dataset path, folder structure, and schema previews. • Expected Output Files: the list of files the gold program produces in pred results/. • Reference Result Files: the reference output contents in gold results/. Structure. The generated script must adhere to the following: 1. Directory layout (hardcoded paths): • Predicted results: ./pred results/ • Reference results: ./reference results/ 2. Function signature: Define a top-level eval() function taking no parameters. It returns a tuple (result, message) where result is a boolean (True/False) indicating pass/fail and message is a string with details. 3. Visual evaluation (for tasks producing plots or images): use an LLM-as-a-judge approach to compare the predicted and reference visual outputs. 4. Main block: if __name__ == "__main__": ok, msg = eval() print(ok, msg) 5. Error handling: Wrap the body of eval() in a try/except so that unexpected errors return (False, f"Error: {e}") rather than crashing. 6. File existence checks: Verify that both predicted and gold files exist before loading. Return (False, "Missing file: ...") if any file is absent. Output Format. Respond with only the Python source code for the evaluation script. Do not include any explanation or markdown formatting.

C.3

LLM Judge Agreement with Human

To validate the reliability of the LLM-as-judge used for output verification (§2), one author independently annotated 52 task outputs and we measured agreement with the LLM judge (GPT-5.2). Table C.1 reports the results. The LLM judge achieves 92.31% raw agreement with the human annotator (48/52) and a Cohen’s κ of 0.85, corresponding to significant agreement. Treating the human labels as ground truth, the judge attains perfect precision (1.00) and 0.87 recall, with an F1 of 0.93. All four disagreements are false negatives, i.e. cases where the LLM rejected outputs that the human accepted, indicating that the judge is conservative rather than permissive. This bias is desirable in our setting, as the judge may discard a small number of valid tasks but does not allow low-quality outputs into D3-Gym tasks. 17

Metric

Value

Raw Agreement Cohen’s κ Precision Recall F1

92.31% 0.85 1.00 0.87 0.93

Table C.1: Agreement between the LLM judge (GPT-5.2) and a human annotator on output verification for 52 task instances. Precision, recall, and F1 treat human labels as ground truth.

D3-Gym Details

D.1

Example Tasks

In this section we present examples of tasks from D3-Gym belonging to the disciplines of geographic information science and computational chemistry. The examples include the task instruction, dataset previews, and excerpts from the reference solution and evaluation script code. Example Task: Geographic Information Science Task Instruction. Perform spatial interpolation of German temperature data using Universal Kriging with a north-south drift model. Use the temperature observation data from temp obs.txt and German border coordinates from de borders.txt. First, estimate the empirical variogram of the temperature measurements and fit a Spherical variogram model to characterize the spatial correlation structure. Then apply Universal Kriging to interpolate temperature values across a regular lat-lon grid covering Germany (47–56.1◦ N, 5–16.1◦ E with 0.1◦ resolution), incorporating a linear north-south drift function based on latitude. Save the fitted variogram plot, the interpolated temperature field and mean drift field as CSV files, temperature field visualizations, a north-south cross-section plot, and the variogram model parameters. Dataset Preview. • temp obs.txt — 73 DWD weather stations (id, latitude, longitude, temperature in ◦ C): # id, lat, 44 52.9336 73 48.6159 91 50.7446 96 52.9437

lon, 8.2370 13.0506 9.3450 12.8518

temp 15.70 13.90 17.00 21.90

• de borders.txt — German border as lon/lat coordinate pairs: 9.52402 9.35000 9.18281

47.5242 47.5989 47.6707

Reference Solution (excerpt). import gstools as gs, numpy as np data = np.loadtxt("benchmark/.../temp_obs.txt") ids, lat, lon, temp = data.T # Empirical variogram + Spherical model fit bin_center, vario = gs.vario_estimate( (lat, lon), temp, latlon=True, geo_scale=gs.KM_SCALE, max_dist=900) model = gs.Spherical(latlon=True, geo_scale=gs.KM_SCALE) model.fit_variogram(bin_center, vario, nugget=False) # Universal Kriging with latitude drift def north_south_drift(lat, lon): return lat uk = gs.krige.Universal(model=model, cond_pos=(lat, lon), cond_val=temp, drift_functions=north_south_drift) g_lat = np.arange(47, 56.1, 0.1) # 92 points g_lon = np.arange(5, 16.1, 0.1) # 112 points uk.set_pos((g_lat, g_lon), mesh_type="structured") uk(return_var=False, store="temp_field") uk(only_mean=True, store="mean_field") Evaluation Script (excerpt). The script validates the variogram model type and fitted parameters, computes RMSE and R2 between the predicted temperature field and the gold standard, and checks physical validity of the interpolated values including the expected latitudinal temperature gradient.

# Grid dimensions must be exactly (92, 112) expected_rows = len(np.arange(47, 56.1, 0.1)) expected_cols = len(np.arange(5, 16.1, 0.1)) assert pred_temp.shape == (expected_rows, expected_cols) # Variogram: must be Spherical; validate fitted parameters assert "spherical" in model_text.lower() assert variance > 0 and nugget >= 0 and nugget < variance assert 10 < len_scale < 5000 # km (gold ˜ 596 km) # Temperature field RMSE vs gold < 2.5; Rˆ2 vs gold > 0.60 temp_rmse = np.sqrt(np.mean((pred_temp - gold_temp)**2)) ss_res = np.sum((pred_temp - gold_temp)**2) ss_tot = np.sum((gold_temp - np.mean(gold_temp))**2) temp_r2 = 1.0 - ss_res / ss_tot assert temp_rmse < 2.5 and temp_r2 > 0.60 # Physical plausibility: temps in [-10, 40] C assert np.min(pred_temp) >= -10 and np.max(pred_temp) <= 40 # North-south gradient: |T_south - T_north| > 0.5 C south = np.mean(pred_mean[:10, :]) # lat ˜47-48 north = np.mean(pred_mean[-10:, :]) # lat ˜55-56 assert abs(south - north) > 0.5

Example Task: Computational Chemistry Task Instruction. Calculate the Madelung constants for a series of ionic crystal structures using the Ewald summation method. Process the crystal structure files NaCl.vasp, CsCl.vasp, ZnO-Hex.vasp, ZnO-Cub.vasp, TiO2.vasp, and CaF2.vasp to determine the electrostatic potential at the reference atom site in each crystal lattice. The Madelung constant quantifies the electrostatic interactions in ionic crystals by summing contributions from both real space and reciprocal space using the Ewald summation technique with appropriate convergence parameters. Save the computed results to a formatted table showing the crystal name, reference atom symbol, and calculated Madelung constant value for each structure. Dataset Preview. • Six VASP POSCAR crystal structure files spanning different crystal symmetries: NaCl.vasp — rock salt (face-centered cubic): Na Cl 1.0000000000000000 0.000000 2.820000 2.820000 2.820000 0.000000 2.820000 2.820000 2.820000 0.000000 Na Cl 1 1 Cartesian 0.000000 0.000000 0.000000 2.820000 0.000000 0.000000 TiO2.vasp — rutile (tetragonal, 6 atoms): Ti O 1.0000000000000000 4.593700 0.000000 0.000000 0.000000 4.593700 0.000000 0.000000 0.000000 2.958700 Ti O 2 4 Cartesian 0.000000 0.000000 0.000000 2.296850 2.296850 1.479350 1.400160 1.400160 0.000000 3.193540 3.193540 0.000000 3.697010 0.896690 1.479350 0.896690 3.697010 1.479350 Reference Solution (excerpt). from ase.io import read import numpy as np ZZ = {'Na': 1, 'Cl': -1, 'Cs': 1, 'Ca': 2, 'F': -1, 'Ti': 4, 'Zn': 2, 'O': -2} class EwaldSum: def __init__(self, atoms, charges): self.cell = atoms.get_cell() self.positions = atoms.get_positions() self.charges = [charges[s] for s in atoms.get_chemical_symbols()]

Workflow Stage

Cost ($)

Candidate Task Collection Filtering & Dataset Preview Creation Code Execution Output Validation Evaluation Planning & Generation

1,094.5 404.5 0.0 46.8 153.3

Total

1,699.1

Table D.2: Cost breakdown ($) by workflow stage for constructing D3-Gym.

def get_madelung(self, eta=None): # Optimal eta from cell volume volume = abs(np.linalg.det(self.cell)) eta = (np.pi * len(self.atoms) / volume)**(1/3) # Sum real-space + reciprocal-space + self-correction potential = self._real_space(eta) + self._recip_space(eta) potential -= 2 * eta / np.sqrt(np.pi) * self.charges[0] return -potential / self.charges[0] for crys in ['NaCl', 'CsCl', 'ZnO-Hex', 'ZnO-Cub', 'TiO2', 'CaF2']: atoms = read(f"benchmark/.../{crys}.vasp") M = EwaldSum(atoms, ZZ).get_madelung() Evaluation Script (excerpt). The script compares computed Madelung constants against established literature reference values for each crystal system (e.g., NaCl = 1.7476, CsCl = 1.7627, TiO2 = 2.408), checking that all six crystals are present, values are positive, and the relative percentage error stays below 5%. # Literature reference values for each crystal system REFERENCE = { 'NaCl': {'value': 1.7476, 'tolerance': 0.05}, 'CsCl': {'value': 1.7627, 'tolerance': 0.05}, 'ZnO-Hex': {'value': 1.6413, 'tolerance': 0.10}, 'ZnO-Cub': {'value': 1.6381, 'tolerance': 0.10}, 'TiO2': {'value': 2.408, 'tolerance': 0.15}, 'CaF2': {'value': 2.5194, 'tolerance': 0.10}, } # All 6 crystals must be present for crystal in ['NaCl','CsCl','ZnO-Hex','ZnO-Cub','TiO2','CaF2']: assert crystal in pred_results # Values must be positive; RPE < 5% vs literature for crystal, (atom, pred_val) in pred_results.items(): assert pred_val > 0 ref = REFERENCE[crystal]['value'] rpe = abs(pred_val - ref) / ref * 100.0 assert rpe < 5.0

D.2

Cost Breakdown

Table D.2 breaks down the cost of each stage of the workflow for constructing D3-Gym. On average, the cost of constructing one environment in D3-Gym is around $3. Most of this cost is incurred by the candidate task collection stage using AutoSDT pipeline Li et al. (2025) due to its multi-step search and filtering for scientifically meaningful tasks across thousands of repository files. This cost remains within reasonable bounds and is far lower than recent work synthesizing training environments for SE tasks Fu et al. (2026). 20

D.3

Quality Validation of D3-Gym Verification Signal

D.3.1

LLM Judge Prompt

Evaluation Logic Agreement Judge Prompt You are a scientific computing evaluation expert. You will be given two evaluation scripts written for the same data analysis task: a gold (human-written) script and a silver (LLM-generated) script. Your job is to rigorously assess whether the silver script makes the same scientific evaluation decisions as the gold. The gold script serves as the authoritative reference; the silver script is being evaluated for alignment with it. Focus on the substance of the evaluation logic—the scientific properties being measured, the pass/fail criteria, and the artifacts being compared—not on superficial differences such as code style, variable naming conventions, or choice of library. Your assessment should reflect whether the two scripts would reach the same pass/fail conclusion on a correctly implemented solution. Score each of the following three aspects independently on a 1–5 Likert scale: • 1 – Completely missing or wrong: the aspect is absent from the silver script, or the silver script’s implementation is entirely incorrect with respect to the gold. • 2 – Present but fundamentally different: the silver script addresses the aspect, but in a way that is scientifically incompatible with the gold (e.g., evaluating a different property altogether). • 3 – Partially aligned: the silver script captures the intent of the gold, but contains differences substantial enough to change pass/fail outcomes on some inputs. • 4 – Well aligned: the silver script is consistent with the gold, with only minor differences unlikely to affect pass/fail outcomes in practice. • 5 – Fully aligned or equivalent: the silver script implements the same evaluation logic as the gold, or a functionally equivalent alternative. Aspect 1: Evaluation Metric. Determine whether the silver script measures the same scientific property as the gold. Consider the metric definition, its mathematical formulation, and the correctness of the implementation. Two scripts that compute the same quantity using different libraries or formulations are aligned. However, an incorrect implementation of the intended metric (e.g., computing concordance without accounting for censoring in survival analysis, or using accuracy on a heavily imbalanced classification task where the gold uses F1) constitutes a misalignment and should be scored accordingly. Aspect 2: Acceptance Criteria. For each pass/fail threshold defined in the gold script, determine whether the silver script applies a corresponding threshold on the same or an equivalent metric. Consider whether the thresholds are calibrated such that a correct solution passing the gold script would also pass the silver, and vice versa. Absent thresholds, dramatically different threshold values, or thresholds applied to the wrong metric all indicate misalignment. Aspect 3: Target Artifact. Determine whether the silver script reads the correct output files, extracts the correct columns, keys, or fields from those files, and loads ground-truth data from the correct reference source. Misalignment includes reading the wrong file, parsing incorrect columns, or comparing against the wrong ground-truth artifact. Output Format. Return only a JSON object: { "evaluation metric": {"score": <1-5>, "rationale": "<one sentence>"}, "acceptance criteria": {"score": <1-5>, "rationale": "<one sentence>"}, "target artifact": {"score": <1-5>, "rationale": "<one sentence>"} }

D.3.2

LLM Judge - Human agreement

To validate the LLM-as-judge used for evaluating silver script quality (§2.3), one author independently scored 20 silver evaluation scripts against their gold counterparts on the same three aspects, i.e. metric choice, threshold & tolerance, and target artifact, using the 1–5 Likert scale in subsubsection D.3.1. Table D.3 reports agreement with the LLM judge (Claude Sonnet 4.5). Across all three dimensions, the judge achieves 85% exact agreement and 98% within-one agreement with the human annotator. Agreement is highest on metric choice (90% exact) and lowest on threshold & tolerance (80% exact), though even in the latter case all scores fall within one point of the human rating. 21

Aspect

Exact Agr.

Within-1 Agr.

Metric Choice Threshold & Tolerance Target Artifact

0.90 0.80 0.85

0.95 1.00 1.00

Overall

0.85

0.98

Table D.3: Agreement between the LLM judge (Claude Sonnet 4.5) and a human annotator on metric choice, threshold & tolerance, and target artifact. D.4

Examples of Gold and Silver Evaluation Scripts

Example 1 Task. Train a model on the Delaney solubility dataset and produce point predictions (y pred) together with calibrated per-molecule uncertainty estimates (y std). Key differences. The gold script checks RMSE ≤ 1.75 and whether |errori | ≤ 2σi for ≥70% of molecules. The silver script tightens RMSE to 1.70 and switches to the standard 95% CI convention (ŷ ± 1.96σ), requiring ≥72% coverage. For wellcalibrated uncertainties the two coverage definitions are nearly equivalent, but the silver thresholds are marginally tighter on both axes. Gold evaluation script. Uses a 2σ envelope for uncertainty calibration and an RMSE threshold of 1.75. # --- thresholds --RMSE_THRESHOLD = 1.75 COVERAGE_THRESHOLD = 0.70

# <-- slightly more lenient

# --- RMSE --rmse = float(np.sqrt(np.mean((y_pred - y_true) ** 2))) # --- uncertainty calibration: 2-sigma envelope --abs_error = np.abs(y_pred - y_true) coverage_2sigma = float(np.mean( # <-- P(|error| <= 2 * std) abs_error <= 2.0 * y_std)) if rmse > RMSE_THRESHOLD: return False, f"RMSE too high: {rmse:.4f}" if coverage_2sigma < COVERAGE_THRESHOLD: return False, f"P(|error| <= 2*sigma)={coverage_2sigma:.3f}" Silver evaluation script. Uses the standard 95% confidence-interval convention (1.96σ) and a tighter RMSE threshold of 1.70. # --- thresholds --rmse_threshold = 1.70 coverage_threshold = 72.0

# <-- 0.05 tighter # (percent scale)

# --- RMSE --rmse = np.sqrt(np.mean((y_true - y_pred) ** 2)) # --- uncertainty calibration: 95% CI --ci_lower = y_pred - 1.96 * y_std # <-- 1.96*sigma CI ci_upper = y_pred + 1.96 * y_std coverage = np.mean( # <-- P(y_true in CI) (y_true >= ci_lower) & (y_true <= ci_upper)) * 100 success = (rmse <= rmse_threshold and coverage >= coverage_threshold)

Example 2) Task. Train a graph convolutional network to predict polymer crystallization tendency from molecular SMILES, evaluated on a held-out test set. Key differences. The gold script uses a single MSE ≤ 350 gate (equivalent to RMSE ≈ 18.7). The silver script replaces this with three complementary metrics: MAE ≤ 25, RMSE ≤ 30, and R2 ≥ 0.1—all must pass. The added R2 gate catches degenerate models that predict near the mean: such a model could have MSE ≈ 300 and pass the gold script, but would fail the silver one due to low variance explained. Gold evaluation script. Uses a single MSE gate with a generous threshold of 350. MSE_THRESHOLD = 350.0

# <-- single gate

y_true = merged["value"].values.astype(float) y_pred = merged["predicted_crystallization_tendency"] \

.values.astype(float) mse = float(np.mean((y_true - y_pred) ** 2)) if mse > MSE_THRESHOLD: return False, f"MSE too high: {mse:.4f}" return True, f"MSE={mse:.4f}"

# <-- only check

Silver evaluation script. Replaces the single MSE gate with three complementary metrics: MAE, RMSE, and R2 . MAE_THRESHOLD = 25.0 RMSE_THRESHOLD = 30.0 R2_THRESHOLD = 0.1

# <-- triple gate

mae = mean_absolute_error(y_true, y_pred) rmse = np.sqrt(mean_squared_error(y_true, y_pred)) r2 = r2_score(y_true, y_pred) all_pass = (mae <= MAE_THRESHOLD and rmse <= RMSE_THRESHOLD and r2 >= R2_THRESHOLD)

D.5

# <-- all three # must pass

Task Difficulty Model GPT 5.4 Claude Opus 4.6 Qwen3 235B A22B Qwen3 32B

Evaluation Success 38.66% 36.00% 29.33% 16.00%

Execution Success 82.66% 78.66% 61.33% 36.00%

Table D.4: Evaluation and execution success of different models on a subset of 75 tasks from D3-Gym, showing the non-triviality of the tasks and the room for improvement.

Training Details

Training Data Collection. For each model, we sample 16 trajectories per task on all tasks in D3-Gym. Each trajectory contains the model’s full reasoning trace and final solution, and is executed against the corresponding silver evaluation script in the environment. We retain only trajectories that successfully pass the silver evaluation and use them as training data. This yields 1,126 successful trajectories for Qwen3-4B, 1,306 for Qwen3-8B, 1,788 for Qwen3-14B, and 2,153 for Qwen3-32B. In RFT-Distill, student models are fine-tuned on the 2,153 successful trajectories generated by Qwen3-32B. In RFT-Self, each model is fine-tuned on its own successful trajectories. For Qwen3-4B-Instruct, we only apply RFT-Distill and use the final solutions without reasoning traces. Supervised Fine-tuning. We perform full-parameter fine-tuning using the LlamaFactory library (Zheng et al., 2024). We use the Qwen template and train with LoRA rank 16, LoRA alpha 32, and dropout 0.05. Training is performed in the standard supervised fine-tuning (SFT) stage over the retained successful trajectories. Training Infrastructure. Training is conducted on NVIDIA H100 GPUs. For smaller models (4B/8B/14B), we use 4 H100 96GB GPUs, while the 32B model is trained on 8 H100 96GB GPUs. we use a learning rate of 5e-5, cosine learning rate scheduling, 1 training epoch, warmup ratio 0.05, bf16 precision, gradient checkpointing, and max gradient norm 1.0. We set both the training cutoff length and maximum sequence length to 32,768 tokens. Training uses per-device batch size 1 and gradient accumulation steps 1. With 4/8 GPUs, this corresponds to an effective batch size of 4/8. We use distributed data parallel training for all runs. Inference and Evaluation. We use vLLM (Kwon et al., 2023) to serve all models for both trajectory collection and evaluation. Unless otherwise specified, we use temperature 0.2, 23

top p 0.95, and max tokens 16k. For each evaluation setting, we run 3 independent inference runs and report the average performance across runs.

ScienceAgentBench-Verified

In this work, we introduce ScienceAgentBench-Verified, a manually revised version of ScienceAgentBench (SAB) that corrects a set of errors and ambiguities in the original benchmark. The revisions cover task instructions, dataset/environment files, gold programs, and gold results. In total, the verified update revises 9 task instructions (Tasks 9, 12, 26, 29, 31, 34, 35, 67, and 92) and updates gold programs and/or gold results for 3 tasks (Tasks 21, 32, and 78). These changes fix issues such as mismatched output specifications, incorrect file references, ambiguous column names, and inconsistencies between the written instruction and the implemented evaluator. Summary of benchmark revisions. The main revisions in SAB-Verified are as follows: (i) instruction fixes for Tasks 9, 12, 26, 29, 31, 34, 35, 67, and 92; and (ii) gold-program and/or gold-result updates for Tasks 21, 32, and 78. Representative example revisions include correcting the requested statistic in Task 9, changing the expected output from SMILES to drug names in Task 12, correcting input file names in Tasks 29, 34, and 35, refining the expected output schema in Task 67, and fixing matrix factor definitions in Task 92. Impact of the benchmark revision. To validate the effectiveness of our verification, we compare several representative frontier models and coding agents on both the original SAB and SAB-Verified. On the original SAB, the strongest success rate among the evaluated settings is achieved by Claude Code (Claude-Sonnet-4.6) at 44.1 SR. Replacing SAB with SAB-Verified while keeping the environment unchanged generally leads to small but nonnegligible shifts in performance, often around 1–2 solved tasks. For example, Claude-Sonnet4.6 with self-debug improves from 35.3 SR / 82.4 VER to 38.2 SR / 86.3 VER, Claude Code (Claude-Sonnet-4.6) improves from 44.1 SR / 79.4 VER to 45.1 SR / 80.4 VER, and Codex CLI (GPT-5.2) improves from 43.1 SR / 89.2 VER to 45.1 SR / 87.3 VER. After additionally incorporating the affected gold-program and gold-result fixes, the resulting performance changes are generally marginal, typically within about one solved task. This suggests that the verified benchmark mainly improves evaluation fidelity by reducing noise and ambiguity, while largely preserving the overall difficulty of the benchmark.

Repository Licenses

We ensure that all repositories from which D3-Gym tasks are adapted are under permissive licenses. We give the full breakdown of licenses in Table G.5 in addition the names of the two repositories that are under custom licenses. There are also 39 repositories that do not provide any license information. We assume that they permit use for research purposes.

License

Repository

MIT GNU (GPL, AGPL, LGPL) None BSD Apache CC ISC

Count 99 43 39 29 22 4 1

Custom

2 BrainIAC DeepDelta

Total

239

Table G.5: Distribution of repository licenses across all source repositories.

Record · ID 146941 · SHA-256 4b7f596bed83d0ac
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.