DataMaster: Towards Autonomous Data Engineering for Machine Learning Yaxin Du1,* , Xiyuan Yang1,* , Zhifan Zhou2 , Wanxu Liu1 , Zixing Lei1 , Zimeng Chen1 , Fenyi Liu1 , Haotian Wu1 , Yuzhu Cai4 , Zexi Liu1 , Xinyu Zhu1 , Wenhao Wang3 , Linfeng Zhang1 , Chen Qian1 , Siheng Chen1,† 1 Shanghai Jiao Tong University
2 Carnegie Mellon University
3 Zhejiang University
4 Beijing University of Aeronautics and Astronautics
arXiv:2605.10906v1 [cs.LG] 11 May 2026
* Equal contribution.
§
† Corresponding author: [email protected]
Code https://github.com/sjtu-sai-agents/DataMaster
Abstract As model families, training recipes, and compute budgets become increasingly standardized, further gains in machine learning systems depend increasingly on data. Yet data engineering remains largely manual and ad hoc: practitioners repeatedly search for external datasets, adapt them to existing pipelines, validate candidate data through downstream training, and carry forward lessons from prior attempts. We study task-conditioned autonomous data engineering, where an autonomous agent improves a fixed learning algorithm by optimizing only the data side, including external data discovery, data selection and composition, cleaning and transformation. The goal is to obtain a stronger downstream solution while leaving the learning algorithm unchanged. To address the open-ended search space, branch-dependent refinement, and delayed validation inherent in autonomous data engineering, we propose DataMaster, a data-agent framework that integrates tree-structured search, shared candidate data, and cumulative memory. DataMaster consists of three key components: a DataTree that organizes alternative data-engineering branches, a shared Data Pool that stores discovered external data sources for reuse, and a Global Memory that records node outcomes, artifacts, and reusable findings. Together, these components allow the agent to discover candidate data, construct executable training inputs, evaluate them through downstream feedback, and carry useful evidence across branches. We evaluate DataMaster on two types of benchmarks, MLEBench Lite and PostTrainBench. On MLE-Bench Lite, it improves medal rate by 32.27% over the initial score; on PostTrainBench, it surpasses the instruct model on GPQA (31.02% vs 30.35%).
1. Introduction Progress in Machine Learning (ML) has historically been driven by better models, better algorithms, and more computation. Yet in an increasing number of settings, model development itself is becoming more standardized: pretrained backbones are widely available, training recipes are easier to reuse, open-source stacks make fine-tuning increasingly turnkey, and evaluation pipelines are more reproducible than before. As a result, performance differences are increasingly shaped by data: what data is used, how it is selected and cleaned, how it is mixed, and how well it matches the downstream objective. This points to a more strongly data-centric phase of ML development [23, 36]. It also changes what autonomous systems should optimize: if data is becoming the main remaining lever for improving downstream performance, then agentic ML systems should move beyond model-centric optimization alone. In practice, current autonomous ML systems remain largely model-centric. Recent ML engi-
neering agents have become increasingly capable at writing code, debugging training pipelines, tuning hyperparameters, and iterating from validation feedback, achieving strong results in benchmarked environments such as MLE-Bench [8, 16, 37]. More recent benchmarks such as PostTrainBench have begun to expand the action space by allowing agents to go online, run experiments, and curate data during training [29]. However, even in these settings, data is still treated primarily as an auxiliary action dimension rather than the central object of optimization. What remains missing is a framework that allows an agent to systematically improve downstream performance by operating on the data side itself. To address this gap, we study task-conditioned autonomous data engineering for ML systems. Given a downstream objective and an initial solution, the agent treats the data state, rather than the model code, as the primary object of optimization and iteratively searches, cleans, adapts, integrates, and reuses data under training-closed-loop feedback. The objective is not to perform an isolated data operation, but to improve the full data pipeline, including discovering candidate sources, cleaning and adapting them, integrating them into training, and validating whether they improve the downstream model. Existing data-centric work addresses important parts of this pipeline, either through autonomous systems for tasks such as dataset discovery [20] and data cleaning [13], or through benchmarks that evaluate these stages in isolation [6, 19]. However, these stages remain fragmented and are rarely linked by downstream training feedback. This problem is harder than simply retrieving a few extra datasets and appending them to the training set. ❶ Data search is open-ended: potentially useful sources are scattered across repositories and the open web, with uneven quality, unclear licensing, and highly variable relevance to the task. ❷ Discovered data is rarely plug-and-play: schema alignment, label mapping, format conversion, filtering, and deduplication often have to be synthesized before the data can even enter training. ❸ Data iteration is inherently cumulative. To avoid repeating failed directions and to build on past successes, the agent must remember where it searched, what transformations were attempted, which adapters worked, and what kinds of data improved performance. To instantiate this idea, we introduce DataMaster, the first data-agent framework that treats the data state as the primary optimization target and unifies external data discovery, executable data refinement, and cumulative memory under downstream training feedback. Its core search structure, DataTree, organizes data engineering as tree-structured search over self-contained data states, each of which includes the current training data and its associated transformations, paired with a shared Data Pool that stores discovered external datasets for reuse and a Global Memory that stores reusable source histories, useful cleaning methods, and training outcomes across nodes. This tree-plus-memory design is natural for data engineering because data iteration is branchable, path-dependent, cumulative, and only verifiable through downstream training. Concretely, DataTree decomposes data engineering into two complementary node types: red nodes, which search for and introduce external data, and black nodes, which clean, adapt, and enhance the current training data. Together, DataTree with Global Memory turn autonomous data engineering from a sequence of isolated data operations into a structured search process over executable data states, enabling the agent to systematically improve downstream performance through data-side decisions. We evaluate DataMaster in two complementary settings: MLE-Bench Lite for controlled classical ML engineering and PostTrainBench for autonomous LLM post-training. On MLE-Bench Lite, DataMaster improves the strongest initial solution by +32.27% in medal rate. On PostTrainBench, DataMaster improves the average score by +19.79 points over the strongest non-human baseline and surpasses official instruct model on GPQA (31.02 vs. 30.35). These results show that autonomous data engineering can deliver substantial gains across both traditional ML engineering and LLM post-training. The main contributions of this paper are as follows: • We formulate task-conditioned autonomous data engineering as a first-class research problem, in which the data state rather than model code is the primary object of optimization. 2
• We introduce DataMaster, a memory-augmented data-agent framework whose DataTree organizes autonomous data engineering as tree-structured search over data states, paired with a Global Memory for cumulative reuse across nodes. • We evaluate DataMaster on two complementary benchmarks, PostTrainBench and MLEBench Lite, showing that structured data-side iteration can improve strong initial solutions across both LLM post-training and classical ML engineering settings.
2. Related Work Data agents and data-centric AI The data-centric AI paradigm [36] motivates a line of work that builds autonomous agents to handle data engineering tasks. CAAFE [14] uses LLMs to generate semantically meaningful features for tabular datasets, focusing on feature-level transformation of a fixed input table. Co-STEER [33] extends this to a full data-centric development loop, focusing on evolving the scheduling and implementation of data operations over a given dataset. Dataforge [32] focuses on cleaning and optimizing tabular data through dual feedback loops; LAMBDA [31] focuses on iterative code generation and debugging for data analysis; and DatawiseAgent [34] focuses on orchestrating the full notebook-centric data science workflow end-to-end. Benchmarks such as Spider2-V [7] and KramaBench [19] reveal that these agents still struggle with complex, heterogeneous real-world pipelines. Across these works, the data sources are given in advance, the agent’s job is to process and transform on the given dataset. DataMaster differs in that the source space itself is open-ended: the agent must discover, evaluate, and adapt external data from scratch, and use downstream training performance as the sole quality signal to guide this process. ML engineering agents and automated learning A growing body of work builds LLM-based agents that autonomously run ML experiments. AIDE [16] and SELA [10] frame ML engineering as a tree search over code solutions, focusing on optimizing model architecture, hyperparameters, and training scripts. MLZero [12], MLE-STAR [24], and ML-Master [37] further push this direction with multi-agent memory, web-search-guided model retrieval, and hierarchical cognitive caching, all targeting stronger performance on MLE-Bench [8]. Across all of these systems, training data is treated as a fixed given—the agent’s action space is confined to model code, architecture, and hyperparameters, and data itself is never optimized. PostTrainBench [29] takes a step toward data-awareness by allowing agents to curate data during post-training, but data curation remains an auxiliary action rather than the central optimization target. Recent progress tends to focus on this data topic [15]. DataMaster is designed for the complementary regime: the model code is fixed, and the agent’s entire action space operates on the data state, discovering external sources, synthesizing data processing pipelines, and accumulating memory across iterations, with downstream training performance as the sole optimization signal.
3. Method 3.1. Problem Formulation We introduce a new problem: task-conditioned autonomous data engineering. Given a downstream ML task and a fixed learning algorithm, an autonomous agent aims to maximize downstream task performance by operating only on the data side. Specifically, the agent may perform dataside operations, including external dataset discovery, data selection and composition, cleaning and transformation, and feature construction, while the algorithm itself remains unchanged. To make this problem precise, we define a data state 𝐷 as the executable configuration produced by such data-side operations under the fixed learning algorithm. It specifies the 3
Input
Data Tree
Task T Fixed algorithm a0 Resource budget B
Red Node Data Explorations
Initialize solution
Initial 0.65
Global Memory
(a0 ,D0 )
Failed Attempts
Successful Outcomes
Reusable Findings
New data discovered!
0.67
Inspecting
0.78
Downloading
for Node v
Update Reuse
v 0.75
Searching
Black Node Data Exploitations
Useful Hints
0.40
0.84
0.82
Current data augmented!
training in budget B fixed algorithm a0
0.87
0.89 Evaluation Feedback
0.93
Add & Read
Data Curation
(a0 ,D* ) Best score!
Data Pool Candidate datasets
## Manifest: - Dataset Info: description, sources - Dataset Preview - Usage Comment by black nodes
Figure 1 | Overview of DataMaster. DataMaster organizes autonomous data engineering as a DataTree, where red nodes broaden the search by discovering external datasets and writing them into a shared Data Pool, while black nodes exploit available candidates to construct executable data states and obtain downstream training feedback. Global Memory stores reusable artifacts, node outcomes, and prior findings, enabling later nodes to reuse discovered data, avoid repeated failures, and coordinate search across branches under a limited budget. available data sources, their selected and composed training subsets, and the processing logic needed to convert them into the final inputs consumed by the algorithm. The input to the problem consists of a downstream task 𝑇 , a fixed learning algorithm 𝑎0 , an initial executable data state 𝐷0 , and a resource budget 𝐵 on downstream training and evaluation, measured in time or compute. For any candidate data state 𝐷, we use 𝐸 (𝑇 , 𝑎0 , 𝐷) to denote the downstream performance. The agent searches over the set Π 𝐵 ( 𝐷0 ) of executable data states reachable from initial data state 𝐷0 under budget 𝐵, and terminates at a refined data state 𝐷∗ with the best downstream performance. Formally, 𝐷∗ = arg max
𝐷 ∈ Π 𝐵 ( 𝐷0 )
𝐸 (𝑇 , 𝑎0 , 𝐷),
(1)
where the budget 𝐵 is computing source or time constraints and 𝐷∗ is used as the final data configuration consumed by the fixed algorithm 𝑎0 to produce the downstream model for task 𝑇 . This problem is challenging for two fundamental reasons. First, it is open-world: useful data may come from arbitrary external repositories and the web, so the search space is not explicitly enumerable in advance. Second, it is long-horizon: the value of a data-side decision often becomes clear only after subsequent refinement and actual downstream training. As a result, solving this problem requires both broad exploration over candidate data sources and cumulative memory over prior discoveries, transformations, and outcomes. 3.2. Framework Overview The formulation above requires searching over a large and open-ended space of reachable executable data states. DataMaster approximates this search by letting an autonomous agent iteratively construct, evaluate, and improve data states for the fixed downstream algorithm. Instead of performing one-shot dataset curation, the agent incrementally searches for new data, tests different refinement choices, and updates later decisions using downstream training feedback. This requires three capabilities: trying multiple next-step choices from the same current state, sharing discovered external datasets across branches, and reusing what earlier attempts have already learned. DataMaster realizes this through three coupled components: a DataTree, a shared Data 4
Pool, and a Global Memory, shown in Fig. 1. The DataTree is the core tree-structured search component of DataMaster, organizing alternative data-engineering branches while preserving branch-local context for later data-side decisions. The Data Pool is a global set of candidate external datasets discovered so far; red nodes write newly discovered entries into it, and black nodes read from it during refinement. The Global Memory stores reusable processing histories, materialized artifacts, training outcomes, and node-level findings so that later nodes can build on earlier discoveries rather than restarting from scratch. In addition, due to the limited budget, the tree growth and scheduling mechanisms make the search efficient: downstream training feedback guides how the tree is expanded, while the scheduling policy decides which frontier nodes should be executed first. Together, these components and their collaboration enable broad data exploration without restarting each branch from scratch, allowing DataMaster to reuse successful choices, avoid repeated failures, and spend limited budget on more promising directions. 3.3. DataTree The DataTree organizes DataMaster’s search over executable data states by representing alternative data-side decisions as branches of a tree. It allows the agent to explore multiple ways of improving the data from shared starting contexts, while preserving the branch-local state and feedback needed for later decisions. Formally, we denote the DataTree as T = (V, E), where node 𝑣0 ∈ V is the root node associated with the initial data state 𝐷0 , V is the set of executed data-engineering nodes, and E is the set of parent-to-child edges. Each node represents one data-engineering step executed by the agent under the current branch context. Each edge represents parent-to-child context inheritance, allowing the child node to refer to the parent’s accumulated data state, artifacts, decisions, and downstream feedback when making the next data-side decision. Specifically, DataTree contains two node types: red nodes broaden the search by discovering new external datasets, while black nodes deepen the search by exploiting available candidates to construct stronger executable training datasets for the downstream task and fixed algorithm. Red Nodes: Data Exploration Red nodes are responsible for expanding the shared Data Pool by discovering external datasets that may be useful for the downstream task. Conditioned on the current search context, the shared Data Pool P, and the retrieved memory M 𝑣 from Global Memory M for node 𝑣, a red node performs open-ended external search and lightweight screening, and outputs a set of newly discovered dataset entries Δ P𝑣 that are appended to P for later exploitation by black nodes. In practice, a red node first generates search queries from the task description, target metric, data schema, failure cases, and prior memory. It then screens returned sources for relevance and usability (e.g., metadata, schema compatibility, modality, scale, label structure, and sample previews), while filtering duplicate or benchmark-restricted sources. Each retained entry stores a compact manifest, including its local path or source pointer, format, schema summary, task relevance, and screening notes. This design turns open-world external search into a reusable candidate layer, separating data acquisition from downstream refinement and allowing later nodes to build on prior discoveries rather than restart from scratch. Black Nodes: Data Exploitation Black nodes exploit the shared Data Pool P to improve the current training-data pipeline. Conditioned on the available candidate datasets in P and the retrieved context M 𝑣 for node 𝑣 from Global Memory M, a black node selects useful sources, adjusts their composition, applies cleaning and transformation rules, constructs features, and
5
adapts the inputs to the fixed downstream training interface. For a black node 𝑣, these decisions are materialized as an executable DataLoader and associated data-processing logic, yielding the node-specific data state 𝐷𝑣 , i.e., the executable data configuration produced by node 𝑣 under the fixed downstream algorithm. The state is then submitted to downstream training through a fixed submit interface, which instantiates the downstream evaluation 𝐸 (𝑇 , 𝑎0 , 𝐷𝑣 ) under the fixed algorithm 𝑎0 . The resulting score 𝑦𝑣 and diagnostics 𝜙𝑣 are written back into M. In this way, black nodes convert candidate datasets into executable training hypotheses and use downstream training to compare, diagnose, and refine data-side decisions under a fixed algorithm, rather than relying on heuristic data selection alone. 3.4. Global Memory & Data Pool: Shared Search State. Global Memory The Global Memory serves as the persistent record layer of DataTree, maintaining a one-to-one record for each executed node to trace branch-level search history, node outputs, downstream outcomes, and reusable findings. Concretely, M stores node-level records of the form ( ( red, Δ P𝑣 , 𝜙𝑣 ), if 𝑣 is a red node, 𝑟𝑣 = ( black, 𝐷𝑣 , 𝑦𝑣 , 𝜙𝑣 ), if 𝑣 is a black node. For a node 𝑣, the default retrieved context is inherited from its parent and sibling nodes: M 𝑣 = {𝑟𝑃𝑎𝑟 ( 𝑣 ) } ∪ {𝑟𝑢 | 𝑢 ∈ Sib( 𝑣)}, where 𝑃𝑎𝑟 ( 𝑣) denotes the parent of node 𝑣 and Sib( 𝑣) denotes the set of its siblings. In addition to this default retrieved context, agents may actively query other red or black node records during execution and may also write newly discovered findings back into M. This makes the search tree coordinated rather than independent: red nodes can avoid redundant discovery and target uncovered directions, while black nodes can reuse validated states, transfer successful processing choices, and use limited training budget more efficiently. Data Pool The Data Pool serves as the shared candidate-data layer of DataMaster, turning external sources discovered by red nodes into reusable candidates for later black-node refinement. Each retained entry stores the information needed for downstream use, including the local path, source description, format, schema, metadata, scale, modality, task relevance, and lightweight screening results. Red nodes append newly discovered entries Δ P𝑣 into P, and black nodes read from P to select, compose, and adapt candidate sources into executable training data for the downstream task and fixed algorithm. While the DataTree maintains branch-local search structure and Global Memory stores node-level outcomes, the Data Pool keeps discovered data sources reusable across branches. 3.5. Tree Growth and Scheduling under Limited Budget With a limited evaluation budget, DataMaster cannot enumerate all reachable executable data states in Π 𝐵 ( 𝐷0 ), since each candidate state may require downstream training and evaluation. We therefore use two budget-control mechanisms: a growth policy that decides whether to expand the tree with a red node or a batch of black nodes, and a scheduling policy that prioritizes which unexecuted node should consume the next evaluation budget. Growth policy Tree growth in DataMaster is dynamic rather than predetermined. A red node expands the candidate data space, but its output is only a set of potential datasets; whether these candidates are actually useful can only be determined after they are selected, adapted, and tested by black nodes. For this reason, each red node is followed by a small batch of black nodes that 6
run in parallel and explore different exploitation hypotheses over the newly expanded Data Pool. After a black node is evaluated, the system decides whether the current branch should continue with further black-node refinement or open a new red node for broader data exploration. This decision is made by a lightweight external controller using the latest training feedback and the current branch context. The advantage of this design is that tree growth remains adaptive: it can deepen when the current data configuration still has room for improvement, and broaden when the branch appears under-covered or its current candidates have been largely exhausted. Scheduling policy While Eq. 1 defines the final data state to be found, scheduling specifies how to allocate the next unit of evaluation budget among pending nodes. The scheduling policy provides this online allocation rule: among pending unexecuted nodes, it selects the next node whose execution is estimated to be most useful for approaching a high-performing data state. Under a limited training budget, DataMaster cannot execute all pending nodes and therefore uses a greedy scheduler to decide which one should consume the next unit of budget. The next executed node is 𝑣next = arg max𝑣 ∈ N Score( 𝑣), where N is the set of nodes that have been generated by the growth policy but not yet executed, and the score balances branch quality against insufficient exploration. In our implementation, we instantiate it with a rule inspired by Upper Confidence Bound (UCB) [4, 18], √︄ log 𝑁𝑃𝑎𝑟 ( 𝑣 ) 𝑅𝑣 + 𝑐𝑡 , (2) Score( 𝑣) = 𝑁𝑣
𝑁𝑣
where 𝑅 𝑣 and 𝑁𝑣 are the cumulative reward and visit count associated with the branch rooted at node 𝑣, 𝑁𝑃𝑎𝑟 ( 𝑣 ) is the corresponding count for its parent, and 𝑐𝑡 is a decaying exploration coefficient. Black-node evaluation scores are propagated upward to update these branch statistics, so that repeatedly successful branches become easier to prioritize while still retaining an incentive to explore insufficiently tested directions. Additional implementation details, including reward backpropagation, initialization of newly generated nodes, and the decay schedule of 𝑐𝑡 , are provided in Appendix C. Overall DataMaster instantiates autonomous data engineering as a closed-loop search system over executable data configurations. DataTree organizes alternative data-side decisions, Data Pool shares discovered external sources, and Global Memory reuses prior outcomes so that data discovery, refinement, and downstream validation reinforce each other rather than remain isolated steps. Fig. 6 shows an example of the whole system. The following experiments evaluate whether this structured data-side search can improve fixed algorithms across both classical machine learning and LLM post-training settings.
4. Experiments 4.1. Setup Benchmarks We evaluate DataMaster on two complementary benchmarks that differ in their data-availability assumptions. MLE-Bench Lite [8] is a held-out subset of MLE-Bench spanning diverse Kaggle competitions across tabular, vision, NLP, and time-series tasks. Each task provides a fixed competition dataset; the agent must improve downstream performance solely through data-side operations (selection, cleaning, feature engineering) while leaving the model training code unchanged. PostTrainBench [29] measures how well AI agents can post-train a base language model given only compute and a time budget, with no training data provided. The agent must discover and curate datasets from scratch, covering seven capabilities (details in Appendix B). 7
Table 1 | Main result comparison on MLE-Bench Lite and PostTrainBench. Unless marked otherwise, agentic systems use GLM-5; ∗ Codex is driven by GPT-5.2-Codex. Best baseline scores are in bold. MLE-Bench Lite
PostTrainBench
System
Medal Rate
Initial Score
35.91%
22.73%
0.00%
Reference Points 0.48% 22.56% 18.75%
9.02%
0.00%
8.53%
8.47%
Human Score
–
–
6.66%
44.84%
63.46% 30.35%
86.95%
35.76%
60.36%
46.91%
Codex*
22.73%
18.18%
0.00%
Baselines 0.15% 40.67% 29.54%
24.31%
9.17%
25.41%
18.46%
Claude Code
36.36%
22.12%
0.00%
2.01%
31.00% 14.29%
2.58%
1.16%
29.27%
11.47%
DatasetResearcher 59.09%
27.27%
0.00%
0.00%
22.64% 18.00%
9.33%
3.56%
6.00%
8.50%
–
–
0.00%
2.07%
22.58% 28.80%
48.37%
6.22%
25.61%
19.09%
ML-Master 2.0
40.91%
27.27%
3.33%
0.00%
31.43% 18.57%
31.45%
8.24%
16.84%
15.69%
DataMaster
68.18%
45.45%
3.33%
21.93%
34.27% 31.02%
49.43%
34.93%
43.29%
31.17%
(+32.27%)
(+22.72%)
DataFlex
Gold Medal AIME ArenaHard Rate 25 Writing
BFCL
GPQA
GSM8K
HealthBench HumanEval Easy
Avg
(+22.70%)
Metrics On MLE-Bench Lite we report four metrics: (i) Medal Rate, the fraction of competitions where the agent’s best node achieves a medal-level submission; (ii) Gold Medal Rate, restricted to gold medals; (iii) Avg. Overcome Rate, the fraction of nodes in DataTree whose submission score exceeds the initial-node submission score; and (iv) Performance Gain, the mean relative improvement of the best score in DataTree over the initial submission. On PostTrainBench we report per-task accuracy for each of the seven capabilities above together with the macro-average (Avg). Baselines We compare DataMaster against four categories of baselines: (i) Frontier agents, including Claude Code CLI [1] and Codex CLI [25]; (ii) Dataset Search Agent, Dataset Research [20]; (iii) End-to-End ML Agent, ML-Master-2 [37]; and (iv) Data-Centric Training Framework, DataFlex [22]. Reference Points: On MLE-Bench Lite, the initial score is obtained by running ML-Master’s code with its data module removed; on PostTrainBench, it corresponds to the base model (Qwen3-1.7B-Base) without fine-tuning. Human score refers to the experttrained instruct model (Qwen3-1.7B), serving as a strong human-curated reference for assessing whether autonomous data engineering can approach expert-designed post-training. See detailed implementations in Appendix B.3. 4.2. Main Results Table 1 presents the main results comparing DataMaster against all baselines on MLE-Bench Lite and PostTrainBench. All systems are powered by GLM-5 [35] as the backbone LLM, with a 12-hour time limit and identical training compute budget per task. Comparison with Initial Score As shown in Table 1, DataMaster substantially outperforms the initial score on both benchmarks. On MLE-Bench Lite, it achieves a medal rate of 68.18% and a gold medal rate of 45.45%, demonstrating that operating exclusively on data while leaving the training code untouched can deliver substantial downstream gains. On PostTrainBench, DataMaster raises the average score from 8.47% to 31.17% (+22.70 percentage points), with particularly large improvements on GSM8K (9.02%→49.43%), HealthBench Easy (0.00%→34.93%), and HumanEval (8.53%→43.29%). 8
Comparison with Other Baselines DataMaster consistently outperforms all baselines on both benchmarks. On MLE-Bench Lite, it achieves the highest medal rate (68.18%) and gold medal rate (45.45%), surpassing DatasetResearch by 9.09 percentage points in medal rate and 18.18 percentage points in gold medal rate, indicating that coupling external data discovery with iterative refinement is more effective than search alone. On PostTrainBench, DataMaster achieves the highest average score (31.17%) among all systems. Compared to Codex (18.46%), which has strong single-task performance on BFCL (40.67%) but struggles with long-horizon refinement, DataMaster achieves more balanced improvements across all seven capabilities. Compared to DataFlex (19.09%), which receives the same curated dataset but only applies training-time optimization, DataMaster outperforms by 12.08 percentage points, demonstrating that data curation quality matters more than training-time scheduling alone. Notably, DataMaster also surpasses the human-expert instruct model on GPQA (31.02% vs. 30.35%), suggesting that autonomous data engineering can approach or exceed expert-curated post-training data on selected capabilities. 4.3. Ablation Study Performance Across Different LLMs To understand how DataMaster varies with backbone LLMs, we evaluate GLM-5, Claude-4.6-sonnet [2], Qwen3.5-Plus [28], and GPT-5.4 [26] on PostTrainBench. Fig. 2 shows per-task accuracy across seven capabilities. Key findings: ❶ GLM5, Claude-4.6-sonnet, and Qwen3.5-plus all achieve around 30% average accuracy, substantially above the Base Model (8.47%), showing that DataMaster is effective across multiple strong backbones. ❷ DataMaster reaches or surpasses the human-expert instruct model on selected capabilities: GLM-5 on GPQA, Claude-4.6-sonnet on HealthBench Easy, and Qwen3.5-plus on HumanEval. ❸ GPT-5.4 performs worse (14.64% average). We observe that it makes far fewer tool calls per node (11.3 on average) than other backbones (40.7 for GLM-5), suggesting that insufficient tool use may limit long-horizon data search and refinement. 90 80
Base Model Claude-4.6-Sonnet Qwen3.5-plus
Instruct Model GLM-5 GPT-5.4
70
Score (%)
60 50 40
+60.6
+57.8 +18.5
30
+12.3
20 10 0
+37.0
+3.3
AIME 25
+4.8
ArenaHard Writing
BFCL
GPQA
GSM8K
HealthBench Easy
HumanEval
Benchmark
Figure 2 | PostTrainBench performance across LLMs. Per-task accuracy of DataMaster powered by four frontier LLMs, compared against Base Model and Instruct Model baselines.
9
Test-Time Scaling We examine test-time scaling on the GPQA task in PostTrainBench by tracking the best accuracy over wall-clock training time. As shown in Fig. 3, DataMaster improves stepwise from the base score of 18.75% to 31.02% by progressively discovering and integrating more relevant science, reasoning and MedQA [17] data. The final node surpasses the instruct model reference, showing that additional search budget can produce stronger task-specific data configurations rather than merely longer execution. 35.0 31.02% Best
32.5
30.0
GPQA Accuracy (%)
Add MMLU-Pro + MedQA
27.5 Add MMLU science subsets
25.0
Add natural-science-reasoning + science_questions
22.5 Increase training data volume (2x samples)
20.0
17.5
PostTrainBench: GPQA
18.75% Base model
0
Instruct model
2
4
6
8
10
12
Training Time (hours)
Figure 3 | Test-time scaling on PostTrainBench. Best-node score as a function of wall-clock time, showing continuous improvement as budget grows.
Component Analysis We ablate DataMaster’s core components: red nodes (data exploration), black nodes (data exploitation), and Global Memory (cross-iteration knowledge store), on 10 MLE-Bench Lite tasks that cover the main task types in the benchmark, including tabular, vision, NLP, time-series, and audio tasks. Each variant disables one component while keeping the other two active. Table 2 shows initial-node and best-node metrics. We find that: ❶ Memory enables cross-iteration learning. Without memory, overcome rate drops from 69.96% to 28.57% and best gold medal rate from 80% to 20%, negating all gains. ❷ Red and black nodes are complementary. Without red node, internal refinement improves (overcome rate 72.40%), but best gold medal rate drops 10%, showing external discovery adds unique value. Without black node, discovered data cannot be adapted (lower medal rate), showing external sources need in-place refinement. ❸ Full system achieves strongest performance. Only with all three components enabled does DataMaster reach a 90% medal rate. Robustness to Algorithm Initialization A practical question is whether DataMaster depends on a specific fixed algorithm implementation. Each MLE-Bench Lite task provides an initial solution with a DataLoader for data loading, preprocessing, and augmentation, and an Algorithm for model construction, training, and inference (Appendix D.4). We compare Full-code, where both parts are provided, with Algo-only, where only the Algorithm skeleton is given and the agent must construct the DataLoader. Table 3 shows that: ❶ DataMaster improves over the 10
Table 2 | Component analysis on MLE-Bench Lite. ✓/✗ indicates whether the component is enabled. Initial-code metrics measure the quality of the initial solution node; best-node metrics reflect the best submission found across the full search tree. Red
Black
Initial-node
Memory
Best-node
Medal Rate
Gold Medal Rate
Medal Rate
Gold Medal Rate
Overcome Rate
✗ ✓ ✓
✓ ✗ ✓
✓ ✓ ✗
50% 60% 50%
20% 20% 20%
80% (+30%) 70% (+10%) 50% (+0%)
70% (+50%) 50% (+30%) 20% (+0%)
72.40% 52.41% 28.57%
✓
✓
✓
50%
30%
90% (+40%)
80% (+50%)
69.96%
initial baseline under both settings, showing that data-side optimization is robust to different fixed-algorithm starting points. ❷ Algo-only starts from a lower initial medal rate (35.91% vs. 45.45%) yet achieves a larger absolute gain (+32.27% vs. +27.28%) and a higher overcome rate (67.31% vs. 60.40%), indicating that DataMaster can compensate for a weaker starting point by constructing better data pipelines. Table 3 | Sensitivity to initial code quality on MLE-Bench Lite. Full-code starts from a complete training pipeline; Algo-only starts from a minimal algorithmic skeleton. All runs use DataMaster (full) with all three components enabled. Green annotations indicate the absolute gain over the corresponding initial-code metric. Initial Code
Full-code Algo-only
Initial-node
Best-node
Medal Rate
Gold Medal Rate
Medal Rate
Gold Medal Rate
45.45% 35.91%
18.18% 22.73%
72.73% (+27.28%) 68.18% (+32.27%)
45.45% (+27.27%) 45.45% (+22.72%)
Overcome Rate
60.40% 67.31%
4.4. Analysis Cost Analysis We analyze DataMaster’s internal cost distribution on MLE-Bench Lite, reporting per-node token usage, API cost, tool calls, and wall-clock time for red and black nodes. Table 4 summarizes the average costs over 641 red nodes and 340 black nodes. We find that: ❶ Black nodes are costlier, using 3.0× more API budget and taking 2.4× longer than red nodes, due to data selection, processing, DataLoader construction, and downstream training/evaluation. ❷ Red nodes dominate search breadth, with nearly twice as many executions as black nodes, reflecting open-ended external data discovery. ❸ This cost profile matches DataMaster’s design: red nodes cheaply expand the candidate space, while black nodes spend more budget validating downstream gains. Data Leakage Analysis A concern in open-ended data search is test-set contamination. We address this through a multi-layer protocol: ❶ source filtering that blocklists benchmark pages, competition discussions, and URLs matching test-split names, preventing test data from entering the training pipeline; ❷ hash-based deduplication that stores test-set hashes and filters out any training samples with exact matches; ❸ provenance tracking that records URL, timestamp, and content hash for every external data point, enabling leakage audits. Fig. 4 audits GPQA leakage over 7,479 discovered training samples. We find zero exact or fuzzy matches and only low 3–5-gram overlap (0.08–1.06%), suggesting terminology overlap rather than test-set copying.
11
Table 4 | Node-wise cost breakdown within DataMaster’s search tree on MLE-Bench Lite.
2.00 1.75
Metric
Node Type Red Black
All
# Nodes
641
340
981
Input tokens (K) Output tokens (K)
55.2 6.9
68.1 24.9
59.4 13.2
API cost ($) Avg. tool calls
0.021 41.3
0.062 39.6
0.036 40.7
Tool time (min) Total time (min)
16.7 21.3
38.5 50.6
24.0 31.6
Train: 7,479 samples Test: 448 samples
14.7% 5.2% 9.4%
16.4%
30.3%
Overlap Rate (%)
1.50 Avg. Tools
1.25
12.7%
per Executed Node
1.06%
RED: 58.52 BLACK: 43.63
1.00 0.75
51.8%
0.50 0.25 0.00
0.26%
49.5%
0.08% 0.00% 0.00% Exact Fuzzy 3-gram 4-gram 5-gram Match ( 90%)
Outer Ring: RED Nodes
|
Inner Ring: BLACK Nodes
Tool Categories
Detection Method
Figure 4 | Test-set leakage check. Test-train overlap on PostTrainBench GPQA.
Inspect / View
Run / Grade
Search
Basic
Edit / Fix
Bash / Env
Memory
Other
Figure 5 | Tool-use composition. Tool-use statistics of Red/Black nodes.
Statistic Analysis We analyze the tool-use composition of red and black nodes in Fig. 5. Key findings: ❶ red nodes use more tools per execution than black nodes on average (58.52 vs. 43.63), reflecting the interaction-heavy nature of external data discovery. ❷ Red-node actions are dominated by Search (49.5%), followed by Bash/Env (30.3%) and Memory (14.7%), showing that red nodes primarily explore candidate data sources, inspect artifacts, and reuse prior search context. ❸ Black-node actions are dominated by Bash/Env operations (51.8%), with non-trivial Run/Grade, Edit/Fix, and Inspect/View usage, indicating that black nodes focus on constructing executable data states and validating them through downstream execution rather than broad source discovery.
5. Conclusion We introduced task-conditioned autonomous data engineering as a new problem where an agent improves a fixed training algorithm by searching over data states. We propose DataMaster to solve this problem through a DataTree for branching data-side search, a shared Data Pool for reusable external sources, and a Global Memory for carrying outcomes, artifacts, and reusable findings across iterations. Experiments on MLE-Bench Lite and PostTrainBench show that 12
DataMaster consistently improves strong initial solutions and outperforms agentic and datacentric baselines, demonstrating that data-side search can be an effective optimization lever beyond model-code modification. These results suggest that future autonomous ML systems should place greater emphasis on data-side improvement, including how data is discovered, selected, transformed, validated, and reused across tasks.
References [1] Anthropic. Claude code: A command-line tool for agentic coding, 2025. URL https: //code.claude.com/docs. [2] Anthropic. Introducing claude sonnet 4.6. https://www.anthropic.com/news/claud e-sonnet-4-6, Feb. 2026. Accessed: 2026-05-07. [3] R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, et al. Healthbench: Evaluating large language models towards improved human health. arXiv preprint arXiv:2505.08775, 2025. [4] P. Auer, N. Cesa-Bianchi, and P. Fischer. Finite-time analysis of the multiarmed bandit problem. Machine learning, 47(2):235–256, 2002. [5] M. Balunović, J. Dekoninck, I. Petrov, N. Jovanović, and M. Vechev. Matharena: Evaluating llms on uncontaminated math competitions. arXiv preprint arXiv:2505.23281, 2025. [6] M. Cai, X. Gao, Y. Li, H. Lin, Z. Liu, Z. Pan, Q. Pei, X. Shang, M. Sun, Z. Tang, et al. Opendataarena: A fair and open arena for benchmarking post-training dataset value. arXiv preprint arXiv:2512.14051, 2025. [7] R. Cao, F. Lei, H. Wu, J. Chen, Y. Fu, H. Gao, X. Xiong, H. Zhang, Y. Mao, W. Hu, et al. Spider2-v: How far are multimodal agents from automating data science and engineering workflows? Advances in Neural Information Processing Systems, 37:107703–107744, 2024. [8] J. S. Chan, N. Chowdhury, O. Jaffe, J. Aung, D. Sherburn, E. Mays, G. Starace, K. Liu, L. Maksin, T. Patwardhan, et al. Mle-bench: Evaluating machine learning agents on machine learning engineering. arXiv preprint arXiv:2410.07095, 2024. [9] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. [10] Y. Chi, Y. Lin, S. Hong, D. Pan, Y. Fei, G. Mei, B. Liu, T. Pang, J. Kwok, C. Zhang, et al. Sela: Tree-search enhanced llm agents for automated machine learning. arXiv preprint arXiv:2410.17238, 2024. [11] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. [12] H. Fang, B. Han, N. Erickson, X. Zhang, S. Zhou, A. Dagar, J. Zhang, A. C. Turkmen, C. Hu, H. Rangwala, et al. Mlzero: A multi-agent system for end-to-end machine learning automation. arXiv preprint arXiv:2505.13941, 2025. [13] X. Gao, X. Wang, Y. Zhu, M. Cai, C. He, and L. Wu. Closing the data loop: Using opendataarena to engineer superior training datasets. arXiv preprint arXiv:2601.09733, 2025. 13
[14] N. Hollmann, S. Müller, and F. Hutter. Large language models for automated data science: Introducing caafe for context-aware automated feature engineering. Advances in Neural Information Processing Systems, 36:44753–44775, 2023. [15] Hugging Face. ml-intern: An open-source ai agent for autonomous ml research. https: //github.com/huggingface/ml-intern, 2026. [16] Z. Jiang, D. Schmidt, D. Srikanth, D. Xu, I. Kaplan, D. Jacenko, and Y. Wu. Aide: Ai-driven exploration in the space of code. arXiv preprint arXiv:2502.13138, 2025. [17] D. Jin, E. Pan, N. Oufattole, W.-H. Weng, H. Fang, and P. Szolovits. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421, 2021. [18] L. Kocsis and C. Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pages 282–293. Springer, 2006. [19] E. Lai, G. Vitagliano, Z. Zhang, O. Chabra, S. Sudhir, A. Zeng, A. A. Zabreyko, C. Li, F. Kossmann, J. Ding, et al. Kramabench: A benchmark for ai systems on data-to-insight pipelines over data lakes. arXiv preprint arXiv:2506.06541, 2025. [20] K. Li, M. Jiang, D. Fu, Y. Wu, X. Hu, D. Wang, and P. Liu. Datasetresearch: Benchmarking agent systems for demand-driven dataset discovery. arXiv preprint arXiv:2508.06960, 2025. [21] T. Li, W.-L. Chiang, E. Frick, L. Dunlap, B. Zhu, J. E. Gonzalez, and I. Stoica. From live data to high-quality benchmarks: The arena-hard pipeline. Blog post.[Accessed 07-02-2025], 2024. [22] H. Liang, Z. Zhao, M. Qiang, M. Chen, L. Ma, R. Yu, H. Feng, S. Sun, Z. Meng, X. Ma, et al. Dataflex: A unified framework for data-centric dynamic training of large language models. arXiv preprint arXiv:2603.26164, 2026. [23] M. Mazumder, C. Banbury, X. Yao, B. Karlaš, W. Gaviria Rojas, S. Diamos, G. Diamos, L. He, A. Parrish, H. R. Kirk, et al. Dataperf: Benchmarks for data-centric ai development. Advances in Neural Information Processing Systems, 36:5320–5347, 2023. [24] J. Nam, J. Yoon, J. Chen, J. Shin, S. Ö. Arık, and T. Pfister. Mle-star: Machine learning engineering agent via search and targeted refinement. arXiv preprint arXiv:2506.15692, 2025. [25] OpenAI. Codex CLI. https://github.com/openai/codex, 2025. Open-source coding agent that runs locally in the terminal. Accessed: 2026-05-06. [26] OpenAI. Introducing gpt-5.4. https://openai.com/index/introducing-gpt-5-4/, Mar. 2026. Accessed: 2026-05-07. [27] S. G. Patil, H. Mao, F. Yan, C. C.-J. Ji, V. Suresh, I. Stoica, and J. E. Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025. [28] Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https: //qwen.ai/blog?id=qwen3.5. [29] B. Rank, H. Bhatnagar, A. Prabhu, S. Eisenberg, K. Nguyen, M. Bethge, and M. Andriushchenko. Posttrainbench: Can llm agents automate llm post-training? arXiv preprint arXiv:2603.08640, 2026.
14
[30] D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023. [31] M. Sun, R. Han, B. Jiang, H. Qi, D. Sun, Y. Yuan, and J. Huang. Lambda: A large model based data agent. Journal of the American Statistical Association, 121(553):1–13, 2026. [32] X. Wang and Y. Fu. Dataforge: A data agent platform for autonomous data engineering. arXiv preprint arXiv:2511.06185, 2025. [33] X. Yang, H. Chen, W. Feng, H. Wang, Z. Ye, X. Shen, X. Yang, S. Sun, W. Liu, and J. Bian. Collaborative evolving strategy for automatic data-centric development. arXiv preprint arXiv:2407.18690, 2024. [34] Z. You, Y. Zhang, D. Xu, Y. Lou, Y. Yan, W. Wang, H. Zhang, and Y. Huang. Datawiseagent: A notebook-centric llm agent framework for automated data science. arXiv e-prints, pages arXiv–2503, 2025. [35] A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xie, et al. Glm-5: from vibe coding to agentic engineering, 2026. [36] D. Zha, Z. P. Bhat, K.-H. Lai, F. Yang, Z. Jiang, S. Zhong, and X. Hu. Data-centric artificial intelligence: A survey. ACM Computing Surveys, 57(5):1–42, 2025. [37] X. Zhu, Y. Cai, Z. Liu, B. Zheng, C. Wang, R. Ye, J. Chen, H. Wang, W.-C. Wang, Y. Zhang, et al. Toward ultra-long-horizon agentic science: Cognitive accumulation for machine learning engineering. arXiv preprint arXiv:2601.10402, 2026.
15
A. Case Study: DataMaster on Random Acts of Pizza To provide a concrete view of the DataMaster workflow, we include a representative walkthrough on the random-acts-of-pizza task. This task requires predicting whether a Reddit pizza request will receive a successful response, using request text and associated metadata. The figure illustrates how DataMaster organizes data engineering as a tree-structured search process, where different branches correspond to alternative data-side decisions under the same fixed algorithm.
Figure 6 | Case walkthrough of DataMaster on the random-acts-of-pizza task. This walkthrough highlights the separation between the three core components. The DataTree provides the tree-based search structure, allowing the agent to explore multiple dataengineering directions. The Data Pool serves as the shared candidate-data layer: Red nodes write newly discovered external data entries into it, and Black nodes later read from it to construct executable training inputs. Global Memory is a separate record layer that stores node outcomes, failed trials, useful hints, and validation feedback for later reuse. Together, these components allow DataMaster to broaden the search over external data sources, refine selected candidates through downstream validation, and carry reusable information across branches without conflating discovered data sources with search memory.
B. Experiment Details B.1. Set up Hardware configuration. We run all MLE-Bench Lite experiments on an 8-GPU machine with NVIDIA H20 GPUs. To enable parallel execution, we allocate one H20 GPU per 4 tasks, allowing multiple competitions to run concurrently while sharing compute resources efficiently. 16
B.2. Benchmark Details B.2.1. MLE-Bench Lite MLE-Bench Lite [8] is a curated subset of MLE-Bench containing 22 Kaggle competitions spanning tabular prediction, computer vision, natural language processing, and time-series forecasting. Each task provides a fixed competition dataset and an initial code solution. The agent operates exclusively on the data side—selecting subsets, engineering features, cleaning outliers, and adjusting data formatting—while the model architecture and training script remain unchanged. Performance is measured by whether the agent’s best submission achieves a Kaggle medal threshold (bronze, silver, or gold), with medal rate and gold medal rate as the primary metrics. B.2.2. PostTrainBench PostTrainBench [29] evaluates autonomous data engineering for post-training a base language model. Unlike MLE-Bench Lite, no training data is provided; the agent must discover, collect, and curate all datasets from scratch. The benchmark measures downstream accuracy after fine-tuning the base model on the agent-curated data. It covers seven diverse capabilities: • AIME 2025 [5] — mathematical reasoning, evaluated on competition-level math problems. • Arena-Hard Writing [21] — instruction following and creative writing, judged by an LLM judge. • BFCL [27] — function calling, testing the model’s ability to generate correct API calls. • GPQA [30] — graduate-level scientific knowledge across physics, chemistry, and biology. • GSM8K [11] — grade-school arithmetic reasoning with multi-step word problems. • HealthBench Easy [3] — medical question answering on clinical scenarios. • HumanEval [9] — code generation, measuring functional correctness of synthesized Python programs. The primary metric is per-task accuracy, together with the macro-average across all seven tasks. B.2.3. Key Differences Between Benchmarks Table 5 summarizes the key differences between the two evaluation settings. Table 5 | Comparison of evaluation settings.
Initial data provided Data discovery required Task domains Number of tasks Fixed component Primary metric
MLE-Bench Lite
PostTrainBench
Yes (competition dataset) No Kaggle ML competitions 22 Training code Medal rate
No Yes (open-world search) LLM post-training capabilities 7 Training script + base model Per-task accuracy
B.3. Baseline Implementation Details
17
B.3.1. Baseline DataFlex DataFlex [22] is a data-centric training framework built on LLaMA-Factory that dynamically controls data selection, mixture ratios, and sample reweighting during training. Unlike DataMaster, it does not perform autonomous dataset discovery; it assumes candidate datasets are already available and optimizes their usage during training. Because DataFlex is built on LLaMA-Factory, it is only applicable to PostTrainBench (language model post-training) and cannot be evaluated on MLE-Bench Lite (heterogeneous ML tasks). For PostTrainBench evaluation, we provide DataFlex with the same curated dataset produced by DataMaster’s best node, isolating DataFlex’s training-time optimization from the data discovery process. In principle, DataMaster (data discovery and curation) and DataFlex (training-time optimization) are orthogonal and could be combined. Claude Code CLI. Claude Code CLI [1] is Anthropic’s command-line agent for agentic coding tasks. It has access to file operations, bash execution, and web search, and can iteratively refine code and data pipelines. We evaluate it on both benchmarks by providing the same initial setup as DataMaster. Codex CLI. For MLE-Bench Lite, we use the Codex CLI implementation provided by the benchmark authors. For PostTrainBench, we report results using GPT-5.2-codex from the original PostTrainBench paper [29]. We do not run our own Codex CLI evaluation on PostTrainBench because: (1) the compact API does not support external model endpoints, and (2) when integrated with GLM-5.1, Codex CLI frequently exceeds the maximum token limit and terminates prematurely, preventing long-horizon iterative refinement. DatasetResearch. DatasetResearch [20] is a pure data-search agent that discovers external datasets but does not perform code adaptation or training. For our evaluation, we run DatasetResearch to discover datasets, then use Claude Code CLI to adapt the discovered data into the training pipeline. To ensure fair comparison, we control the total training data volume to match DataMaster’s best node. ML-Master 2.0. ML-Master 2.0 [37] is an end-to-end machine learning agent that can modify both data and code. We evaluate it on MLE-Bench Lite using the authors’ released implementation. It is not evaluated on PostTrainBench due to implementation constraints. B.4. Reference Points Initial Score. The initial score represents the baseline performance before any autonomous data engineering. • On MLE-Bench Lite, the initial score is obtained by running ML-Master 2.0’s code with its data module removed. This isolates the contribution of the fixed training pipeline (model architecture, hyperparameters, training script) from any data-side improvements. The resulting medal rate (35.91%) and gold medal rate (22.73%) serve as the starting point for measuring data engineering gains. • On PostTrainBench, the initial score corresponds to the base model Qwen3-1.7B-Base evaluated directly on the seven downstream tasks without any fine-tuning. This represents the zero-shot or few-shot performance of the pretrained model before any task-specific 18
data curation or training. The average score of 8.47% reflects the base model’s limited capability on specialized tasks without post-training. Human Score. The human score represents the performance ceiling achieved by human experts through manual data curation and training. On PostTrainBench, the human score corresponds to Qwen3-1.7B, the instruct-tuned version of the base model, trained by Alibaba’s expert team on carefully curated instruction-following datasets. The average score of 46.91% represents the state-of-the-art achievable through expert human data curation and serves as the target for autonomous data engineering systems. B.5. Per-Task Performance Analysis We analyze DataMaster’s per-task performance on 17 MLE-Bench Lite tasks (5 tasks excluded due to invalid initial submissions). We report normalized gain, which measures the best-node improvement relative to the median-to-gold score range, and overcome rate, the fraction of DataTree nodes that surpass the initial-node score. Fig. 7 visualizes both metrics across all tasks. Key findings: ❶ DataMaster achieves non-negative gain on all 17 tasks, never degrading performance. Top performers show both high gain and high overcome rate (e.g., 102.33% gain with 85.71% overcome on an audio classification task). ❷ Gain and overcome rate are not always aligned: some tasks exhibit high gain with low overcome rate, where few nodes discover exceptional configurations; others show low gain with high overcome rate, where the initial score is already near-optimal yet DataMaster still improves steadily. ❸ Performance varies by domain: audio and sequence-to-sequence tasks achieve the highest gains, followed by tabular tasks, while image/text classification tasks show more variable gains—particularly when external data is scarce or train/test distributions are near-identical. Normalized gain computation. For each competition, we compute a normalized gain that measures the best-node improvement relative to the score range between the median and gold thresholds: best − initial |gold − median| × 100 higher-is-better, Gain% = (3) initial − best × 100 lower-is-better. |gold − median| This normalization accounts for the varying difficulty across competitions: a 0.01 improvement on a tightly clustered metric may represent a larger relative achievement than a 0.05 improvement on a widely spread one. Table 6 reports the full per-task results, including the initial-node score, best-node score, relative improvement, normalized gain, and overcome rate. B.6. Cost Analysis To assess the practical cost of DataMaster, we measure wall-clock time and API token consumption (input and output separately) across all MLE-Bench Lite runs. Table 7 compares DataMaster against baselines in terms of compute cost and resulting performance.
19
Audio Classification Image Classification Image To Image
Normalized Gain (%)
100
Seq->Seq Tabular Text Classification
80 60 40 20
aptos
mlsp -blind 2013-bird s nessd etect i o text-n n nom ormengli ad2018 sh-la ngua g plant e -path leaf text-n s orm- pooky-au thor russi an-la ngu dogs age -vs-c ats pizza dogdeno breed ising tps-m docs ay-20 22 whal e-red ux tps-d ec-2 aeria 021 l-ca insul ctus ts-de tect
0
Figure 7 | Task-wise normalized gain and overcome rate on 17 MLE-Bench Lite tasks. All tasks achieve non-negative gain; top performers span Audio, Tabular, Image Classification, and Seq->Seq domains.
Table 6 | Per-task performance on 17 MLE-Bench Lite tasks with valid initial scores.
Competition
Dir.
Initial
Best
Rel. Impr.
Gain%
Overcome
mlsp-2013-birds aptos2019-blindness nomad2018 text-norm-english plant-pathology leaf-classification spooky-author text-norm-russian dogs-vs-cats random-acts-of-pizza dog-breed denoising-documents tps-may-2022 whale-redux tps-dec-2021 aerial-cactus insults-detect
Higher Higher Lower Higher Higher Lower Lower Higher Lower Higher Lower Lower Higher Higher Higher Higher Higher
0.86753 0.92106 0.06033 0.99285 0.98142 0.02367 0.23456 0.97232 0.01028 0.62265 0.46888 0.03622 0.99477 0.98777 0.96330 0.99999 0.95637
0.93870 0.96239 0.05207 0.99575 0.99350 0.00004 0.19931 0.97393 0.00331 0.64955 0.44098 0.03341 0.99583 0.99109 0.96338 1.00000 0.95637
+0.07117 +0.04133 +0.00826 +0.00290 +0.01208 +0.02363 +0.03525 +0.00161 +0.00697 +0.02690 +0.02790 +0.00281 +0.00106 +0.00332 +0.00008 +0.00001 +0.00000
102.33% 99.36% 59.04% 42.21% 40.48% 21.81% 13.89% 11.33% 8.36% 7.09% 5.92% 5.08% 4.15% 2.67% 2.52% 0.00% 0.00%
85.71% 20.00% 88.89% 86.70% 80.00% 92.90% 60.00% 60.00% 87.50% 76.92% 57.89% 14.30% 81.25% 88.00% 58.80% 40.00% 0.00%
20
Table 7 | Efficiency comparison on MLE-Bench Lite. Tokens are reported as average per-task input/output tokens; time is average wall-clock minutes per task. Performance columns reuse the main-table results. Input Tokens (K)
Output Tokens (K)
API Cost ($)
Time (min)
Medal Rate
Gold Rate
Claude Code Codex ML-Master 2.0 DatasetResearch
522.5 97.3 349.0 38.5
13.3 19.0 113.9 64.6
0.09 0.05 0.29 0.15
127.6 95.4 669.6 71.7
36.36% 22.73% 40.91% 59.09%
22.12% 18.18% 27.27% 27.27%
DataMaster
2690.3
596.3
1.61
718.7
68.18%
45.45%
Method
Table 8 | Task-wise MLE-Lite test scores and overcome rates. Direction indicates whether higher or lower scores are better for each competition metric. Competition
Direction
aerial-cactus-identification aptos2019-blindness-detection mlsp-2013-birds new-york-city-taxi-fare-prediction plant-pathology-2020-fgvc7 ranzcr-clip-catheter-line-classification spooky-author-identification detecting-insults-in-social-commentary dog-breed-identification dogs-vs-cats-redux-kernels-edition histopathologic-cancer-detection jigsaw-toxic-comment-classification-challenge nomad2018-predict-transparent-conductors random-acts-of-pizza text-normalization-challenge-english-language denoising-dirty-documents leaf-classification siim-isic-melanoma-classification tabular-playground-series-dec-2021 tabular-playground-series-may-2022 text-normalization-challenge-russian-language the-icml-2013-whale-challenge-right-whale-redux
Higher Higher Higher Lower Higher Higher Lower Higher Lower Lower Higher Higher Lower Higher Higher Lower Lower Higher Higher Higher Higher Higher
Test Initial Score
Test Best Score
Overcome Rate
0.99999 0.92106 0.86753
1.00000 0.96239 0.93870 4.73358 0.99350 0.89733 0.19931 0.95637 0.44098 0.00331 0.99788 0.98663 0.05207 0.64955 0.99575 0.03341 0.00004 0.91516 0.96338 0.99583 0.97393 0.99109
40% 20% 85.71% 100% 80%
None 0.98142
None 0.23456 0.95637 0.46888 0.01028
None None 0.06033 0.62265 0.99285 0.03622 0.02367 0.90303 0.96330 0.99477 0.97232 0.98777
None 60% 0.00% 57.89% 87.50% 100%
None 88.89% 76.92% 86.70% 14.30% 92.90% 92.86% 58.80% 81.25% 60% 88%
Table 9 | Average MLE-Lite search-tree and trajectory-step statistics by competition category. Metric
node_count red_node_count black_node_count initial_node_count avg_steps_per_node red_avg_steps_per_node black_avg_steps_per_node
Image Cls.
Tabular
Text Cls.
Audio Cls.
Image Reg.
Seq-to-Seq
Image-to-Image
29.6250 16.0000 12.6250 1.0000 30.7981 28.7049 35.5963
34.5000 17.7500 15.7500 1.0000 35.1650 31.9904 40.5089
89.5000 66.5000 22.0000 1.0000 22.5211 22.1431 25.3585
83.5000 61.5000 21.0000 1.0000 20.9376 18.0587 28.5417
22.0000 14.0000 7.0000 1.0000 36.6818 37.8571 36.7143
33.0000 15.5000 16.5000 1.0000 39.0038 39.8364 39.5919
16.0000 8.0000 7.0000 1.0000 40.7500 39.3750 46.2857
21
C. Scheduling Policy This appendix provides the full specification of the UCB1-based scheduling policy introduced in Section 3.5. C.1. Reward Design and Backpropagation The DataTree contains two types of executable nodes with fundamentally different outputs: red nodes produce candidate datasets (no training score), while black nodes produce downstream evaluation metrics. To unify them under a single scheduling framework, we define the immediate reward as 𝜖, 𝑟 𝑣 = 𝑦𝑣 , 0,
if 𝑣 is a red node and completes without error, if 𝑣 is a black node with evaluation score 𝑦𝑣 , otherwise (buggy or failed nodes),
where 𝜖 is a small positive constant (we use 𝜖 = 0.01 in all experiments). This ensures that successful data-discovery steps receive non-zero credit without dominating the exploitation signal from actual training outcomes. Upon completion of node 𝑣 with reward 𝑟𝑣 , the statistics are updated along the entire root path. Let Path( 𝑣) = ( 𝑣, 𝑝 ( 𝑣), 𝑝 ( 𝑝 ( 𝑣)), . . . , root) denote the sequence of nodes from 𝑣 to the root. For every 𝑢 ∈ Path( 𝑣): 𝑁𝑢 ← 𝑁𝑢 + 1,
(4)
𝑅𝑢 ← 𝑅𝑢 + 𝑟 𝑣 .
(5)
That is, both 𝑣 itself and all of its ancestors receive the same reward increment and visit increment. This backpropagation mechanism allows parent and grandparent nodes to accumulate evidence about the quality of their subtrees, enabling the scheduler to compare branches at different depths on a common scale. C.2. Node Initialization When a new frontier node 𝑣 is created by the growth policy, its statistics are initialized as 𝑁𝑣 = 0,
𝑅 𝑣 = 0.
Since 𝑁𝑣 = 0 would make the UCB1 score in Eq. (2) undefined (division by zero), we handle unvisited nodes as a special case: +∞, if 𝑁𝑣 = 0, √︄ Score( 𝑣) = 𝑅 log 𝑁 𝑝 ( 𝑣 ) 𝑣 , if 𝑁𝑣 ≥ 1. 𝑁 𝑣 + 𝑐𝑡 𝑁𝑣 This optimistic initialization guarantees that every newly generated frontier node is selected for execution before any previously visited node is revisited, ensuring that no branch is left completely unexplored. Once a node has been visited at least once ( 𝑁𝑣 ≥ 1), both the exploitation term 𝑅 𝑣 / 𝑁𝑣 and the exploration term are well-defined, and the standard UCB1 formula applies.
22
C.3. Exploration Coefficient Decay The exploration coefficient 𝑐𝑡 controls the trade-off between exploiting high-reward branches and exploring under-visited ones. Let 𝑡 denote the number of node executions completed so far (i.e., the global step counter). We adopt a piecewise linear decay schedule that transitions through three phases over the total budget of 𝑇 rounds: 𝑐0 , 𝑡 < 𝑡1 , 𝑐𝑡 = max 𝑐0 − 𝛼 ( 𝑡 − 𝑡1 ), 𝑐min , 𝑡1 ≤ 𝑡 ≤ 𝑡2 , 𝑐 , 𝑡 > 𝑡2 , min where: • 𝑐0 is the initial exploration constant, • 𝛼 is the linear decay rate per step during the transition phase, • 𝑐min is the lower bound preventing purely greedy behavior, • 𝑡1 = ⌊ 𝑝1 · 𝑇 ⌋ and 𝑡2 = ⌊ 𝑝2 · 𝑇 ⌋ are the phase boundaries, defined as fractions of the total budget. The three phases serve distinct purposes: 1. Exploration phase (𝑡 < 𝑡1 ): 𝑐𝑡 remains at its maximum value 𝑐0 , encouraging broad coverage of the frontier when little information is available. 2. Transition phase (𝑡1 ≤ 𝑡 ≤ 𝑡2 ): 𝑐𝑡 decays linearly, gradually shifting priority toward branches that have already demonstrated strong performance. 3. Exploitation phase (𝑡 > 𝑡2 ): 𝑐𝑡 is clamped at 𝑐min , concentrating the remaining budget on the most promising branches while retaining a minimal exploration incentive to avoid premature convergence. C.4. Hyperparameter Settings Table 10 lists the default hyperparameters used in all experiments. Table 10 | Default scheduling hyperparameters. Symbol
Description
Value
𝑐0 𝑐min 𝛼 𝑝1 𝑝2 𝜖 𝑇
Initial exploration constant Lower bound on 𝑐𝑡 Linear decay rate Exploration phase ratio Transition end ratio Red-node reward Maximum rounds (budget)
√ 1.414 (≈ 2) 0.5 0.01 0.3 0.7 0.01 40
Tree structure parameters
num_red Red nodes per expansion 1 num_black Initial black nodes per red 5 max_black_per_red Maximum black nodes per red 5 √ The choice of 𝑐0 = 2 follows the theoretical recommendation for UCB1 when rewards lie in [0, 1]. The phase ratios ( 𝑝1 , 𝑝2 ) = (0.3, 0.7) allocate roughly the first 30% of the budget to pure 23
exploration, the middle 40% to transition, and the final 30% to exploitation. The lower bound 𝑐min = 0.5 ensures that even in the late exploitation phase, a branch with significantly fewer visits than its siblings retains a meaningful chance of being selected. C.5. Alternative Decay Schedules While we use the piecewise schedule by default, our implementation also supports two alternative decay strategies for ablation purposes: Linear decay.
𝑐𝑡 = max 𝑐0 − 𝛼 · 𝑡 , 𝑐min .
This provides a steady, monotonic decrease from the first step onward, without a dedicated exploration phase. Exponential decay.
𝑐𝑡 = max 𝑐0 · 𝛾 𝑡 , 𝑐min ,
where 𝛾 ∈ (0, 1) is the decay factor (default 𝛾 = 0.99). Exponential decay reduces exploration aggressively in early steps but converges more slowly to 𝑐min compared to linear decay. C.6. Priority Queue Implementation The scheduler maintains a max-priority queue over the frontier N . Each time a node completes and new children are generated by the growth policy, the new nodes are inserted with Score( 𝑣) = +∞. After node completion triggers backpropagation (Eqs. 4–5), existing frontier nodes are not re-scored eagerly; instead, scores are computed lazily at selection time. The next node 𝑣next is popped from the queue, and execution proceeds. This design keeps scheduling overhead constant per step regardless of frontier size.
D. Case Study D.1. Branch Concentration Bias in Leaf Classification We further analyze tree-growth asymmetry using the leaf-classification task as a case study. We reconstruct the search tree from the UCT node logs using each node’s identifier and parent identifier. After excluding the artificial root and initial node, we treat the children of the initial node as root-level branches. For each search step 𝑡 , let 𝑛𝑖 ( 𝑡 ) be the number of expanded nodes assigned to branch 𝑖, and let 𝐾 be the total number of root-level branches. We first convert branch sizes into proportions: 𝑛𝑖 ( 𝑡 ) 𝑝𝑖 ( 𝑡 ) = Í 𝐾 . 𝑗=1 𝑛 𝑗 ( 𝑡 )
We then measure branch concentration using the normalized Herfindahl-style index: Í𝐾 1 2 𝑖=1 𝑝𝑖 ( 𝑡 ) − 𝐾 Bias( 𝑡 ) = . 1 − 𝐾1 This value ranges from 0 to 1, where 0 indicates uniform expansion across branches and 1 indicates that all expanded nodes are concentrated in a single branch. 24
Leaf Classification: Tree Growth Bias Over Search Steps
Branch Concentration Bias
1.0 0.8 0.6 0.4 0.2 0.0
0
5
10
15
20 Search step
25
30
35
Figure 8 | Branch Concentration Bias over search steps for the leaf-classification task. Higher values indicate more asymmetric tree growth. The curve initially decreases as the search expands across multiple branches, but later rises as one branch receives a larger share of subsequent expansions. The leaf-classification tree contains 38 extracted nodes and 5 root-level branches after removing the artificial root and initial node. The final branch sizes are 23, 6, 3, 2, and 2 nodes, showing a clear imbalance in how expansion is allocated. As shown in Fig. 8, the bias decreases during the early search steps, suggesting initial diversification across branches. However, the later increase indicates that the agent increasingly concentrates expansion on a dominant branch. This supports the observation that the search process is not uniform; instead, it exhibits branch-level concentration as the tree develops. D.2. Red–Black Exploration Ratio Analysis To further examine the structural behavior of the search process, we compare Red-node and Black-node executions across competition categories. For each task, we compute two ratios. The first is the executed-node ratio: 𝑁Red 𝑅node = , 𝑁Black
where 𝑁Red and 𝑁Black denote the numbers of executed Red and Black trajectory nodes. The second is the tool-intensity ratio: 𝑇Red / 𝑁Red 𝑅tool = , 𝑇Black / 𝑁Black where 𝑇Red and 𝑇Black denote the total numbers of tool calls made by executed Red and Black nodes. Ratios are first computed at the task level and then averaged within each competition category. Overall, most competition categories show executed-node ratios above 1, suggesting that the agent tends to allocate more executed nodes to Red-node exploration than to Black-node implementation. The tool-intensity ratio is also generally near or above 1, but it is usually less pronounced than the executed-node ratio. This indicates that the Red–Black imbalance is primarily structural, reflected in the number of executed nodes, while per-node tool-use intensity provides a secondary and more task-dependent source of imbalance.
25
4.0 Executed-node ratio Tool-intensity ratio
RED / BLACK Ratio
3.5 3.0 2.61
2.60
2.5 2.00
2.0 1.5
1.86
1.28
1.25
1.40
1.37
1.24
1.22
1.29 1.14
1.10 0.93
1.0 0.5 0.0 .
e Cls Imag
la Tabu
r
ls.
C Text
.
io Cls
Aud
Imag
e Re
g.
eq
to-S
Seq-
Imag
e-to
ge
-Ima
Figure 9 | Red–Black exploration ratios by competition category. The blue bars show the executed-node ratio, while the pink bars show the tool-intensity ratio. The dashed horizontal line at 𝑦 = 1 indicates parity between Red and Black nodes. Values above 1 indicate Red-node dominance, whereas values below 1 indicate Black-node dominance. Error bars denote the standard error of the mean across tasks within each category. D.3. Text Embedding Distribution for Spooky Author Identification To examine the semantic alignment between the available training/validation data and the public test set, we visualize the spooky-author-identification task using a t-SNE projection. We sample 1,000 examples from the training/validation data and 1,000 examples from the public test set, using only the text field. Each text instance is embedded with Qwen3-Embedding-0.6B, L2-normalized, and projected into two dimensions using t-SNE with perplexity 30 and a fixed random seed.
26
60
Train / Validation Test Set
40
Dimension 2
20 0 20 40 60
40
20
0
Dimension 1
20
40
Figure 10 | Text embedding t-SNE visualization for the spooky-author-identification task. Each point corresponds to one text example, with colors indicating whether the example comes from the training/validation data or the public test set. The substantial overlap between the two groups suggests that the public test set is broadly aligned with the available training/validation distribution in the embedding space. This alignment also helps explain why task-relevant public text resources can potentially support downstream modeling and evaluation. The projection shows that training/validation and test examples are highly intermixed, with no clear separation between the two groups. This pattern suggests that the public test distribution is close to the available training/validation distribution under the chosen embedding model. More broadly, it indicates that publicly available text data may contain useful signals for downstream tasks, which helps motivate retrieval- and search-based strategies in agentic model development. D.4. Initial Code Examples We present the initial code used in the dog-breed-identification task as a representative example of the two code settings in Section 4.3. Listing 1 shows the DataLoader component, which handles data loading, label encoding, train/validation splitting, and data augmentation and Algorithm component, which implements model construction, the training loop and validation. In the Full-code setting, both components are provided; in the Algo-only setting, only the Algorithm component is included and the agent must construct the DataLoader from scratch. Listing 1 | Initial code structure for dog-breed-identification (both DataLoader and Algorithm components). 1 # ==================== DataLoader Component ==================== 2 class DogDataset ( Dataset ) : 3 def __init__ ( self , ids , labels , img_dir , transform , is_train ) : 4 ... 5 def __getitem__ ( self , idx ) : 6 image = Image . open (...) . convert ( " RGB " ) 27
7 8
if self . transform : image = self . transform ( image ) return ( image , self . labels [ idx ]) if self . is_train else image
9
class MyDataLoader ( BaseDataLoader ) : def __init__ ( self , batch_size =32 , num_workers =8 , 12 input_dir = " ./ input / " , ** kwargs ) : ... 10 11
13 14 15 16
17 18
19 20 21
22 23 24 25
def setup ( self ) : # Load and encode labels labels_df = pd . read_csv ( os . path . join ( self . input_dir , " labels . csv " ) ) le = LabelEncoder () labels_df [ " breed_encoded " ] = le . fit_transform ( labels_df [ " breed " ]) # Train / val split ( fixed val . csv or stratified 80/20) ... # Augmentation : Resize (384) , RandomCrop (320) , Flip , ColorJitter self . train_transform = transforms . Compose ([...]) self . val_transform = transforms . Compose ([...]) # Create DogDataset + DataLoader for train / val / test ...
26
# = = = = = = = = = = = = = = = = = = = = Algorithm Component ==================== 28 class MyAlgorithm ( BaseAlgorithm ) : 29 def __init__ ( self , model_name = ’ efficientnet_b3 ’ , 30 lr =1 e -4 , epochs =10 , ** kwargs ) : ... 27
31 32 33 34 35 36 37 38 39 40 41 42 43
def fit ( self , train_data ) : # Build pretrained EfficientNet - B3 via timm self . model = timm . create_model ( self . model_name , pretrainedTrue = , num_classes = num_classes ) # AdamW optimizer + CosineAnnealingLR scheduler ... # Training loop with validation ; save best checkpoint for epoch in range ( self . epochs ) : # forward → loss → backward → step ... # validate and save best model ...
44 45 46 47 48 49 50 51
52
53 54 55
def predict ( self , test_data ) : self . model . eval () with torch . no_grad () : for images in test_loader : # TTA : average ( original , horizontal flip ) outputs = self . model ( images ) outputs_flip = self . model ( torch . flip ( images , [3]) ) probs = ( softmax ( outputs ) + softmax ( outputs_flip ) ) / 2 ... # Create submission DataFrame return pd . DataFrame ({ ’ id ’: test_ids , ’ breed ’: predicted })
28