TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development

[2608.26086] TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development Skip to main content Search Submit Donate Log in Computer Science > Machine Learning arXiv:2608.26086 (cs) [Submitted on 26 Aug 2026 ( v1 ), last revised 28 Sep 2026 (this version, v3)] Title: TraceML: What Auto-Research Agents Miss in Long-Horizon ML Development Authors: Jiarui Yan , Weiwei Sun , Sijie Li , Wenhan Li , Yiming Yang View PDF HTML (experimental) Abstract: Auto-research agents now run machine-learning development unattended for hours, revising data pipelines, models, and validation from their own feedback, yet on most competitions they still finish below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final submission and discard the development process behind it. We introduce TraceML, which pairs human and agent work on the same competitions under one version-level schema: 4{,}465 human Kaggle trajectories across 134 competitions, seven of which are also worked by two agent scaffolds, giving 430 paired human and 207 agent trajectories. Every code version carries its score, its timestamp, and labels for the action taken, its intent, the edit size, and the score effect. Read this way, the gap becomes concrete. Experts alternate data work, validation, model changes, and ensembling, and return to approaches they had set aside. Each agent scaffold instead collapses into a narrow loop: Codex spends its steps re-weighting ensembles and tuning submissions, MLEvolve mutates its model in place, and neither pivots at the human rate nor reopens abandoned work. A short planning prompt distilled from human practice moves the behaviors it names toward the human profile and lifts scores, but the effort profile stays agent-shaped: instruction closes only the part of the gap that reduces to instructions. We release the corpus, the schema, and the labeling models, together with an open-source toolkit that turns a run from any command-line agent into a TraceML trajectory and reads it against the human cohorts. Subjects: Machine Learning (cs.LG) ; Artificial Intelligence (cs.AI) Cite as: arXiv:2608.26086 [cs.LG] (or arXiv:2608.26086v3 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2608.26086 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Jiarui Yan [ view email ] [v1] Wed, 26 Aug 2026 17:50:13 UTC (801 KB) [v2] Thu, 27 Aug 2026 21:22:31 UTC (798 KB) [v3] Mon, 28 Sep 2026 22:20:24 UTC (811 KB) Full-text links: Access Paper: View PDF HTML (experimental) TeX Source view license Additional Features Audio Summary Current browse context: cs.LG < prev | next > new | recent | 2026-08 Change to browse by: cs cs.AI References & Citations SciX Google Scholar Semantic Scholar export BibTeX citation Bookmark Bibliographic Tools Bibliographic and Citation Tools Bibliographic Explorer Toggle Bibliographic Explorer ( What is the Explorer? ) Connected Papers Toggle Connected Papers ( What is Connected Papers? ) Litmaps Toggle Litmaps ( What is Litmaps? ) scite.ai Toggle scite Smart Citations ( What are Smart Citations? ) Code, Data, Media Code, Data and Media Associated with this Article alphaXiv Toggle alphaXiv ( What is alphaXiv? ) Links to Code Toggle CatalyzeX Code Finder for Papers ( What is CatalyzeX? ) DagsHub Toggle DagsHub ( What is DagsHub? ) GotitPub Toggle Gotit.pub ( What is GotitPub? ) Huggingface Toggle Hugging Face ( What is Huggingface? ) ScienceCast Toggle ScienceCast ( What is ScienceCast? ) Demos Demos Replicate Toggle Replicate ( What is Replicate? ) Spaces Toggle Hugging Face Spaces ( What is Spaces? ) Spaces Toggle TXYZ.AI ( What is TXYZ.AI? ) Related Papers Recommenders and Search Tools Link to Influence Flower Influence Flower ( What are Influence Flowers? ) Core recommender toggle CORE Recommender ( What is CORE? ) IArxiv recommender toggle IArxiv Recommender ( What is IArxiv? ) Author Venue Institution Topic About arXivLabs arXivLabs: experimental projects with community collaborators arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website. Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them. Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs . Which authors of this paper are endorsers? | Disable MathJax ( What is MathJax? ) We gratefully acknowledge support from our major funders , member institutions , and all contributors. About · Help · Contact · Subscribe · Copyright · Privacy · Accessibility · Operational Status (opens in new tab) Major funding support from (opens in new tab) (opens in new tab) (opens in new tab)

Related documents

• TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development#779366
• Toward Latent Language Model Skills Steering and Optimization: An Empirical Study#794243
• Toward Latent Language Model Skills Steering and Optimization: An Empirical Study#790224
• AI Harness Engineering: A Runtime Substrate for Foundation-Model Software Agents#180720
• A Lean and Spec-Driven AI-Assisted Software Development Lifecycle for Applied AI Education: The AI-SDLC Approach#1050527