MLEvolve: A Self-Evolving Framework for Automated Machine Learning Algorithm Discovery Shangheng Du1 2 , Xiangchao Yan1 ♣ , Jinxin Shi1 2 , Zongsheng Cao1 , Shiyang Feng1 , Zichen Liang1 , Boyuan Sun1 , Tianshuo Peng1 , Yifan Zhou1 , Xin Li1 , Jie Zhou1 2 , Liang He1 2 , Bo Zhang1 ♣ and Lei Bai1 ♣
arXiv:2606.06473v1 [cs.AI] 4 Jun 2026
1 Shanghai Artificial Intelligence Laboratory, 2 East China Normal University
Large language model (LLM) agents are increasingly applied to long-horizon tasks such as scientific discovery and machine learning engineering (MLE), where sustained self-evolution becomes a key capability. However, existing MLE agents suffer from inter-branch information isolation, memoryless search, and lack of hierarchical control, which together hinder long-horizon optimization. We present MLEvolve, an LLM-based self-evolving multi-agent framework for end-to-end machine learning algorithm discovery. By extending tree search to Progressive MCGS, MLEvolve enables cross-branch information flow through graph-based reference edges and gradually shifts the search from broad exploration to focused exploitation with an entropy-inspired progressive schedule. To allow the agent to evolve with accumulated experience, we introduce Retrospective Memory, which combines a cold-start domain knowledge base with a dynamic global memory for task-specific experience retrieval and reuse. For stable long-horizon iteration, we further decouple strategic planning from code generation with adaptive coding modes. Evaluation on MLE-Bench shows that MLEvolve achieves state-of-the-art performance across multiple dimensions including average medal rate and valid submission rate under a 12-hour budget (half the standard runtime). Moreover, MLEvolve also outperforms specialized algorithm discovery methods including AlphaEvolve on mathematical algorithm optimization tasks, demonstrating strong cross-domain generalization. Our code is available at https://github.com/InternScience/MLEvolve.
1. Introduction Artificial intelligence (AI) is reshaping scientific research and complex engineering, leading to the paradigm of AI for Science [1]. With the continued advancement of large language models (LLMs), LLM-based agent systems [2] are now being applied to long-horizon autonomous tasks such as scientific discovery [3, 4], automated experimentation [5], and end-to-end algorithm design [6]. Unlike single-turn reasoning, these scenarios involve open search spaces and limited time budgets, where agents must continually generate solutions, execute code, evaluate outcomes, and adjust strategies based on feedback. During this process, the agent continuously evolves: accumulating experience from past trials, adaptively adjusting exploration strategies, and progressively refining implementations according to the current search stage. This sustained self-evolving capability is becoming central to long-horizon autonomous agents. Machine Learning Engineering (MLE) is one of the most representative scenarios for such long-horizon self-evolving tasks. Designing high-performance AI systems still relies heavily on expert knowledge and extensive manual iteration [7]. Although recent advances in AutoML [8, 9] have achieved significant progress in optimizing discrete stages such as data processing and model selection, they often fall short of covering the entire end-to-end MLE pipeline, i.e., from data preparation to model training and inference. Recently, LLM-based coding agents have been applied to MLE scenarios [10, 11, 12, 13, 14], using the planning and code generation capabilities of LLMs to iteratively optimize within open search spaces. These agents typically employ greedy or evolutionary search [12, 15],
♣: Please send correspondence regarding this report to [email protected], [email protected], and [email protected]
Monte Carlo Tree Search [14, 16], or multi-agent collaboration [13] to explore candidate solutions. Despite these advances, existing MLE agents still face three key challenges that hinder self-evolution over long horizons. First, existing search mechanisms are limited by information isolation between branches and lack adaptive exploration strategies. Most methods adopt linear or tree-structured search [12, 14, 16], confining information within individual branches and making it difficult to transfer successful strategies across different search trajectories. Moreover, these methods generally employ fixed exploration strategies throughout the optimization process, leading to inefficient resource allocation under limited time budgets. Second, most search frameworks are memoryless and unable to accumulate experience from past interactions [17, 18]. Current search frameworks propagate only scalar rewards, resulting in each planning decision being made in isolation without reusing insights from similar attempts earlier in the search. While some recent methods explore memory mechanisms [17, 18, 19], they require extra LLM calls or provide only static knowledge, lacking automatic experience accumulation during search. Third, most existing methods couple planning and code implementation into one-shot generation, lacking hierarchical control. A reasonable design requires distinguishing what to modify from how to implement, yet many methods [14, 20] rewrite the entire solution at every iteration, resulting in low iteration efficiency and uncontrollable modifications. Retrospective Memory
Limitations of Existing Methods Branch Isolation
Knowledge Base (domain priors)
Plan
non-adaptive exploration
Global Experience (search records)
Adaptive Code Generation Memoryless Search
Evolve
Generate
no experience reuse
Planner (why)
Coder (how) detailed plan (what)
Stepwise (module by module)
Diff (patch edit)
Execute lacking hierarchical control
• solution.py • submission.csv • top-k candidates
coding modes (by search state) Base (full rewrite)
One-shot Generation
Best solution
Progressive MCGS
MLE-Bench
(75 Kaggle competitions)
AlphaEvolve Math
(Math algorithm problems)
• Graph-based exploration • Cross-branch sharing • Adaptive exploration -> exploitation over time
Figure 1 | Overview of MLEvolve that summarizes its core components and supported tasks. Existing MLE agents suffer from inter-branch isolation, memoryless exploration, and lack of hierarchical control. MLEvolve addresses these through Progressive MCGS, Retrospective Memory, and Hierarchical Planning with Adaptive Code Generation, supporting long-horizon iterative optimization tasks, such as end-to-end MLE and mathematical algorithm discovery. To bridge this gap, we present MLEvolve (Figure 1), an LLM-based self-evolving multi-agent framework for MLE tasks. MLEvolve unifies three core components: (1) Progressive Monte Carlo Graph Search (MCGS), which addresses isolation and limited reuse in tree search through graph-based cross-branch information flow, and introduces an entropy-inspired progressive exploration schedule that adaptively steers the search from broad exploration to focused exploitation over time; (2) Retrospective Memory, pairing a curated domain knowledge base for cold-start initialization with a dynamic global memory that automatically accumulates and retrieves task-specific experience throughout the search; and (3) Hierarchical Planning with Adaptive Code Generation, which separates strategic planning from code generation and selects among full rewrite, stepwise, and diff-based editing modes according to the current search state. Consequently, MLEvolve achieves more stable and self-evolving exploration of end-to-end ML pipelines, leading to stronger solutions for challenging MLE tasks. Experimental results
2
show that MLEvolve achieves a 65.3% average medal rate on MLE-Bench under a 12-hour budget (half the standard runtime), establishing state-of-the-art performance, and further outperforms specialized algorithm discovery methods including AlphaEvolve [6] on mathematical optimization tasks. Our key contributions are as follows: • We propose MLEvolve, a self-evolving multi-agent framework for end-to-end MLE tasks, which unifies progressive graph search, retrospective memory, and hierarchical adaptive code generation to support long-horizon iterative optimization. • We introduce Progressive MCGS and Retrospective Memory for self-evolving optimization. Progressive MCGS resolves inter-branch isolation through graph-based cross-branch information flow and a progressive exploration schedule, while Retrospective Memory enables automatic experience accumulation and retrieval throughout the search. • Extensive experiments show that MLEvolve achieves a 65.3% average medal rate on MLE-Bench under a 12-hour budget, achieving the best among all existing methods, and further outperforms AlphaEvolve [6] and AlphaEvolve-v2 [21] on mathematical optimization tasks, demonstrating cross-domain generalization.
2. Related Work 2.1. Automated Machine Learning Algorithm Discovery To address the unique challenges of MLE, a dedicated class of coding agents has been developed [12, 22, 23, 14], with many evaluated on benchmarks such as MLE-Bench [24]. These agents primarily frame the problem as a search for an optimal code-based solution. Early works like AIDE [12] employ greedy search, which is susceptible to local optima. Subsequent frameworks adopt more structured exploration. ML-Master [14] and AIRA-Dojo [16] use MCTS, MARS [18] introduces budget-aware MCTS with contrastive reflection, and FM-Agent [15] applies evolutionary multi-island parallel search. Other works explore agent collaboration, such as R&D-Agent [13] with researcher-developer combination and AIBuildAI [25] with hierarchical multi-agent coordination. Several methods also incorporate external knowledge or memory. AutoMind [17] and Leeroo [26] ground search with domain knowledge bases, while ML-Master 2.0 [19] introduces hierarchical cognitive caching for crosstask knowledge distillation. However, these methods commonly suffer from inter-branch information isolation and the inability to accumulate and reuse experience from past trials. Our method addresses these limitations from a self-evolving perspective, enabling the agent to continuously adapt its search behavior, accumulate experience, and refine solutions during long-horizon optimization.
2.2. Graph-based Planning and Search Early methods that combine graph structures with MCTS, often referred to as MCGS [27, 28], were primarily developed for planning and reinforcement learning tasks with well-defined state spaces, where identical states are merged to compress the search space. Recent graph-based frameworks such as LocAgent [29] and CodexGraph [30] use graphs as static dependency representations for retrieval or localization, but these graphs do not evolve during search. In contrast, our MCGS targets open-ended LLM-based code generation, where each node represents a distinct candidate solution. The graph structure is not used to compress the state space, but to enable cross-branch information flow, trajectory reuse, and solution composition through dynamic reference edges.
3
Retrospective Memory
Progressive Monte-Carlo Graph Search
Knowledge Base
ROOT
Early Search (broad exploration)
(cold-start priors, domain knowledge)
Inter-Node Pipeline
Global Experience (plan, code, metric, analysis, ...)
Plan Generation
Code Generation & Review
Code Execution
Results Update
Hierarchical Planning & Adaptive Code Generation Base (full write)
Hybrid retrieval: BM25 ⊕ FAISS→RRF→Top-K records
Cross-Branch Reference
Multi-Branch Aggregation
"branch stagnation
continued branch stagnation
"global stagnation
...
gradually decays
Past Attempts
N*
N*
A-root
New Solution
Improve
Evolution
Fusion
ET: Primary Edge
Debug A-root
N*
Top/Best
Aggregation
N*
N*
N*
N*
N*
...
N* N*
N*
Top Nodes and Trajectory from Other Branches
Parent Node
I’ve failed many times. Let’s check the past attempts.
N*
New Solution
The performance cannot be further improved. I need to learn from the solutions of other branches.
Ensemble
Graph-level Top Nodes
Eref: Intra-branch Evolution
N*
Top Nodes from Other Branches
Parent Node
Late Search (focused exploitation)
Draft
Diff (patch edit)
Intra-Branch Evolution w(t)
N*
Stepwise (module by module)
Submission
The ideas from different branches can be fused. Let’s try a new solution from root.
A-root
New Branch Draft Solution
Draft
Improve
Code Review
Evolution
Aggregation
Debug
Data Leakage
Fusion
Eref: Cross-branch Reference
Eref: Multi-branch Aggregation
Figure 2 | Framework of MLEvolve. The framework consists of three components. (i) Progressive MCGS extends MCTS with graph-based cross-branch information flow and a progressive exploration schedule. (ii) Retrospective Memory pairs a cold-start knowledge base with a dynamic global memory for experience accumulation and retrieval. (iii) Hierarchical Planning with Adaptive Code Generation decouples strategic planning from code implementation and selects among different coding modes according to the search state.
2.3. Memory and Experience Mechanisms for LLM Agents Memory mechanisms have been explored to improve LLM agent performance in iterative tasks [31]. Recent work on long-term episodic memory [32] enables agents to accumulate and retrieve experiential records across extended horizons, supporting more informed subsequent decisions. In the MLE domain, recent works further explore experience reuse. ROME [33] introduces “reasoning gradients” as structured optimization directions and stores successful trajectories as momentum memory, MARS [18] extracts insights through contrastive reflection over historical attempts, and ML-Master 2.0 [19] introduces hierarchical cognitive caching for cross-task knowledge distillation. While these methods advance experience reuse, most require additional LLMs for reflection or summarization. Our retrospective memory automatically accumulates and retrieves experience without requiring additional LLMs for explicit reflection, and further incorporates a static domain knowledge base for cold-start initialization.
3. MLEvolve In automated algorithm discovery, strong solutions often arise from careful design, accumulated experience, and reference to multiple candidate pathways, rather than from a single linear refinement. To this end, we introduce MLEvolve, a self-evolving multi-agent framework for MLE tasks. As shown in Figure 2, the design combines three key components: (1) Progressive MCGS (§3.2), which extends MCTS with graph-based cross-branch information sharing and a progressive exploration schedule to transition from broad exploration to focused exploitation; (2) Retrospective Memory (§3.3), combining a static domain knowledge base for cold-start initialization with a dynamic global memory 4
that automatically accumulates and retrieves historical experience during search; and (3) Hierarchical Planning with Adaptive Code Generation (§3.4), which separates strategic planning from code generation and selects different coding modes according to the current search state.
3.1. Problem Formulation Our objective is to automate the search, design, and optimization of end-to-end ML pipelines. We formalize the task as identifying the optimal solution within a structured search space [12], where each node represents a complete candidate solution covering preprocessing, feature engineering, model training, and prediction. The goal is to find the optimal solution for a given task: 𝑠∗ = arg max ℎ (𝑇, 𝑠) , 𝑠∈ S
(1)
where ℎ (𝑇, 𝑠) denotes the evaluation of candidate solution 𝑠 on task 𝑇 , which may vary by task (e.g., accuracy, AUC, or loss). The solution space S is organized as a directed graph and explored through iterative search.
3.2. Progressive MCGS The search strategies in existing MLE methods face limitations such as branch information isolation and overly fixed search behavior. Greedy and evolutionary algorithms are prone to becoming trapped in local optima, while tree-search-based methods often spend substantial resources exploring lowvalue branches under limited time budgets, leading to inefficient resource allocation in later stages. To address these limitations, we propose Progressive MCGS, which introduces a graph structure that enables cross-branch information sharing and a progressive exploration schedule that adaptively balances exploration and exploitation over time. 3.2.1. Graph-based Search Space To realize the optimization objective in Eq. (1), we organize the search process as a directed graph: 𝐺 = (𝑉, 𝐸) ,
𝐸 = 𝐸𝑇 ∪ 𝐸ref ,
(2)
where each node 𝑣 ∈ 𝑉 maps to a candidate solution 𝑠 ( 𝑣) ∈ S. Directed edges capture both generative and referential relationships: • Primary edges 𝐸𝑇 : ( 𝑢, 𝑣) ∈ 𝐸𝑇 means that 𝑣 is derived from 𝑢 by applying an operator 𝑜, i.e., 𝑣 = 𝑔𝑜 ( 𝑢, 𝑅). These edges preserve the parent–child generative order and are used for selection and backpropagation. • Reference edges 𝐸ref : ( 𝑟, 𝑣) ∈ 𝐸ref denotes that 𝑣 additionally incorporates information from node 𝑟 beyond its parent node. These edges connect nodes across branches or non-adjacent levels, enabling cross-branch knowledge flow and compositional transfer, but do not participate in backpropagation. When 𝐸ref = ∅, the search reduces to standard MCTS. 3.2.2. Progressive MCGS-based Exploration The MCGS process follows the classical MCTS loop of selection, expansion, simulation, and backpropagation, with a progressive exploration schedule in the selection phase and graph-based expansion types. Selection with Progressive Exploration Scheduling. Although the overall search space is formulated as a graph, the selection stage operates solely on the tree backbone formed by the primary edges 𝐸𝑇 . 5
At each iteration, the selection policy traverses 𝐸𝑇 in a top-down manner to identify a node 𝑣𝑡 for expansion using the UCT criterion: √︄ ln( 𝑁𝑣 + 1) , (3) 𝜋sel ( 𝑣) = arg max UCT( 𝑖) , where UCT( 𝑖) = 𝑄 𝑖 + 𝑐 ( 𝑡 ) 𝑁𝑖 + 𝜀 𝑖 ∈ C ( 𝑣) where 𝑄 𝑖 denotes the average reward of child node 𝑖, 𝑁𝑖 is its visit count, 𝑁𝑣 is the visit count of the parent, and 𝜀 > 0 is a smoothing constant. The exploration constant 𝑐 ( 𝑡 ) is gradually reduced over time following a piecewise schedule (𝑐0 → 𝑐min ). Inspired by entropy-based exploration principles [34], we introduce an entropy-inspired progressive exploration schedule that transitions the search from broad exploration toward focused exploitation. Within a local time window, the branch selection frequencies form an empirical distribution 𝜋𝑡 , Í whose Shannon entropy 𝐻 ( 𝜋𝑡 ) = − 𝑖 𝜋𝑡 ( 𝑖) log 𝜋𝑡 ( 𝑖) quantifies the dispersion of search effort. The core mechanism is a probabilistic soft switch between UCT-based exploration (higher entropy) and Elite-Guided exploitation (lower entropy). At each step, the system chooses between these strategies according to a time-dependent weight: 𝑃 ( 𝑆𝑡 = UCT) = 𝑤 ( 𝑡 ) ,
𝑃 ( 𝑆𝑡 = Elite) = 1 − 𝑤 ( 𝑡 ) ,
(4)
where 𝑆𝑡 denotes the selection strategy at step 𝑡 , 𝑤 ( 𝑡 ) gradually decreases from 1.0 to a minimum threshold 𝑤min as search time progresses. The schedule 𝑤 ( 𝑡 ) is designed so that the empirical branchselection entropy 𝐻 ( 𝜋𝑡 ) progressively decreases over time, concentrating computation on promising branches. In the Elite-Guided exploitation mode, the system bypasses local tree traversal and selects from an elite set of top- 𝐾 globally best-performing nodes, weighted by inverse rank: 1/rank( 𝑣𝑖 ) , 𝑗=1 1/rank( 𝑣 𝑗 )
𝑃 ( 𝑣𝑖 | elite set) = Í 𝐾
(5)
where rank( 𝑣𝑖 ) is the position of node 𝑣𝑖 when all valid nodes are sorted by metric. This allows the search to directly exploit high-value nodes regardless of their position in the graph, while the probabilistic transition retains exploration capacity even in later stages. Expansion. To incorporate information flow and compositional reuse into the search process, we extend the standard MCTS expansion with graph-based operations. All expansion types are unified under a single formulation: 𝑣new = 𝑔𝑜 ( 𝑣𝑡 , 𝑅) ,
( 𝑣𝑡 , 𝑣new ) ∈ 𝐸𝑇 , {( 𝑟, 𝑣new ) | 𝑟 ∈ 𝑅 } ⊆ 𝐸ref ,
(6)
where 𝑅 denotes the reference set. We instantiate this formulation with four expansion types (formal definitions in Appendix B): (1) Primary expansion ( 𝑅 = ∅). The new node is generated solely from its parent without referencing other nodes. This constitutes the baseline expansion against which the graph-based variants extend. (2) Intra-branch evolution (𝑅 = R hist ( 𝑣𝑡 , 𝑘)). Inspired by human problem-solving strategies, this mode emphasizes reflecting on past attempts instead of blind trial and error. The agent takes the nearest 𝑘 nodes within the same branch to form a local trajectory as the reference set, reviewing which changes improved outcomes or caused failures. Through self-reflection, the agent reinforces effective patterns while avoiding repeated mistakes. (3) Cross-branch reference (𝑅 = R cross ( 𝑁 )). In ML competitions, contestants often draw inspiration from community-shared solutions when progress stalls. Similarly, when a branch shows signs of stagnation, MCGS selects the top- 𝑁 nodes across all evaluated branches as references, enabling the agent to draw on strong solutions discovered in other branches. 6
(4) Multi-branch aggregation (𝑅 = R agg ). For complex tasks, progress often requires synthesizing complementary insights from multiple strong solutions. This resembles a form of collective intelligence: trajectories from different branches are merged and fragments of useful insights are combined to spark novel directions. A new branch root is created beneath 𝑣0 , serving as a fresh starting point. Representative cases are provided in Appendix G. Simulation. After generating a candidate 𝑣new , its code is executed in an interpreter. The execution outputs are parsed to extract the task-specific metric and execution logs. An immediate reward 𝑅 ( 𝑣) is designed to reflect execution validity and performance contribution: −1, 𝑅 ( 𝑣) = 1, 2,
if execution fails or no valid metric is obtained if execution succeeds but does not improve the branch best if execution succeeds and refreshes the branch best metric.
(7)
This structure distinguishes failed runs, feasible but non-improving attempts, and actual improvements, yielding stable credit assignment during MCGS. Backpropagation. After simulation, the reward 𝑅 ( 𝑣) is propagated to the root only along primary edges 𝐸𝑇 . Reference edges 𝐸ref are excluded because they represent auxiliary information reuse rather than parent–child generation, and therefore should not participate in credit assignment. For each ancestor node 𝑢 on the primary path, we update its visit count 𝑁𝑢 and cumulative reward 𝑊𝑢 : 𝑁𝑢 ← 𝑁𝑢 + 1,
𝑊𝑢 ← 𝑊𝑢 + 𝑅 ( 𝑣) ,
(8)
and compute the average value estimate: 𝑄 𝑢 = 𝑊𝑢 /( 𝑁𝑢 + 𝜀) .
(9)
Multi-Level Stagnation Detection. While the soft-switch schedule governs the global explorationexploitation transition, the graph-based operators introduced above are triggered by explicit stagnation conditions to prevent branches from falling into unproductive loops: • Branch-level stagnation: triggered when a branch produces 𝜏branch consecutive expansions without improving its best metric. The system first attempts intra-branch evolution; in later stages when other branches have accumulated strong solutions, cross-branch reference is further activated to incorporate external knowledge. • Global-level stagnation: triggered when the global best metric has not improved for 𝜏global steps, activating multi-branch aggregation.
3.3. Retrospective Memory To enable experience accumulation during search, we introduce a retrospective memory that retrieves relevant historical experience before each planning decision, transforming the search into experiencedriven decision-making. The memory comprises a static domain knowledge base for cold-start initialization and a dynamic global memory for runtime experience accumulation. 3.3.1. Domain Knowledge Base Effective ML solution design typically relies on domain priors and hands-on experience. LLM internal knowledge alone is often insufficient for specialized tasks, leading to a high rate of cold-start errors. To mitigate this, we curate a lightweight domain knowledge base of candidate models, organized by task type. For different task types (e.g., image classification, natural language processing, tabular 7
regression), the knowledge base provides suitable models together with concise usage guidelines, synthesized from open-source repositories and competition platforms. Given a task 𝑇 , the system retrieves relevant entries 𝑅 𝐾 𝐵 (𝑇 ) by matching the task description against domain keywords, treated as an optional signal during initial solution generation: 𝑠init = Init(𝑇, 𝑅 𝐾 𝐵 (𝑇 )) ,
(10)
where Init(·) denotes the initialization procedure that generates the first plan and code. 3.3.2. Dynamic Global Memory During search, the global memory accumulates structured records after each valid node execution including the plan, outcome, analysis, and feedback signal. Hybrid retrieval. Records are retrieved via a combination of lexical keyword matching and FAISS [35]based semantic search, fused through Reciprocal Rank Fusion (RRF): score( 𝑑 ) = 𝛼 ·
1 𝑘 + 𝑟lex ( 𝑑 )
+ (1 − 𝛼) ·
1 𝑘 + 𝑟vec ( 𝑑 )
,
(11)
where 𝑟lex ( 𝑑 ) and 𝑟vec ( 𝑑 ) denote the ranks of record 𝑑 in the lexical and vector retrieval results, respectively; 𝑘 is a smoothing constant; and 𝛼 balances the two signals. Stage-aware retrieval. Agents retrieve memory records with stage-specific queries and filters: • Planning stage: After generating an initial free-text plan, the agent uses it as a query to retrieve relevant successful and failed experiences. These records guide the refinement of the plan into a structured module-level specification, helping the agent reuse effective strategies while avoiding previously unsuccessful directions. • Debugging stage: When encountering an execution error, the agent uses the error message as a query to retrieve similar resolved errors from memory, providing helpful debug strategies.
3.4. Hierarchical Planning and Adaptive Code Generation To address the lack of hierarchical control in one-shot code generation, we introduce a hierarchical generation pipeline that decouples strategic planning from code implementation and adaptively selects among different code generation modes according to the current search state. 3.4.1. Planner-Coder Decoupling We decouple strategic planning from code generation to separate global reasoning from local implementation. The planner operates at the module level, using execution feedback, branch trajectories, and retrieved memory to decide what to modify and why. The coder then implements the planned changes at the code level, focusing on how to realize the modification while preserving the existing code structure and working functions. 3.4.2. Adaptive Code Generation Modes Rather than applying a single code generation mode, the coder applies three coding modes with different granularity, selected according to the current search state and task requirements: • Base mode: Full code generation from scratch. This mode constructs a complete solution when no reliable solution is available, especially during initial drafting. 8
• Stepwise mode: Module-by-module generation following the planner’s specification. This mode is used for complex tasks that require multi-stage pipelines, where decomposing the solution into modules helps reduce generation difficulty. • Diff mode: Targeted diff edits on the existing code. When a working solution already exists, this mode enables localized refinements with more stable and controlled modifications. The framework is realized through a team of specialized agents, each tailored to a specific search phase or operator type. Detailed agent descriptions are provided in Appendix A.
4. Experiments 4.1. Experiment Setup Benchmarks. We evaluate MLEvolve on two benchmarks. The primary benchmark is MLE-Bench [24], introduced by OpenAI for end-to-end machine learning engineering, comprising 75 Kaggle tasks across three complexity levels (low, medium, and high), with full details and evaluation metrics in Appendix C. To assess cross-domain generalization, we also use 15 open-ended mathematical optimization tasks from AlphaEvolve [6]. Implementation details. We adopt Gemini-3.1-Pro-preview as the backbone LLM for all agents, with temperature set to 1.0. Each task is assigned a maximum of 500 expansion steps and a 12-hour runtime, executed on 21 vCPUs, 234 GB of RAM, and a single NVIDIA H200 GPU. Full hyperparameter settings are listed in Appendix D. Baselines. We compare MLEvolve with a series of MLE agents, including both proprietary and opensource agent frameworks. The proprietary methods include FM-Agent [15], MLE-STAR-Pro-1.5 [22], MARS [18], MARS+ [18], and AIBuildAI [25]. The open-source methods include AIDE [12], R&DAgent [13], ML-Master [14], AIRA-Dojo [16], Leeroo [26], and ML-Master 2.0 [19]. The baseline results in Table 1 are taken from the MLE-Bench leaderboard or the corresponding papers.
4.2. Main Results MLEvolve achieves state-of-the-art performance on MLE-Bench. As shown in Table 1, under a 12-hour budget, MLEvolve attains an average medal rate of 65.3% and a gold medal rate of 34.7%, achieving the best overall performance among all compared MLE agents. The results are consistent across difficulty levels, with medal rates of 80.3%, 64.0%, and 46.7% on low, medium, and high complexity tasks, respectively. In addition, MLEvolve achieves a 100% valid submission rate and a 76.0% above-median rate, meaning that its submissions surpass the human median Kaggle score in more than three quarters of the tasks. MLEvolve outperforms both open-source and proprietary baselines at half the standard 24-hour budget. MLEvolve generalizes to mathematical algorithm optimization. To further evaluate the generalization ability of MLEvolve in self-evolving optimization scenarios, we apply it to the 15 mathematical optimization tasks from AlphaEvolve. These tasks differ from end-to-end MLE pipelines, but share a similar iterative optimization structure: the agent repeatedly proposes candidate solutions, evaluates their quality, and refines them through continued search. As shown in Table 2, MLEvolve achieves the best result on 11 of 15 tasks when compared with specialized algorithmic discovery methods, including AlphaEvolve [6], AlphaEvolve-v2 [21], SimpleTES [36], TTT-Discover [37], and OpenEvolve [38]. These results suggest that our self-evolving mechanism is not limited to the MLE domain, but can generalize to broader algorithmic optimization problems that require iterative optimization.
9
Table 1 | Main results on MLE-Bench (75 tasks, full set). Medal rates are reported across three complexity levels and overall, along with valid submission rate, above-median rate, and gold medal rate. Results are mean ± SEM over 3 seeds. We group methods by whether their code is publicly available. Best results are in bold; second best is underlined. Medal rate by complexity Agent
Time (h)
Low (%)
Medium (%) High (%)
Other evaluation dimensions All (%)
Valid (%)
Med+ (%)
Gold (%)
Proprietary Methods FM-Agent [15] Gemini-2.5-Pro
24
62.1±1.5
36.8±1.5
33.3±0.0
43.6±0.9
96.9±1.2
51.6±1.2
22.7±0.8
MLE-STAR-Pro-1.5 [22] Gemini-2.5-Pro
24
68.2±2.6
34.2±1.5
33.3±0.0
44.0±1.3
93.8±0.4
52.9±1.6
19.1±1.8
MARS [18] Gemini-3-Pro-preview
24
74.2±1.5
52.6±3.0
37.8±2.2
56.0±1.5
98.7±0.0
65.8±1.6
31.1±0.4
MARS+ [18] Gemini-3-Pro-preview
24
78.8±1.5
60.5±1.5
44.4±2.2
62.7±0.8
100.0±0.0
74.2±0.9
33.8±0.4
AIBuildAI [25] Claude-Opus-4.6
24
77.3±0.0
61.4±0.9
46.7±0.0
63.1±0.4
100.0±0.0
71.1±1.2
25.8±0.4
Open-Source Methods AIDE [12] o1-preview
24
35.9±1.9
8.5±0.4
11.7±1.3
17.1±0.6
82.8±1.1
29.4±1.3
9.4±0.8
R&D-Agent [13] gpt-5
12
68.2±2.6
21.1±1.5
22.2±2.2
35.1±0.4
53.3±0.0
40.4±0.9
16.4±0.9
ML-Master [14] DeepSeek-R1
12
48.5±1.5
20.2±2.3
24.4±2.2
29.3±0.8
93.3±1.3
44.9±1.2
17.3±0.8
AIRA-Dojo [16] o3
24
55.0±1.5
22.0±1.2
21.7±1.1
31.6±0.8
97.5±0.3
45.5±0.8
17.3±0.4
Leeroo [26] Gemini-3-Pro-preview
24
68.2±2.6
44.7±1.5
40.0±0.0
50.7±1.3
50.7±1.3
50.7±1.3
21.3±2.0
ML-Master 2.0 [19] DeepSeek-V3.2-Speciale
24
75.8±1.5
50.9±3.5
42.2±2.2
56.4±2.5
95.6±1.2
63.1±1.2
19.6±0.9
MLEvolve (ours) Gemini-3.1-Pro-preview
12
80.3±1.5
64.0±0.9
46.7±0.0
65.3±0.8
100.0±0.0
76.0±2.3
34.7±0.0
4.3. Ablation Study To evaluate the effectiveness of proposed components, we conduct ablation experiments on MLE-Bench Lite (22 tasks) by removing one component at a time while keeping all others unchanged. As shown in Table 3, removing any single component leads to a clear performance decline, indicating that all three components help alleviate the existing limitations. Specifically, removing Progressive MCGS causes the largest drop in both medal rate and beat ratio. Without this module, the search reverts to standard tree-based MCTS with a fixed strategy that wastes resources on low-value branches in later stages. Removing Retrospective Memory also leads to a 13.64% drop in medal rate. In this setting, the agent can still occasionally discover strong solutions through search alone, but it lacks experience feedback and guidance in long-horizon tasks. Replacing adaptive code generation with one-shot generation similarly reduces overall performance, since the absence of planner-coder decoupling and diff-based editing weakens the stability of iterative code refinement. We further analyze the individual mechanisms within Progressive MCGS and Retrospective Memory in Appendix E. The results show that intra-branch evolution is the most critical factor within Progressive MCGS, while Elite-Guided exploitation mainly improves leaderboard ranking by further refining already competitive solutions
10
Table 2 | Comparison on 15 mathematical programming tasks grouped by problem type. ↑ / ↓ means higher/lower is better. Values are displayed with task-dependent precision. Best results are in bold; second best is underlined. “–” indicates the result is not reported. Problem
↑ /↓
Geometric packing / regions Packing hexagons in hexagons Circle packing in a square Circle packing in a rectangle Heilbronn convex regions Heilbronn triangles Kissing number dimension 11
AlphaEvolve AlphaEvolve-v2 SimpleTES
TTT-Discover OpenEvolve
MLEvolve
↓ ↑ ↑ ↑ ↑ ↑
3.930092 2.6358627564 2.3658321334 0.0309368890 0.03652988988003016 593
3.931 – – 0.0309 0.0365 593
3.931 2.635983 – – – –
– – – – – –
– 3.9284759302 – 2.6359830395 – 2.3658323759 – 0.0309372079 – 0.03652988988003020 – 592
Additive combinatorics Sums differences problems 1 Sums and differences problems 2
↑ ↑
1.1479889651 1.1584172816
1.1479 1.1584
1.143975 –
– –
– –
1.1901774219 1.1585457700
Autocorrelation / inequalities An uncertainty inequality First autocorrelation inequality Third autocorrelation inequality variant Third autocorrelation inequality Second autocorrelation inequality
↓ ↓ ↓ ↓ ↑
0.3520991044225 1.5052939684 1.4687620697 1.4556427954 0.8962799442
0.3521 1.5032 – 1.4557 0.961
– – 1.503871 1.5028628983 – – 1.453675 – 0.962694 0.959100
– 1.507190 – 1.460000 0.944900
0.3520991044160 1.5028628749 1.4587698922 1.4548507482 0.9054217971
Ratio / overlap optimization Max-to-min ratios Minimum Overlap Problem
↓ ↓
12.88926611203 0.3809230351
12.889266112 0.380924
– – 0.380868 0.3808753232
– 0.380965
12.8892299077 0.3808968496
Table 3 | Component-level ablation on MLE-Bench Lite (22 tasks). Each row removes one core component of MLEvolve. Beat Ratio is the average percentage of human Kaggle competitors outperformed. Best performances are marked in bold. Configuration
Medal (%)
Gold (%)
Beat Ratio (%)
81.82 68.18 68.18 72.73
54.55 40.91 50.00 40.91
88.39 79.91 81.90 84.14
MLEvolve w/o Progressive MCGS w/o Retrospective Memory w/o Adaptive Code Generation
toward higher-performing ones.
4.4. Further Analysis 4.4.1. Progressive Search Entropy Dynamics To empirically validate the progressive transition from exploration to exploitation described in §3.2.2, we measure the effective number of active branches during search. Specifically, within a sliding window at search progress 𝑡 , we compute the empirical distribution 𝜋𝑡 of branches selected for solution iteration, and use exp( 𝐻 ( 𝜋𝑡 )) to quantify the number of branches over which search effort is effectively distributed, where 𝐻 ( 𝜋𝑡 ) is the Shannon entropy of 𝜋𝑡 . As shown in Figure 3, MLEvolve gradually reduces the effective number of active branches from 4.8 in the early exploration stage to 2.8 in the later exploitation stage, indicating that the soft switch schedule progressively concentrates computation on more promising candidates. In contrast, Vanilla MCTS remains almost uniform throughout, continuing to spread resources across branches even after promising directions emerge. The observed entropy trend is consistent with the scheduling behavior in Eq. (4), showing the effectiveness of the progressive exploration schedule and Elite-Guided exploitation.
11
Effective # of Active Branches exp(H( _t))
6
MLEvolve (Ours) Vanilla MCTS
4.8
5
4.8
4.3
4
3
2.8
2
Elite-Guided Exploitation
Exploration 1
0.0
0.2
0.4
0.6
Search Progress t / T
0.8
1.0
Figure 3 | Effective branch count exp( 𝐻 ( 𝜋𝑡 )) over search progress. MLEvolve progressively reduces from 4.8 to 2.8, empirically validating the soft-switch schedule (Eq. (4)). Vanilla MCTS with a fixed exploration constant remains near 4.3 throughout. Gemini-3.1-Pro-preview GPT-5.5
DeepSeek-v4-Pro Kimi-K2.6
Beat Ratio (%)
100 80 60 40 20 0
Image
NLP
Audio
Figure 4 | Performance of MLEvolve across different backbone LLMs on representative tasks covering Image, NLP, and Audio domains. Beat ratios are reported per domain, with full per-task scores provided in Appendix. 4.4.2. Performance with Different LLMs We further evaluate MLEvolve with four LLM backbones, including Gemini-3.1-Pro-preview, GPT-5.5, DeepSeek-v4-Pro, and Kimi-K2.6, on representative MLE-Bench tasks covering Image, NLP, and Audio domains. As shown in Figure 4, the four backbones exhibit clearly different domain strengths. For example, GPT-5.5 reaches the highest beat ratio on NLP tasks with 96.2%, while Kimi-K2.6 leads on the evaluated Audio tasks with 99.2%. Despite these per-domain differences, all four LLMs achieve competitive results under the same MLEvolve pipeline, suggesting that the framework is not tightly coupled to a specific LLM backbone. These results show that MLEvolve remains effective across different backbone LLMs and task domains. Full per-task scores are provided in Appendix F.
12
98.2%
100
Average Beat Ratio (%)
80
70.7% 60
40
20
MLEvolve (Ours) Vanilla MCTS 0
0
2
4
6
Time (hours)
8
10
12
Figure 5 | Beat ratio over the 12-hour search budget on representative tasks. MLEvolve converges faster and continues improving in late stages, whereas the baseline plateaus early. 4.4.3. Performance Over Time To examine how performance evolves during the search, Figure 5 reports the beat ratio (the percentage of human Kaggle participants outperformed by the current best submission) as a function of elapsed time. MLEvolve improves rapidly in the early stage and continues to make gains throughout the middle and late stages, reaching a final beat ratio of 98.2% on the representative tasks. In contrast, Vanilla MCTS plateaus much earlier, ending at ∼70% and struggling to further refine solutions once the early promising directions have been explored. This trend indicates that MLEvolve can sustain improvement over a longer search horizon, supporting the effectiveness of the self-evolving design.
5. Conclusion In this work, we present MLEvolve, an LLM-based self-evolving multi-agent framework for long-horizon MLE tasks. By integrating Progressive MCGS, Retrospective Memory, and Hierarchical Planning with Adaptive Code Generation, MLEvolve enables adaptive search, sustained experience accumulation, and flexible code generation within a unified optimization process. Experiments show that MLEvolve achieves state-of-the-art performance on MLE-Bench, attaining a 65.3% average medal rate under a 12-hour budget and outperforming all existing baselines. Ablation studies verify the effectiveness of each component. Results on AlphaEvolve mathematical optimization tasks further show that MLEvolve generalizes beyond MLE to broader algorithmic optimization problems. In future work, we will extend MLEvolve to more general AI for Science scenarios, including automated scientific experimentation, cross-disciplinary algorithm discovery, and autonomous research workflows.
References [1] Richard Van Noorden and Jeffrey M Perkel. “AI and science: what 1,600 researchers think”. In: Nature 621.7980 (2023), pp. 672–675. [2] Shangheng Du et al. “A survey on the optimization of large language model-based agents”. In: ACM Computing Surveys 58.9 (2026), pp. 1–37.
13
[3] Chris Lu et al. “Towards end-to-end automation of AI research”. In: Nature 651.8107 (2026), pp. 914–919. [4] NovelSeek Team et al. “NovelSeek: When Agent Becomes the Scientist–Building Closed-Loop System from Hypothesis to Verification”. In: arXiv preprint arXiv:2505.16938 (2025). [5] Shiyang Feng et al. “Internagent-1.5: A unified agentic framework for long-horizon autonomous scientific discovery”. In: arXiv preprint arXiv:2602.08990 (2026). [6] Alexander Novikov et al. “Alphaevolve: A coding agent for scientific and algorithmic discovery”. In: arXiv preprint arXiv:2506.13131 (2025). [7] Saleema Amershi et al. “Software engineering for machine learning: A case study”. In: 2019 IEEE/ACM 41st International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP). IEEE. 2019, pp. 291–300. [8] Xin He, Kaiyong Zhao, and Xiaowen Chu. “AutoML: A survey of the state-of-the-art”. In: Knowledge-based systems 212 (2021), p. 106622. [9] Matthias Feurer et al. “Auto-sklearn 2.0: Hands-free automl via meta-learning”. In: Journal of Machine Learning Research 23.261 (2022), pp. 1–61. [10]
Xingyao Wang et al. “Openhands: An open platform for ai software developers as generalist agents”. In: International Conference on Learning Representations. Vol. 2025. 2025, pp. 65882– 65919.
[11]
Qian Huang et al. “Mlagentbench: Evaluating language agents on machine learning experimentation”. In: arXiv preprint arXiv:2310.03302 (2023).
[12]
Zhengyao Jiang et al. “Aide: Ai-driven exploration in the space of code”. In: arXiv preprint arXiv:2502.13138 (2025).
[13]
Xu Yang et al. “R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science”. In: arXiv preprint arXiv:2505.14738 (2025).
[14]
Zexi Liu et al. “ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoning”. In: arXiv preprint arXiv:2506.16499 (2025).
[15]
Annan Li et al. “The fm agent”. In: arXiv preprint arXiv:2510.26144 (2025).
[16]
Edan Toledo et al. “AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench”. In: arXiv preprint arXiv:2507.02554 (2025).
[17]
Yixin Ou et al. “AutoMind: Adaptive Knowledgeable Agent for Automated Data Science”. In: arXiv preprint arXiv:2506.10974 (2025).
[18]
Jiefeng Chen et al. “MARS: Modular Agent with Reflective Search for Automated AI Research”. In: arXiv preprint arXiv:2602.02660 (2026).
[19]
Xinyu Zhu et al. “Toward ultra-long-horizon agentic science: Cognitive accumulation for machine learning engineering”. In: arXiv preprint arXiv:2601.10402 (2026).
[20]
Shangheng Du et al. “AutoMLGen: Navigating Fine-Grained Optimization for Coding Agents”. In: arXiv preprint arXiv:2510.08511 (2025).
[21]
Bogdan Georgiev et al. “Mathematical exploration and discovery at scale”. In: arXiv preprint arXiv:2511.02864 (2025).
[22]
Jaehyun Nam et al. “MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement”. In: arXiv preprint arXiv:2506.15692 (2025).
[23]
Haoyang Fang et al. “Mlzero: A multi-agent system for end-to-end machine learning automation”. In: arXiv preprint arXiv:2505.13941 (2025). 14
[24]
Jun Shern Chan et al. “MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering”. In: The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. 2025.
[25]
Ruiyi Zhang et al. “AIBuildAI: An AI Agent for Automatically Building AI Models”. In: arXiv preprint arXiv:2604.14455 (2026).
[26]
Alireza Nadafian, Alireza Mohammadshahi, and Majid Yazdani. “KAPSO: A Knowledgegrounded framework for Autonomous Program Synthesis and Optimization”. In: arXiv preprint arXiv:2601.21526 (2026).
[27]
Johannes Czech, Patrick Korus, and Kristian Kersting. “Monte-Carlo graph search for AlphaZero”. In: arXiv preprint arXiv:2012.11045 (2020).
[28]
Edouard Leurent and Odalric-Ambrym Maillard. “Monte-carlo graph search: the value of merging similar states”. In: Asian Conference on Machine Learning. PMLR. 2020, pp. 577–592.
[29]
Zhaoling Chen et al. “Locagent: Graph-guided llm agents for code localization”. In: Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025, pp. 8697–8727.
[30]
Xiangyan Liu et al. “Codexgraph: Bridging large language models and code repositories via code graph databases”. In: Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2025, pp. 142–160.
[31]
Zeyu Zhang et al. “A survey on the memory mechanism of large language model-based agents”. In: ACM Transactions on Information Systems 43.6 (2025), pp. 1–47.
[32]
Wujiang Xu et al. “A-mem: Agentic memory for llm agents”. In: Advances in Neural Information Processing Systems 38 (2026), pp. 17577–17604.
[33]
Yifei Zhang et al. “Reasoning as Gradient: Scaling MLE Agents Beyond Tree Search”. In: arXiv preprint arXiv:2603.01692 (2026).
[34]
Edwin T Jaynes. “Information theory and statistical mechanics”. In: Physical review 106.4 (1957), p. 620.
[35]
Jeff Johnson, Matthijs Douze, and Hervé Jégou. “Billion-scale similarity search with GPUs”. In: IEEE transactions on big data 7.3 (2019), pp. 535–547.
[36]
Haotian Ye et al. “Evaluation-driven Scaling for Scientific Discovery”. In: arXiv preprint arXiv:2604.19341 (2026).
[37]
Mert Yuksekgonul et al. “Learning to discover at test time”. In: arXiv preprint arXiv:2601.16175 (2026).
[38]
Asankhaya Sharma. OpenEvolve: an open-source evolutionary coding agent. 2025. url: https:
//github.com/algorithmicsuperintelligence/openevolve.
15
Appendix A. Agent Descriptions MLEvolve is realized through a team of specialized agents, each tailored to a specific search phase or operator type. We summarize their roles: • Draft Agent. Generates initial candidate solutions at the root node, which can retrieve model priors from the cold-start knowledge base (§3.3.1). • Improve Agent. Iteratively refines a runnable solution. It usually obtains structured guidance from the planner and applies controlled revisions through the Diff mode. • Debug Agent. This agent is triggered only when execution fails. It repairs faulty solutions based on error traces (e.g., missing dependencies, tensor shape mismatches), applying minimal modifications until the issue is fixed or the retry limit is reached. • Evolution Agent. Corresponds to intra-branch evolution by aggregating recent consecutive nodes along the same branch. It extracts experience from the past trajectory and uses it to propose targeted refinements for the current solution. • Fusion Agent. Performs cross-branch reference when a branch stagnates. It aggregates strong solutions from other branches as additional references, supplying reusable strategies for the current solution. • Aggregation Agent. Triggered by global stagnation, it aggregates top trajectories from multiple branches to create a new branch starting point. • Code Review Agent. After each code generation step, it reviews the generated code for naming or import errors, suspicious patterns, and metric consistency before execution. • Data Leakage Agent. Checks for potential leakage between training and evaluation splits to prevent overfitting to evaluation artifacts and avoid inflated scores. • Result Parse Agent. Parses execution logs to extract task-specific metrics, execution status, and key insights, and transfers structured information back into the search loop.
B. Expansion Type Formulations In §3.2.2 we introduce a unified expansion rule 𝑣new = 𝑔𝑜 ( 𝑣𝑡 , 𝑅) ,
( 𝑣𝑡 , 𝑣new ) ∈ 𝐸𝑇 , {( 𝑟, 𝑣new ) | 𝑟 ∈ 𝑅 } ⊆ 𝐸ref ,
parametrized by the reference set 𝑅. This appendix gives the precise instantiation of 𝑅 for each of the four expansion types. (1) Primary expansion (𝑅 = ∅). The new node is generated solely from its parent, without referencing other nodes: 𝑣new = 𝑔𝑜 ( 𝑣𝑡 , ∅) , ( 𝑣𝑡 , 𝑣new ) ∈ 𝐸𝑇 . (12) This is the baseline expansion against which the graph-based variants extend, and corresponds to operators such as Draft, Improve, and Debug. (2) Intra-branch evolution ( 𝑅 = R hist ( 𝑣𝑡 , 𝑘)). The reference set consists of the nearest 𝑘 ancestor nodes of 𝑣𝑡 within the same branch, forming a local trajectory: 𝑣new = 𝑔𝑜 ( 𝑣𝑡 , R hist ( 𝑣𝑡 , 𝑘)) ,
( 𝑣𝑡 , 𝑣new ) ∈ 𝐸𝑇 , {( 𝑟, 𝑣new ) | 𝑟 ∈ R hist ( 𝑣𝑡 , 𝑘)} ⊆ 𝐸ref .
(13)
The primary edge preserves the parent–child relation, while the reference edges record information flow from intra-branch history. Selection and backpropagation are conducted exclusively along 𝐸𝑇 . 16
(3) Cross-branch reference (𝑅 = R cross ( 𝑁 )). When the current branch stagnates, the system constructs a reference set from the top- 𝑁 nodes selected across evaluated branches according to their performance: 𝑣new = 𝑔𝑜 ( 𝑣𝑡 , R cross ( 𝑁 )) ,
( 𝑣𝑡 , 𝑣new ) ∈ 𝐸𝑇 , {( 𝑟, 𝑣new ) | 𝑟 ∈ R cross ( 𝑁 )} ⊆ 𝐸ref .
(14)
These reference edges allow the new node to reuse effective designs discovered in other branches, providing external guidance for improving the current solution. (4) Multi-branch aggregation ( 𝑅 = R agg ). Triggered by global stagnation, this operator creates a top new branch beneath the root 𝑣0 by aggregating top trajectories from multiple branches. Let T𝑏 Ð top denote the best-performing trajectories in branch 𝑏 ∈ B; then R agg = 𝑏 ∈ B T𝑏 , and the expansion is: 𝑣new = 𝑔𝑜 ( 𝑣0 , R agg ) ,
( 𝑣0 , 𝑣new ) ∈ 𝐸𝑇 , {( 𝑢, 𝑣new ) | 𝑢 ∈ R agg } ⊆ 𝐸ref .
(15)
Unlike incremental refinement along a single branch, aggregation pools information from multiple branches and opens an independent exploration trajectory.
C. MLE-Bench Benchmark and Evaluation Metrics C.1. MLE-Bench We evaluate MLEvolve on MLE-Bench [24], a benchmark introduced by OpenAI for assessing autonomous machine learning engineering. MLE-Bench comprises 75 carefully curated Kaggle competitions spanning natural language processing, computer vision, signal processing, and tabular data analysis. These competitions are selected from 586 candidates through manual screening by ML engineers, ensuring each task represents authentic and challenging ML engineering work. The dataset includes competitions of varying complexity: 22 low-complexity tasks (solvable by experienced engineers in under 2 hours), 38 medium-complexity tasks (2–10 hours), and 15 high-complexity tasks (over 10 hours), covering 15 distinct problem categories. Each competition includes the original problem description, datasets with reconstructed train-test splits, local grading code, and human baseline performance from Kaggle leaderboards. This setup enables direct comparison between AI agents and human competitors while maintaining evaluation integrity. The benchmark employs medal achievement rates as the primary metric, where agents must reach bronze, silver, or gold medal thresholds based on their performance relative to human participants. Agents must work autonomously within time constraints (24-hour time limit) to produce valid submission files.
C.2. Evaluation Metrics We use the following metrics to evaluate performance on MLE-Bench. All thresholds and percentile data are officially provided by Kaggle and MLE-Bench. • Medal Rate (All, in %): the percentage of tasks on which the submission earns a medal (gold, silver, or bronze). We additionally report the medal rate stratified by task complexity (Low / Medium / High). • Gold Medal Rate (Gold, in %): the percentage of tasks on which the submission earns a gold medal. • Valid Submission Rate (Valid, in %): the percentage of tasks that produce a valid submission passing format and correctness checks.
17
• Above Median Rate (Med+, in %): the percentage of tasks on which the submission beats half of the human competitors. • Beat Ratio (in %): the average percentage of human competitors whose performance is surpassed by the agent’s submission.
D. Hyperparameters Table 4 lists the key hyperparameters used in all MLEvolve experiments. Values are kept fixed across all 75 MLE-Bench tasks unless otherwise stated. Table 4 | Default hyperparameter configuration of MLEvolve. Hyperparameter
Description
Default
General Search steps time_limit parallel_search_num initial_drafts max_drafts max_fusion_drafts temperature
Max search steps Total time limit per task Parallel branches Initial drafts Max branches from primary expansion Max additional branches from aggregation LLM decoding temperature
500 12 h 3 3 5 2 1.0
Progressive MCGS √ exploration_constant lower_bound phase_ratios explore_switch_start explore_switch_end min_exploration_weight elite_topk
UCT exploration constant 𝑐0 UCT lower bound 𝑐min UCT decay phase ratios Soft-switch start time ratio Soft-switch end time ratio Minimum exploration weight 𝑤min Top- 𝐾 candidates in elite-guided exploitation
2 0.5 (0.3, 0.7) 0.5 0.7 0.2 3
Stagnation Detection branch_stagnation_threshold topk_stagnation_threshold
Branch-level stagnation 𝜏branch Global-level stagnation 𝜏global
3 6
memory_similarity_threshold Retrieval similarity threshold memory_embedding_model Sentence embedding model
0.7 BGE-base-en-v1.5
Retrospective Memory
E. Detailed Component Analysis To complement the component-level ablation in §4.3, we further isolate individual mechanisms within Progressive MCGS and Retrospective Memory on a 9-task subset of MLE-Bench, disabling one mechanism at a time while keeping all others unchanged. Among the internal mechanisms of Progressive MCGS, removing intra-branch evolution causes the largest drop, reducing the medal rate from 66.67% to 33.33%. This indicates that reusing recent branch history is critical for preventing the agent from repeatedly making similar mistakes across iterations. Removing cross-branch reference and Elite-Guided exploitation leads to milder medal-rate decreases, but they affect different aspects of performance. Cross-branch reference provides external guidance from high-performing solutions discovered in other branches, helping the agent escape 18
Table 5 | Detailed component analysis on 9 representative tasks. Each row disables one mechanism within Progressive MCGS or Retrospective Memory while keeping all others unchanged. Configuration
Medal (%)
Beat Ratio (%)
MLEvolve
66.67
82.43
Progressive MCGS w/o Evolution w/o Cross-branch w/o Elite-Guided
33.33 55.56 55.56
74.95 75.93 71.39
Retrospective Memory w/o Knowledge Base w/o Global Memory
44.44 44.44
76.07 73.58
Table 6 | Score comparison of MLEvolve across four LLM backbones on 8 representative MLE-Bench tasks. Best result for each task is highlighted in bold. Task
Metric
Gemini-3.1-Pro-preview
GPT-5.5
DeepSeek-v4-Pro
Kimi-K2.6
Accuracy ↑ AUC ↑ AUC ↑
0.8984 0.9638 0.9252
0.8999 0.9568 0.9045
0.9032 0.8375 0.8662
0.8905 0.8880 0.9294
Jaccard ↑ Logloss ↓ AUC ↑
0.7136 0.2175 0.6992
0.7216 0.2324 0.7888
0.7113 0.2298 0.7782
0.7195 0.2305 0.7649
AUC ↑ AUC ↑
0.9947 0.9486
0.9947 0.9274
0.9938 0.9490
0.9934 0.9363
Image Tasks cassava-leaf-disease-classification ranzcr-clip-catheter-line-classification siim-isic-melanoma-classification NLP Tasks tweet-sentiment-extraction spooky-author-identification random-acts-of-pizza Audio Tasks the-icml-2013-whale-challenge-right-whale-redux mlsp-2013-birds
stagnant local trajectories. Elite-Guided exploitation mainly improves leaderboard ranking by further refining already competitive solutions toward higher-performing ones, as reflected by the lowest beat ratio after its removal. For the memory system, removing either the Knowledge Base or Global Memory reduces the medal rate to 44.44%, showing that both sources of experience contribute to performance. However, removing Global Memory leads to a lower beat ratio than removing the Knowledge Base, suggesting that dynamically accumulated experience has a stronger impact on overall solution quality during long-horizon search. The Knowledge Base mainly provides task-relevant priors for cold-start initialization, while Global Memory continuously accumulates and reuses task-specific experience throughout the search process.
F. Detailed Results with Different LLMs To provide a detailed view of per-task performance across different LLM backbones, we report the scores of Gemini-3.1-Pro-preview, GPT-5.5, DeepSeek-v4-Pro, and Kimi-K2.6 on 8 representative MLE-Bench tasks covering Image, NLP, and Audio domains. As shown in Table 6, each model has its own strengths across different tasks and domains, with no single model dominating all tasks. All four backbones produce competitive results under the same MLEvolve pipeline, confirming that the framework is not tied to a specific LLM.
19
G. Case Study We present representative cases illustrating the three graph-based expansion operators. Each case is drawn from an actual run on MLE-Bench.
G.1. Intra-branch Evolution Figure 6 shows an intra-branch evolution case on the aptos2019-blindness-detection task. After six successive draft and improvement steps, the branch reaches a plateau where prior regularization attempts (EMA, Mixup, cross-validation) all failed to improve the score. The Evolution Agent reviews the local trajectory, identifies the bottleneck as architectural, and proposes fusing a DINOv3 backbone with a ResNet50. The coder implements the change via diff-mode edits.
G.2. Cross-branch Reference Figure 7 shows a cross-branch reference case on the mlsp-2013-birds task. When the current branch stagnates with a symmetric Focal Loss, the Fusion Agent identifies an alternative loss design (Asymmetric Loss) from a strong solution in another branch. The coder replaces FocalLoss with AsymmetricLoss via diff-mode edits.
G.3. Multi-branch Aggregation Figure 8 shows a multi-branch aggregation case on the same mlsp-2013-birds task. After global stagnation is detected, the Aggregation Agent synthesizes successful components from multiple branches (EfficientNet-B1 with GeM pooling, bandpass filter, Multi-Label Focal Loss, 5-fold crossvalidation) into a new branch starting point.
20
### Case of Intra-branch Evolution "Task": "aptos2019-blindness-detection” “Planner”: “Your Past Evolution trajectory:
Step 1: Branch ID: 4 Stage: DRAFT Design: I have successfully integrated the three steps into a unified Python script. The resulting code first preprocesses the training and test images in parallel—applying Minkowski Norm color constancy and cropping uninformative borders—and caches them as `.npy` arrays. It then constructs the `DINOv3SquaredEMD` model with the exact prescribed DINOv3 backbone and trains it for 10 epochs using Automatic Mixed Precision (AMP) and the custom Squared EMD Loss. ... Step2 ... Step 6 I will analyze this trajectory to identify successful patterns and failed approaches, then use these insights to guide my improvement plan. { "reason": "Applying a Tier 3 Systemic Paradigm Shift based on trajectory analysis: Previous regularization attempts (EMA, Mixup, CV) failed to improve the single-fold peak, indicating the model is bounded by its architectural inductive bias rather than optimization noise. ViTs process 16x16 patches which can dilute microscopic retinal lesions. Fusing the DINOv3 backbone with a CNN explicitly injects missing high-frequency, translation-invariant local features, breaking the representation plateau.", "module": ["model_design"], "plan": { “model_design”: “WHAT: Implement a Hybrid ViT-CNN architecture (Tier 3 change) by integrating a ResNet50 backbone alongside DINOv3. WHY: ViT models lack the local inductive bias required to optimally detect tiny microaneurysms, which CNNs naturally capture. A hybrid model synergizes global context and local precision. HOW: In `DINOv3SquaredEMD`, `import timm` and add `self.cnn = timm.create_model(‘resnet50’, pretrained=True, num_classes=0)`. In `forward`, extract `cnn_feat = self.cnn(x)` (2048 dims) and concatenate it with the DINOv3 features (`cls_token` + `gem_pool`) along dim=1. Update the classification head's `LayerNorm` and first `Linear` layer to accept 4096 dimensions. In `build_model_and_optimizer`, append `{'params': model.cnn.parameters(), 'lr': 3e-5}` to the AdamW optimizer parameter groups. INTERFACE constraints: Maintain the class name `DINOv3SquaredEMD`, frozen backbone logic, and the exact input/output signatures."} }”
“Coder”: An elegant way to overcome the representations plateau caused by the pure ViT architecture's lack of local
inductive bias is to introduce a CNN model acting alongside it. This Hybrid ViT-CNN approach complements the global receptive field of the vision transformer with the high-frequency and translation-invariant local processing capability of a convolutional neural network (like ResNet), ensuring small structures—like microaneurysms—are explicitly preserved and detected. Here is the implementation of the specified improvement plan. ### Explaining the Enhancements 1. **Import `timm`**: Add the package required to load the ResNet backbone. 2. **Hybrid Architecture Instantiation**: Within `DINOv3SquaredEMD.__init__`, inject `self.cnn = timm.create_model('resnet50', pretrained=True, num_classes=0)` to create the ResNet50 backbone. 3. **Head Adaptation**: Adjust the `LayerNorm` and the first `Linear` layer inside `self.head` to accept `4096` dimensions since DINOv3 provides `2048` dimensions (cls_token + gem_pool) and ResNet50 yields `2048` dimensions. 4. **Feature Fusion**: In the `forward` method, pass the image `x` through `self.cnn`, extract `cnn_feat`, and concatenate it with the ViT features alongside the channel dimension. 5. **Optimizer Update**: Finally, register `model.cnn.parameters()` with a learning rate of `3e-5` to the `AdamW` optimizer inside `build_model_and_optimizer()` to ensure the convolutional pathway updates efficiently.
<<<<<<< SEARCH import scipy.optimize class OptimizedRounder: ======= import scipy.optimize import timm class OptimizedRounder: >>>>>>> REPLACE
<<<<<<< SEARCH for param in block.parameters(): param.requires_grad = False self.head = nn.Sequential(...) ======= for param in block.parameters(): param.requires_grad = False
...
self.cnn = timm.create_model('resnet50', pretrained=True, num_classes=0) self.head = nn.Sequential(...) >>>>>>> REPLACE
Figure 6 | An intra-branch evolution case on the aptos2019-blindness-detection task. 21
### Case of Cross-branch Reference "Task": "mlsp-2013-birds” “Planner”: “I'll compare the reference solutions and selectively incorporate the best ideas. Now I will output my analysis in
JSON format { "reason": "The current Focal Loss formulation symmetrically scales positive and negative errors, which is suboptimal for extreme multi-label sparsity where negative examples overwhelmingly dominate. By transitioning to Asymmetric Loss (ASL), we can decouple the focusing parameters and aggressively zero out easy negatives. Updating both model_design and training_evaluation ensures this powerful multi-label loss is properly implemented and utilized.", “module”: [“model_design”, "training_evaluation"], "plan": { "model_design": "CHANGES (list ALL modifications, one or multiple): Change #1: [Category: Loss Function] - What: Replace the FocalLoss class with an AsymmetricLoss class using gamma_neg=4, gamma_pos=1, clip=0.05. - Why: ASL independently tunes the decay of positive and negative gradients, applying hard clipping to easy negatives to punish background dominance without degrading sparse positive bird calls. - Source: Reference Solution A. WHY current solution limited (and which reference addressed it best): - Root cause: Focal Loss treats probability decay symmetrically, allowing the massive sum of easy negatives to overwhelm rare positive signals. - Evidence: Models using ASL in extreme multi-label audio tasks consistently outperform those using BCE or Focal Loss. - Best reference: Reference Solution A. HOW selected techniques apply to MY solution: - Mechanism: By defining AsymmetricLoss, the loss from highly confident negatives is aggressively diminished. - Compatibility: Fits as a drop-in multi-label loss module replacing FocalLoss. - Expected improvement: Better handling of the 19-class imbalance, boosting AUC. KEEP UNCHANGED (must explicitly list): - Random seed: 42 - Data split: 5-fold CV - Core architecture: HybridPoolBirdModel", "training_evaluation": "CHANGES (list ALL modifications, one or multiple): Change #1: [Category: Training Strategy] What: Instantiate criterion = AsymmetricLoss() instead of FocalLoss() in the fold setup. - Why: To apply the decoupled gamma penalties defined in the new model design during the training loop. - Source: Reference Solution A. WHY current solution limited (and which reference addressed it best): - Root cause: The existing loop is hardcoded to FocalLoss, preventing the utilization of asymmetric penalization. - Evidence: Reference A successfully integrates ASL directly into the PyTorch loop for audio multi-label tasks. - Best reference: Reference Solution A. HOW selected techniques apply to MY solution: - Mechanism: Simply replaces the criterion assignment so the gradients are computed asymmetrically. - Compatibility: AsymmetricLoss accepts the identical inputs signature. - Expected improvement: Allows the model to effectively converge on rare bird species without interface breakage. "} }”
“Coder”: An issue with symmetrically scaling positive and negative errors via standard Focal Loss is that the sheer volume of negative examples in highly imbalanced, multi-label audio tasks can overwhelm the sparse but critical positive signals from bird calls. To address this, we replace the existing `FocalLoss` with `AsymmetricLoss` (ASL). ASL explicitly decouples the focusing parameters (`gamma_pos` vs `gamma_neg`), applying an aggressive decay specifically to the numerous easy negatives without degrading the rare positive signals. Furthermore, by defining a `clip` value, we entirely nullify gradients from high-confidence negatives, effectively prioritizing the model's focus on learning challenging patterns and sparse labels. Here are the code modifications to implement this plan:
<<<<<<< SEARCH class FocalLoss(nn.Module): def forward(self, inputs, targets): bce = F.binary_cross_entropy_with_logits(inputs, targets, reduction='none’) pt = torch.exp(-bce) return ((1 - pt)**2.0 * bce).mean() <<<<<<< SEARCH ======= scheduler = class AsymmetricLoss(nn.Module): def __init__(self, gamma_neg=4.0, gamma_pos=1.0, clip=0.05, eps=1e-8): torch.optim.lr_scheduler.CosineAn nealingLR(optimizer, T_max=40) super().__init__() criterion = FocalLoss() self.gamma_neg = gamma_neg ======= self.gamma_pos = gamma_pos scheduler = self.clip = clip torch.optim.lr_scheduler.CosineAn self.eps = eps nealingLR(optimizer, T_max=40) def forward(self, inputs, targets): criterion = AsymmetricLoss() probs = torch.sigmoid(inputs) >>>>>>> REPLACE probs_pos = probs probs_neg = 1.0 – probs ... loss_pos = loss_pos * torch.pow(1.0 - probs_pos, self.gamma_pos) loss_neg = loss_neg * torch.pow(1.0 - probs_neg, self.gamma_neg) return (loss_pos + loss_neg).mean() >>>>>>> REPLACE
Figure 7 | A cross-branch reference case on the mlsp-2013-birds task. 22
### Case of Multi-branch Aggregation "Task": "mlsp-2013-birds” “Thinking and plan”: “
I have access to multiple successful approaches from different independent branches. I'll synthesize these diverse insights and create a completely new solution that combines the best ideas in an innovative way. This synthesized solution integrates the most powerful strategies across different branches to create an exceptionally robust audio classification pipeline. ### Key Innovations Synthesized: 1. **Dynamic Spatial Representation (Branch 5 & 3)**: Utilizing `efficientnet_b1` and completely replacing standard Global Average Pooling with a custom `GeMPooling` (Generalized Mean Pooling) module. This parametrically bridges max and average pooling, perfectly suited for preserving peak, sparse acoustic transient events (bird calls) without discarding contextual acoustic background. 2. **Frequency-Targeted Bandpass Filter (Branch 6)**: During `librosa` feature extraction, we explicitly bind `fmin=500` and `fmax=8000`. Heavy environmental noise such as wind, rain, and vehicles natively resides in lower frequencies, so filtering focuses the Log-Mel Spectrogram entirely on the target frequency range of bird vocalizations. 3. **Multi-Label Imbalance Handling (Branch 6)**: Transitioning from vanilla BCE to a custom Multi-Label `Focal Loss` prevents the optimizer from being swamped by simple 'absent' negative predictions across the 19 species. 4. **Resilient Ensembling & Safety Protocols (Branch 1 & 4)**: Enforcing a robust 5-Fold Cross-Validation strategy dynamically scales each fold individually to prevent data leakage, yielding reliable Out-Of-Fold (OOF) predictions. Robust fail-safes (`np.nan_to_num`) aggressively sterilize extreme values prior to any scaling operation to ensure stable convergence across all folds. }”
“Code”: ```python
import os, numpy as np, pandas as pd, librosa, torch, torch.nn as nn import timm, torchaudio from sklearn.model_selection import KFold from sklearn.metrics import roc_auc_score # 1. Load CV folds, Labels, and Filenames labels_dict, train_rec_ids, test_rec_ids, rec_id2filename = {...}, [...], [...], {...}... ... # 2. Extract Bandpass-Filtered Log-Mel Spectrograms def extract_log_mel(wav_path): y, sr = librosa.load(wav_path, sr=16000) y = y[:160000] if len(y) >= 160000 else np.pad(y, (0, 160000 - len(y))) S = librosa.feature.melspectrogram(y, sr, n_fft=1024, hop_length=320, n_mels=128, fmin=500, fmax=8000) return np.expand_dims(librosa.power_to_db(S, ref=np.max), 0) X_train, Y_train = [...] # 19-class multi-hot X_test = {rec_id: extract_log_mel(...) for rec_id in test_rec_ids} ... # 3. Custom Dataset with SpecAugment and Sterilization class BirdDataset(Dataset): def __getitem__(self, idx): x = torch.tensor(np.nan_to_num((self.X[idx]-self.mean)/self.std)) if self.is_train: x = self.freq_mask(self.time_mask(x)) return x, torch.tensor(self.Y[idx]) ... # 4. Neural Network Design (EfficientNet-B1 + GeM Pooling + Focal Loss) class BirdModel(nn.Module): def forward(self, x): return self.fc(self.pool(self.backbone.forward_features(x))) # 5. Model Training Pipeline & 5-Fold Evaluator for fold, (tr_idx, va_idx) in enumerate(KFold(5, shuffle=True, rs=42).split(X_train)): model = BirdModel().cuda() best_state = train_fold(..., mixup=True, focal_loss=True, patience=10) oof_preds[va_idx], test_preds[fold] = predict(val), predict(test) ... # 6. Ensemble Submissions Generation sub_df['Probability'] = map_ids(test_preds.mean(0)) sub_df.to_csv(‘./submission/submission.csv’, index=False) ... ```
Figure 8 | A multi-branch aggregation case on the mlsp-2013-birds task. 23