Official Review · ICLR 2025

The authors present MLE-bench, a benchmark designed to evaluate AI agents’ capabilities in machine learning engineering. This benchmark is constructed from publicly available Kaggle challenges, with AI agent performance assessed based on the percentage achieving medal-level scores comparable to real human submissions. The paper releases the benchmark’s data and code, includes three open-source agents as baselines, and evaluates state-of-the-art foundational models using the AIDE agent. 1. Given recent advancements in AI agents for coding, engineering, and scientific discovery, the proposed benchmark serves as a highly valuable testbed for evaluating foundational models and agents in real-world machine learning engineering contexts. 1. The paper is clearly written and easy to follow, providing necessary details on both the dataset and technical implementation. 1. The study includes solid empirical evaluations across different agents, foundational models, scaling factors, and contamination issues, offering valuable insights into AI agents’ current capabilities and limitations in the field. 1. The authors provide a thorough discussion of the benchmark’s limitations and ethical considerations. ### Accessibility of the Benchmark As discussed by the authors in Section 6 L505, the benchmark is costly to run. Based on my estimates, using the authors’ cost descriptions and current rates on OpenAI and Lambda Cloud, a single full evaluation with AIDE and o1-preview would require approximately 4,000 USD (around 2,600 USD for API queries and 1,300 USD for an A10 GPU server). Considering the costs for development and extensive experiments would be substantially higher than a single evaluation, this benchmark may be inaccessible to small to medium-scale academic labs. The authors may consider providing a lighter dataset split to improve accessibility, similar to the approach in SWE-Bench. ### Evaluation Reliability and Contamination One particular challenge in benchmarks for AI agents is establishing reliable detection for potential cheating behaviors (e.g., hacking the evaluation function or accessing private test data). I appreciate the authors for addressing this with a tool designed to flag such behaviors. However, Table 6 in Appendix A.3 indicates a high false positive rate, which may hinder practical reliability. The rate is sufficiently high that manually checking all flagged submissions would be demanding. Additionally, while the authors provide a thorough discussion and empirical analysis of contamination issues, this remains a critical limitation for this and other similar benchmarks. For instance, Figure 5 shows GPT-4o familiarity scores above 0.4 across all problems. Does this suggest that these problems are included in the model’s training set? Also, the conclusions drawn from the correlation between familiarity and performance could be significantly impacted by confounders, such as problem difficulty. Furthermore, while the obfuscated datas