Official Review · ICLR 2025

This paper introduces GFMBench, a framework for standardizing and automating benchmarking of Genomic Foundation Models (GFMs), which integrates four large-scale benchmarks containing 42 million genomic sequences across 75 datasets. The framework addresses several challenges in genomic modeling by providing unified protocols, consistent metrics, and automated evaluation pipelines that work across both DNA and RNA tasks from multiple species. GFMBench is released as open-source software with user-friendly interfaces, built-in tools for genomic tasks, and includes a public leaderboard and online hub for sharing benchmark results and resources with the research community. The paper integrates 4 large genomics benchmarks from other groups into a unified framework. They also create a way to run a pipeline with these benchmarks with GFMs. The paper exhibits several significant limitations. First, the technical innovation appears limited in scope. While the work integrates multiple existing benchmarks into a unified framework, it does not create new benchmarking methodologies or metrics. The automation and standardization approaches described largely follow established machine learning practices, without introducing fundamental methodological advances in benchmarking or evaluation techniques. The work's focus is predominantly engineering-oriented rather than research-driven. The main contributions center on developing practical tools and interfaces for the genomics community. While valuable from a practical perspective, the emphasis on standardization over novel scientific insights raises questions about its suitability for a top-tier machine learning conference. Furthermore, similar benchmark aggregation platforms already exist in other machine learning domains, making this approach less innovative. What specific technical innovations in benchmarking does GFMBench introduce? What novel metrics or evaluation approaches are introduced? What specific reproducibility issues does it solve that other frameworks don't? Why were these 4 benchmarks and then GFMs selected among others? In Table 4, why is GFMBench listed as a model?