Official Review · ICLR 2025

This paper introduced a benchmark suite for genomic foundation models(gFMs) called GeneBench that systematically evaluates gFMs on an wide array of datasets across a range of tasks for both short and long range sequence prediction. This work also presented a new method called GenHybrid that leverages both SSM and attention based model architectures. 1. Through and comprehensive dataset curation for gLM evaluation that covers both short and long range tasks. 2. A wide range on gFMs are benchmarked across various model architectures and parameter sizes. 3. The introduction of new hybrid method that leverages both attention based models and state-space models and outperformed existing models in most of the datasets evaluated. 4. In-depth analysis of the benchmarking results and provided insights into the current state of gFMs and their performance differential in various tasks. 1. Lacking on tasks beyond classification and regression. One of the most promising application of gFMs is to predict zero-shot mutation effects and generative modeling of genomic sequences. This benchmark effort misses in both aspects. Many mutation effect databases are available and a comprehensive curation of a benchmarking dataset will be of vast interest to the community. 2. Lacking vertical comparison of different model architectures on model sizes and pre-training schemes. I understand this will be computationally costly but as a benchmarking effort, this is necessary to paint a more complete picture of the model landscape. 3. Missing naive benchmarks and ab initio models for comparison. It has been shown in many recent studies that gFMs do not outperform ab initio models trained on task specific datasets. Adding both ab initio models and native benchmarks will be very important as a benchmark suite. 1. What’s the rationale behind the collection of tasks used in this study? they seem to be very similar in terms of task type and it will be more interesting to see more variation in tasks such as zero-shot mutational effect prediction and generative sequence modeling. 2. The paper is presented as a benchmark suite but the introduction of GenHybrid seems to be the main emphasis throughout the results section. However, the details of such model is missing from the main text of the paper. What’s the main focus of this paper? 3. Why not include ab initio models trained on tasks specific data and naive benchmark in this suite? The numers presented in the paper is without context can hardly can be used as a standardized benchmark for future methods.