Official Review · ICLR 2025

This paper proposes a set of five benchmarking tasks for genomic language models, with a focus on tasks that involve modeling long-range DNA dependencies, including enhance-target gene interaction and 3D genome organization. These tasks represent a diverse set of biologically important characteristics. The authors benchmark five baseline methods on each task - task-specific expert models, CNN-based models, and three recent DNA language models, and find that the DNA language models they benchmark do not outperform the task-specific models. - The tasks included in the benchmark are biologically significant and diverse, especially the focus on tasks with different dimensionalities (1D or 2D) and granularities (binned or base pair level). In addition, the paper does a good job explaining the biological significance of each task. Both the increased focus on long-range tasks and base pair resolution tasks represent novel aspects of this benchmark compared to other previously published benchmarks. - The results are presented in a clear and concise manner - The performance of the three evaluated DNA language models — HyenaDNA, Caduceus-Ph, and Caduceus-PS — on these tasks reveal important limitations of current DNA language models at modeling long range dependencies, even when the models are able to incorporate long context lengths. - In my opinion, the main weakness of this paper is novelty. Although the benchmarking tasks are important and biologically motivated, all five of the tasks and corresponding datasets have already been used to benchmark models in previous publications and in some previous benchmarks, such as BEND (Marin, et al. 2023) and LRB (Kao, et al. 2024). The “Regulatory sequence activity” task directly uses the training, validation, and test set sequences from Enformer. Therefore, the main contribution of this paper is consolidating these tasks into one resource and comparing the performance of DNA language models and more traditional supervised approaches. One potential way to increase the novelty and contribution of this work could be to incorporate more datasets that weren’t used in previous publications to increase the amount of benchmarking data for each task. For example, for the enhancer-target gene task, the dataset is relatively small, and different experimental datasets often suffer from their own technical and experimental biases. Since a number of similar datasets exist and are publicly available, it would improve and increase the impactfulness of the benchmark to aggregate multiple similar datasets. - The term “Expert model” is misleading. For example, L369-370 refers to “expert models tailored to each task,” but in some cases, such as using Enformer as an expert model for eQTL prediction, the expert model has not actually been tailored to this task. An alternative could be to refer to these models at “State of the art models.” In addition, in some cases the choice of Expert/SOTA model could be revisited. For example, f