Official Review · ICLR 2025

This paper proposes a new benchmarking suite to evaluate the ability of protein and genomic language models to accurately represent cellular function. The authors claim that most existing benchmarks focus on structural properties instead of function, and highlight several challenges with evaluating accurate functional representation. The authors then define six categories of tasks in their benchmarking suite, and aggregate 18 datasets falling into these categories that capture aspects of cellular function. Tasks involve predicting properties of individual sequences as well as pair-wise or multi-sequence interactions. The authors then benchmark several protein and genomic foundation models on the previous tasks, and highlight comparative performance as well as the presence of scaling law behaviour. NOTE: Raised score from 5->6 in response to rebuttals. - Selection of datasets capture meaningful aspects of protein function and phylogeny, and cover a wide range of prediction tasks. Organization of datasets into hierarchy based on prediction task and relationship conceivably would help extension of the dataset by integration of future datasets. - Authors highlight the importance of controlling data leakage that might occur through sequence homology, and construct train / test splits that avoid this using several sequence identity thresholds. This contributes towards the rigour of models that would be benchmarked on DGEM, and raises the current standard in the genomics space. An additional contribution could be to include code to regenerate splits based on a random seed; this would be useful for generating CIs. - Benchmarking of existing foundation models on the selected tasks reveal interesting empirical evidence on the presence of scaling laws and performance gaps between existing AA and NA models. - Although the selected tasks / datasets do represent aspects of biological function, I am concerned that the overall claims of the paper are too large in its current state. Many aspects of biological function that could be represented by a foundation model embedding are not tested for in this dataset, with most of the proposed tasks evaluating enzymatic function or evolutionary similarity between proteins. In particular, attention to tasks specific to NA sequences that could be used to evaluate gLMs are relatively lacking in comparison to protein tasks. For example, the classification of sequence into annotation classes as defined by the Gene Ontology resource is commonly used, or classification of RNA splice isoform function or tissue-specific expression could be a task related to function that evaluates gLMs. I believe the paper should either seek to restrict the breadth of their claims, or incorporate other commonly used measures of biological function into their dataset. - As a benchmarking paper, I believe the authors are well positioned to further increase the relevance of their work by inclusion of baselines that are not derived from a foundat