arXiv CSopen access

MIMIC: A Generative Multimodal Foundation Model for Biomolecules

MIMIC: A Generative Multimodal Foundation Model for Biomolecules Siavash Golkar†1,2 , Jake Kovalic∗†1,3 , Irina Espejo Morales∗†1,2 , Samuel Sledzieski∗†4,5,9 , Minhuan Li∗†4,5 , Ksenia Sokolova†5,9 , Géraud Krawezik†1,7 , Alberto Bietti†1,4 , Claudia Skok Gibbs2 , Roman Klypa16 , Shengwei Xiong17 , François Lanusse1,10 , Liam Parker1,11 , Kyunghyun Cho1,2 , Miles Cranmer1,12,13 , Tom Hehir1,13 , Michael McCabe1,2 , Lucas Meyer1 , Rudy Morel1,4 , Payel Mukhopadhyay1,12 , Mariel Pettee1,15 , Helen Qu1,6 , Jeff Shen1,18 , David Fouhey1,2 , Hadi Sotoudeh13,14 , Vikram Mulligan5 , Pilar Cossio4,5 , Sonya M. Hanson4,5 , Alisha N. Jones17 , Olga G. Troyanskaya5,8,9,19 , Shirley Ho1,2,6,18,20 1

Polymathic AI, 2 Center for Data Science, New York University, 3 Department of Applied Physics, Yale University, Center for Computational Mathematics, Flatiron Institute, 5 Center for Computational Biology, Flatiron Institute, 6 Center for Computational Astrophysics, Flatiron Institute, 7 Scientific Computing Core, Flatiron Institute, 8 Department of Computer Science, Princeton University, 9 Princeton Precision Health, Princeton University, 10 AIM, Université Paris-Saclay, Université Paris Cité, 11 Department of Physics, University of California, 12 Department of Applied Mathematics and Theoretical Physics, University of Cambridge, 13 Institute of Astronomy, University of Cambridge, 14 Kavli Institute for Cosmology, University of Cambridge, 15 Department of Physics, University of Wisconsin-Madison, 16 Université Grenoble Alpes, CNRS, Grenoble INP, LJK, 17 Department of Chemistry, New York University, 18 Department of Astrophysical Sciences, Princeton University, 19 Lewis-Sigler Institute for Integrative Genomics, Princeton University, 20 Department of Physics, New York University

arXiv:2604.24506v1 [cs.AI] 27 Apr 2026

4

∗

Equal contribution

†

Core contributors

Abstract Biological function emerges from coupled constraints across sequence, structure, regulation, evolution, and cellular context, yet most foundation models in biology are trained within one modality or for a fixed forward task. We present MIMIC, a generative multimodal foundation model trained on our newly curated and aligned dataset, LORE, linking nucleic acid, protein, evolutionary, structural, regulatory, and semantic/contextual modalities within partially observed biomolecular states. MIMIC uses a split-track encoder-decoder architecture to condition on arbitrary subsets of observed modalities and reconstruct or generate missing components of molecular state across the genome, transcriptome, and proteome. Multimodal conditioning consistently improves MIMIC’s sequence reconstruction relative to sequence-only inputs, while its learned representations enable state-of-the-art performance on RNA and protein downstream tasks. MIMIC achieves state-of-the-art splicing prediction, and its joint generative formulation enables isoform-aware inference that further improves performance. Beyond prediction, the same generative framework supports constrained design. For RNA, MIMIC identifies corrective edits in a clinically relevant HBB splice-disrupting mutation without reverting it by using evolutionary and structural signals. For proteins, jointly conditioning on shape and surface chemistry of PD-L1 and hACE2 binding sites produces diverse, high-confidence sequences with strong in silico support for target binding. Finally, MIMIC uses experimental context as semantic conditioning to model assay-dependent RNA chemical probing, rather than treating context as a fixed output. Together, these results position MIMIC’s aligned multimodal generative modeling as a strong foundation for unifying representation learning, conditional prediction, and constrained biomolecular design within a single model.

Corresponding authors: Siavash Golkar, [email protected]; Olga Troyanskaya, [email protected]; Shirley Ho, [email protected]

1

1

Introduction

The central dogma of molecular biology is often visualized as a linear, deterministic flow of information [1]. However, the biological reality is a probabilistic web of feedback loops where downstream functions constrain upstream sequences, and environmental contexts dictate phenotypic outcomes. The function of a gene is not solely encoded in its coding sequence and promoter but is modulated by a shifting landscape of epigenetics and trans-acting factors [2–4]; similarly, while the amino acid sequence of a protein carries much of the information needed to specify its fold [5] both folding and function are realized within a crowded cellular milieu shaped by chaperones and quality-control systems [6]. Evolution further couples these layers: measures of evolutionary constraint reflect selection over joint molecular states rather than any single modality in isolation [7–9]. To truly understand a biological system, one must model not just the forward flow of information, but the full joint probability distribution of its molecular states. Despite this interdependence, the current landscape of artificial intelligence in biology remains fragmented. We have witnessed a proliferation of “siloed experts”—models that master a single molecular modality but lack the capacity to operate across the broader central dogma. By focusing almost exclusively on forward prediction (mapping a sequence to a specific phenotype), these models fail to capture the full joint distribution of biological states. Consequently, they leave complex inverse problems unsolved: for example, given a desired protein structure, mRNA stability, and splicing pattern, what upstream nucleotide sequence is most likely to have produced them? This limitation is widespread, characterizing large language models trained on biomedical [10–17], protein sequences [18–28], or nucleic acid [29–33] sequences. It is equally prevalent in forward regulatory genomic models [34–40], structure prediction systems for both RNA [41–45] and proteins [46–49], and narrowly scoped phenotype predictors such as those for splicing [50–52]. Constructing a framework to capture this complex joint distribution has previously been difficult not simply because biology is multimodal, but because its measurements are heterogeneous in form, scale, and provenance: sequences, structures, evolutionary signals, and functional assays have different data types, are distributed across disparate resources, and are represented in incompatible coordinate systems. Different biological properties are only sparsely measured together; thus most samples will feature only a subset of possible modalities, requiring an architecture that can model observed data and infer missing components. These challenges are compounded by the range of context lengths required to capture both regulatory and biomolecular dependencies, which can span from local amino acid environments to kilobase-scale genomic neighborhoods. Existing biomolecular foundation models typically comprise a single-stream architecture, rendering them poorly suited to handle the task of learning across the joint distribution of heterogeneous biological data. Recent “omnimodal” scientific architectures instead treat heterogeneous measurements as first-class tokens via modality-specific tokenization [53, 54], then learn a unified latent representation across diverse modalities via an encoder-decoder architecture. Our key insight is that by leveraging a flexible omnimodal architecture adapted to handle the heterogeneous context lengths and alignment structures of biological data, a single unified model can approximate the joint distribution of molecular states across the central dogma. We introduce MIMIC, a generative foundation model with a novel split-track architecture designed to unify biomolecular data. MIMIC learns to translate fluently between the languages of the genome, the transcriptome, and the proteome while also interfacing with context-aware text representations that emphasize cellular specificity (Figure 1a). Rather than solely predicting a quantity of interest or a single-track effect from sequence alone, MIMIC can process a rich molecular phenotype (e.g., uniting mRNA sequences with splice patterns, chemical reactivity scores, and a description of experimental conditions) to make conditioned predictions, including sequence edits 2

and designs (Figure 1c). This capability provides a rich opportunity to interrogate the interdependence between modalities across the central dogma of biology, offering new avenues for discovery of regulatory mechanisms, protein design, and the interpretation of non-coding variants in the context of human health (Figure 1d). A key obstacle to this kind of modeling has been the lack of aligned multimodal training objects: biological measurements are distributed across incompatible resources, are unevenly observed, and rarely co-occur in a form that allows direct learning over shared molecular state. To address these limitations, we introduce LORE, a curated dataset released alongside MIMIC. LORE aligns heterogeneous, multi-scale biological data into unified per-entity snapshots, providing the critical cross-modal supervision necessary for joint distribution training (Figure 1b). Together, MIMIC and LORE move biological foundation modeling beyond siloed predictors and toward a holistic computational framework. In this work, we demonstrate that MIMIC is a state-of-the-art foundation model for predicting biomolecular properties, outperforming specialized models for proteins, nucleic acids, variant prediction, and splice site prediction. We show that multimodal conditioning reduces the search space of molecular design, enabling generation of high-quality proteins from a desired molecular surface and recovery of wild-type splicing patterns in the presence of pathogenic mutation. By integrating natural language descriptions of cellular and experimental context directly into the latent space, MIMIC shifts the traditional modeling paradigm from rigid, track-specific output toward flexible semantic conditioning; we find that context-conditioned predictions of chemical probing scores improve the computational modeling of RNA 2D structures. Despite operating at a comparable size to other single-modality foundation models (1 billion parameters), MIMIC demonstrates the value of diverse and heterogeneous data as an alternative to pure scaling for improving model performance. This work represents a step toward a unified computational model of cellular biomolecules: one capable of learning bidirectionally across the molecular layers of the central dogma, and of translating directly between primary sequences and biological outcomes.

2

MIMIC: A multimodal foundation model for molecular biology

To address the structural and contextual complexity of the central dogma, we developed MIMIC: a generative foundation model designed to jointly represent and reason over the heterogeneous molecular data of the central dogma. Because biological measurements span fundamentally different coordinate systems from nucleotide transcripts to amino acid sequences, MIMIC employs a novel split-track encoder-decoder transformer architecture. This design organizes biological inputs into distinct track groups, each suited to the coordinate system and molecule of the underlying modality, and learns a unified latent representation across them. We train a 1 billion parameter version of MIMIC, (see Table S7 for architecture details) on sequences up to 10,000 tokens using this splittrack tokenization scheme. The encoder processes a unified token sequence assembled from several distinct track groups: a nucleic acid track, an amino acid track, and separate token groups for semantic and contextual information, alongside a set of learnable register tokens that aggregate global state across all tracks (Section B). Within the nucleic acid and amino acid tracks, features sharing a common coordinate system (for example, phyloP conservation scores aligned to nucleotide positions, or DSSP secondary structure labels aligned to residue positions) are each embedded and summed element-wise into a single track representation. This split-track summation keeps total sequence length tractable regardless of how many modalities are present, while preserving the positional alignment between co-localized signals. Unaligned global information, including natural language descriptions of cellular context and functional annotations, occupies separate token groups concatenated to the biological tracks. 3

Figure 1: Overview of the MIMIC framework. (A) Molecular biology data is highly heterogeneous, spanning genomic, transcriptomic, and proteomic sequences, each with multiple assays and measurements that describe their function. (B) We curate LORE, a multimodal dataset that integrates and aligns data from multiple repositories into a set of unified but partially observed training examples. (C) Using this data, we build MIMIC, a multimodal foundation model for heterogeneous molecular biology. MIMIC models these examples using a split-track encoderdecoder architecture that combines co-aligned within-track signals while preserving distinct semantic inputs. (D) MIMIC enables multimodal representation learning, highly accurate property and splicing prediction, and constrained generative design across molecular biology.

4

Positional encoding uses Rotary Position Embeddings (RoPE) with a local group-reset strategy in which the position index restarts at zero for each track group, so that attention patterns reflect within-track distances rather than absolute token offsets across the full sequence. A cross-attention decoder with a 1,000-token context window queries the encoder’s latent representation to reconstruct or generate arbitrary target modalities. This decoupling of observed context from predicted outputs is central to MIMIC; the asymmetric design (a wide receptive field for context, a focused window for generation) allows MIMIC to condition on any observed subset of modalities and generate any target subset without architectural modification or task-specific output heads, enabling tasks as distinct as generating a protein sequence from surface geometry, inferring a splice pattern from transcript boundaries, or predicting a phyloP conservation profile from sequence context alone. MIMIC is trained using a staged curriculum that progressively scales the encoder context window from 1,000 to 10,000 tokens, allowing the model to first learn local sequence features before resolving long-range regulatory and structural dependencies. To handle the combinatorial heterogeneity of available modalities across samples, training is organized into a set of approximately 25 heuristic pathways, each specifying a required and optional set of input and target modalities. This pathway sampling ensures that rare but high-value combinations (for example, jointly observed structure and splice data) are adequately represented during training relative to more abundant sequenceonly samples. Register tokens are trained via a reconstruction objective in which random token dropout forces them to encode sufficient information for the decoder to recover masked inputs, encouraging the registers to function as compressed global state vectors rather than simple copy mechanisms (Section C).

3

LORE: a massively multimodal aligned biomolecular dataset

Training a multimodal architecture like MIMIC requires a fundamental shift in how biological data is curated: observations cannot be siloed by modality, but must be explicitly aligned to reflect their biological interdependencies. To provide this critical infrastructure, we curated LORE, a massively multimodal dataset of genomic, transcriptomic, and proteomic measurements containing approximately 15.5 million proteins and 13 million RNA transcripts from over 6000 organisms spanning all domains of life, along with over 4 billion tokens of biomedical, functional, and contextual text (Section A). The central design principle of LORE is alignment; this design is distinct from massive sequence-only corpora such as OpenGenome2 [55] or BFD [56] that do not provide aligned or multimodal samples. Each row of the LORE dataset is an observation of either a transcript, a protein, or a pairing of the two, containing the subset of the modalities that are available for these biomolecules. Rather than treating different biomolecules as disconnected observations, we organize them into unified samples so that measurements referring to the same underlying genomic entity can be modeled jointly in their natural coordinate system. Because available data are inherently incomplete, LORE is intentionally constructed as a partially observed multimodal dataset: no example is required to contain all modalities. Instead, we preserve high-value overlapping subsets wherever alignment is available, while reducing redundancy through sequence clustering and representative selection [57] (Section A.6). At the nucleic acid level, LORE includes tracks aligned with the unspliced RNA sequence [58, 59] such as evolutionary conservation [8], RNA chemical probing [60], and regulatory assays [4,61] (Section A.3). At the protein level, it includes amino acid sequence [62], structural [63] and surface features [64], functional descriptions [65], and measurements of abundance [66] (Section A.4). Crucially, LORE also contains observations with semantic context; epigenetic tracks, chemical probing data, and abundance measurements are paired with textual descriptions of the relevant organism, tissue, cell line, or experimental condition (A.5). These aligned multimodal ob5

servations make LORE a uniquely comprehensive substrate for training models that must reason jointly across the molecular layers of the central dogma, rather than learning each layer in isolation.

4

Evaluating MIMIC across diverse biomolecular tasks

4.1

Multimodal sequence completion

Self-supervised pretraining is commonly instantiated as sequence completion: predicting missing tokens from context. This objective underlies both masked and autoregressive language models, and therefore provides a natural starting point for comparing uni-modal baselines to a multimodal framework (Figure 2a). To evaluate the performance of MIMIC over proteomic modalities, we likewise evaluate sequence completion on amino acid sequences, reporting per-residue reconstruction accuracy. We compare against several sequence-only protein language models (ProtBERT, ESM2, ESM-C, and ESM3-open); MIMIC conditioned with structural descriptors achieves the highest reconstruction accuracy (Figure 2a). To evaluate whether MIMIC has learned the distribution over genomic modalities, we formulated a sequence completion task on unspliced transcripts from MIMIC’s test set, masking 100 nucleotides and reporting per-nucleotide top-1 accuracy at masked positions, stratified by exonic and intronic regions. We compared MIMIC with the full set of aligned genomic modalities against two long-context genomic foundation models trained on OpenGenome2: NTv3 [30] (masked reconstruction) and Evo 2 [29] (autoregressive). MIMIC likewise achieves the highest reconstruction accuracy in both intronic (Figure 2b) and exonic (Figure 2c) regions. To clarify the contribution of multimodal conditioning, we ablated subsets of aligned auxiliary modalities (Section D.1). Even in the sequence-only configuration, MIMIC is competitive with the top sequence-only models, and conditioning on more modalities leads to consistent improvements to sequence recovery. We find that multimodal conditioning is especially useful in cases where additional modalities are ambiguous rather than easily predictable from the primary sequence.

4.2

Predicting downstream RNA and protein properties

One of the most powerful uses of biological foundation models is their ability to enable transfer learning to downstream tasks [67–69]. We next investigated whether MIMIC’s learned representations were informative for predicting several downstream properties of both nucleic acid and amino acid sequences, using two foundation model benchmarking suites: PFMBench [70] and mRNABench [71]. Following the protocol for each benchmark, we trained a supervised probe on MIMIC embeddings for each task (Section D.2, Section D.3). PFMBench [70] covers protein function, structure, interaction, and developability tasks (Figure 2d). Across these categories, MIMIC consistently matches or exceeds strong protein-only baselines including ESM3 [25], ESM-C [72], ProTrek [13], and SaProt [73] (Figure 2d). We show the head-to-head win rate for MIMIC against each of the models evaluated in the benchmark. MIMIC outperforms each other model on at least 7/11 tasks. MIMIC achieves especially strong performance on the protein-ligand binding tasks (BindingDB, PDBbind), suggesting that jointly pretraining on surface chemical properties improves protein representations for predicting binding. MIMIC is also the best performing method for all developability tasks. We next evaluated MIMIC on mRNABench [71], which includes several tasks covering prediction of mRNA function, localization, transcriptional regulation, and variant effect prediction (Figure 2e, f). Here too, MIMIC is a state-of-the-art RNA foundation model. MIMIC outperforms Evo 2 [29] and Orthrus [74] on 4/7 of the tasks, Dilated ResNet [71] on 6/7 tasks, and outperforms all other methods on all tasks. MIMIC displays the strongest relative performance for tasks that benefit from training on aligned protein modalities, such as GO function prediction (+2% vs. Evo2, +9% 6

Figure 2: MIMIC achieves state-of-the-art performance across RNA and protein sequence property prediction benchmarks. (A) Per-residue top-1 amino acid inpainting accuracy at 100 masked positions. MIMIC (with structural and surface conditioning) outperforms all sequence-only protein language model baselines including ESM3-open, ESM-C, ESM-2 (650M), and ProtBERT. (B-C) Per-nucleotide top-1 inpainting accuracy at 100 masked intronic (B) and exonic (C) positions. MIMIC (with RASP2, phyloP, and splicing conditioning) outperforms Evo 2 and NTv3 despite the latter two being evaluated on their training distribution (OpenGenome2). (D) PFMBench protein benchmark evaluation across function, structure, interaction, and developability tasks. MIMIC, even with sequence only input, is consistently among the top performing models. Below, we show the head-to-head win rate of all models against MIMIC; no model outperforms MIMIC on more than half the tasks. (E) mRNABench evaluation across RNA function, localization, and translation regulation tasks. MIMIC is the top performing model, outperforming Evo2 and Orthrus more than half the time and outperforming other models on nearly every task. (F) On mRNABench complex variant effect prediction MIMIC is already the best performing method. Additionally, the phyloP VEP (Section 4.3) greatly outperforms all other methods. On Mendelian variants, phyloP VEP improves performance, but MIMIC does not surpass the best-performing baselines on this task.

vs. Orthrus) and protein localization prediction (+2% vs. Evo2, +5% vs. Orthrus). Among the translation-related tasks in mRNABench, MIMIC shows its clearest gain on the Sugimoto halflife data set [75], with a relative improvement of 8% (vs. Orthrus). Because the Sugimoto data measures mean ribosome load for endogenous human transcript isoforms, this result suggests that MIMIC captures features relevant to isoform-level translational output. In contrast, MIMIC does not surpass the previous best model on the LBKWK half-life data set [76] (−8% vs. Orthrus), a benchmark derived from synthetic mRNA constructs in which stability-related effects are likely 7

to play a larger role. These results suggest that MIMIC is particularly strong for endogenous transcript-level translation prediction. We note that with 566M encoder parameters (only the encoder is used to generate embeddings), MIMIC achieves state-of-the-art performance while being one of the smaller networks evaluated here.

4.3

Variant Effect Prediction with MIMIC-predicted constraint

For a sequence-only foundation model, downstream use is necessarily mediated through its latent representation: in the absence of an explicit aligned output modality, task transfer must proceed through probing or fine-tuning. In contrast, MIMIC can expose task-relevant signal directly through generation itself, rather than forcing every downstream task to be learned from a generic embedding. Variant effect prediction (VEP) provides a clear example. We report the performance of MIMIC in the mRNAbench VEP tasks under two inference scenarios. In the first (denoted as MIMIC in Figure 2f), we follow the mRNAbench protocol to train a linear probe over the latent state. In the second, denoted as “MIMIC (phyloP VEP)”, we use the model’s wild-type phyloP prediction together with a smoothed mutant-wild-type phyloP difference and train a logistic regressor on top of this two dimensional feature space (Section D.4). The two signals are complementary: reference-site conservation is most informative in regions already under detectable evolutionary constraint, whereas the mutation-induced signal remains informative in weakly conserved sequence, where pathogenicity depends more directly on mutation-specific disruption of local regulatory logic. This extremely low-dimensional regressor is more interpretable than representation-based alternatives and performs substantially better in practice than the embedding-based approach (Figure 2f). On complex variants, the embedding-based version of MIMIC’s VEP is already the strongest (+33% vs. SpliceBERT [77], the next-best model), but the phyloP-based VEP vastly outperforms all embeddingbased approaches (+82% vs. embedding-based MIMIC). On Mendelian variants, switching from embedding-based to phyloP-based VEP improves MIMIC performance by 90%.

5

From splicing prediction to splice-conditioned RNA design

5.1

Predicting transcript-specific splice patterns

Alternative splicing is a central mechanism of transcript diversity and gene regulation, and its disruption is a major contributor to human disease [50, 52, 78]. To evaluate MIMIC as a splice site predictor, we conditioned the model on DNA sequence alone and asked it to predict donor (5’) and acceptor (3’) splice sites across a held-out set of human genes (Figure 3a). We compare with SpliceAI [79], AlphaGenome [35], and NTv3 [30], and report performance stratified by coding and non-coding genes as all models perform significantly better in coding regions. MIMIC outperforms these models (Figure 3a, +1% AUPR vs. AlphaGenome, +3% vs. SpliceAI, +11% vs. NTv3 in coding regions; +12% vs. AlphaGenome, +14% vs. SpliceAI, +311% vs. NTv3 in non-coding regions), demonstrating that a generative multimodal model can match dedicated forward predictors and genomic-only foundation models on classical splice-site detection. However, per-site evaluation offers an incomplete picture of splicing mechanisms. Prior work has shown that collapsing evaluation into per-site targets introduces noise into training and reduces clinically relevant predictive performance [79], and long-read transcriptomic studies have reinforced that biologically informative splicing patterns are only visible when transcript elements are considered jointly [80–83]. To circumvent this, MIMIC is jointly trained on full transcript realizations; it learns where splice sites occur within valid transcripts.

8

Figure 3: MIMIC accurately predicts splice sites and designs RNA sequences with predictable splice patterns. (A) Gene-level splice site prediction: MIMIC takes a genomic region as input and predicts donor and acceptor positions. Across coding (left) and non-coding (right) regions, MIMIC outperforms AlphaGenome, SpliceAI, and NTv3. (B) Transcript-conditioned splice prediction: providing transcript context in terms of transcription start (TSS) and end (TES) sites enables transcript-specific inference. MIMIC sees larger gains in performance over previous models when conditioning on TSS and TES. (C) An example of splice prediction with and without transcription specific conditioning. Without TSS and TES conditioning, MIMIC predicts an additional exon inclusion in SPRY1. After conditioning, the correct splice architecture is predicted. The predicted probability of several false positive acceptor and donor sites in SPRY1 is substantially reduced following TSS and TES conditioning, leading to prediction of only the true splice junctions. (D) Starting from a mutant sequence with the pathogenic variant fixed, MIMIC conditions on the wild-type splice pattern and wild-type phyloP to re-design a nucleotide sequence that suppresses inclusion of the cryptic exon. (E) We apply this design pipeline to the HBB IVS-II-654 C > T mutation. (Top.) Wild-type conservation score, annotated with the cryptic acceptor and donor sites induced by the mutation. (Middle.) MIMIC phyloP VEP localized around the cryptic exon changes significantly between the wild-type sequence and the splice-altering C > T variant. In contrast, a the non-splice-altering C > A control shows no difference in predicted conservation from the wild type. (Bottom.) We show the predicted probability of pathogenic acceptor or donor sites for 30-nucleotide windows designed by MIMIC. MIMIC designs are significantly less likely to result in the inclusion of the cryptic exon. To avoid circularity, we use SpliceAI as an independent oracle for acceptor/donor probability at the cryptic sites.

9

We therefore additionally evaluated all models at the transcript level, prompting MIMIC to recover the full internal splice structure of held-out human isoforms. At this level of evaluation, MIMIC is clearly the strongest splicing model (Figure 3b, +6% AUPR vs. AlphaGenome, +5% vs. SpliceAI, +11% vs. NTv3 in coding regions; +26% vs. AlphaGenome, +16% vs. SpliceAI, +146% vs. NTv3 in non-coding regions). Furthermore, because MIMIC is a multimodal model, it can be conditioned explicitly on transcript boundary information. We show that explicitly providing the transcript start site (TSS) and transcript end site (TES) as input to MIMIC alongside the genomic sequence further improves recovery of internal splice structure (+3% in coding regions, +5% in non-coding regions, light blue in Figure 3b), consistent with evidence that transcription start site usage shapes downstream isoform selection [84, 85]. As a representative example, we show that transcript conditioning improves MIMIC predictions of splicing in SPRY1 (Figure 3c). Prior to conditioning, MIMIC predicts an extra exon not included in the ground truth, as well as several alternative donor sites. After conditioning, all false positive splice junctions are predicted with significantly lower probability, so only the true exons remain. These results position MIMIC as a state-of-the-art splicing model that moves beyond single-locus prediction to global modeling of transcript architecture.

5.2

Multimodal conditioned design of aberrant splicing mutations

One of the central motivations for MIMIC is that a multimodal foundation model should support generative design tasks, not only forward prediction. Our results on sequence completion and splicing prediction suggest that MIMIC uses multimodal context to tighten the conditional distribution over biologically plausible sequences. We sought to investigate whether the same conditional machinery could be repurposed for an inverse problem: starting from a pathogenic regulatory context, can the model generate sequences that restore a wild-type-like state? We first show that MIMIC preferentially regenerates the wild-type allele when the pathogenic position is masked, and that phyloP conditioning significantly improves this recovery, confirming that the model’s conditional distribution is biased toward reference-compatible states (Section D.4). To move beyond this single-nucleotide demonstration, we next asked whether MIMIC could support a more demanding form of design: restoring wild-type function even when the pathogenic allele is held fixed. We investigated this in the intronic β-globin (HBB ) variant c.316-197C>T (IVS-II-654 C>T), a canonical splice-altering mutation that promotes pseudoexon inclusion through aberrant splice-site gain [86, 87] (Figure 3d). This setting provides a stringent test of whether the model can use multimodal context to identify compensatory sequence rewrites outside the causal locus, rather than merely regenerating the reference base. Furthermore, although wild-type phyloP conservation in this region provides no indication of the pathogenic nature of this mutation, MIMIC’s phyloP VEP score clearly distinguishes the pathogenic C>T substitution from a matched non-splice-altering C>A control, localizing the signal to the cryptic acceptor and donor (Figure 3e). We therefore prompted MIMIC with both the wild-type splice pattern and the pathogenic sequence with the C>T variant fixed (Figure 3e), generating designs for editable windows of 30 or 50 nucleotides centered at varying distances from the mutation (Section E.1). Designed sequences were evaluated using SpliceAI as an independent oracle to determine the probability of cryptic site gain at the pathogenic position.1 MIMIC produces designs that substantially reduce the likelihood of pathogenic splicing while maintaining the causal mutation. Furthermore, guided by the observation that deviation from the wild-type phyloP is a marker of pathogenicity, we also prompted MIMIC to produced design that would recreate not only the wild-type splice pattern but also the 1 AlphaGenome’s splice prediction did not correctly predict the cryptic Acceptor for the mutant sequence and was therefore not suitable for evaluating the corrective effect of MIMIC designs.

10

wild-type phyloP. In this more demanding case, again MIMIC produces designs that significantly and consistently reduce the likelihood of the cryptic exon inclusion (Figure 3e). That effective corrections can be found even when the editable window does not directly overlap the cryptic splice sites demonstrates that multimodal conditioning supports indirect regulatory rewiring, through the modification of exonic regulatory splicing elements. A

Target Binding Complex

C

Binder

Binding Site

D Receptor

hACE2, Design Alignment

MIMIC Design

SARS-Cov-2-RBD – hACE2 PDB ID: 6VW1

MIMIC Design

MIMIC Design Pipeline Binder Surface Feature (MaSIF)

F

MIMIC Designs (n=37)

hACE2, Design Diversity

MIMIC Designs (n=20)

MIMIC Designs (n=37)

Designed Binder Sequence ...TSVDEQFKTFLDKFNHESQDLFYNNTIATNNYNTNLTDENVQ NMNNAGDKWTAFFKEQASLAQMYPLQEIQNLTIKINAQAIQYNGT SVLSEDKTKRLNSILNTISTITSTAKVCNPDNPQDCLVLEPGLNE INANSLDYNERLWAYDSYYSEIGKQLNPLYDELIVLKNEYARANH YEDSADYYRSEFELNGIDGYEYTRGQWIDDYEHWMEELKPIYEQY YAYTRAKLMQAYPSYLSPIGPIPTPFLGDATGRFPSELYSITLPF PQKPHLDATEAIIEQAWDAHRIRKET...

G

PD-L1, Design Diversity

Sequence Identity (%)

Binder Backbone

Sequence Identity (%)

B

E

PD-L1, Design Alignment

MIMIC Designs (n=20)

Figure 4: MIMIC designs recover target-binder properties with high sequence diversity. (A) Schematic of the target binding complex (binder in blue, receptor in grey, binding site in red), use SARS-Cov-2-RBD - hACE2 (PDB ID: 6VW1) as an example. (B) Overview of the MIMIC design pipeline. The model generates novel sequences conditioned on the wild-type (WT) binder’s backbone coordinates, MaSIF surface fingerprints, or both. (C) AlphaFold2 confidence (pLDDT) of designs across five conditioning strategies for a small PD-L1 (PDB ID: 4ZQK) and large hACE2 (PDB ID: 6VW1) target. Diamonds indicate the mean; the dashed line indicates the pLDDT = 85 quality threshold. (D, E) In silico evaluation of high-confidence designs (pLDDT > 85). Metrics shown are backbone TM-score versus WT (top), MaSIF surface similarity (middle), and AlphaFold3 cofolding iPTM (bottom). Red dots highlight the exemplar designs shown in the structural insets. (F, G) Pairwise sequence identity matrices for highconfidence designs and the WT sequence (red, top-left), ordered by hierarchical clustering on sequence dissimilarity.

6

Multimodal conditioned design of structured binders

Having demonstrated that multimodal conditioning improves our ability to generate nucleotide sequences with specific regulatory properties, we reasoned that an analogous approach could reduce the search space within a protein design framework. Our approach here builds directly upon the rich 11

literature of structure- and surface-centric design [88–92]. Rather than treating backbone scaffolding and surface complementarity as separate design tasks, MIMIC’s joint sequence-structure-surface representation allows simultaneous conditioning on both, explicitly encoding the physical, chemical, and electrostatic environment that mediates protein-protein interactions. To test this, we targeted two therapeutically relevant proteins with well-characterized binding interfaces: Programmed death-ligand 1(PD-L1, PDB: 4ZQK), a major checkpoint immunotherapy target that engages Programmed Cell Death Protein 1 (PD-1) to suppress T-cell activation, and human Angiotensin-converting enzyme 2 (hACE2, PDB: 6VW1), the host receptor for the SARSCoV-2 spike protein. For each target, we generated sequence ensembles under five conditioning strategies (20 per strategy totaling 100 designs each): MaSIF surface features alone (covering 40% or 100% of residues proximal to the binding site), backbone geometry alone, or backbone combined with partial or full surface features (Figure 4a). We evaluated design quality using the AlphaFold2 predicted local distance difference test (pLDDT). Across both targets, design confidence reached the maximum when both backbone structure and surface chemistry were provided. Notably, the impact of multimodal conditioning scaled with target complexity: while providing the surface alone was sufficient to achieve high-confidence designs (pLDDT ≥ 85) for the smaller PD-L1 target, the larger hACE2 target relied much more heavily on additional backbone conditioning to reach comparable confidence (Figure 4b). To assess structural fidelity and sequence diversity among high-confidence designs (pLDDT ≥ 85; PD-L1: n = 37, hACE2: n = 20), we evaluated their backbone recovery, surface similarity, and predicted binding competence relative to the wild-type (WT) structures (Figure 4d, e). Across both targets, the designed sequences consistently recovered the WT fold (median TM-score > 0.85) and surface chemistry (median MaSIF similarity > 0.90). Furthermore, in silico co-folding with AlphaFold3 provided evidence of retained binding activity: for the smaller PD-L1 target, 35 out of 37 high-confidence designs achieved iPTM scores > 0.75 (median iP T M = 0.81). The hACE2 target presented a more challenging design task due to its larger size and complexity; only two of the 20 designs with high-confidence fold were predicted to bind the SARS-CoV-2 spike RBD (AlphaFold3 iP T M > 0.75). Crucially, MIMIC did not simply memorize or produce trivial variations of the WT sequence. Pairwise sequence identity matrices revealed that the generated designs maintain only about 50% sequence identity to the WT (Figure 4f, g, red boxes). While the designs exhibit local clustering (resulting in blocks of higher mutual sequence identity among themselves as they converge on shared functional motifs) the design ensembles remain globally distinct from the original WT sequence. These results illustrate a central advantage of multimodal generative modeling for protein design: the ability to constrain generative search using complementary structural and chemical priors. By simultaneously conditioning on both backbone geometry and molecular surface features, MIMIC efficiently narrows the vast sequence space toward an ensemble of diverse solutions that are structurally stable, chemically faithful, and highly compatible with the target interaction, all without collapsing into trivial wild-type memorization. This targeted reduction in design freedom, analogous to the transcript-boundary constraints demonstrated earlier for RNA splicing, suggests that MIMIC has internalized a sequence fitness landscape tightly aligned with the true biological and physicochemical constraints of the interaction interface.

12

Figure 5: MIMIC leverages experimental context for accurate RNA reactivity prediction and RNA structure modeling. (A-B) MIMIC accurately predicts transcriptome-wide chemical probing reactivity (RASP2 scores) and adapts to condition-specific contexts. (A) Pearson correlation coefficients (r) between MIMIC-predicted and experimentally measured RASP2 scores for coding and non-coding RNAs. The context-aware generation (blue) significantly outperforms the sequence-only baseline (grey). Error bars represent standard error of the mean. (B) Representative reactivity tracks illustrating the model’s sensitivity to cellular environment and probing approach. Top: MIMIC captures structural differences between in vivo and in vitro (cell-free) conditions for transcript ENST00000663097.1. Bottom: Predictions for transcript ENSMUST0000065709.13 demonstrate the model’s ability to differentiate between chemical probes, capturing the broad sensitivity of icSHAPE compared to the A/C-restricted sensitivity of DMS-seq. Blue tracks represent MIMIC predictions; grey tracks represent experimental measurements.(C-D) MIMIC-predicted reactivity improves RNA 2D structure modeling. (C) Structure modeling workflow using ViennaRNA evaluated on an 800 nt window of transcript ENSMUST00000128851.8. Incorporating MIMICpredicted reactivity (blue, F1 = 0.987) yields a 2D structure nearly identical to the experimentally-guided reference (grey), vastly outperforming sequence-alone folding (orange, F1 = 0.404). (D) Quantitative evaluation of structural improvement across the test set (n = 1, 000 RNA sequences). Left: Box plot comparing F1 scores of sequence-alone modeling versus MIMIC-guided modeling (*** P = 6.25 × 10−47 , one-sided paired Wilcoxon signed-rank test). Error bars represent standard error of the mean. Right: Histogram of the change in F1 score (∆F1) after incorporating MIMIC predictions. The rightward shift demonstrates robust structural improvement across the dataset (median ∆F1 = +0.061, mean = +0.095 ± 0.194 s.d.).

7

Context-conditioned RNA reactivity prediction to guide structure modeling

Biological processes are inherently dynamic, governed as much by their surrounding cellular environment and physiological state as by their underlying molecular sequence. Consequently, as we 13

aggregate large-scale experimental datasets to train foundation models, we must recognize that these measurements do not exist in a vacuum. They are inextricably linked to the specific biological context, cellular state, and probing approach used during data collection. To faithfully capture this complexity, MIMIC is designed to model experimental conditions explicitly. Rather than relying on fixed output heads for different assays, MIMIC processes natural language descriptions of experimental conditions directly alongside molecular data. By incorporating these semantic inputs into its latent representation, the model learns to treat experimental context not as generic auxiliary metadata, but as a primary modality that dictates how a molecular state should be realized. Crucially, we demonstrate that this context-aware capability translates directly into practical benefits for downstream analytical tasks. By accurately predicting condition-specific reactivity profiles, MIMIC provides high-fidelity constraints that improve computational RNA 2D structure modeling. We grounded this evaluation in chemical probing data from the RASP2 (RNA Atlas of Structure Probing) database [60]. Because chemical probing serves as the primary experimental workhorse for investigating RNA architecture, it provides an ideal and highly stringent computational test case; reactivity measurements for identical transcripts vary substantially depending on the probing approach and the cellular environment. We found that conditioning MIMIC on the correct natural language experimental context significantly improves transcriptome-wide RASP2 predictions relative to models conditioned on sequence alone (Figure 5a). This performance gap demonstrates that MIMIC successfully learns to leverage semantic context to resolve biological ambiguities that sequence alone cannot address. To confirm that these improvements reflect biologically meaningful adaptations rather than statistical artifacts, we inspected individual MIMIC predictions against experimentally measured RASP2 reactivity scores across diverse contexts (Figure 5b). The model demonstrates a high degree of sensitivity to environmental variables; for example, it successfully captures the distinct structural profiles of the same transcript probed in vivo versus in vitro (cell-free). Furthermore, the model accurately adjusts its output based on the specified probing approach. When comparing an in vivo icSHAPE assay (click selective 2-hydroxyl acylation and profiling experiment, carried out with the NAI-N3 probe) to dimethyl sulfate sequencing (DMS-seq), MIMIC successfully differentiates between the broad sensitivity of icSHAPE and the A/C-restricted sensitivity of DMS, generating the appropriate condition-specific reactivity tracks. This contextual conditioning extends beyond experimental parameters. We found that MIMIC can also leverage other biological modalities to uncover structural insights. For instance, conditioning the model on evolutionary conservation scores (PhyloP) yields a distinct uplift in RASP2 prediction accuracy (Figure S6). This predictive synergy highlights an underlying biological correlation between RNA structure and evolutionary conservation, conceptually analogous to how coevolutionary information derived from multiple sequence alignments (MSAs) drives structural predictions in the protein folding field [46]. Beyond predicting reactivity profiles, we investigated whether MIMIC’s context-aware predictions provide tangible utility for downstream biological tasks. Specifically, we evaluated the use of MIMIC-predicted reactivity scores to guide computational RNA 2D structure modeling (Figure 5c, details in Appendix D.7). Integrating MIMIC predictions into standard folding algorithms, ViennaRNA [93], generates plausible RNA secondary structures that are qualitatively nearly identical to structures guided by experimental reference data, vastly outperforming sequence-alone folding. Quantitatively, incorporating MIMIC-predicted reactivity profiles robustly improves structural modeling accuracy across a representative population of the test set (Figure 5d). The resulting folds exhibit a significant increase in F1 scores compared to sequence-alone baselines. Together, these results demonstrate that MIMIC successfully internalizes diverse biological modalities-ranging from semantic experimental descriptions to evolutionary conservation—as integral components of biological modeling. By leveraging these contextual inputs to specify the 14

appropriate realization of a molecular measurement, MIMIC provides a powerful framework for generating and interpreting complex biological data, ultimately driving more accurate modeling and analysis across a wide array of downstream tasks.

8

Discussion

This study introduces MIMIC, a generative multimodal foundation model for heterogeneous biological data. Across the results presented here, a consistent pattern emerges: aligned auxiliary context improves model representations and performance, consistent with prior findings from “omnimodal” scientific models [54]. To enable the training of MIMIC, we developed LORE, an aligned data set spanning diverse biomolecular modalities. We show that multimodal conditioning enhances sequence completion, yields transferable representations for RNA and protein downstream tasks and supports transcript-level modeling of splicing. We apply MIMIC for multimodal biomolecular design, providing stabilizing constraints that increase design efficiency in both nucleic acid and protein settings. MIMIC supports semantic context by modeling it as a conditioning variable within the same framework, rather than treating it as metadata, imposing a fixed output schema, or requiring separate track-specific outputs. Sequence, structure, regulation, evolutionary constraint, and experimental context can be understood as distinct but related observations of an underlying molecular state. This perspective helps explain why a single model can support tasks that are often studied separately, and our results indicate that biological modeling may benefit from frameworks that move flexibly between representation learning, conditional prediction, and constrained generation within a shared model. Several limitations should, however, guide the interpretation of these findings. First, LORE captures only a subset of the measurements that shape molecular phenotype. Many important facets of molecular biology remain absent or only weakly represented. Even among the modalities that are included, coverage is highly uneven, so MIMIC is trained on a partial and imbalanced view of molecular biology, which may limit its performance in some applications relative to specialized models designed for specific tasks. In parallel, the current context scale remains limited relative to the biological processes one would ultimately wish to capture, and the restricted decoder window further limits the extent of direct generation over long outputs. A natural next step is to broaden modality coverage while improving alignment quality and metadata standardization. Particularly important additions include long-range chromatin and regulatory measurements, richer transcript-resolved assays, and more complete measurements of biomolecular function and interaction. Additionally, longer and explicitly multiscale architectures may be needed to connect local biochemical constraints with cell-state-scale regulation. On the conditioning side, semantic context would likely benefit from moving beyond sparse free text toward more structured representations of experimental state that still interface naturally with language. We show that MIMIC unifies heterogeneous molecular measurements within a single generative framework, enabling missing components to be inferred from available context and allowing prediction and design to be framed as different forms of inference over the same underlying biomolecular system. More broadly, jointly modeling the interconnected layers of molecular biology through flexible multimodal representations of aligned molecular state represents a powerful new abstraction for biological foundation models.

15

Contributions. The author contributions are summarized below. The order of the authors within each category is randomized. • Project Lead: Siavash Golkar • Core Team: Ksenia Sokolova, Jake Kovalic, Minhuan Li, Samuel Sledzieski, Alberto Bietti, Geraud Krawezik, Siavash Golkar, Irina Espejo Morales • Advisory: Alisha N. Jones, Shirley Ho, Sonya M. Hanson, Olga G. Troyanskaya, Pilar Cossio MIMIC Contributors • Model architecture: Jake Kovalic, Siavash Golkar, Irina Espejo Morales, François Lanusse Lanusse, Liam Parker • Pretraining: Irina Espejo Morales, Siavash Golkar, Jake Kovalic, Geraud Krawezik • Downstream evaluations: Siavash Golkar, Minhuan Li, Jake Kovalic, Ksenia Sokolova, Irina Espejo Morales, Shengwei Xiong, Alisha Jones, Samuel Sledzieski • RNA design task: Shengwei Xiong, Siavash Golkar, Alisha Jones • Protein design task: Minhuan Li, Samuel Sledzieski, Siavash Golkar, Vikram Mulligan • RNA 2D structure modeling task: Minhuan Li, Irina Espejo Morales, Siavash Golkar • Manuscript writing: Samuel Sledzieski, Siavash Golkar, Minhuan Li, Ksenia Sokolova, Alisha Jones, Jake Kovalic, Claudia Skok Gibbs, Irina Espejo Morales LORE Contributors • Modality alignment and merging: Jake Kovalic, Irina Espejo Morales, Siavash Golkar, Samuel Sledzieski • RNA chemical probing (RASP): Minhuan Li, Alisha N. Jones, Roman Klypa • Epigenetics: Ksenia Sokolova, Siavash Golkar, Claudia Skok Gibbs • Protein sequence, structure and surface features: Samuel Sledzieski, Minhuan Li • Gene and transcript sequence and annotations: Jake Kovalic • Text processing and biomedical corpus: Irina Espejo Morales

Other This section includes contributions ranging from code review and problem solving, compute and HPC support, as well as discussions and feedback. • Payel Mukhopadhyay, Miles Cranmer, Rudy Morel, Lucas Meyer, Helen Qu, Jeff Shen, Tom Hehir, Hadi Sotoudeh, Kyunghyun Cho, Mariel Pettee, David Fouhey

16

Acknowledgments We would like to acknowledge the support of the Simons Foundation and of Schmidt Sciences. This work was supported in part by the AI2050 program at Schmidt Sciences (Grant G-25-70028). A.N.J. acknowledges support from NIH grant 1R35GM160278-01. Additionally, computations were run at facilities supported by the Scientific Computing Core at the Flatiron Institute. The Flatiron Institute is a division of the Simons Foundation. The authors thank Justin Kinney, Doug Renfrew, and Bargeen Turzo for helpful discussions, and Lucy Reading-Ikkanda and Chi-Dat Lam for assistance with figures. Data and Model Availability We are currently preparing MIMIC code and weights for public release. We are also preparing LORE for public release, including data clustering, splits, and precomputed tokenization. Upon release, both the MIMIC model and the LORE data set will be available on the Polymathic AI GitHub (https://github.com/PolymathicAI).

17

References [1] Francis H. C. Crick. Central dogma of molecular biology. Nature, 227(5258):561–563, 1970. [2] Roadmap Epigenomics Consortium. Integrative analysis of 111 reference human epigenomes. Nature, 518(7539):317–330, 2015. [3] ENCODE Project Consortium. Expanded encyclopaedias of dna elements in the human and mouse genomes. Nature, 583(7818):699–710, 2020. [4] Jason D. Buenrostro, Paul G. Giresi, Lisa C. Zaba, Howard Y. Chang, and William J. Greenleaf. Transposition of native chromatin for fast and sensitive epigenomic profiling of open chromatin, dna-binding proteins and nucleosome position. Nature Methods, 10(12):1213–1218, 2013. [5] Christian B. Anfinsen. Principles that govern the folding of protein chains. 181(4096):223–230, 1973.

Science,

[6] F. Ulrich Hartl, Andreas Bracher, and Manajit Hayer-Hartl. Molecular chaperones in protein folding and proteostasis. Nature, 475(7356):324–332, 2011. [7] Adam Siepel, Gill Bejerano, Jakob S. Pedersen, Angie S. Hinrichs, Minmei Hou, Kate Rosenbloom, Hiram Clawson, John Spieth, LaDeana W. Hillier, Stephen Richards, George M. Weinstock, Richard K. Wilson, Richard A. Gibbs, W. James Kent, Webb Miller, and David Haussler. Evolutionarily conserved elements in vertebrate, insect, worm, and yeast genomes. Genome Research, 15(8):1034–1050, 2005. [8] Katherine S. Pollard, Melissa J. Hubisz, Kate R. Rosenbloom, and Adam Siepel. Detection of nonneutral substitution rates on mammalian phylogenies. Genome Research, 20(1):110–121, 2010. [9] Eugene V. Davydov, Daniel L. Goode, Marina Sirota, Gregory M. Cooper, Arend Sidow, and Serafim Batzoglou. Identifying a high fraction of the human genome to be under selective constraint using GERP++. PLoS Computational Biology, 6(12):e1001025, 2010. [10] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234–1240, 2020. [11] Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Charles Lau, Ryutaro Tanno, Ira Ktena, et al. Towards generalist biomedical ai. Nejm Ai, 1(3):AIoa2300138, 2024. [12] Adibvafa Fallahpour, Andrew Magnuson, Purav Gupta, Shihao Ma, Jack Naimer, Arnav Shah, Haonan Duan, Omar Ibrahim, Hani Goodarzi, Chris J. Maddison, and Bo Wang. Bioreason: Incentivizing multimodal biological reasoning within a dna-llm model, 2025. [13] Zicheng Ma, Chuanliu Fan, Zhicong Wang, Zhenyu Chen, Xiaohan Lin, Yanheng Li, Shihao Feng, Jun Zhang, Ziqiang Cao, and Yi Qin Gao. Prottex: Structure-in-context reasoning and editing of proteins with large language models, 2025.

18

[14] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Trans. Comput. Healthcare, 3(1), October 2021. [15] Juan Manuel Zambrano Chaves, Eric Wang, Tao Tu, Eeshit Dhaval Vaishnav, Byron Lee, S. Sara Mahdavi, Christopher Semturs, David Fleet, Vivek Natarajan, and Shekoofeh Azizi. Tx-llm: A large language model for therapeutics, 2024. [16] Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), 9 2022. [17] Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. Biomistral: A collection of open-source pretrained large language models for medical domains, 2024. [18] Tristan Bepler and Bonnie Berger. Learning protein sequence embeddings using information from structure. arXiv preprint arXiv:1902.08661, 2019. [19] Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):1315–1322, 2019. [20] Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C Lawrence Zitnick, Jerry Ma, et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the national academy of sciences, 118(15):e2016239118, 2021. [21] Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Fazel-Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of atomiclevel protein structure with a language model. Science, 379(6637):1123–1130, 2023. [22] Bo Chen, Xingyi Cheng, Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, et al. xtrimopglm: unified 100b-scale pre-trained transformer for deciphering the language of protein. arXiv preprint arXiv:2401.06199, 2024. [23] Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, et al. Prottrans: toward understanding the language of life through self-supervised learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):7112–7127, 2021. [24] Timothy Truong Jr and Tristan Bepler. Poet: A generative model of protein families as sequences-of-sequences. Advances in Neural Information Processing Systems, 36:77379–77415, 2023. [25] Thomas Hayes, Roshan Rao, Halil Akin, et al. Simulating 500 million years of evolution with a language model. Science, 2025. [26] Timothy Fei Truong Jr and Tristan Bepler. Understanding protein function with a multimodal retrieval-augmented foundation model. arXiv preprint arXiv:2508.04724, 2025.

19

[27] Jin Su, Yan He, Shiyang You, Shiyu Jiang, Xibin Zhou, Xuting Zhang, Yuxuan Wang, Xining Su, Igor Tolstoy, Xing Chang, et al. A trimodal protein language model enables advanced protein searches. Nature Biotechnology, pages 1–7, 2025. [28] Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. Protst: Multi-modality learning of protein sequences and biomedical texts. In International Conference on Machine Learning, pages 38749–38767. PMLR, 2023. [29] Garyk Brixi, Matthew G. Durrant, Jerome Ku, Michael Poli, et al. Genome modeling and design across all domains of life with Evo 2. bioRxiv, 2025. [30] Juan Boshar, Richard Evans, Thomas Pierrot, Peter K. Koo, Alexander Stark, Volodymyr Kuleshov, et al. A foundational model for joint sequence-function multi-species modeling at scale for long-range genomic prediction. bioRxiv, 2025. [31] Eric Nguyen, Michael Poli, Matthew G. Durrant, et al. HyenaDNA: Long-range genomic sequence modeling at single nucleotide resolution, 2023. NeurIPS 2023. [32] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range DNA sequence modeling, 2024. [33] Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. DNABERT-2: Efficient foundation model and benchmark for multi-species genome, 2023. [34] Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R. Ledsam, Agnieszka GrabskaBarwinska, Kyle R. Taylor, Yannis Assael, et al. Effective gene expression prediction from sequence by integrating long-range interactions. Nature Methods, 18(10):1196–1203, 2021. [35] Žiga Avsec et al. Advancing regulatory variant effect prediction with AlphaGenome. Nature, 649(8099):1206–1218, 2026. [36] David R. Kelley, Yakir A. Reshef, Maxwell Bileschi, David Belanger, Cory Y. McLean, and Jasper Snoek. Sequential regulatory activity prediction across chromosomes with convolutional neural networks. Genome Research, 28(5):739–750, 2018. [37] Jian Zhou and Olga G. Troyanskaya. Predicting effects of noncoding variants with deep learning–based sequence model. Nature Methods, 12(10):931–934, 2015. [38] Kathleen M. Chen, Aaron K. Wong, et al. A sequence-based global map of regulatory activity for deciphering human genetics. Nature Genetics, 54(7):940–949, 2022. [39] Johannes Linder, Divyanshi Srivastava, Han Yuan, Vikram Agarwal, and David R. Kelley. Predicting RNA-seq coverage from DNA sequence as a unifying model of gene regulation. Nature Genetics, 57(4):949–961, 2025. [40] David R. Kelley. Cross-species regulatory sequence activity prediction. PLoS Computational Biology, 16(7):e1008050, 2020. [41] Haopeng Yu, Heng Yang, Wenqing Sun, Zongyun Yan, Xiaofei Yang, Huakun Zhang, Yiliang Ding, and Ke Li. An interpretable rna foundation model for exploring functional rna motifs in plants. Nature Machine Intelligence, 6(12):1616–1625, 2024.

20

[42] Nicola Calonaci, Alisha Jones, Francesca Cuturello, Michael Sattler, and Giovanni Bussi. Machine learning a model for rna structure prediction. NAR genomics and bioinformatics, 2(4):lqaa090, 2020. [43] Tao Shen, Zhihang Hu, Siqi Sun, Di Liu, Felix Wong, Jiuming Wang, Jiayang Chen, Yixuan Wang, Liang Hong, Jin Xiao, et al. Accurate rna 3d structure prediction using a language model-based deep learning approach. Nature Methods, 21(12):2287–2298, 2024. [44] Raphael JL Townshend, Stephan Eismann, Andrew M Watkins, Ramya Rangan, Masha Karelina, Rhiju Das, and Ron O Dror. Geometric deep learning of rna structure. Science, 373(6558):1047–1051, 2021. [45] Marek Justyna, Maciej Antczak, and Marta Szachniuk. Machine learning for rna 2d structure prediction benchmarked on experimental data. Briefings in Bioinformatics, 24(3):bbad153, 2023. [46] John Jumper, Richard Evans, Alexander Pritzel, et al. Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873):583–589, 2021. [47] Minkyung Baek, Frank DiMaio, Ivan Anishchenko, et al. Accurate prediction of protein structures and interactions using a three-track neural network. Science, 373(6557):871–876, 2021. [48] Josh Abramson, Jonas Adler, Jack Dunger, et al. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 630(8016):493–500, 2024. [49] Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRxiv, pages 2022–07, 2022. [50] Kishore Jaganathan, Sofia Kyriazopoulou Panagiotopoulou, James F. McRae, Shawn F. Darbandi, David Knowles, Yang I. Li, Jack A. Kosmicki, Juan Arbelaez, Wei Cui, Guy B. Schwartz, Eric D. Chow, Elias Kanterakis, Han Gao, Amir Kia, Serafim Batzoglou, Stephan J. Sanders, and Kyle K.-H. Farh. Predicting splicing from primary sequence with deep learning. Cell, 176(3):535–548.e24, 2019. [51] Tony Zeng and Yang I. Li. Predicting RNA splicing from DNA sequence using Pangolin. Genome Biology, 23(1):103, 2022. [52] Aviya Litman, Zhicheng Pan, Ksenia Sokolova, Joyce Fang, Tess Marvin, Natalie Sauerwald, Christopher Y Park, Chandra L Theesfeld, and Olga G Troyanskaya. Variant-resolved prediction of context-specific isoform variation with a graph-based attention model. Cell Genomics, 2026. [53] David Mizrahi, Roman Bachmann, Oğuzhan Fatih Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4M: Massively multimodal masked modeling. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. [54] Liam Parker, Francois Lanusse, Jeff Shen, Ollie Liu, Tom Hehir, Leopoldo Sarra, Lucas Meyer, Micah Bowles, Sebastian Wagner-Carena, Helen Qu, Siavash Golkar, Alberto Bietti, Hatim Bourfoune, Nathan Casserau, Pierre Cornette, Keiya Hirashima, Geraud Krawezik, Ruben Ohana, Nicholas Lourie, Michael McCabe, Rudy Morel, Payel Mukhopadhyay, Mariel Pettee, 21

Bruno Regaldo-Saint Blancard, Kyunghyun Cho, Miles Cranmer, and Shirley Ho. AION-1: Omnimodal foundation model for astronomical sciences, 2025. [55] Arc Institute. OpenGenome2: a database of nearly 9 trillion base pairs of curated DNA from across all domains of life. Hugging Face dataset, 2025. Dataset: arcinstitute/opengenome2. [56] Martin Steinegger and Johannes Söding. Clustering huge protein sequence sets in linear time. Nature Communications, 9(1):2542, 2018. [57] Martin Steinegger and Johannes Söding. MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature Biotechnology, 35(11):1026–1028, 2017. [58] Nuala A. O’Leary et al. Reference sequence (RefSeq) database at NCBI: current status, taxonomic expansion, and functional annotation. Nucleic Acids Research, 44(D1):D733–D745, 2016. [59] Jonathan M. Mudge et al. GENCODE 2025: reference gene annotation for human and mouse. Nucleic Acids Research, 53(D1):D966–D975, 2025. [60] Kunting Mu, Yuhan Fei, Yiran Xu, and Qiangfeng Cliff Zhang. Rasp v2.0: an updated atlas for rna structure probing data. Nucleic Acids Research, 53(D1):D211–D219, 11 2025. [61] Rimantas Kodzius et al. CAGE: cap analysis of gene expression. Nature Methods, 3(3):211– 222, 2006. [62] The UniProt Consortium. UniProt: the universal protein knowledgebase in 2025. Nucleic Acids Research, 53(D1):D609–D617, 2025. [63] Mihaly Varadi et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research, 50(D1):D439–D444, 2022. [64] Pablo Gainza et al. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature Methods, 17(2):184–192, 2020. [65] Emmanuel Boutet, Damien Lieberherr, Michael Tognolli, Michel Schneider, and Amos Bairoch. UniProtKB/Swiss-Prot. Methods in Molecular Biology (Clifton, N.J.), 406:89–112, 2007. [66] Qingyao Huang, Damian Szklarczyk, Mingcong Wang, Milan Simonovic, and Christian von Mering. Paxdb 5.0: curated protein quantification data suggests adaptive proteome changes in yeasts. Molecular & Cellular Proteomics, 22(10), 2023. [67] Samuel Sledzieski, Meghana Kshirsagar, Minkyung Baek, Rahul Dodhia, Juan Lavista Ferres, and Bonnie Berger. Democratizing protein language models with parameter-efficient finetuning. Proceedings of the National Academy of Sciences, 121(26):e2405840121, 2024. [68] Robert Schmirler, Michael Heinzinger, and Burkhard Rost. Fine-tuning protein language models boosts predictions across diverse tasks. Nature Communications, 15(1):7407, 2024. [69] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Liang Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Iclr, 1(2):3, 2022. 22

[70] Zhangyang Gao, Hao Wang, Cheng Tan, Chenrui Xu, Mengdi Liu, Bozhen Hu, Linlin Chao, Xiaoming Zhang, and Stan Z Li. Pfmbench: Protein foundation model benchmark. arXiv preprint arXiv:2506.14796, 2025. [71] Ruian Shi, Taykhoom Dalal, Philip Fradkin, Divya Koyyalagunta, Simran Chhabria, Andrew Jung, Cyrus Tam, Defne Ceyhan, Jessica Lin, Kaitlin U. Laverty, Ilyes Baali, Bo Wang, and Quaid Morris. mrnabench: A curated benchmark for mature mrna property and function prediction. bioRxiv, 2025. [72] ESM Team. Esm cambrian: Revealing the mysteries of proteins with unsupervised learning, 2024. [73] Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. SaProt: Protein Language Modeling with Structure-aware Vocabulary. bioRxiv, page 2023.10.01.560349, 2023. [74] Philip Fradkin, Ruian Shi, Taykhoom Dalal, Keren Isaev, Brendan J Frey, Leo J Lee, Quaid Morris, and Bo Wang. Orthrus: towards evolutionary and functional rna foundation models. BioRxiv, pages 2024–10, 2025. [75] Yoichiro Sugimoto and Peter J Ratcliffe. Isoform-resolved mrna profiling of ribosome load defines interplay of hif and mtor dysregulation in kidney cancer. Nature structural & molecular biology, 29(9):871–880, 2022. [76] Kathrin Leppek, Gun Woo Byeon, Wipapat Kladwang, Hannah K Wayment-Steele, Craig H Kerr, Adele F Xu, Do Soon Kim, Ved V Topkar, Christian Choe, Daphna Rothschild, et al. Combinatorial optimization of mrna structure, stability, and translation for rna-based therapeutics. Nature communications, 13(1):1536, 2022. [77] Ken Chen, Yue Zhou, Maolin Ding, Yu Wang, Zhixiang Ren, and Yuedong Yang. Selfsupervised learning on millions of primary rna sequences from 72 vertebrates improves sequence-based rna splicing prediction. Briefings in bioinformatics, 25(3):bbae163, 2024. [78] Malgorzata Ewa Rogalska, Claudia Vivori, and Juan Valcárcel. Regulation of pre-mrna splicing: roles in physiology and disease, and therapeutic prospects. Nature Reviews Genetics, 24(4):251–269, 2023. [79] Y. Strauch, J. Lord, M. Niranjan, and D. Baralle. Ci-spliceai—improving machine learning predictions of disease causing splicing variants using curated alternative splice sites. PLOS ONE, 17(6):e0269159, 2022. [80] Natan Belchikov, Justine Hsu, Xiang Jennie Li, Julien Jarroux, Wen Hu, Anoushka Joglekar, and Hagen U. Tilgner. Understanding isoform expression by pairing long-read sequencing with single-cell and spatial transcriptomics. Genome Research, 34(11):1735–1746, 2024. [81] Chenchen Zhu, Jingyan Wu, Han Sun, Francesca Briganti, Benjamin Meder, Wu Wei, and Lars M. Steinmetz. Single-molecule, full-length transcript isoform sequencing reveals diseaseassociated rna isoforms in cardiomyocytes. Nature Communications, 12:4203, 2021. [82] Anoushka Joglekar, Wen Hu, Bei Zhang, Oleksandr Narykov, Mark Diekhans, Jordan Marrocco, Jennifer Balacco, Lishomwa C. Ndhlovu, Teresa A. Milner, Olivier Fedrigo, Erich D.

23

Jarvis, Gloria Sheynkman, Dmitry Korkin, M. Elizabeth Ross, and Hagen U. Tilgner. Singlecell long-read sequencing-based mapping reveals specialized splicing patterns in developing and adult mouse and human brain. Nature Neuroscience, 27:1051–1063, 2024. [83] Jun Inamo, Akari Suzuki, Mahoko Takahashi Ueda, Kensuke Yamaguchi, Hiroshi Nishida, Katsuya Suzuki, Yuko Kaneko, Tsutomu Takeuchi, Hiroaki Hatano, Kazuyoshi Ishigaki, Yasushi Ishihama, Kazuhiko Yamamoto, and Yuta Kochi. Long-read sequencing for 29 immune cell subsets reveals disease-linked isoforms. Nature Communications, 15:4285, 2024. [84] Carlos Alfonso-Gonzalez and Valérie Hilgers. (alternative) transcription start sites as regulators of rna processing and transcript diversity. Trends in Cell Biology, 34(12):1018–1028, 2024. [85] Carlos Alfonso-Gonzalez, Ivano Legnini, Sarah Holec, Lucia Arrigoni, H. C. Ozbulut, Fernando Mateos, David Koppstein, Agnieszka Rybak-Wolf, Ulrich Bönisch, Nikolaus Rajewsky, and Valérie Hilgers. Sites of transcription initiation drive mrna isoform selection. Cell, 186(11):2438–2455.e22, 2023. [86] T. C. Cheng, S. H. Orkin, S. E. Antonarakis, M. J. Potter, J. P. Sexton, A. F. Markham, P. J. Giardina, A. Li, and Jr. Kazazian, H. H. beta-thalassemia in chinese: Use of in vivo rna analysis and oligonucleotide hybridization in systematic characterization of molecular defects. Proceedings of the National Academy of Sciences of the United States of America, 81(9):2821– 2825, 1984. [87] Dan Lu, Xiuli Gong, Yudan Fang, Xinbing Guo, Yanwen Chen, Fan Yang, Guijun Zhao, Qingwen Ma, Yitao Zeng, and Fanyi Zeng. Correction of rna splicing defect in β654-thalassemia mice using CRISPR/Cas9 gene-editing technology. Haematologica, 107(6):1427–1437, 2022. [88] Pablo Gainza, Freyr Sverrisson, Frederico Monti, Emanuele Rodola, Davide Boscaini, Michael M Bronstein, and Bruno E Correia. Deciphering interaction fingerprints from protein molecular surfaces using geometric deep learning. Nature methods, 17(2):184–192, 2020. [89] Pablo Gainza, Sarah Wehrle, Alexandra Van Hall-Beauvais, Anthony Marchand, Andreas Scheck, Zander Harteveld, Stephen Buckley, Dongchun Ni, Shuguang Tan, Freyr Sverrisson, et al. De novo design of protein interactions with learned surface fingerprints. Nature, 617(7959):176–184, 2023. [90] Jue Wang, Sidney Lisanza, David Juergens, Doug Tischer, Joseph L Watson, Karla M Castro, Robert Ragotte, Amijai Saragovi, Lukas F Milles, Minkyung Baek, et al. Scaffolding protein functional sites using deep learning. Science, 377(6604):387–394, 2022. [91] Justas Dauparas et al. Robust deep learning-based protein sequence design using ProteinMPNN. Science, 378(6615):49–56, 2022. [92] Joseph L. Watson et al. De novo design of protein structure and function with RFdiffusion. Nature, 620(7976):1089–1100, 2023. [93] Ronny Lorenz, Stephan H Bernhart, Christian Höner zu Siederdissen, Hakim Tafer, Christoph Flamm, Peter F Stadler, and Ivo L Hofacker. Viennarna package 2.0. Algorithms for molecular biology, 6(1):26, 2011.

24

[94] Eric W. Sayers, Mark Cavanaugh, Linda Frisse, Kim D. Pruitt, Valerie A. Schneider, Beverly A. Underwood, Linda Yankie, and Ilene Karsch-Mizrachi. Genbank 2025 update. Nucleic Acids Research, 53(D1):D56–D61, 2025. [95] wwPDB consortium. Protein data bank: the single global archive for 3d macromolecular structure data. Nucleic Acids Research, 47(D1):D520–D528, 2019. [96] Xinying Song, Alex Salcianu, Yang Song, Dave Dopson, and Denny Zhou. Fast wordpiece tokenization, 2021. [97] UCSC Genome Browser. phyloP conservation scores (100-way) for hg38 (readme). http:// hgdownload.cse.ucsc.edu/goldenPath/hg38/phyloP100way/README.txt, 2026. Accessed 2026-02-18. [98] UCSC Genome Browser. phyloP conservation scores (30-way) for mm9 (readme). http: //hgdownload.cse.ucsc.edu/goldenPath/mm9/phyloP30way/README.txt, 2026. Accessed 2026-02-18. [99] Xi-Wen Wang, Chu-Xiao Liu, Ling-Ling Chen, and Qiangfeng Cliff Zhang. Rna structure probing uncovers rna structure-dependent biological functions. Nature chemical biology, 17(7):755– 766, 2021. [100] A. S. Hinrichs, D. Karolchik, R. Baertsch, G. P. Barber, G. Bejerano, H. Clawson, M. Diekhans, T. S. Furey, R. A. Harte, F. Hsu, et al. The ucsc genome browser database: update 2006. Nucleic Acids Research, 34(Database issue):D590–D598, January 2006. [101] Hao Zhao, Zhifu Sun, Jing Wang, Haojie Huang, Jean-Pierre Kocher, and Liguo Wang. Crossmap: a versatile tool for coordinate conversion between genome assemblies. Bioinformatics, 30(7):1006–1007, 2014. [102] W James Kent, Ann S Zweig, G Barber, Angie S Hinrichs, and Donna Karolchik. Bigwig and bigbed: enabling browsing of large distributed datasets. Bioinformatics, 26(17):2204–2207, 2010. [103] A. Shrake and J. A. Rupley. Environment and exposure to solvent of protein atoms: lysozyme and insulin. Journal of Molecular Biology, 79(2):351–371, 1973. [104] Patrick Kunzmann, Tom David Müller, Maximilian Greil, Jan Hendrik Krumbach, Jacob Marcel Anter, Daniel Bauer, Faisal Islam, and Kay Hamacher. Biotite: new tools for a versatile python bioinformatics library. BMC bioinformatics, 24(1):236, 2023. [105] Steven Grudman, J Eduardo Fajardo, and Andras Fiser. Optimal selection of suitable templates in protein interface prediction. Bioinformatics, 39(9):btad510, 2023. [106] Eric W. Sayers et al. Database resources of the national center for biotechnology information in 2026. Nucleic Acids Research, 54(D1):D20–D27, 2026. [107] Phil Wang (lucidrains). X transformers. https://github.com/lucidrains/x-transformers. Accessed 2026-04-17. Version 2.16.1. [108] Siavash Golkar, Alberto Bietti, Mariel Pettee, Michael Eickenberg, Miles Cranmer, Keiya Hirashima, Geraud Krawezik, Nicholas Lourie, Michael McCabe, Rudy Morel, Ruben Ohana, Liam Holden Parker, Bruno Régaldo-Saint Blancard, Kyunghyun Cho, and Shirley Ho. Contextual counting: A mechanistic study of transformers on a quantitative task, 2024. 25

[109] Hugging Face. Data collator (transformers documentation). https://huggingface.co/doc s/transformers/en/main_classes/data_collator, 2026. Accessed 2026-02-17. [110] Lukas. Efficient llm pretraining: Packed sequences and masked attention. https://hugg ingface.co/blog/sirluk/llm-sequence-packing, 2024. Published 2024-10-07; accessed 2026-02-17. [111] NVIDIA. Sequence packing (nemo framework user guide 24.09). https://docs.nvidia.co m/nemo-framework/user-guide/24.09/nemotoolkit/features/optimizations/sequence _packing.html, 2026. Accessed 2026-02-17. [112] NVIDIA. Speeding up variable-length training with dynamic context parallelism and nvidia megatron core. https://developer.nvidia.com/blog/speeding-up-variable-lengt h-training-with-dynamic-context-parallelism-and-nvidia-megatron-core, 2026. Published 2026-01-28; Accessed 2026-02-17. [113] Xinyou Wang, Zaixiang Zheng, Fei Ye, Dongyu Xue, Shujian Huang, and Quanquan Gu. Diffusion language models are versatile protein learners. arXiv preprint arXiv:2402.18567, 2024. [114] Noelia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is a deep unsupervised language model for protein design. Nature Communications, 13(1):4348, 2022. [115] Michael Heinzinger, Konstantin Weissenow, Joaquin Gomez Sanchez, Adrian Henkel, Martin Steinegger, and Burkhard Rost. ProstT5: Bilingual Language Model for Protein Sequence and Structure, July 2023. Pages: 2023.07.23.550085 Section: New Results. [116] Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein language modeling with structure-aware vocabulary. BioRxiv, pages 2023–10, 2023. [117] Yang Tan, Chen Liu, Jingyuan Gao, Banghao Wu, Mingchen Li, Ruilin Wang, Lingrong Zhang, Huiqun Yu, Guisheng Fan, Liang Hong, et al. Venusfactory: A unified platform for protein engineering data retrieval and language model fine-tuning. arXiv preprint arXiv:2503.15438, 2025. [118] Shuxian Zou, Tianhua Tao, Sazan Mahbub, Caleb N Ellington, Robin Algayres, Dian Li, Yonghao Zhuang, Hongyi Wang, Le Song, and Eric P Xing. A large-scale foundation model for rna function and structure prediction. bioRxiv, pages 2024–11, 2024. [119] Zhihan Zhou, Weimin Wu, Harrison Ho, Jiayi Wang, Lizhen Shi, Ramana V Davuluri, Zhong Wang, and Han Liu. Dnabert-s: Pioneering species differentiation with species-aware dna embeddings. Bioinformatics, 41(Supplement_1):i255–i264, 2025. [120] Weijie Yin, Zhaoyu Zhang, Shuo Zhang, Liang He, Ruiyang Zhang, Rui Jiang, Gan Liu, Jingyi Wang, Xuegong Zhang, Tao Qin, et al. Ernie-rna: an rna language model with structureenhanced representations. Nature Communications, 16(1):10076, 2025. [121] Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P De Almeida, Hassan Sirelkhatim, et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, 22(2):287–297, 2025.

26

[122] Rafael Josip Penić, Tin Vlašić, Roland G Huber, Yue Wan, and Mile Šikić. Rinalmo: Generalpurpose rna language models can generalize well on structure prediction tasks. Nature Communications, 16(1):5671, 2025. [123] Zishuo Zeng, Ariel A. Aptekmann, and Yana Bromberg. Decoding the effects of synonymous variants. Nucleic Acids Research, 49(22):12673–12691, 2021. [124] Rita Vaz-Drago, Nelson Custodio, and Maria Carmo-Fonseca. Deep intronic mutations and human disease. Human Genetics, 136(9):1093–1111, 2017. [125] Ryo Kurosawa, Kei Iida, Masahiko Ajiro, Tomonari Awaya, Mamiko Yamada, Kenjiro Kosaki, and Masatoshi Hagiwara. PDIVAS: Pathogenicity predictor for deep-intronic variants causing aberrant splicing. BMC Genomics, 24(1):601, 2023. [126] Kerstin Lindblad-Toh, Manuel Garber, Or Zuk, Michael F Lin, Brian J Parker, Stefan Washietl, Pouya Kheradpour, Jason Ernst, Gregory Jordan, Evan Mauceli, et al. A highresolution map of human evolutionary constraint using 29 mammals. Nature, 478(7370):476– 482, 2011. [127] Katherine S. Pollard, Melissa J. Hubisz, Kate R. Rosenbloom, and Adam Siepel. Detection of nonneutral substitution rates on mammalian phylogenies. Genome Research, 20(1):110–121, 2010. [128] Michael J. Cormier, Brent S. Pedersen, Pinar Bayrak-Toydemir, and Aaron R. Quinlan. Combining genetic constraint with predictions of alternative splicing to prioritize deleterious splicing in rare disease studies. BMC Bioinformatics, 23(1):482, 2022. [129] Kyle K. H. Farh, Alexander Marson, Jiang Zhu, et al. Genetic and epigenetic fine mapping of causal autoimmune disease variants. Nature, 518(7539):337–343, 2015. [130] Katherine E Deigan, Tian W Li, David H Mathews, and Kevin M Weeks. Accurate shapedirected rna structure determination. Proceedings of the National Academy of Sciences, 106(1):97–102, 2009.

27

Materials and Methods A The LORE dataset 29 A.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 A.2 Tokenization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 A.3 Nucleic acid track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 A.4 Protein track . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 A.5 Semantic & contextual modalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 A.6 Cross-track alignment, clustering, and splits . . . . . . . . . . . . . . . . . . . . . . . 35 B MIMIC architecture 36 B.1 Unified encoder-decoder backbone via cross-attention . . . . . . . . . . . . . . . . . . 36 B.2 Split-Track summation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 B.3 Register tokens and downstream representations . . . . . . . . . . . . . . . . . . . . . 38 B.4 Shared semantic embedding space . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 B.5 Localized Rotary Positional Embeddings (RoPE) . . . . . . . . . . . . . . . . . . . . 38 C Training 38 C.1 Heuristic pathway sampling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 C.2 Asymmetric budgeting and target packing . . . . . . . . . . . . . . . . . . . . . . . . 39 C.3 Staged curriculum learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 C.4 Dynamic workload balancing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 C.5 Register token self-supervision . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 C.6 Training hyperparameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 D Model evaluation 42 D.1 Sequence completion using multimodal conditioning . . . . . . . . . . . . . . . . . . 42 D.2 Protein embedding evaluation with PFMBench . . . . . . . . . . . . . . . . . . . . . 43 D.3 RNA embedding evaluation with mRNABench . . . . . . . . . . . . . . . . . . . . . 44 D.4 Model performance on variant effect prediction benchmarks . . . . . . . . . . . . . . 44 D.5 Splice site prediction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 D.6 RASP2 and context evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 D.7 SHAPE-Guided RNA secondary structure modeling and evaluation . . . . . . . . . . 48 E Multimodal generative design 49 E.1 RNA splicing design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 E.2 Protein sequence design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

28

A

The LORE dataset

Table S1: LORE modality composition summary. The number of samples for contextual modalities include sequence × context pairs. The number of sequences is reported after reduction of redundancy based on sequence similarity. Modality

Samples

Nucleic Acid Track Nucleic acid sequence Transcript annotation Splice junctions Coding sequence mask Coding sequence junctions Evolutionary conservation (phyloP) Promoter usage (CAGE) Chromatin accessibility (ATAC-Seq) RNA chemical probing (RASP2)

12,967,153 12,967,153 12,967,153 12,967,153 11,906,378 518,480 15,073,298 14,021,301 1,644,404

Protein Track Amino acid sequence Secondary structure (DSSP) Tertiary structure (backbone) Solvent accessible surface area (SASA) Chemical surface (MaSIF) Protein abundance Functional captions

15,607,838 15,607,838 15,607,838 15,607,838 1,671,206 1,803,028 176,626

Semantic & contextual modalities Taxonomic classification 12,967,153 Biomedical corpus 3,797,568

A.1

Overview

A significant challenge in multimodal biological modeling is the scarcity of aligned data. Public repositories such as GenBank [94], UniProt [62], and the PDB [95] contain extensive data, but these resources are often disjoint. To train MIMIC, we curated LORE, a multimodal dataset that aligns heterogeneous signals into a unified coordinate system. The modality sources in LORE were to provide multi-view and context-specific information across genomic, transcriptomic and proteomic layers of the central dogma. In the following sections, we provide a detailed description of the data sources, preprocessing, alignment and curation efforts that were necessary to build LORE.

A.2

Tokenization

Unless specified otherwise in their individual sections, all modalities used one of the following tokenization strategies. Character tokenization Sequences of discrete characters were tokenized at the character level, with a distinct token for each possible value (e.g. {A, C, G, U } for nucleotide sequences) plus pad, mask, and unknown tokens. Boolean/mask tracks were considered a special case of character 29

tokenization with a vocabulary of {0, 1}. Modalities with character tokenization include: nucleic acid sequence (n = 4), splicing mask (n = 2), splice junctions (n = 3), coding sequence mask (n = 2), coding sequence junctions (n = 3), coding sequence codons (n = 64), chromatin accessibility (n = 12), amino acid sequence (n = 24), secondary structure (n = 9). Continuous tokenization Several modalities are sequences of continuous values. In this case, we randomly sampled 20, 000 training samples and computed percentile bins, which were saved with the tokenizer. New samples were assigned to one of n = 100 bins and assigned the corresponding token. We also included pad, mask, and unknown tokens for missing values. Bin centers were also saved and used for detokenization. Modalities with continuous tokenization include: evolutionary conservation, RNA chemical probing, promoter usage, solvent accessible surface area, MaSIF n_vertices, MaSIF si_index, MaSIF charge, MaSIF hbond, MaSIF hydrophobicity, protein abundance Class tokenization A few modalities appy a single label to the whole sample, rather than containing a sequence. In this case the data were tokenized as a single token, corresponding to the number of label options, plus pad, mask, and unknown tokens. Modalities with class tokenization: transcript annotation (n = 7), taxonomic classification (kingdom, n = 11). Text tokenization Free text modalities were tokenized using the pre-trained WordPiece [96] tokenizer from Biobert [10] (HuggingFace: dmis-lab/biobert-base-cased-v1.2). Modalities with text tokenization include: contexts for {RASP2, ATAC-Seq, CAGE, Protein Abundance}, protein functional captions, biomedical corpus.

A.3

Nucleic acid track

LORE integrates physicochemical, evolutionary, and semantic modalities alongside standard sequencestructure pairs. The nucleic acid track aligns the RNA/DNA sequence (including dynamic flanking regions) with structural and evolutionary priors. Nucleotide sequence Along with full human and mouse genomes and accompanying annotations from GENCODE [59], we integreate nucleic acid data from over 6000 additional organisms from NCBI RefSeq [58]. We selected common model organisms including Drosophila melanogaster (fruit fly), Caenorhabditis elegans (roundworm), and Saccharomyces cerevisiae (yeast), as well as all genomes tagged as complete reference genomes by RefSeq [58]. These collected organisms span all domains of life and contain over 25 million transcripts and 123 billion nucleotides. We limit LORE to transcripts of at most 10,000 base pairs, along with any aligned associated modalities such as splice patterns, experimental assays, corresponding protein sequence and structure (see A.6). All transcripts are padded with genomic flanking regions (200 BP on either side for Bacteria/Archaea, 1000 BP upstream/300 BP downstream otherwise) so that important regulatory context regarding is available in our transcript-level data. Transcript Annotation To distinguish genomic elements, we included discrete classification labels for coding status (coding, non-coding) and feature type [58, 59]. Because these feature types are non-standard across the varied annotators hosted by RefSeq [58], we merge synonyms and significantly condense our vocabulary to produce a tractable subset: {lncRNA, miscRNA, protein_coding, pseudogene, rRNA, tRNA, other}.

30

Nucleotide-Level Annotation (Splicing) We additionally provide annotation of functional regions within the genomic transcripts assembled above [58, 59]. Binary masks for coding sequence (CDS) and spliced exons are extracted directly from annotated genomes. Explicitly labeled untranslated regions (UTRs), start codons and stop codons from GENCODE [59] are collected in a similar manner, but we do not currently use them for training MIMIC. We then construct splice junction tracks from the exon masks, separately labeling donors and acceptors to align with previous splice prediction work [30, 35, 50]. We combine donors, acceptors, transcription start sites (TSS) and transcription end sites (TES) into a single multiclass track, but we evaluate only acceptor and donor prediction when comparing. Transcript coding sequence In addition to the nucleotide-level tacks discussed above, for protein coding regions we provide a spliced RNA representation which is tokenized per 3-mer. Each codon is aligned to the associated amino acid sequence and is thus summed with other protein modalities in the split-track architecture discussed in B.2 below. Evolutionary conservation (phyloP) Conservation scores derived from phyloP to [8] describe nucleotide-level evolutionary conservation by testing whether the substitution rate at each position deviates from neutral drift under a phylogenetic model. We utilized 100-way alignments for human sequences and 30-way alignments for mouse sequences to capture lineage-specific constraints [97,98]. Chemical probing (RASP2) Transcriptome-wide chemical probing data [99] curated in RASP2 [60] provides probabilistic measures of RNA base-pairing accessibility. These assays serve as indicators of secondary structure, across various experimental contexts and can be used as a bias to in-silico RNA secondary structure prediction algorithms. Since the published chemical probing data in RASP2 is aligned to genome releases that may be different from what is published in RefSeq [58], we use the LiftOver tool [100] to correctly align experimental tracks to genomic sequence. Chromatin accessibility (ATAC-Seq) Chromatin accessibility (ATAC-seq [4]) data were obtained from the ENCODE Project [3] portal by querying all released narrowPeak BED files associated with ATAC-seq experiments. For each file, we extracted and stored accompanying metadata, including genome assembly, biosample term, release information, and accession identifier, and standardized these into a unified metadata table. Human experiments from hg38 were retained directly, whereas mouse experiments from mm10 were lifted over to mm39 using CrossMap [101]. Metadata were updated so that successfully converted mouse files were reassigned to mm39, and metadata tables were then split by species to maintain a consistent organization across downstream processing steps. To generate continuous chromatin accessibility tracks suitable for transcript-level integration, narrowPeak files were converted to bigWig [102] format using the signalValue field provided by ENCODE. Within each experiment, peak signal values were discretized into ten quantile-bins, assigning integer values from 1 to 10 to accessible regions, where higher values corresponded to stronger accessibility relative to other peaks in the same experiment. Prior to bigWig generation, peaks were sorted, overlapping intervals were merged using the maximum signal across overlaps, and intervals were clipped to chromosome boundaries using the appropriate genome-specific chromosome sizes. We then extracted per-base ATAC signal across transcript coordinates for human and mouse transcripts. For each transcript and experiment, bases overlapping an accessible region retained their corresponding rank-binned accessibility value, bases within mappable regions but lacking a peak were assigned 0 to indicate closed chromatin in that experiment, and bases that could not be 31

mapped because of missing chromosomes or transcript coordinates extending beyond chromosome boundaries were recorded as missing values. This produced a transcript-by-experiment representation of chromatin accessibility that preserved experiment-specific signal without averaging across contexts. Promoter Usage (CAGE) Cap Analysis of Gene Expression (CAGE) [61] peak data was downloaded from the FANTOM5 repository at RIKEN phase 2.6. We used TPM (Transcripts Per Million) normalized CAGE peak data for the following species and genome builds: Human: hg19, Mouse: mm9, Rat: rn6, Rhesus Macaque: rheMac8, Chicken: galGal5, Dog: canFam3. In the case where a different build was required, we mapped CAGE peak locations between genome builds using LiftOver. Metadata (cell type, replicate, sample ID) was extracted from raw CAGE data file column names. As it was unstructured, we used the following regular expressions to identify donors and biological replicate (if donor info was not found) • Donor: re.match w/ r"(.+?)[,\s]+(donor\d+|pool\d+|day\d+)$ • Replicate: r’\b(biol\_rep[l]?\d+|tech\_rep\d+|donor\d+|pool\d+|donation\d+)\b’

A.4

Protein track

LORE also contains multiple views on proteins, aligning the amino acid sequence with several structural and physicochemical properties, as well as descriptions of its function. Amino acid sequence To ensure that we had matching structures for the protein sequences in our data set, we started with ≈ 200M structures download from AlphaFoldDB [63] version 4 and extracted amino acid sequences from these structures. The UniProt ID was used as the primary identifier for joining and mapping aligned modalities within the protein track. Secondary structure Secondary structure predictions were made using the mkdssp package (https://github.com/PDB-REDO/dssp). We use the canonical 8-class secondary structure plus an unknown/loop class: α helix (H), 310 helix (G), 5 helix (I), extended strand (E), β bridge (B), turn (T), bend (S), coil (C), loop (L). Tertiary structure We retained entries from AlphaFoldDB with pLDDT > 70. Tertiary structures were tokenized using the pretrained ESM3 structure tokenizer [25] with 4096 classes based on local backbone geometry. The ESM3 structure tokenizer is a vector-quantized variational autoencoder (VQ-VAE) trained to reconstruct protein structures from AlphaFoldDB, ESMAtlas [21], and antibody-antigen complexes from the PDB. Solvent accessible surface area The solvent accessible surface area was computed on the AlphaFoldDB structures using the Shrake-Rupley “rolling-probe” algorithm [103], as implemented in the biotite Python package (version 1.4) [104]. MaSIF surface features We computed MaSIF (Molecular Surface Interaction Fingerprinting) features [88] on a subset (∼ 10%) of our protein structures due to compute limits ( 50,000 cpu hours for 1.6M datasets), prioritizing both sequence and structural diversity. These geometric fingerprints encode the electrostatic potentials, hydrophobicity and curvature of the protein surface, explicitly teaching the model to recognize interaction hotspots. To ensure consistency with residue-level 32

tokens from other modalities, we aggregate surface vertex features to the corresponding residue. Specifically, a vertex is mapped to a residue if it lies within 2.8 Å of any of its atoms, a threshold corresponding to the diameter of a solvent molecule as established in the literature [105]. Based on these proximity mappings, we construct a residue-level feature vector by computing the mean for shape index and hydrophobicity, and the sum for electrostatic charge and hydrogen bonding. Additionally, the total number of mapped vertices is recorded as a proxy for solvent-accessible surface area. Completely buried residues lacking any proximal surface vertices are assigned NaN values. Abundance We included experimental measurements of protein abundance from PaxDB v5.0 [66] as a context-dependent protein modality. The paxdb-uniprot-links-v5.0 file was used to map from PaxDB record IDs to UniProt IDs; any non-matching records were discarded. As abundance values are already normalized across different experimental conditions in PaxDB, we were able to collate records from different species into a single unified data set. Functional text We aligned sequences with natural language descriptions, including functional protein captions and gene family descriptions. The functional annotations were obtained from Swiss-Prot [62] for the fields of function, family and subcellular location. Preprocessing for captions modality removed database and publication IDs from text.

A.5

Semantic & contextual modalities

Biomedical corpus We curated a standalone text corpus derived from exclusively from biomedical literature (PubMed) [106]. While this corpus is not aligned to specific sequences, it is tokenized using the same vocabulary as the semantic tracks WordPiece [96]. This enables the model to transfer semantic reasoning learned from the broad literature to the specific gene and protein descriptions. Overall after tokenization the corpus amounts to 3,797,568 samples and 4 billion tokens. Taxonomic classification We provide each gene’s taxonomic family as plain text conditioning to MIMIC. Rather than attempting to create a standardized vocabulary for more than 600 families in our final dataset, we allow our natural language module described above to process the taxonomic information semantically. Cell line, tissue, and experimental context • RASP2: The context fields for this modality are extracted from the RASP2 database [60] and are: approach, reagent, condition, cell line and description. The description field mainly contains a summary of the other fields, sometimes misplaced fields or extra information, this field always goes last in the context string to prevent the model from looking into the future. Table S5 shows all the field combinations for human (the condition is standardized here so that the table is readable, but condition has not been standardized in our LORE dataset). • CAGE: In the CAGE peaks dataset, human represents 67.9% of the dataset and mouse 32% with 0.2% of the samples having a null species. The descriptions contain information about the tissue, cell types, cell lines and other descriptive fields about donors such as age or sex, examples are shown in Table S2.

33

• ATAC: In the ATAC dataset, human represents 85% and mouse 15% approximately. Examples of sub-contexts are shown in Table S3. • Protein Abundance: The contexts for this modality have a free text format with minimum processing directly form the dataset source. The type of information they contain are: organism, tissue, donor information, cell line or condition. Not all contexts include all of the later information. A subsample of contexts is shown in Table S4. Table S2: Context descriptions for human and mouse within the CAGE peaks dataset. We provide sample counts for each species, as well as the number of sub-contexts and several randomly sampled example contexts. Organism

Count

# Sub-Contexts

Homo sapiens (human)

32.8M

488

Mus musculus (mouse)

15.4M

106

Examples of Sub-Contexts Heart, CD34+ stem cells, Light melanocytes, HTMMT mixed Mullerian tumor cell line, Hs 5.T leiomyosarcoma cell line Intestine, Medulla oblongata, Striatal neurons, Granule cells, Spiral ganglion neurons

Table S3: Context descriptions for human and mouse within the ATAC-seq dataset. We provide sample counts for each species, as well as the number of sub-contexts and several randomly sampled example contexts. Organism Homo sapiens (human) Mus musculus (mouse)

Count

# Sub-Contexts

30,618,542 5,381,573

206 36

Examples of Sub-Contexts A549, Gm18907, Hg02884, Mcf-7, Lung Regulatory T Cell, Dendritic Cell, Heart, Megakaryocyte, Stomach

Table S4: Context descriptions within the Protein Abundance dataset. We provide the number of samples from the five top species species and 5 varied sub-context examples that are sampled randomly to illustrate diversity. This table is not exhaustive, as a total of 155 normalized species and 637 unique contexts were included. Context

Count

Homo sapiens (human)

1,752,363

Mus musculus (mouse)

483,996

Arabidopsis thaliana (thale cress) Glycine max (soybean)

367,701

Danio rerio (zebrafish)

84,782

99,396

Examples of Sub-context liver, HEK293, esophagus mucosa, A431, vitreous humor brain, kidney cortex, midbrain, placenta, spleen B cells silique, leaf, flower bud, flower, 1 d, rosette, circadian seed, matured seed, shoot, shoot, nitrogen-fed, nodule, nitrogen-fed whole organism, testis, embryo 120 h, embryo 48 hpf, whole body

34

Table S5: Unique RASP2 context combinations. We show combinations of approach, reagent, condition, and cell line in the full dataset of 118 human normed datasets. The description field is dropped from the table for readability reasons. approach

condition

cell line

in vivo

H9 HELA HEPG2 K562

in vitro chromatin in vitro chromatin in vivo cytoplasm in vitro cytoplasm in vivo nucleoplasm in vitro nucleoplasm in vivo

-

icSHAPE

NAI-N3

icLASER

Naz-N3

in vitro in vivo

-

smartSHAPE

NAI-N3

in vivo 125ng in vivo 1ng in vivo 25ng in vivo 5ng

HEK293 HEK293 HEK293 HEK293

Keth-seq

N3-kethoxal

kethoxal vs no-treat kethoxal vs no-treat in vitro

HELA HELA

denatured

FBL K562

in vitro

FBL K562

in vivo

FBL K562

in vivo

-

DMS-seq

DMS-MaPseq

A.6

reagent

DMS

DMS

Cross-track alignment, clustering, and splits

Sequence Clustering & Selection To mitigate redundancy and data leakage, we clustered the protein and RNA sequences based on sequence similarity using MMseqs2 version 87e7103d289029dc3345f85ea9a4c4c6d6416e46 [57]. We used -c 0.8 and –min-seq-id 0.3 or –min-seq-id 0.7 to obtain clusters of varying levels of similarity. 30% clusters were used for creating splits, while 70% clusters were used to ensure diversity in the training data. We selected a single representative sample per cluster, resulting in a final dataset of approximately 15.5 million protein samples and 13 million RNA samples. We show the sizes of each cluster in Table S6. Alignment We mapped UniProt IDs to their corresponding RefSeq/Ensembl IDs where possible [58, 59, 62] using the idmapping.dat file provided by UniProt release 2024_04. This yielded a core subset of approximately 2 million aligned transcript/protein pairs. For gene clusters in this aligned set, we included all documented splice isoforms to capture alternative splicing patterns [58, 59].

35

Table S6: Protein and RNA clustering sizes used in the data selection pipeline. Modality Protein Protein RNA RNA

Identity

Unique Clusters

Total Members

30% 70% 30% 70%

42,943,944 111,588,657 10,861,904 11,804,593

299,566,796 299,566,796 25,001,445 25,001,445

Table S7: MIMIC Architecture Hyperparameters. Summary of most relevant architectural choices. Architecture parameter

Value

Encoder depth Decoder depth Hidden width Attention heads Self Attention Mask

20 layers 12 layers 1536 24 per stack 50% bidirectional, 25% causal, 25% anti-causal 10,000 1,000 5 0.75

Encoder token budget Decoder token budget Register tokens RoPE fraction

Data splits Train, validation, and test splits were performed at the cluster level for both protein and RNA sequences [57]. This ensures that no sequence in the validation set shares significant similarity with the training set. All aligned modalities (e.g., structure, conservation, expression) associated with a given sequence were assigned to the same split as their parent sequence.

B

MIMIC architecture

Standard transformer architectures, designed for single-stream text or protein data, are poorly suited to efficiently model heterogeneous multimodal data. MIMIC addresses this challenge via a novel “Split-Track” architecture that sums aligned modalities while concatenating distinct molecular entities.

B.1

Unified encoder-decoder backbone via cross-attention

MIMIC is built upon the 4M/AION encoder-decoder backbone [53, 54] with hyperparameterers shown in Table S7. All the experiments in this paper use the x-transformers configuration implemented in [107] both for the encoder and decoder. MIMIC employs a full decoder that probes the encoder’s latent representation via cross-attention. The self-attention blocks of both the decoder and the encoder have 50% bidirectional and 25% causal and 25% anti-causal attention masks following the insight of [108] that causal attention can significantly outperform bidirectional attention in certain tasks.

36

Figure S1: The MIMIC Framework Architecture. (left) Split-Track Input Construction: MIMIC manages multimodal heterogeneity by grouping biologically aligned features into “Summed Tracks.” The Nucleic Acid track element-wise sums sequence sequences (Lseq ), structural data, and conservation scores into a single representation. Similarly, the Protein track utilizes the ESM3 tokenizer to sum amino acid residues with discretized 3D structure tokens, preserving vertical alignment. These biological tracks are concatenated with discrete semantic modalities (which occupy a Shared Semantic Embedding Space) and learnable Register Tokens to create the final unified token sequence. (right) Encoder-Decoder with Latent Dropout: Following a 4M-inspired backbone, the Transformer Encoder processes the unified input. A random token dropout is applied to the resulting latent state. The Transformer Decoder then probes this latent representation via cross-attention using target/query tokens to handle reconstruction objectives and generate global state summaries.

B.2

Split-Track summation

A core innovation of MIMIC is the efficient packing of biological data. Rather than flattening all modalities into a single prohibitively long sequence, we group biologically aligned features into Summed Tracks. • Nucleic Acid Track: We align the RNA/DNA sequence (Lseq ) with its corresponding structural data and conservation scores. These are projected into the same latent dimension and summed element-wise. • Protein Track We perform the same operation for the protein track with DSSP secondary structure and SASA naturally aligned with the protein sequence. For the backbone structure, we utilize the ESM3 structure tokenizer [25] to discretize 3D coordinates which are also aligned with the amino acid residues. Finally, we align MaSIF surface features [64] by assigning to each residue the sum of each surface feature’s values associated to grid points that are closer than 2.8 angstrom to the residue’s Cα . • Discrete Modalities: Semantic modalities such as cellular context descriptions or taxonimic information are treated as distinct, un-summed tracks. These are concatenated to the 37

biological tracks, allowing the model to process “metadata” and “molecular data” as separate but interacting information streams.

B.3

Register tokens and downstream representations

To aggregate a global representation of the biological state, we append learnable Register Tokens to the sequence. These tokens serve as information “sinks,” pooling features from across the diverse tracks (DNA, Protein, Context) into a compact vector summary. These register tokens also provide a convenient anchor for downstream representation learning. We recommend using register tokens together with a mean-pooled relevant track when a task requires a single vector representation, and using register tokens together with the full relevant track when the downstream task is token-level, such as residue-level or nucleotide-level prediction.

B.4

Shared semantic embedding space

To bridge the gap between hard experimental data and human-interpretable concepts, we unify all text-based modalities into a single shared embedding space. The context, functional captions, and taxonomy share weights with the biomedical corpus. This weight tying enables the model to transfer semantic reasoning learned from the large-scale corpus to the sparse context tokens.

B.5

Localized Rotary Positional Embeddings (RoPE)

We implement Rotary Positional Embeddings (RoPE) with a local group reset strategy. The positional index is reset to 0 at the start of each distinct group. Crucially, this rotation is calculated relative to the element’s position within its group, not its absolute index in the tensor. When tokens are dropped out during training, the remaining tokens retain their original positional indices, ensuring that the model preserves correct residue-to-residue distances regardless of input sparsity.

C

Training

While the MIMIC architecture provides the tensor capability to process multimodal data, the core challenge lies in optimization: how to train a single model to simultaneously predict diverse biological tasks such as protein folding structure, genomic variant effect predictor, regulatory behavior. This is complicated by the extreme heterogeneity of biological data availability. A specific gene in the LORE dataset might have high-confidence AlphaFold structure and deep evolutionary alignments but lack cell-specific ATAC-seq data; another might have extensive transcriptomic profiling but no solved protein structure. In total, we observe approximately 150 distinct “modality presence signatures,” or combinations of which tracks are available for a given sample, across the dataset. We employ a heuristic-based training strategy that simplifies these combinatorial possibilities into defined Training Pathways, optimized via an asymmetric token budget and a staged curriculum.

C.1

Heuristic pathway sampling

Rather than having a monolithic objective function, we define training as a stochastic sampling process over a set of ≈ 25 heuristic pathways. Each pathway Pk is a tuple defined as: Pk = (Ireq , Iopt , Treq , Topt , wk )

38

where I denotes Input modalities, T denotes Target modalities (to be predicted), and wk is the sampling weight. This heuristic approach allows us to group scientifically distinct tasks into simplified training routines. For example, to teach the model regulatory logic, we define pathways that mandate the presence of cellular context tags (Ireq = {RNA, Context}). However, to ensure the model remains robust to missing metadata, we also include “context-naive” pathways where the model must predict properties solely from sequence (Ireq = {RNA}). During a training step, a sample is drawn from the dataset. We then verify which pathways are valid for this sample’s available modalities. A specific pathway is selected based on the normalized weights wk , effectively up-sampling rare but high-value tasks (e.g., experimental structure prediction) relative to abundant tasks (e.g., simple sequence modeling).

C.2

Asymmetric budgeting and target packing

Biological modeling requires a massive receptive field to capture long-range interactions (e.g., enhancer-promoter loops), but the generation of dense structural data is computationally expensive. We address this with an asymmetric context window: 10,000 tokens for the Encoder and 1,000 tokens for the Decoder. To maximize training efficiency within this 1,000 token decoder limit, we employ a Target Packing strategy. Many biological targets are sparse or short (e.g., splice junction tracks or singlevalue abundance measurements). Rather than training on these in isolation, we “fill the budget” by packing multiple disjoint targets into a single decoder pass until the 1,000-token limit is reached. For instance, if the primary objective of a pathway is to predict Splice Junctions (≈ 200 tokens), and the sample also possesses evolutionary data, the data loader will automatically append the phyloP track (≈ 800 tokens) to the target sequence. This acts as a powerful regularizer, forcing the model to learn joint dependencies between splicing and conservation even when the explicit task was merely splicing prediction.

C.3

Staged curriculum learning

Training stability is critical when optimizing across such diverse modalities. We implemented a staged curriculum that progressively scales the context window from 1k → 2k → 4k → 8k → 10k tokens. This allows the model to learn local feature associations before attempting to resolve global dependencies. Crucially, the data handling strategy differs by modality during this scaling: • RNA/DNA (Crop Strategy): Nucleic acid sequences are randomly cropped to fit the current context window. Because RNA folding is often driven by local base-pairing, windowed crops remain valid training signals. • Proteins (Drop Strategy): The fold of a protein and therefore its function is a holistic global property; a truncated protein sequence cannot reliably be used for structure prediction due to missing long-range contacts. Therefore, if a protein’s structure track exceeds the current stage’s token budget, the sample is dropped entirely rather than cropped. Additionally, for the protein track, we explicitly align the coding sequence codons to the amino acid residues, treating rna_codons as a distinct track.

39

Figure S2: Workload balancing strategies for multi-scale biological sequences. (A) Naive padding incurs significant computational waste (grey tokens) due to heavy-tailed distributions, where short peptides (cool colors) are padded to match long RNAs (warm colors). (B) Sequence packing reduces padding but compromises biological integrity by truncating meaningful long-range context. (C) Dynamic Workload Balancing (ours) buckets samples by length with adaptive batch sizes. This preserves single-entity semantics and coordinate frames without truncation while maximizing memory efficiency and I/O locality via bucket affinity.

C.4

Dynamic workload balancing

Our sequence lengths span multiple biological scales, from short peptides to long non-coding RNAs. With this heavy-tailed distribution, padding-based batching (padding each batch to its longest example, Figure S2a) is extremely wasteful: a small fraction of long samples forces most batches to pay the memory and compute cost of the maximum length, dominated by padding rather than biological signal [109]. In large-scale NLP, this is often mitigated with sequence packing (example/sample packing, Figure S2b): concatenating multiple shorter examples into fixed-length packs with boundary-aware masking to reduce padding. [110, 111] In practice, packing pipelines frequently truncate or drop overflow. [111] This is incompatible with MIMIC: packing mixes multiple biological entities into one sequence, while our register tokens, targets, and multi-track alignment assume a single coherent entity and coordinate frame (e.g., CDS codons ↔ residues; coordinate-tied regulatory tracks). Truncation is also not benign: it deletes meaningful long-range context (global folding contacts; distal regulation), creating systematic supervision corruption. We therefore use length-bucketed dynamic batching aligned with the staged learning curriculum (Figure S2c). At each stage, samples are bucketed by total token length (encoder+decoder), and batch size is chosen per bucket to balance peak memory against step time. Importantly, equal-token budgets do not imply equal compute: attention cost grows superlinearly with sequence length, so longer buckets are compute-heavier even at similar token counts. This emphasis differs from Dynamic Context Parallelism [112], which addresses variable-length efficiency by adapting the degree of context parallelism at the systems level; our method is a data-side complement that reduces padding and preserves per-sample multimodal semantics. To improve input throughput, we enforce bucket affinity: each GPU (and thus each node) is pinned to a length bucket, repeatedly streaming from the same bucket-specific shard set. This 40

improves I/O locality and cache reuse (page cache / file system cache and object-store read-ahead), reducing cross-bucket thrash and stabilizing step time.

C.5

Register token self-supervision

MIMIC utilizes learnable “register tokens” to aggregate global information. To prevent these tokens from degenerating into trivial copy mechanisms, we apply a Random Token Dropout of 0–10% across all input tracks. The register tokens are trained via a reconstruction objective: they must encode sufficient information to allow the decoder to recover the masked input tokens. This forces the registers to act as compressed “state vectors” for the biological entity, capturing high-level semantic and structural properties that are robust to local noise.

C.6

Training hyperparameters

Batching follows the length-bucketed dynamic batching strategy described in Section C.4, with perbucket batch sizes chosen to balance memory usage and throughput. Training is distributed, by default, across 92 NVIDIA H200 GPUs using NCCL, although the number of GPUs used at each stage varied occasionally due to resources availability. Training proceeds for multiple epochs over the dataset with a staged curriculum that increases the maximum context length from 1k to 10k tokens. All training hyperparameters are summarized in Table S8 for shared parameters between curriculum learning stages and in Table S9 for the individual stages. Table S8: MIMIC training hyperparameters shared across stages. Summary of optimization, batching, and distributed training settings used commonly across all stages of context-length curriculum training. Shared training hyperparameter

Value

Optimizer AdamW betas AdamW epsilon Weight decay Learning-rate schedule Warmup / cooldown Numerical precision

AdamW (0.9, 0.95) 1 × 10−8 0.05 Cosine decay 8 / 10 epochs bfloat16

Table S9: MIMIC training hyperparameters by curriculum learning stages. Summary of optimization, batching, and distributed training settings used across all stages of context-length curriculum training. The minimum effective learning rate is 100× smaller than maximum effective learning rate for all stages. Training hyperparameter

1k stage

2k stage

4k stage

8k stage

10k stage

Effective max learning rate Total batch size

2.21 × 10−4 ∼ 2,000

1.89 × 10−4 ∼ 2,300

1.44 × 10−4 ∼ 2,600

1.05 × 10−4 ∼ 3,000

4.66 × 10−5 ∼ 3,100

41

D

Model evaluation

D.1

Sequence completion using multimodal conditioning

To perform the inpainting task, MIMIC is conditioned on the amino acid or nucleotide sequence, as well as additional auxiliary modalities for the length of a protein or unspliced transcript. A 100token window in the center of the sequence is masked, and MIMIC is prompted to reconstruct this missing window. Sequences are reconstructed in a single model pass, by sampling from a softmax over the model’s predicted logits (temperature = 1e − 8). All models are evaluated based on a per-token reconstruction accuracy; thus random performance for nucleotide reconstruction ≈ 0.25, and for amino acid reconstruction ≈ 0.04 Although multimodal sequence completion is a controlled benchmark rather than an biologically relevant goal, it provides a useful test of whether the model has learned the shared latent factors that constrain sequence under auxiliary observations, which informs decisions about MIMIC for both representation learning and design. Let A denote the masked nucleotide or amino-acid sequence and O a diverse set of auxiliary biophysical observations. When these observations substantially reduce uncertainty over sequence identity, the inpainting task is no longer a purely local denoising problem (i.e. when H(A | O) ≪ H(A), predicting A under conditioning by O where H(·) is the Shannon entropy). Rather, it requires inferring the latent biochemical and structural factors that jointly constrain both sequence and observation. This view suggests that success on multimodal sequence completion should translate into more informative sequence representations for other downstream targets conditioned on sequence, provided those targets depend on overlapping latent factors even if they are not directly supervised during pretraining.

Figure S3: Ablation on conditioning inputs to MIMIC. Adding additional modalities as conditioning to MIMIC significantly increases its performance on the sequence completion task, demonstrating the value of a model that can ingest a multimodal context. (C) For amino acid sequences, MIMIC sees performance gains from each structure or chemical modality, and the largest increase in performance when all are provided as conditioning. (B) For intronic sequences, the strongest inpainting performance comes from including evolutionary conservation, splicing patterns, and epigenetic information. We note that RASP data is only available for exonic sequences, which is why performance does not meaningfully change. (C) For exonic sequences where mRNA 2D structure plays a meaningful role to constrain evolution, RASP data significantly improves inpainting performance.

42

Ablation of modalities We explored the effect of ablating different input modalities in the sequence completion task to see which modalities contribute the most to model performance. For amino acid sequences (Figure S3a), the most effective single modalities are the MaSIF hydrophobicity and tokenized backbone structure, with the combination of all structural and chemical modalities leading to a significant improvement over the addition of any single modality. For intronic sequences (Figure S3b), the largest increase in performance comes from including evolutionary information (phyloP), splicing information, and epigenetics (either ATAC-seq or CAGE peaks). For exonic sequence (Figure S3c), the largest increase from an individual modality is the RASP score, which provides structural information for mature mRNA molecules. Across all sequence types, while each individual modality provided a bump in performance, we saw continual gains from adding additional modalities, indicating that each provides complimentary information allowing the model to succeed at the inpainting task. Exploring variable performance gain The improvement in performance from conditioning on an auxiliary modality is heterogeneous across different samples: in some cases, adding a modality yields substantial improvements, while in others the effect is negligible. To probe the source of this variability, we asked whether the multimodal conditioning performance increase is explained by how well the model can predict the same modality from the nucleotide or amino acid sequence. Concretely, for each masked segment used in the inpainting task, we additionally evaluate a matched modality-prediction task on the same sequence interval. For a masked 100 amino acid (or nucleotide) span of the sequence, instead of conditioning on the sequence and auxiliary modality to reconstruct the masked segment, we provide the full sequence context and prompt the model to predict the modality in the masked interval. For protein sequence completion we use MaSIF hydrophibicity as the auxiliary modality, while for nucleic acid sequence we use RASP2; these two displayed the largest individual increase in performance in our previous ablation. We then compare MIMIC’s modality-conditioned inpainting accuracy to its modality prediction accuracy. We find that multimodal conditioning is most useful where the auxiliary modality is ambiguous from sequence alone. That is, in regimes where the model has not learning anything about the sequence → modality relationship additional conditioning is unhelpful, but it is equally unhelpful when that relationship is trivial and little additional information is provided. We quantify this “ambiguous” regime by computing the top-1 and top-10 accuracy of MIMIC predicted bins for hydrophibicity (Figure S4a) and RASP2 (Figure S4b). The largest increase in inpainting accuracy appears for samples where top-10 accuracy is high while top-1 accuracy remains low. In this setting the model can narrow the auxiliary modality to a small set of plausible configurations but cannot resolve the exact state; providing the true measurement therefore removes residual ambiguity and yields the greatest improvement in reconstruction. Conversely, a high top-1 accuracy, either because the information is already present or the pattern is trivial, renders multimodal conditioning largely redundant and the performance increase diminishes.

D.2

Protein embedding evaluation with PFMBench

For each task in PFMBench [70], we trained a supervised probe on representations from MIMIC. Specifically, we prompted MIMIC with the amino acid sequence, and provided the full concatenation of the register tokens along with and protein-track outputs (preserving token-level information). We followed the training procedure outlined in the benchmark manuscript, including for hyperparameter selection. Performance metrics for competing models are taken from the benchmark manuscript. We report the 11 “representative tasks” sub-selected by the original authors. Full performance metrics for all tasks and models are reported in Table S10. 43

Figure S4: Performance gains from multimodality depends on modality predictability. (A) Each bin groups spans by AA→hydrophobicity prediction accuracy (x: top-1; y: top-10) and is colored by AA inpainting accuracy increase (uplift) from conditioning on hydrophobicity. Uplift peaks when top-10 is high but top-1 is moderate, indicating that MaSIF hydrophobicity a provides complementary (non-redundant) constraint when it is only partially predictable from sequence alone. (B) We see the same trends for RNA inpainting. We show RNA→RASP2 top-1 vs top-10 accuracy colored by accuracy uplift. Largest gains occur when top-10 is high but top-1 remains low, indicating complementary (non-redundant) constraint between sequence and RASP2.

D.3

RNA embedding evaluation with mRNABench

Likewise, for mRNABench [71] we trained a supervised probe on MIMIC representations. Because mRNABench does not have any nucleotide-level tasks, we trained the probe on the concatenation of the register tokens and a mean-pooled RNA-track representation. MIMIC was trained on unspliced transcripts, while mRNABench contains mature (spliced) mRNA sequences and, where relevant, a corresponding coding sequence track. To address this difference, we leverage the multimodal nature of MIMIC to generate representations under several conditioning cases. In the non-coding case, we generate embeddings using only nucleotide (NT) sequence. In the coding case, we generate embeddings using (1) NT sequence only, (2) NT and protein (AA) sequence (3) NT sequence, codons, and AA sequence. For each case, we then augment the mRNABench training data with additional samples containing each subset of input modalities (e.g. NT, AA, and NT+AA for case 2 above). At inference time, each subset is used to generate embeddings per sample, and the average probe prediction across all subsets of modalities is used as the model prediction for that sample. The mRNABench validation set is used to select the best inference case per task. Performance metrics for competing models are taken from prior work and are not recomputed here. Full performance metrics for all tasks and models are reported in Table S11.

D.4

Model performance on variant effect prediction benchmarks

Because MIMIC is given only a nucleotide sequence and its local multimodal context, its phyloP output should not be interpreted as a reconstruction of the underlying phylogenetic calculation. This predicted quantity is better interpreted as a sequence-derived estimate of expected evolutionary constraint: a learned mapping from local sequence context to the conservation level with which that context is most often associated in the training data. Under this interpretation, predicted values near 44

zero correspond to sequence neighborhoods that resemble weakly constrained sites, positive values correspond to neighborhoods characteristic of constrained positions, and negative values correspond to neighborhoods associated with faster-than-neutral or weakly maintained sequence classes [8]. Using predicted phyloP for variant effects Given this constraint signal, we hypothesized that MIMIC-predicted phyloP would be useful to evaluate the functional impact of genomic sequence variants. To construct an informative multimodal signal of this impact, we predict not only the phyloP for the variant sequence, but the change induced by mutation. Let pbwt bmut denote the i and p i model-predicted phyloP values at position i for the wild-type and mutant sequence, respectively, and define the mutation-response profile ∆b pi = pbmut − pbwt i i . For scalar variant scoring, we summarize this local response by the mean absolute perturbation in a centered window W = 30 nucleotides around the variant, s∆p (W ) =

1 X |∆b pi | . |W | i∈W

This variant-level formulation is important because raw conservation is not synonymous with pathogenicity. A synonymous substitution at a conserved position need not be pathogenic if it does not materially perturb splicing or other regulatory logic, whereas pathogenic splice-altering variants can arise in intronic sequences whose background conservation is only modest [123–125]. A mutation-dependent, context-aware quantity is therefore more directly aligned with variant effect than the reference-site score alone. We refer to this score throughout the manuscript as “MIMIC (phyloP VEP),” which is used for prediction of wild-type alleles in ClinVar variants (Figure S5), for the VEP benchmarks of mRNABench (Figure 2f, Table S12), and to evaluate whether MIMIC detects a splice-altering pathogenic variant (Figure 3e, middle). Pathogenicity signal in conserved and weakly conserved sequence To illustrate the value of multimodal conditioning and the MIMIC phyloP VEP score, we performed a study of variants in ClinVar. For each pathogenic location we mask the allele at this location and prompt MIMIC to predict the wild type allele, either from the local context only, or from the sequence context and phyloP score. Providing evolutionary constraint to MIMIC significantly increases the rate of correctly recovering the wild type allele (Figure S5a). This improvement is even more pronounced for pathogenic variants, suggesting that constraint is an especially strong signal in these cases. Thus, we next compared the wild type phyloP constraint with the MIMIC phyloP VEP predictor as a signal of variant pathogenicity, measured using normalized AUPR. We stratify these results by positions under positive constraint (phyloP ≥ 0) versus those under weak constraint or accelerated evolution (phyloP < 0) (Figure S5b). In the former regime, true phyloP outperforms MIMIC phyloP VEP, confirming that positive constraint is a strong pathogenicity signal. However, in the latter true phyloP loses nearly all its discriminative power, while MIMIC phyloP VEP is still able to accurately predict pathogenicity (although less so than in the highly-constrained regime). This pattern indicates complementarity rather than redundancy. Reference-site conservation is most informative in regions already under detectable evolutionary constraint, as expected from comparative genomics [126, 127]. In contrast, the mutation-induced response contributes most in sequence where background conservation is weak and pathogenicity depends more directly on 45

Figure S5: MIMIC predicted phyloP variant effect prediction provides complementary signal to raw conservation. (A) Wild-type allele exact recovery rate from ClinVar pathogenic and benign SNVs, comparing sequence-only and sequence+phyloP conditioning; phyloP conditioning yields significant gains for both variant classes. (B) Normalized AUPRC for pathogenicity prediction stratified by the sign of observed phyloP. Raw phyloP is most informative at positively constrained positions (phyloP ≥ 0), while the MIMIC phyloP VEP score retains significant predictive value at weakly conserved or accelerated positions (phyloP < 0), demonstrating complementarity between conservation and mutation-induced perturbation signals. mutation-specific disruption of local regulatory logic. This is precisely the regime in which splicealtering and other non-coding regulatory variants are often difficult to prioritize from conservation alone, despite the broad importance of non-coding disease-associated variation [50, 51, 124, 125, 128, 129]. The result therefore supports a broader interpretation of the phyloP head: it does not simply recover the observed conservation track, but learns sequence features whose perturbation is informative for functional variant effect. This mutation-aware interpretation also provides a rationale for phyloP conditioning during generation (Section 5.2). If pathogenic variants are associated with departure from the local constraint regime expected for the wild-type sequence, then conditioning design toward a wildtype-like, nonpathogenic phyloP profile should bias generation away from deleterious regulatory configurations.

D.5

Splice site prediction

We evaluate MIMIC’s ability to predict splice junctions and compare to SpliceAI [50], NTv3 [30] and AlphaGenome [35]. To enable a fair comparison despite differing native input lengths, each model is supplied with the context required by its architecture: SpliceAI receives 5kb of flanking sequence on each side of the evaluation region, AlphaGenome is queried on a 16,384 bp genomic interval containing the evaluation window, NTv3 is queried on a genomic window sized to ⌈pred_len/0.375⌉ (rounded up to the next multiple of 128 bp) and centered on the target region, and MIMIC consumes the full sequence up to 10kb directly. MIMIC’s splice junction head outputs a 5-class distribution over non-splice, donor, acceptor, TSS, TES at each position. Since the prediction region excludes TSS and TES anchor positions by construction, we compute per-position donor and acceptor probabilities via a 2-class softmax 46

restricted to non-splice, donor and non-splice, acceptor logits respectively. For both tasks described below, we evaluate model performance using Area Under the Precision and Recall Curve (AUPR). Since this task exhibits extreme class imbalance (∼99.85% of all nucleotide positions are not splice sites), we require a metric that can measure how well the models identify rare events. Additionally, a threshold-independent ranking metric like AUPR is appropriate because we are interested not only in whether the model assigns higher scores to true splice sites than to non-splice sites, but also in how well those scores support discrimination across decision thresholds. Since all models are trained with different held out portions, finding an overlapping test set greatly reduces the number of samples available. Therefore, all results presented here are on MIMIC’s test set, but not necessarily the test sets of SpliceAI, NTv3, and Alphagenome. Gene-level Splice Prediction For the gene-level splicing task, we assemble a dataset of human coding and non-coding transcripts, and collect all possible splice junctions from every transcript belonging to a gene. For each gene, the evaluation region spans from the earliest transcription start site (TSS) to the latest transcription end site (TES) across all of its transcripts, and every position within this span is labeled as a donor, an acceptor, or neither based on the union of junctions observed in any transcript. Because many genes exceed MIMIC’s context length, we restrict this exploration to genes with length < 25kb, and for genes longer than 10kb we evaluate all models on a common 10kb region with the most splice junctions. Samples are stratified by gene length to ensure balanced coverage across length bins, and stratified bootstrapping is used to construct error bars. Results are shown in Figure 3a above and Table S13 below. Transcript-level Conditional Splice Prediction For the transcript-level task, we group transcripts that share the same TSS and TES into a single mimic-gene, treating each mimic-gene as one evaluation unit. Donor and acceptor labels are defined as the union of junctions across all transcripts in the group, restricted to the interval between the shared TSS and TES. We first evaluate all models on this new dataset in the same unconditioned mode as in the gene-level task. Then, MIMIC is explicitly conditioned on the TSS and TES positions: the boundary tokens marking the start and end of the transcript are provided as input, along with 200 bp of flanking genomic context on each side. As before, samples are stratified by transcript length to balance coverage across length bins and stratified bootstrapping provides error bars. Detailed results are given in Figure 3b3 above and Table S13 below.

D.6

RASP2 and context evaluation

We further analyze the contribution of contextual and multimodal conditioning to RASP2 score prediction in a setting where sequence information is partially observed and complemented by auxiliary tracks, including phyloP and splicing patterns (Figure S6). In contrast to the analysis in Figure 5, we evaluate on a modified dataset in which each input RNA sequence is restricted to a contiguous window covering 50% of its full length. This controlled reduction in sequence information enables a more direct assessment of how non-sequence modalities contribute to prediction performance. In this regime, the model must rely on phyloP and splicing signals to compensate for missing sequence context when inferring RASP2 scores. This setup amplifies the relative contribution of auxiliary modalities compared to the full-sequence setting, where sequence information alone can dominate prediction. We observe that conditioning on phyloP and splicing jointly with the available sequence context yields the largest improvement in RASP2 prediction accuracy, highlighting the complementary nature of evolutionary and regulatory signals in constraining RNA structure. 47

Figure S6: MIMIC-predicted RASP2 scores under partial sequence observation and multimodal conditioning. Predictions are conditioned on a contiguous window covering 50% of the input RNA sequence, together with auxiliary modalities. Conditioning on phyloP and splicing, in addition to sequence context, yields the largest performance improvement over sequence-only baselines. All sequences are filtered to length < 10k nucleotides.

D.7

SHAPE-Guided RNA secondary structure modeling and evaluation

To evaluate the biological utility of our predicted per-nucleotide reactivity scores, we assessed whether they can serve as effective surrogate experimental data for guiding thermodynamic RNA secondary structure prediction. We utilized RNAfold from the ViennaRNA package [93] incorporating SHAPE (Selective 2’-hydroxyl acylation analyzed by primer extension) reactivity constraints, comparing the resulting predicted structures against structures generated using experimental measured reactivities. Crucially, in this evaluation, the reference is defined as the minimum free energy (MFE) secondary structure computed by RNAfold when constrained by actual experimental chemical probing data, rather than experimentally derived 3D structures. Evaluation Dataset We randomly sampled 1,000 RNA transcripts from a held-out validation set. For a transcript of length L, each sample consists of the raw RNA sequence; ground-truth per-nucleotide reactivity values, denoted rgt ∈ RL , derived from experimental measurements (e.g., icSHAPE or related chemical probing technologies); and predicted per-nucleotide reactivity values, denoted r̂ ∈ RL , output by MIMIC for the same transcript conditioned on the corresponding experimental context. Prior to folding, both the ground-truth and predicted reactivity scores are strictly normalized to a scale of 0.0 to 1.0. SHAPE-Guided Folding Procedure For each of the 1,000 transcripts, three independent secondary structure predictions were performed using RNAfold (version 2.7.2): 1. Ground-truth SHAPE run (run_gt): RNAfold was invoked with the ground-truth reactivity file via the –shape flag. The reactivity file was formatted as a two-column tab-delimited text file containing 1-based nucleotide indices and reactivity values. 2. Predicted SHAPE run (run_pred): RNAfold was invoked identically, but the –shape flag pointed to MIMIC’s predicted reactivity values (r̂) in the same format. 3. Sequence-only baseline (run_seq_alone): RNAfold was invoked with no SHAPE data, 48

folding purely on the basis of thermodynamic nearest-neighbor free energies. This serves as the baseline against which SHAPE-guided improvements are measured. The –shape flag instructs RNAfold to incorporate per-nucleotide reactivities as pseudo-energy penalties using the model proposed by Deigan et al. [130]. This model adds a pseudo-energy term ∆GSHAPE (i) to the folding free energy of each paired nucleotide i: ∆GSHAPE (i) = m · ln(ri + 1) + b

(1)

where ri is the reactivity, and m and b are empirically determined slope and intercept parameters (using ViennaRNA default values). The minimum free energy (MFE) secondary structure is extracted from each RNAfold output as the dot-bracket string. Base Pair Extraction and Evaluation Metrics Secondary structures in dot-bracket notation are parsed into 0-based base pair sets P ⊂ {(i, j) | i < j} using a standard stack-based algorithm. We evaluated the predicted (P̂ from run_pred) and sequence-only (P seq from run_seq_alone) base pair sets against the ground-truth reference (P gt from run_gt). For a given prediction P, True Positives (TP), False Positives (FP), and False Negatives (FN) are defined as: TP = |P ∩ P gt |,

FP = |P \ P gt |,

FN = |P gt \ P|

(2)

Performance is quantified using Positive Predictive Value (PPV) and Sensitivity: PPV =

TP , TP + FP

Sensitivity =

TP TP + FN

(3)

yielding the harmonic mean, the F1 score (where PPV, Sensitivity, and F1 default to 0 if denominators are 0): 2 · PPV · Sensitivity F1 = (4) PPV + Sensitivity The structural improvement provided by MIMIC is measured as the difference in F1 scores: ∆F1 = F1 (P̂, P gt ) − F1 (P seq , P gt )

(5)

where ∆F1 > 0 indicates that incorporating predicted reactivities yields a structure closer to the experimental reference than sequence-alone folding. To assess whether the improvement is statistically significant at the pop 1000 ulation level, we applied the Wilcoxon signed-rank test to the paired F1 scores F1pred , F1seq i=1 across all 1,000 transcripts. We report a one-sided p-value testing the alternative hypothesis that F1pred > F1seq . This was obtained by halving the two-sided p-value (applicable given that the observed median of ∆F1 is positive). Statistical Testing

E

Multimodal generative design

E.1

RNA splicing design

Leaning on the strong RNA multimodal inpainting performance, we adopt a similar methodological approach for RNA design. We evaluate designed windows of both 30 nucleotides (Figure 3e) and 49

50 nucleotides (Figure S7). For a given window of width w centered at position x, we masked the interval [x − w2 , x + w2 ] from the full unspliced transcript sequence and provided this sequence as conditioning to MIMIC. In addition, we provided the full splice junctions track, with the wild type acceptor and donor specified, and (optionally) the full wild-type phyloP as conditioning. Designs were generated in a single model pass, where nucleotides were sampled from a softmax over the model predicted logits at each position (temperature = 1e − 8). We sample designs from windows centered every 5 nucleotides from −300 to −130, every 1 nucleotide from −130 to +50, and every 5 nucleotides from +50 to +200 from the pathogenic site. In order to avoid trivially reverting the pathogenic mutation, we avoid designing in a w2 -width window around the mutation location. Designs are evaluated using SpliceAI [50] as an independent oracle, since MIMIC-predicted splicing would be circular with MIMIC designs. For each designed sequence, we query SpliceAI for the probability of an acceptor at the cryptic donor (x = 0) and acceptor (x = −75) sites.

Figure S7: Splicing design with a 50 nucleotide designable window.

E.2

Protein sequence design

To explore the value of multimodal conditioning, we design amino acid sequences under five different conditioning cases: backbone only, surface features (MaSIF) for the 40% of residues closest to the binding site (measured in 3D distance), all surface features (MaSIF), and the full backbone plus 40%/100% of MaSIF features. Backbone and surface features were computed using the downloaded 50

PDB structures (PD-L1, 4ZQK chain A; hACE2, 6VW1 chain A). We adopt an iterative design strategy wherein MIMIC is used for both generation and verification. In the generation step, a sequence is sampled from a softmax over logits from a single forward pass (temperature = t). We then prompt MIMIC with the designed sequence to predict the auxiliary modalities M̂ (either backbone, surface features, or both). At each position x, we compute whether M̂i matches the true conditioning modality Mi with acceptance threshold a. For all positions such that |M̂i − Mi | > a, these positions are marked for re-generation in the following iteration. We also randomly sample r additional satisfied tokens for re-generation. Finally, with probability p we resample positions in regions where the same amino acid is repeated ≥ 2 times. This allows MIMIC flexibility to explore the design space without getting trapped in local maxima. Iteration proceeds for a maximum of 10 cycles, with the possibility to exit early if all positions are satisfied. We treat this design problem as a hyperparameter search over temperature t, acceptance threshold a, and resampling rate r, and repeat resample probability p. We uniformly sample over {linear, cosine, exponential} annealing strategies from tstart ∈ {1, 1.5, 2, 3} to tend ∈ {0.001, 0.01, 0.05, 0.1, 0.5}, a ∈ {0.15, 0.2, 0.25, 0.3}, and {linear, cosine, exponential} annealing strategies from rstart ∈ {0.25, 0.3, 0.4, 0.5, 0.7, 0.9} to rend ∈ {0.02, 0.05, 0.08, 0.1}, and p ∈ {0.99, 1}. For each of the five conditioning cases described above, we randomly sample 200 hyperparameter sets and generate a designed sequence per set, then use ESMFold [21] as a fast proxy of quality to filter to the top 20 designs for each case. Designs are evaluated along several axes. We first use AlphaFold2 to predict the fold for the generated sequence, filtering for high-confidence designs (average pLDDT > 85). To assess overall structural fidelity, we compute the TM-score of the AlphaFold2-predicted structure against the native structure. Alongside backbone topology, we quantify the conservation of surface properties by defining a MaSIF similarity score between the sequence-aligned predicted and native structures. To compute this score, we isolate jointly surface-exposed residues, retaining only positions mapped to at least one surface vertex in both structures. Across these shared surface residues, we calculate the Pearson correlation coefficient for each of the four biophysical features (shape index, electrostatic charge, hydrogen-bonding, and hydrophobicity). The final similarity score is the arithmetic mean of these four correlations, linearly rescaled to a range of 0 to 1. Consequently, a score of 1.0 indicates perfectly correlated surface chemistry, 0.0 indicates perfect anti-correlation, and 0.5 denotes no net correlation (this neutral value is also assigned by default if fewer than five jointly exposed residues are available for comparison). Finally, for these high-confidence designs, we evaluate their predicted binding using the AlphaFold3 web API and report the interface predicted template modeling score (iPTM) between chains.

51

52

Antibiotic Resistance

0.688

0.687 0.634 0.584 0.673 0.673 0.684 0.690 0.634 0.593 0.687 0.658 0.646

Model

MIMIC

DPLM [113] ESM2 [21] ESM3 [25] ESM-C [72] PGLM [22] ProtGPT2 [114] ProstT5 [115] ProtST [28] ProTrek [27] ProtT5 [23] SaProt [116] VenusPLM [117]

0.755 0.736 0.648 0.717 0.747 0.697 0.768 0.718 0.764 0.762 0.751 0.752

0.595

EC #

Function

0.758 0.762 0.659 0.754 0.748 0.703 0.732 0.749 0.839 0.726 0.740 0.738

0.746

Localization

0.757 0.764 0.813 0.768 0.728 0.494 0.814 0.685 0.774 0.780 0.824 0.716

0.808

Secondary Structure

0.294 0.321 0.157 0.300 0.331 0.148 0.130 0.066 0.049 0.186 0.248 0.339

0.307

Stability

Structure

0.174 0.137 0.225 0.207 0.169 0.172 0.166 0.189 0.192 0.197 0.166 0.168

0.267

BindingDB

0.137 0.147 0.156 0.147 0.169 0.135 0.183 0.195 0.173 0.201 0.155 0.165

0.204

PDBbind

Interaction

0.701 0.712 0.703 0.702 0.745 0.712 0.721 0.515 0.804 0.721 0.763 0.702

0.736

Metal Ion Binding

0.812 0.806 0.774 0.810 0.836 0.777 0.799 0.807 0.826 0.785 0.812 0.832

0.849

Clone CLF

0.801 0.812 0.775 0.810 0.795 0.768 0.816 0.693 0.815 0.801 0.811 0.820

0.835

Material Production

0.828 0.845 0.781 0.842 0.822 0.789 0.819 0.820 0.834 0.787 0.844 0.828

0.857

Solubility

Developability

Table S10: Model performance on PFMBench. For each task, the best performing model is in bold and the second best is italic. Metrics: Antibiotic resistance (Accuracy), Enzyme Commission (EC) number (F1 score), Localization (F1 score), Secondary structure (Accuracy), Stability (AUROC), BindingDB (Spearman), PDBbind (Spearman), Metal Ion Binding (Accuracy), Cloning CLF (AUROC), Material Production (Accuracy), Solubility (AUROC).

Table S11: Model performance on mRNABench. For each task, the best performing model is in bold and the second best is italic. Metrics: eCLIP (AUPR), GO (AUPR), Protein Localization (AUPR), RNA Localization (AUPR), MRL LBKWK (Pearson), MRL Sugimoto (Pearson), RNA Half Life (Pearson). Function

Localization

Translation Regulation

Model

eCLIP

GO

Protein Localization

RNA Localization

MRL (LBKWK)

MRL (Sugimoto)

RNA Half Life

MIMIC

0.470

0.475

0.415

0.756

0.585

0.496

0.623

AIDO-RNA [118] Dilated ResNet [71] DNABERT2 [33] DNABERT-S [119] ERNIE-RNA [120] Evo2 [29] HyenaDNA [31] NT-v2 [121] Orthrus [74] RiNALMo [122] SpliceBERT [77]

0.420 0.421 0.371 0.374 0.373 0.473 0.374 0.391 0.465 0.393 0.377

0.366 0.155 0.326 0.318 0.341 0.467 0.308 0.350 0.435 0.347 0.382

0.343 0.169 0.325 0.311 0.338 0.408 0.311 0.336 0.396 0.327 0.360

0.734 0.645 0.675 0.685 0.691 0.767 0.670 0.719 0.789 0.690 0.726

0.534 0.411 0.534 0.393 0.503 0.507 0.506 0.512 0.633 0.466 0.514

0.360 0.448 0.286 0.292 0.328 0.455 0.287 0.297 0.460 0.416 0.336

0.555 0.647 0.475 0.467 0.518 0.661 0.462 0.530 0.696 0.532 0.531

Table S12: Variant effect prediction tasks from mRNABench. For each task, the best performing model is in bold and the second best is italic. All metrics reported are AUPR. Model

Complex Variants

Mendelian Variants

MIMIC MIMIC (phyloP VEP)

0.142 0.253

0.263 0.500

AIDO-RNA [118] Dilated ResNet ERNIE-RNA [120] Evo2 [29] DNABERT2 [33] DNABERT-S [119] HyenaDNA [31] NT-v2 [121] Orthrus [74] RiNALMo [122] SpliceBERT [77]

0.092 0.080 0.097 0.081 0.093 0.070 0.064 0.064 0.071 0.086 0.104

0.540 0.585 0.481 0.561 0.435 0.542 0.320 0.540 0.574 0.546 0.460

53

Table S13: Model performance on human splice-site prediction. For each task, the best performing model is in bold and the second best is italic. All metrics reported are AUPR. Gene-level Model MIMIC (unconditioned) MIMIC (conditioned) AlphaGenome NTv3 SpliceAI

Transcript-level

Coding

Non-coding

Coding

Non-coding

0.905 ± 0.009 – 0.898 ± 0.009 0.818 ± 0.015 0.879 ± 0.014

0.641 ± 0.015 – 0.572 ± 0.014 0.156 ± 0.014 0.562 ± 0.014

0.847 ± 0.009 0.874 ± 0.008 0.803 ± 0.009 0.766 ± 0.011 0.804 ± 0.009

0.669 ± 0.011 0.701 ± 0.011 0.531 ± 0.015 0.272 ± 0.018 0.577 ± 0.012

54