M3 -Gen: Interpretable Multimodal Generation of Gene Expression Profiles Using Clinical and Imaging Data Francesca Pia Panaccione1[0009−0005−8007−963X] , Carlo Sgaravatti1[0009−0001−4962−5365] , and Marco Venere1[0009−0002−8991−1443]
arXiv:2607.21343v1 [cs.LG] 23 Jul 2026
DEIB - Dipartimento Elettronica, Informazione e Bioingegneria, Politecnico di Milano, Milan, 20133, Italy, {francescapia.panaccione,carlo.sgaravatti,marco.venere}@polimi.it
Abstract. Integrating heterogeneous biomedical data—including clinical metadata, histopathology images, and molecular profiles—is crucial for comprehensive disease understanding. However, gene expression data acquisition remains constrained by high costs and privacy concerns, limiting its use in multimodal research and AI-driven applications. We present MultiModal Molecular Generation (M3 -Gen), a novel framework for the generation of gene expression profiles by conditioning a Generative Adversarial Network on histopathology images and clinical metadata. M3 -Gen learns a unified latent representation from the clinical variables and the images, leveraging contrastive learning, and exploits the embeddings of the two modalities to guide a generative model in producing biologically coherent gene expression profiles. Evaluations on the TCGA dataset demonstrate that M3 -Gen generates realistic and functionally meaningful gene expression data. Importantly, by integrating multiple modalities in an attention-based mechanism, M3 -Gen provides intrinsic explainability: it allows the identification of which regions of the histopathology images most strongly influenced the generation of specific gene expression profiles, making the model’s decisions interpretable by design. Code will be available at: https://github.com/CarloSgaravatti/M3-Gen. Keywords: Generative AI · Multimodal Learning · Synthetic Gene Expression · Explainability · LLM · Deep Learning · Computer Vision.
1
Introduction
Biomedical research increasingly relies on integrating heterogeneous data types— molecular profiles, clinical metadata, and medical images—to build richer models of biological systems and disease mechanisms. In practice, however, these modalities are unevenly available: while clinical records and histopathology slides are routinely collected and standardized, gene expression profiles remain costly, privacy-sensitive, and limited in scale [9]. This imbalance not only constrains the scope of multimodal studies but also prevents researchers from fully exploiting the rich contextual information contained in clinical and morphological data.
2
F.P. Panaccione et al.
Generative AI—from Generative Adversarial Networks (GANs) to denoising diffusion models—presents a promising solution. By learning complex, highdimensional data distributions, these methods can produce realistic synthetic transcriptomic profiles that augment scarce cohorts and safeguard patient privacy [2]. Despite early successes in generating gene-expression data (e.g., Panaccione et al. [11], Vinas et al. [15], Lacan et al. [6]), most approaches rely on limited inputs and overlook critical clinical and histological context. To overcome these limitations, we introduce M3 -Gen (MultiModal Molecular Generation), a unified framework that learns a shared latent representation of clinical variables and histopathology images to produce biologically coherent gene expression profiles in silico. To the best of our knowledge, we are the first to address the problem of conditioning a generative model of gene expressions on both images and text. In contrast, existing methods such as [14,17] focus on directly predicting gene expression from pathology images, without leveraging textual clinical information or generative modeling. This distinction is important: while prediction is inherently a deterministic task, generative modeling provides a mechanism to explore previously unseen combinations of clinical and histopathological inputs, enabling the generation of multiple plausible omics profiles corresponding to the same conditions. Furthermore, as demonstrated in this work, M3 -Gen inherently supports explainability. By integrating multimodal inputs, the model captures the most relevant visual patterns in the histopathology images in relation to the clinical context when generating a specific gene expression profile. This allows tracing which aspects of the input data contributed most to the output, and observing the correlations and interconnections across the three data modalities (clinical variables, imaging, and gene expression), providing insights into the underlying biological mechanisms. In our parallel line of work, we explored a more complex fusion architecture aimed at increasing representational capacity [12]. In contrast, the present manuscript deliberately focuses on interpretability through an attention-based fusion mechanism, enabling patch-level analysis of histopathology images in the gene expression generation process. We evaluate M3 -Gen using standard metrics from the generative modeling literature—such as distributional alignment and downstream predictive performance—to demonstrate its ability to produce realistic, functionally meaningful gene expression profiles.
2
Data and Methods
At a high level, M3 -Gen consists of three main stages: (i) a preprocessing phase, where we extract and filter image patches from tissue slides and summarize clinical metadata into compact textual descriptions; (ii) a contrastive pretraining step, where we align visual and textual representations in a shared embedding space; and (iii) a generative model, where a Wasserstein GANs with Gradient Penalty (WGAN-GP) [4] is conditioned on these multimodal embeddings to synthesize gene expression profiles. As illustrated in Figure 1b, an Image Encoder embeds patches of tissue slides, whose mean representation is aligned, via
M3 -Gen: Multimodal Generation of Gene Expression Profiles
3
contrastive learning, with the embedding produced by a Text Encoder from the clinical description. These embeddings are then integrated through an attention mechanism to produce a single multimodal embedding. Finally, this multimodal embedding is used to condition the WGAN-GP, by concatenating it to the input noise of the generator and to the real or generated gene expression in input to the discriminator.
Tile Embeddings Eimg
Otsu Thresholding
Mean eimg
Image Encoder
Contrastive Loss
Text Embedding etext
{ “info” : { “gender”: …, “age”: …, }, “disease_type”: …, “diagnosis”: { “disease”: …, “stage”: …, }
“Generate a gene expression for a patient with a diagnosis of …”
Bio-Medical Llama
“Generate a gene expression for a patient with a diagnosis of …”
Text Encoder
}
(a) Preprocessing
(b) Contrastive Pretraining Generator
Gaussian Noise z
Tile Embeddings Eimg
Generated Samples
Concat
...
...
...
Keys & Values Attention𝐺
G êimg
Text Embedding etext
Concat Score ...
...
Attentionⅅ
...
Query
Concat
D êimg
Multimodal Embeddings Real Samples
Discriminator
(c) Conditional generation with WGAN-GP
Fig. 1: The overall pipeline of M3 -Gen. In the preprocessing step (a), we extract patches from tissue slides using the Otsu thresholding algorithm to select the patches with a percentage of background pixels higher than 80%, and we extract textual descriptions from patients’ metadata with an LLM. We then pre-train an image encoder and a text encoder to align their embeddings with contrastive learning (b). Finally, an attention mechanism is used to combine the embeddings of the patches and of the text to condition a WGAN-GP (c).
4
F.P. Panaccione et al.
2.1
Data
The input data for our method consist of paired histopathology whole-slide images (WSIs), clinical metadata, and matched gene expression profiles, all retrieved from TCGA public repository 1 and focused on twelve different tumor types. WSIs are ultra-high-resolution images of tissue sections, which are computationally infeasible to analyze in full, and are therefore typically subdivided into smaller tiles for downstream processing. Clinical metadata includes patient-specific information such as demographics, cancer subtype, and treatment history, all related to the disease condition. Gene expression profiles, obtained from RNA sequencing, are quantified using FPKM (Fragments Per Kilobase of transcript per Million mapped reads), providing normalized measures of gene activity specific to the disease context. 2.2
Preprocessing
To prepare data for multimodal generation, we segment tissue regions from pathology slide thumbnails using Otsu Thresholding [10] and extract multi-scale high-resolution tiles (128–1024 px), keeping only those tiles with more than 20% of tissue content. For clinical context, we convert structured metadata into concise case summaries using a quantized instruction-tuned language model. We used a version of Llama3-8B fine-tuned on medical data 2 [1]. Irrelevant fields are removed, and the remaining data are serialized into prompts, resulting in 200-word descriptions that capture disease site, demographics, and experimental conditions. 2.3
Constrastive Pretraining
To enable effective multimodal conditioning without the need for end-to-end training of the entire model, we first align visual and textual representations in a shared embedding space using contrastive pretraining. Contrastive learning is a technique that aims to bring the embeddings of semantically similar data points from different modalities (e.g., an image and its corresponding text) closer together in a shared d-dimensional latent space, while pushing unrelated pairs further apart. Since image patches and clinical descriptions originate from different modalities but are correlated to the same patient, we learn modality-specific encoders with semantically aligned representations. We fed as input to the Image Encoder a random subset of N patches extracted from the tissue slides, producing as output an embedding matrix Eimg ∈ RN ×d . The Text Encoder, instead, embeds the clinical textual description into etext ∈ Rd . We then compute the mean eimg ∈ Rd of the patch embeddings and align etext and eimg using the Information Noise-Contrastive Estimation (InfoNCE) loss 1 2
https://www.cancer.gov/ccg/research/genome-sequencing/tcga https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B
M3 -Gen: Multimodal Generation of Gene Expression Profiles
function, as defined in CLIP [13]: 1 sim(eimg , etext ) sim(etext , eimg ) L= CE + CE , 2 τ τ
5
(1)
where sim(a, b) = a⊤ b denotes cosine similarity after ℓ2 normalization, and τ is a temperature hyperparameter. The cross-entropy loss encourages each image to match its paired text, and vice versa. 2.4
Generative Model
To synthesize gene expression profiles conditioned on both visual and textual patient information, we employ a Conditional WGAN-GP architecture, currently a standard for transcriptomic data synthesis due to its training stability and capacity to model complex biological distributions. The generator takes as input a latent noise vector z ∼ N (0, I) concatenated with a multimodal embedding derived from the contrastive pretraining step. Specifically, we reuse patch-level embeddings Eimg ∈ RN ×d and clinical text embeddings etext ∈ Rd . WSIs might contain patches that refer to the diagnosed disease and other patches that are not relevant. Thus, to capture the most relevant visual patterns in relation to the clinical context, we apply a multi-head attention mechanism, using etext as the query and Eimg as keys and values. This produces an attention-weighted image d embedding êG img ∈ R , which is concatenated with the random latent vector to form the input to the generator. For the discriminator, we apply the same attention mechanism, conditioning on êD img . The discriminator takes as input both real and generated gene expression profiles and is trained to distinguish whether a sample is real or generated. This makes our method inherently interpretable, as we can compute attention maps that can be directly inspected to quantify the contribution of each image patch in conditioning the generated transcriptomic profiles. This provides a biologically meaningful explanation of which tissue regions drive the synthesis process. Attention maps are directly connected to the attention weights given as output by the multi-head attention layer of the generator. Given a multi-head attention layer with H heads, the weights α(h) ∈ RN for head h are computed as: ! (WhQ etext )⊤ (WhK Eimg ) (h) √ α = softmax , (2) dh where WhQ , WhK ∈ Rdh ×d are learnable projection matrices that map, respectively, the textual query etext and the visual keys Eimg into a common subspace of dimension dh . To obtain a single interpretable relevance score per patch, we compute the mean of the attention weights across all heads: H
α=
1 X (h) α , H h=1
(3)
6
F.P. Panaccione et al.
resulting in α ∈ RN , where each element αi represents the relative importance of the i-th patch embedding in conditioning the generator. These weights can be visualized as an attention map over the WSI patches, highlighting the most influential regions for the synthesized gene expression profile.
3
Experiments
We test our method on the TCGA dataset, comparing our results with standard generative models. The dataset provides both tissue slides and clinical descriptions of each patient, associated with a gene expression profile derived from RNA-seq. For storage and computational purposes, we select only a subset of TCGA by keeping only the samples having a tissue slide of dimension less than 70 MB, for a total of 1224 clinical cases. 3.1
Experimental Setup
We performed our experiments on a machine with the AMD Ryzen 1950X CPU, with 128 GB of RAM, using two NVIDIA A6000 GPUs with 48 GB of VRAM. We employed PyTorch 3 version 2.6.0 to define our model and perform the experimental evaluation, and CUDA version 12.4 for model training on the GPU devices. We use UNI [3] as an Image Encoder, which offers a pretrained Vision Transformer for histopathology tiles. As a Text Encoder, we exploit Clinical ModernBERT [8] 4 5 . We add a linear projection layer to the embeddings of these two models to have the same embedding size for both modalities. In our experiments, we use an embedding size of 128. 3.2
Evaluation Metrics
We evaluate our approach with three classes of metrics: (i) unsupervised metrics (Precision, Recall [5] and Correlation [7]), (ii) detectability, which consist of training a classifier to detect whether samples are real or generated (lower Accuracy and F1-score means high similarity between real and generated data), and (iii) utility, which assesses the performance (Accuracy and F1-score) of models trained on synthetic data and tested on real data to classify the disease type of a patient from RNA-seq. 3.3
Contrastive Pretraining Results
Our feature extractors, UNI and Clinical ModernBERT, are pretrained, respectively, on WSI data and medical descriptions. Thus, to align their representations with contrastive learning, we freeze the pretrained layers and train only the linear https://pytorch.org/ https://huggingface.co/Simonlee711/Clinical_ModernBERT 5 Both UNI and Clinical ModernBERT are accessible through Huggingface 3 4
M3 -Gen: Multimodal Generation of Gene Expression Profiles
7
projection. At each training step we select 16 random patches to be fed as input to the Image Encoder. We train the model for 20 epochs using a batch size of 8 and the Adam optimizer, with a learning rate of 0.0001. Figure 2 shows the effectiveness of our contrastive training approach, using t-SNE as a dimensionality reduction tool for visualization, where we analyze the separability of different disease types in our embedding space. Specifically, it shows that text embeddings can separate well the different diseases, while image embeddings, computed as the mean of all the patches, are noisier. Differently, by selecting the 32 most similar patches to the text embedding and computing the mean of these embeddings we can still separate well the embedding space using images. This insight gives an additional motivation to use attention in our conditional model. Image embeddings
Text embeddings
32 patches most similar to text
40 20 0 20 40 40
20
0
20
40
40
Squamous Cell Neoplasms Adenomas and Adenocarcinomas Cystic, Mucinous and Serous Neoplasms Acinar Cell Neoplasms
20
0
20
40
Ductal and Lobular Neoplasms Gliomas Paragangliomas and Glomus Tumors Transitional Cell Papillomas and Carcinomas
40
20
0
20
40
Nevi and Melanomas Mature B-Cell Lymphomas Germ Cell Neoplasms Mesothelial Neoplasms
Fig. 2: TSNE visualizations of the embeddings after pre-training with contrastive learning. Different colors indicate different disease types.
Furthermore, in Figure 6, we provide a more detailed visualization of the image embeddings after contrastive pretraining, including representative patches for each class, which highlights how the contrastive learning objective structures the latent space according to morphological similarity, grouping patches with comparable tissue organization or cellular composition while maintaining clear separation between different disease classes. 3.4
Conditional WGAN-GP Results
We train our Conditional WGAN-GP for 500 epochs using a latent dimension of 256 and the Adam optimizer with a starting learning rate of 0.0005, which decreases by a factor of 0.5 every 50 epochs. Table 1 presents the results for the unsupervised and detectability metrics, comparing a vanilla WGAN-GP model without conditioning against our proposed strategy, which employs a multi-head attention mechanism to integrate multimodal information. To further assess the effectiveness of this approach, we conduct an ablation study considering three variants: single-modality conditioning
8
F.P. Panaccione et al.
Fig. 3: t-SNE visualization of image embeddings obtained after contrastive pretraining. The figure not only shows the distribution of embedding points in the latent space but also includes representative image patches for each class. This visualization highlights how the contrastive learning approach effectively separates images according to both class-specific and morphological features, producing well-separated clusters that capture meaningful relationships within the data.
(image-only or text-only) and a simple multimodal configuration that uses the mean of the text and image embeddings obtained after contrastive learning. From the unsupervised metrics, the benefits of multi-head attention are particularly evident in the detectability results, where the logistic regression classifier reaches approximately 71% accuracy and 76% F1 score. This indicates that attention-based conditioning produces samples that are more realistic and less easily distinguishable from real data. The model conditioned on the mean embedding, while simpler, achieves the best balance between precision and recall—maintaining a recall of around 86% without compromising precision—suggesting that it effectively captures a broader portion of the data distribution. However, as shown in the subsequent tables, our attention-based conditioning strategy demonstrates its full potential in real-world utility evaluations. The relatively lower performance on unsupervised metrics may reflect the model’s ability to generate realistic yet novel samples that do not merely replicate the training data but instead provide meaningful variations, which is particularly valuable from both a privacy and ethical standpoint.
M3 -Gen: Multimodal Generation of Gene Expression Profiles
9
Table 1: Results of unsupervised and detectability metrics (Accuracy and F1Score), computed as the mean of 5 runs. We compare M3 -Gen with a WGAN-GP conditioned on the disease type. We denote with Image Only and Text Only the single-modal models trained by concatenating to the noise only the image or text embedding, while in Mean we compute the mean of the two. Best results are in bold. Precision ↑ WGAN-GP 0.890 M3 -Gen(Img Only) 0.820 M3 -Gen(Text Only) 0.885 M3 -Gen(Mean) 0.822 3 M -Gen 0.763 Method
Recall ↑ 0.778 0.834 0.781 0.859 0.820
Correlation Accuracy ↑ (MLP) ↓ 0.871 0.927 0.887 0.862 0.886 0.814 0.880 0.840 0.880 0.869
F1-Score (MLP) ↓ 0.926 0.851 0.804 0.829 0.858
Accuracy (LR) ↓ 0.851 0.758 0.762 0.776 0.713
F1-Score (LR) ↓ 0.869 0.804 0.804 0.816 0.767
Table 2: Utility evaluation with Random Forest (RF) and Multi Layer Perceptron (MLP), classifying the disease type given the gene expression. TRTR means training only on the real data, and TSTR involves training on the generated data. T(S+R)TR involves using both data sources to train. The Augmentation Factor (Aug. Factor ) represents how many samples are generated for each real sample. Training Data Aug. Factor TRTR TSTR TSTR T(S+R)TR T(S+R)TR
1 10 1 10
Accuracy (MLP) ↑ 0.880 0.883 0.897 0.889 0.894
F1-Score (MLP) ↑ 0.870 0.871 0.883 0.878 0.883
Accuracy (RF) ↑ 0.902 0.901 0.908 0.908 0.909
F1-Score (RF) ↑ 0.882 0.880 0.893 0.892 0.894
We further evaluate the practical utility of our model by training Random Forest (RF) and Multi-Layer Perceptron (MLP) classifiers to predict disease type from gene expression profiles. We adopt three training strategies: TRTR (training only on real data), TSTR (training only on generated data), and T(S+R)TR (training on both real and synthetic data). Results in Table 2 show that models trained on generated data alone achieve comparable performance to those trained on real data, and that augmenting the dataset with ten synthetic samples per real instance further improves predictive accuracy. The combined T(S+R)TR setup consistently attains the highest scores across classifiers compared toTRTR, suggesting that the inclusion of synthetic data can provide modest but consistent benefits for downstream prediction, even when real-world labelled data is available. Finally, within the TSTR setting (Table 3), we assess the influence of the different conditioning strategies, as in the unsupervised evaluation. The attentionbased model consistently achieves the highest accuracy and F1 scores across classifiers, outperforming both the unconditioned WGAN-GP and the singlemodality variants. In contrast, the simple mean-based fusion performs worse than using either modality alone, confirming that effective multimodal integration requires a mechanism that can selectively weight complementary information
10
F.P. Panaccione et al.
Table 3: Comparison of the utility (TSTR) of M3 -Gen with the same benchmarks of Table 1. Accuracy (MLP) ↑ WGAN-GP 0.853 M3 -Gen(Img Only) 0.863 M3 -Gen(Text Only) 0.874 M3 -Gen(Mean) 0.870 M3 -Gen 0.883 Method
F1-Score (MLP) ↑ 0.847 0.851 0.863 0.861 0.871
Accuracy (RF) ↑ 0.891 0.894 0.901 0.899 0.901
F1-Score (RF) ↑ 0.867 0.872 0.879 0.880 0.880
Table 4: Agreement between differentially expressed (DE) genes identified from real and generated data for the contrast Squamous Cell Neoplasms vs Adenomas and Adenocarcinomas. Regulation Top-N genes Overlap Jaccard similarity Up-regulated 100 89 0.80 Down-regulated 100 86 0.75
rather than collapsing it into an averaged representation. Overall, these findings highlight the importance of attention-driven fusion in leveraging both modalities to improve the quality, realism, and downstream utility of the generated data. 3.5
Gene- and Pathway-Level Biological Coherence
To further support the biological coherence of the generated transcriptomic profiles, we performed an additional gene- and pathway-level comparison between real and synthetic data on a representative disease-type contrast (Squamous Cell Neoplasms vs Adenomas and Adenocarcinomas) using the held-out test split (containing 83 and 102 samples for the two diseases). Differential expression consistency. For each gene, we compared the two disease groups using a two-sided non-parametric Mann–Whitney U test on logtransformed expression values (log2 (FPKM + 1)), with multiple-testing correction performed using the Benjamini–Hochberg procedure. We selected the top-100 most deregulated genes separately for up- and down-regulation and repeated the same analysis on synthetic profiles generated by M3 -Gen for the same disease contrast. A quantitative summary of the agreement between real and generated data is reported in Table 4, showing a strong overlap for both up-regulated (89/100, Jaccard = 0.80) and down-regulated genes (86/100, Jaccard = 0.75). These results indicate that disease-associated transcriptional differences are largely preserved in the generated profiles. Pathway-level agreement. To assess whether this gene-level consistency translates into coherent biological processes, we performed enrichment analysis on the derived gene sets using a standard pathway gene-set library. The comparison of the
M3 -Gen: Multimodal Generation of Gene Expression Profiles
11
Top pathways (Enrichr) Squamous Cell Neoplasms vs Adenomas and Adenocarcinomas Real upregulated Generated upregulated Estrogen signaling pathway Staphylococcus aureus infection Amoebiasis Renin secretion Phototransduction Melanogenesis Signaling pathways regulating pluripo... Long-term potentiation Amphetamine addiction p53 signaling pathway
Estrogen signaling pathway Staphylococcus aureus infection Renin secretion Phototransduction Tyrosine metabolism Amoebiasis Long-term potentiation Glycolysis / Gluconeogenesis Amphetamine addiction p53 signaling pathway
0.0
0.5
1.0 1.5 -log10(adj p)
2.0
0
Real downregulated
1
2 -log10(adj p)
3
Generated downregulated
Maturity onset diabetes of the young Proximal tubule bicarbonate reclamation Metabolism of xenobiotics by cytochro... Bile secretion Fat digestion and absorption Pancreatic secretion Glycolysis / Gluconeogenesis Retinol metabolism PPAR signaling pathway Gastric cancer
Maturity onset diabetes of the young Fat digestion and absorption Glycolysis / Gluconeogenesis Proximal tubule bicarbonate reclamation PPAR signaling pathway Butanoate metabolism Bile secretion Fructose and mannose metabolism Starch and sucrose metabolism Tyrosine metabolism 0
1
2
3 4 5 -log10(adj p)
6
0
1
2
3 4 5 -log10(adj p)
6
Fig. 4: Top enriched pathways for the contrast Squamous Cell Neoplasms vs Adenomas and Adenocarcinomas, computed from the top-100 up-regulated (top row) and down-regulated (bottom row) genes. Results are shown for real data (left) and synthetic profiles generated by M3 -Gen (right). Bar lengths represent pathway significance as − log10 adjusted p-values, highlighting a strong agreement between real and generated data.
top enriched pathways obtained from real and generated data reveals a substantial overlap among the top-10 pathways (8/10, Jaccard = 0.67). Representative results for both up- and down-regulated genes are reported in Figure 4, highlighting that synthetic data capture pathway-level trends consistent with those observed in real samples. 3.6
Explainability Analysis
To demonstrate the intrinsic interpretability of our multimodal architecture, we explore the generator’s inner workings through its multi-head attention mechanism. As visualized in Figure 5, the attention weights provide a direct visualization of how the model integrates histopathological context when synthesizing gene expression profiles. This analysis provides insight into the model’s decision-making, highlighting the regions of the WSI that have contributed most to the generation process. We examined the attention weights for ten randomly selected tissue slides to assess how the model prioritizes different patches. The resulting maps highlight distinct hotspots (shown in warm colors), indicating the regions that the generator relies on most heavily. Importantly, this confirms that the attention
12
F.P. Panaccione et al.
Fig. 5: Attention maps generated by the multi-head attention mechanism for ten representative histopathology slides. Warmer colors (yellow–red) highlight tissue regions assigned higher attention weights, indicating greater influence on the synthesis of the corresponding gene expression profiles. These visualizations illustrate the interpretability of M3 -Gen, revealing how the model selectively focuses on biologically relevant structures within each slide when generating molecular data.
Table 5: Robustness evaluation of M3 -Gen under common histopathology image artifacts: out-of-focus (OOF), stain variation (SV) and tissue folding (TF). Results are the mean of 5 independent runs. Artifact OOF
SV
TF
✓ ✓ ✓
✓
✓ ✓
Unsupervised Detectability Precision Recall Correlation Accuracy F1-Score ↑ ↑ ↑ (LR) ↓ (LR) ↓ 0.763 0.820 0.880 0.713 0.767 0.806 0.778 0.875 0.757 0.803 0.812 0.755 0.864 0.742 0.788 0.812 0.749 0.873 0.729 0.773 0.841 0.702 0.851 0.707 0.751
Utility Accuracy F1-Score (RF) ↓ (RF) ↓ 0.901 0.880 0.900 0.878 0.899 0.876 0.900 0.878 0.899 0.876
mechanism can selectively identify biologically relevant tissue structures that drive gene expression. By establishing a direct, traceable link between microscopic image features and molecular outputs, this approach enhances the interpretability and transparency of our generative framework.
3.7
Robustness to Common Histopathology Artifacts
In real-world clinical settings, histopathology whole-slide images are often affected by acquisition and preparation artifacts, such as out-of-focus regions, staining variability, or tissue folding. These artifacts may introduce noise or distort morphological patterns, potentially impacting the reliability of downstream models. To assess the robustness of M3 -Gen under such non-ideal conditions,
M3 -Gen: Multimodal Generation of Gene Expression Profiles
13
Fig. 6: Examples of simulated histopathology image artifacts used in the robustness analysis. From left to right: no artifacts (clean image), out-of-focus blur, stain variation, tissue folding, and combined artifacts (out-of-focus, stain variation, and tissue folding). Artifacts are synthetically applied at the patch level to mimic common real-world degradations observed in clinical whole-slide images.
we conducted a robustness analysis by evaluating the model in the presence of common histopathological artifacts. Specifically, we considered three widely observed artifact types: out-of-focus blur, staining variations, and tissue folding. Each artifact was synthetically introduced at the image level prior to feature extraction, while keeping the clinical metadata and gene expression profiles unchanged. For stain variation and tissue folding, we followed the same procedures of [16] and applied them to each patch of the WSI independently. To simulate, instead, out-of-focus regions, we apply a Gaussian blur to random regions of the patches. ?? reports some visualizations of the simulated artifacts. We designed five experimental settings: (i) no artifacts (i.e. clean images), (ii) out-of-focus only, (iii) stain variation only, (iv) tissue folding only, and (v) a combined scenario where all three artifacts are present simultaneously. For each setting, we evaluated M3 -Gen following exactly the same protocol adopted in Tables 1 and 2, evaluating the model with unsupervised metrics, detectability and utility. All metrics were computed as the mean over five independent runs, ensuring direct comparability with the main experimental results. Table 5 reports the results and indicates that M3 -Gen is highly robust to moderate image degradations. Across all artifact configurations, performance remains largely stable, with only marginal variations compared to the artifactfree baseline. A closer inspection of Table 5 highlights a few consistent trends.
14
F.P. Panaccione et al.
Precision and recall exhibit a trade-off across different artifact configurations, with improvements in one metric typically accompanied by slight decreases in the other, indicating minor shifts in distribution coverage rather than a loss of overall fidelity. Among individual artifacts, tissue folding has the strongest impact, although the degradation remains limited. As expected, the configuration combining all three artifacts yields the lowest performance. Nevertheless, even in this most challenging setting, results remain superior to those obtained with the unconditioned WGAN-GP reported in Tables 1 and 2. This suggests that histopathology images remain informative even when moderately corrupted, and that the multimodal design of M3 -Gen, leveraging both visual and clinical textual information, provides additional robustness under realistic imaging conditions. These findings support the applicability of M3 -Gen in realistic clinical scenarios, where image imperfections are unavoidable, and further strengthen its potential for deployment in practical biomedical research pipelines.
4
Conclusions
This work introduced a multimodal generative framework that conditions gene expression synthesis on both clinical metadata and histopathology images. Our aim was to generate gene expression profiles that are not only statistically realistic but also clinically coherent for real-world applications. Our results demonstrate that the model performs strongly across standard generative evaluation metrics, producing synthetic gene expression profiles that closely match real data distributions. Compared to existing generative approaches for gene expression profiles, our model not only improves realism and utility in downstream predictive tasks but also introduces an interpretable conditioning mechanism providing clear insights into which regions of the images are most influential in producing specific synthesized profiles. This interpretability is particularly valuable, as it opens the door for future studies to generate pathological gene expression profiles and investigate the morphological features most relevant to their manifestation. From a clinical and translational standpoint, M3 -Gen is not intended to replace molecular profiling, but rather to support research and development workflows in which gene expression data are scarce, costly, or constrained by privacy considerations. By enabling the generation of biologically coherent transcriptomic profiles conditioned on routinely available data, namely histopathology images and clinical metadata, the proposed framework facilitates in silico experimentation, data augmentation for downstream models, and exploratory analyses without requiring additional wet-lab assays. Moreover, the intrinsic interpretability of the multimodal attention mechanism provides a principled way to relate tissue morphology to molecular variation, supporting hypothesis generation in computational pathology and precision medicine settings. Future work will focus on benchmarking M3 -Gen against additional generative models and further leveraging its multimodal foundation. A promising direction is bidirectional synthesis: generating virtual histopathology images from gene expression inputs and studying how changes in one modality affect the other.
M3 -Gen: Multimodal Generation of Gene Expression Profiles
15
This approach enables in silico experiments that are privacy-preserving and cost-effective, offering new opportunities to explore the molecular consequences of patient context and tissue morphology.
Acknowledgements Model training and testing were possible thanks to the HPC grant from by the Ministry of Education, Youth and Sports of the Czech Republic through the e-INFRA CZ (ID:90254)
References A high-performance biomedical language model. 1. Bio-medical: https://huggingface.co/ContactDoctor/Bio-Medical-Llama-3-8B (2024) 2. van Breugel, B., Liu, T., Oglic, D., van der Schaar, M.: Synthetic data in biomedicine via generative artificial intelligence. Nature Reviews Bioengineering 2(12), 991–1004 (2024) 3. Chen, R.J., Ding, T., Lu, M.Y., Williamson, D.F., Jaume, G., Chen, B., Zhang, A., Shao, D., Song, A.H., Shaban, M., et al.: Towards a general-purpose foundation model for computational pathology. Nature Medicine (2024) 4. Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., Courville, A.C.: Improved training of wasserstein gans. Advances in neural information processing systems 30 (2017) 5. Kynkäänniemi, T., Karras, T., Laine, S., Lehtinen, J., Aila, T.: Improved precision and recall metric for assessing generative models. Advances in neural information processing systems 32 (2019) 6. Lacan, A., Sebag, M., Hanczar, B.: GAN-based data augmentation for transcriptomics: survey and comparative assessment. Bioinformatics 39(Supplement_1), i111–i120 (2023) 7. Lacan, A., Sebag, M., Hanczar, B.: Gan-based data augmentation for transcriptomics: survey and comparative assessment. Bioinformatics 39(Supplement_1), i111–i120 (06 2023). https://doi.org/10.1093/bioinformatics/btad239, https://doi. org/10.1093/bioinformatics/btad239 8. Lee, S.A., Wu, A., Chiang, J.N.: Clinical modernbert: An efficient and long context encoder for biomedical text. arXiv preprint arXiv:2504.03964 (2025) 9. Liu, H., Zhang, Y., Luo, J.: Contrastive learning-based histopathological features infer molecular subtypes and clinical outcomes of breast cancer from unannotated whole slide images. Computers in Biology and Medicine 170, 107997 (2024) 10. Otsu, N., et al.: A threshold selection method from gray-level histograms. Automatica 11(285-296), 23–27 (1975) 11. Panaccione, F.P., Mongardi, S., Masseroli, M., Pinoli, P.: Biogan: Enhancing transcriptomic data generation with biological knowledge. Bioengineering 12(6), 658 (2025) 12. Panaccione, F.P., Sgaravatti, C., Pinoli, P.: Gemm-gan: A multimodal generative model conditioned on histopathology images and clinical descriptions for gene expression profile generation. In: Rodolà, E., Galasso, F., Masi, I. (eds.) Image Analysis and Processing - ICIAP 2025 Workshops. pp. 393–404. Springer Nature Switzerland, Cham (2026)
16
F.P. Panaccione et al.
13. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PmLR (2021) 14. Schmauch, B., Romagnoni, A., Pronier, E., Saillard, C., Maillé, P., Calderaro, J., Kamoun, A., Sefta, M., Toldo, S., Zaslavskiy, M., et al.: A deep learning model to predict rna-seq expression of tumours from whole slide images. Nature communications 11(1), 3877 (2020) 15. Viñas, R., Andrés-Terré, H., Liò, P., Bryson, K.: Adversarial generation of gene expression data. Bioinformatics 38(3), 730–737 (2022) 16. Wang, N.C., Kaplan, J., Lee, J., Hodgin, J., Udager, A., Rao, A.: Stress testing pathology models with generated artifacts. Journal of Pathology Informatics 12(1), 54 (2021). https://doi.org/https://doi.org/10.4103/jpi.jpi_6_21, https://www. sciencedirect.com/science/article/pii/S2153353922001766 17. Zheng, Y., Pizurica, M., Carrillo-Perez, F., Noor, H., Yao, W., Wohlfart, C., Marchal, K., Vladimirova, A., Gevaert, O.: Digital profiling of cancer transcriptomes from histology images with grouped vision attention. BioRxiv pp. 2023–09 (2024)