arXiv CSopen access

Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

arXiv:2604.08333v1 [cs.CV] 9 Apr 2026

Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification Xun Zhu

Fanbin Mo

Xi Chen

Department of Electronic Engineering Tsinghua University Beijing, China [email protected]

School of Artificial Intelligence BUPT Beijing, China [email protected]

Department of Electronic Engineering Tsinghua University Beijing, China [email protected]

Kaili Zheng

Shaoshuai Yang

Yiming Shi

Department of Electronic Engineering Tsinghua University Beijing, China [email protected]

Jian Gao

Miao Li∗

Ji Wu∗

College of AI Tsinghua University Beijing, China [email protected]

Department of Electronic Engineering Tsinghua University Beijing, China [email protected]

Department of Electronic Engineering College of AI, Tsinghua University Beijing, China [email protected]

ABSTRACT

CCS CONCEPTS

The rise of multimodal large language models (MLLMs) has sparked an unprecedented wave of applications in the field of medical imaging analysis. However, as one of the earliest and most fundamental tasks integrated into this paradigm, medical image classification reveals a sobering reality: state-of-the-art medical MLLMs consistently underperform compared to traditional deep learning models, despite their overwhelming advantages in pre-training data and model parameters. This paradox prompts a critical rethinking: where exactly does the performance degradation originate? In this paper, we conduct extensive experiments on 14 open-source medical MLLMs across three representative image classification datasets. Moving beyond superficial performance benchmarking, we employ feature probing to track the information flow of visual features module-by-module and layer-by-layer throughout the entire MLLM pipeline, enabling explicit visualization of where and how classification signals are distorted, diluted, or overridden. As the first attempt to dissect classification performance degradation in medical MLLMs, our findings reveal four failure modes: 1) quality limitation in visual representation, 2) fidelity loss in connector projection, 3) comprehension deficit in LLM reasoning, and 4) misalignment of semantic mapping. Meanwhile, we introduce quantitative scores that characterize the healthiness of feature evolution, enabling principled comparisons across diverse MLLMs and datasets. Furthermore, we provide insightful discussions centered on the critical barriers that prevent current medical MLLMs from fulfilling their promised clinical potential. We hope that our work provokes rethinking within the community—highlighting that the road from high expectations to clinically deployable MLLMs remains long and winding.

• Computing methodologies → Machine learning; Computer vision; • Networks → Network performance evaluation; • Applied computing → Life and medical sciences.

∗ Corresponding authors.

INTRODUCTION

The emergence of multimodal large language models (MLLMs) has marked a paradigm shift in artificial intelligence, seamlessly integrating visual understanding with linguistic reasoning [22, 23]. Leveraging the vast knowledge embedded in pre-trained language models, coupled with the capacity to interpret complex visual patterns through supervised fine-tuning (SFT), recent advances have extended MLLMs into the domain of medical imaging analysis. In this field, medical MLLMs have demonstrated remarkable adaptability and competitive performance across diverse clinical tasks, ranging from medical visual question answering (VQA) [20], radiology report generation [43], lesion detection [24] to multi-task joint optimization [50]. These achievements have ignited unprecedented enthusiasm for their potential to revolutionize clinical workflows, bringing the vision of democratizing access to expert-level diagnostic capabilities tantalizingly within reach. As one of the earliest and most fundamental tasks integrated into the MLLM paradigm, medical image classification sits at the core of computational diagnosis, serving as the prerequisite for lesion detection, disease staging, and treatment planning [25]. Historically, traditional deep learning models such as convolutional neural networks (CNNs), multi-layer perceptrons (MLPs) and vision transformers (ViTs) have established robust baselines on specialized datasets. Naturally, researchers have developed high-yield performance expectations (hype) for medical MLLMs: the belief that massive pre-training data and billions of parameters should

ArXiv, Preprint, 2026

Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, and Ji Wu (a) BUSI

95.00 90.00 85.00 80.00 75.00 70.00 65.00 94.00

(b) COVID19-CT

90.00 86.00 82.00 78.00 74.00 70.00 98.00

93.00 88.00 83.00 78.00 73.00

Re M sne o Ef b i le t5 0 fc n i e et nt V2 ne tR B0 C o epV nv GG ne In C x c e on t-S pt vM io i nN xe r e F a Xt M ste S L P rN -M e ix t er S2 -B H -M ir L C y eM P cle LP M -B C r LP os - B sV 4 BiF i T o - 18 Gr F rm ou as er pM te r B ix ViT Fo rm 2 er -B S Co CN nv et M ML oC P Tr oV S an i T sX 1 .0 N et H iF - B C o Me us nv dV e-S M S d M am H Me L P b a ua dV M H tuo LM i xe ua gp r tu t- -R1 og V -2 B pt isi - V on isi - 7B on Li - 3 ng 4 B Sh Li sh iz ng u-7 B Sh he sh iz nG u-3 he PT 2 nG -7 B PT BV H -3 2 L ulu BV H Me L ulu dH - M 4B ulu e d H Me 7B ulu d M Me 14 B ed dM Ge 32 m e B M dG m a e d em -4 Ge m B m a-2 m a- 7 B 1. 54B

68.00

Figure 1: Performance comparison of state-of-the-art medical MLLMs and traditional deep learning methods across three medical image classification scenarios. Model categories are distinguished by background color: CNN-based (green), MLP-based (yellow), ViT-based (blue), hybrid-based (red), and MLLM-based (gray).

inherently yield superior discriminative power. However, emerging evidence and discussions reveal an unsettling discrepancy that medical MLLMs consistently underperform compared to traditional deep learning methods. The performance gap has been observed across diverse classification scenarios, including thoracic pathology diagnosis [10], knee osteoarthritis grading [38], intracranial hemorrhage subtyping [41], skin cancer classification and respiratory disease diagnosis [28]. Our preliminary experiments further corroborate these findings. As illustrated in Figure 1, current state-ofthe-art medical MLLMs struggle to match the average performance of traditional deep learning methods. Crucially, this degradation persists across medical imaging modalities, suggesting a systematic architectural limitation rather than dataset-specific anomalies. The paradox mentioned above calls for a systematic investigation into the genesis and progression of performance degradation. In the general domain, Zhang et al. [49] attribute the deficiency to the scarcity of fine-grained alignment data during multimodal pretraining, while Fu et al. [11] diagnose brittleness in task prompts and underutilization of visual representations by the LLM. In the medical domain, Jeong et al. [17] question the efficacy of medical adaptation. However, these pioneering works fall short of systematically dissecting how failures originate, propagate, and are either mitigated or exacerbated as visual information traverses the MLLM pipeline. Without a fine-grained, layer-wise diagnosis that traces information flow across each module, efforts to improve the classification capability of medical MLLMs risk treating symptoms rather than root causes.

In this paper, we present the first systematic dissection of classification performance degradation in medical MLLMs. Through extensive experiments on 14 open-source medical MLLMs across three representative classification tasks—ultrasound breast cancer classification, computed tomography (CT) COVID-19 diagnosis, and chest X-ray pneumonia screening—we move beyond superficial output-level benchmarking to diagnose the root causes of failure. Specifically, we design a suite of probing experiments that track the information flow of visual features module-by-module and layer-bylayer throughout the entire MLLM pipeline, from the vision tower, through the connector, to the LLM and its final semantic output. This enables us to explicitly visualize where and how classification signals are distorted, diluted, or overridden. Our diagnostic journey reveals four critical failure modes: 1) quality limitation in visual representation; 2) fidelity loss in connector projection; 3) comprehension deficit in LLM reasoning; and 4) misalignment of semantic mapping. To enable quantitative comparisons, we introduce wellcrafted scores that characterize the healthiness of feature evolution throughout the MLLM pipeline. This quantitative lens enables objective cross-model comparisons of information preservation or loss across diverse modules and datasets, transforming subjective performance variations into measurable diagnostic criteria. Beyond these diagnostic findings, we provide insightful discussions on three counter-intuitive barriers that critically hinder medical MLLMs including the limited impact of medical adaptation, the suboptimal state of vision tower, and the comprehension-utilization entanglement in LLM reasoning. We hope that our discussions can redirect the field’s focus from the scale-centric hype of amassing data and parameters toward targeted innovations in architecture and mechanism design, thereby unlocking the transformative potential of MLLMs in clinical medicine. In a nutshell, the contributions of this paper can be summarized as follows: • A systematic diagnosis. We conduct the first layer-wise, modulelevel dissection of classification performance degradation in medical MLLMs, revealing exactly where and how visual information is distorted throughout the pipeline. • A taxonomy of failure modes. We identify and distill our observations into four critical failure modes, providing a unified vocabulary for diagnosing medical MLLMs. • Apples-to-apples comparisons. We design novel scores that characterize the healthiness of feature evolution, offering a unified standard for fair cross-model evaluation. • Insightful discussions. We reveal three intriguing barriers, with the hope of redirecting the field’s focus from scaling toward architectural and mechanistic innovations that unlock clinical potential.

2 RELATED WORK 2.1 Medical MLLM MLLMs typically consist of three core components [52]: a vision tower for raw visual feature extraction; a connector for cross-modal alignment; and an LLM backbone for auto-regressive text generation. This architecture enables MLLMs to function as a generalpurpose interface for diverse visual and linguistic tasks. The success of general-domain MLLMs has spurred a surge of interest in adapting them to the medical domain [46], broadly following two development routes. The first route assembles off-the-shelf vision towers

Lost in the Hype

ArXiv, Preprint, 2026

Medical Image Classification Datasets BUSI

Normal

COVID19-CT

Benign

Malignant

Target

Modality

Non-COVID

Split

Normal

COVID train test

630 150

train test

598 148

Target Modality Split

Large Language Model

Target Modality

5216 640

MedVLM-R1-2B Huatuogpt-Vision-7B/34B Lingshu-7B/32B ShizhenGPT-7B/32B-VL Hulu-Med-4B/7B/14B/32B MedGemma-4B/27B MedGemma-1.5-4B

HireMLP-B CycleMLP-B4

ViTbased

CrossViT-18 BiFormer-B

FasterViT-2 GroupMixFormer-B

CNNbased

Resnet50 MobilenetV2 FasterNet RegVGG

Convnext-S ConvMixer InceptionNeXt-S Efficientnet-B0

Hybridbased

SCNet MoCoViT1.0 HiFuse-S MedVMamba

ConvMLP-S TransXNet-B Conv-SdMLPMixer

Split

Failure Modes 1 Quality Limitation in Visual Representation

P(C, start) P(C, low)

P(V, start) P(V, min)

P(L, end)

P(V, end)

(Ⅰ) FHSV

P(C, end)

FHSC

MLP Probing

Vision Tower

Pfinal

(Ⅳ)

(Ⅲ) FHSL

（Ⅱ）

P P

MLP-Mixer-B S2-MLP

Layer-wise, Module-level Dissection

MLPbased

Feature Probing for Systematic Diagnosis

Vision Tower

Traditional Deep Learning Models

Medical MLLMs

Pneumonia train test

Connector

Models for Comparison & Analysis

Chest-Xray

Connector

Fidelity Loss in Connector Projection

3 Comprehension Deficit in LLM Reasoning

P(L, low) P(L, start)

Large Language Model

Output

Misalignment of Semantic Mapping

Figure 2: Evaluation framework for dissecting classification degradation in medical MLLMs. and LLMs with custom-designed connectors, offering architectural flexibility for specific medical scenarios. For example, XrayGPT [33] is designed for chest X-ray analysis, combining frozen MedClip [42] with Vicuna [7] via a linear projection connector. Uni-Med [50] employs a connector mixture-of-experts to bridge EVA-CLIP [9] and LLaMA2 [35] for multi-task joint learning. The second route directly fine-tunes general MLLMs on medical data, capitalizing on the rapid iteration and ever-strengthening capabilities of foundation models. For instance, LLaVA-Med [20] adapts LLaVA through curriculum learning for medical VQA. Several models built upon the QwenVL series [39]-such as HuatuoGPT-Vision [5], Lingshu [44], and ShizhenGPT-VL [4]—target distinct medical capabilities, ranging from medical multimodal understanding and reasoning to Traditional Chinese Medicine diagnostics. While these developments signal exciting progress toward generalist medical AI, the field has been surprisingly silent on a critical question: given their overwhelming advantages in pre-training data and model parameters, do medical MLLMs actually deliver tangible gains over traditional methods on foundational tasks? The answer, as we will show, is far from straightforward.

2.2

Medical Image Classification

Medical image classification serves as a fundamental capability in medical image interpretation and computational diagnosis. Over the past decade, four types of deep learning architectures have been developed to address this task. CNN-based methods [6, 13, 32, 48] have been widely adopted due to their strong local feature extraction and parameter efficiency. MLP-based methods [34, 47] replace convolutions with pure multi-layer perceptrons, achieving global receptive fields at the cost of local inductive bias. ViT-based methods [3, 12] leverage self-attention to capture long-range dependencies, yet require substantial data for learning. Hybrid-based methods [16, 21, 29] aim to integrate the advantages of different network architectures by balancing local detail and global context. Rather than employing dedicated classification heads, medical

MLLMs reformulate image classification as language-based VQA. By leveraging the reasoning and instruction-following capabilities of the LLM, classification results are generated either as a choice among predefined options [31] or as an open-ended response directly specifying the category [51]. Despite the intuitive appeal of handling both classification and more complex medical tasks within the unified framework, emerging observations reveal that medical MLLMs consistently underperform compared to traditional deep learning methods across diverse classification tasks [28, 38, 41]. While these studies provide valuable discussions, they largely remain at the phenomenological level. Our work provides the first systematic diagnosis of classification performance degradation in medical MLLMs, revealing four failure modes and three critical barriers that previous studies have overlooked.

2.3

Feature Probing

By training lightweight classifiers on frozen intermediate activations, probing reveals which information is encoded at specific network depths and serves as a diagnostic tool for representation quality [2, 8]. Probing originates in natural language processing, where it reveals that language models encode hierarchical linguistic information in a layer-wise fashion [14, 36]. In computer vision, probing demonstrates a layer-wise progression from edges and textures to object parts and holistic concepts [27]. With the advent of multimodal models, probing has been extended to analyze crossmodal representations. Several studies have investigated whether visual concepts survive projection into the language space [37] and how multimodal fusion affects representational quality [40]. Zhang et al. [49] employ linear probes to show that classification-critical information persists in vision tower outputs but remains inaccessible to the LLM. Despite these advances, existing probing studies in medical MLLMs typically focus on isolated or specific layers, driven by individual study objectives. This fragmentation prevents a holistic understanding of how classification signals transform across the pipeline, while quantitative metrics to characterize such

ArXiv, Preprint, 2026

Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, and Ji Wu

evolutionary dynamics remain absent. In this work, we conduct a pipeline-wide, layer-wise probing analysis in diverse medical MLLMs. Within each module, we further introduce quantitative metrics to characterize the healthiness of feature evolution, enabling principled comparisons of different models and components.

EVALUATION PROTOCOL

As illustrated in Figure 2, this section presents the evaluation setup for systematically dissecting classification performance of medical MLLMs, including the selected medical image classification datasets, the models for comparison and analysis, the implementation details of feature probing, and the evaluation metrics.

3.1

examine how supervised adaptation alters the information flow and whether it strengthens or weakens representational capabilities. We use the AdamW optimizer with a learning rate of 1e-4. A cosine learning rate scheduler with a warmup ratio of 0.05 is adopted. All experiments are trained twice for 20 epochs with a batch size of 4, separately using 8 NVIDIA A800 GPUs and 8 Ascend 910B NPUs.

Datasets

Ultrasound for Breast Cancer Classification. The Breast Ultrasound Images (BUSI) dataset [1] is collected from 600 female patients at Baheya Hospital, Cairo, Egypt. It contains 780 breast ultrasound images with an average resolution of 500*500 pixels. BUSI is divided into three categories: normal, benign, and malignant, containing 133, 437, and 210 images respectively.

3.3

CT for COVID-19 Diagnosis. The COVID19-CT dataset [45] is constructed by extracting CT images from 760 COVID-19 preprints on medRxiv and bioRxiv. It comprises 349 COVID-19 positive CT images from 216 patients and 397 negative samples. Xray for Pneumonia Screening. The Chest-Xray dataset [19] is derived from a retrospective cohort of pediatric patients aged one to five years at Guangzhou Women and Children’s Medical Center. It contains 5,856 chest X-ray images, labeled as normal (1,583 images) and pneumonia (4,273 images). Detailed instruction templates and examples for each classification task are provided in Supplementary Materials A.

3.2

Models and Implementation Details

We follow the comparison protocol of Ren et al. [29] and include four types of traditional deep learning models: 8 CNN-based, 4 MLP-based, 4 ViT-based, and 7 hybrid-based methods. For medical MLLMs, we select 14 open-source models spanning six representative series: MedVLM-R1 [26], HuatuoGPT-Vision-7B/34B [5], Lingshu-7B/32B [44], ShizhenGPT-7B/32B-VL [4], Hulu-Med-4B/7B /14B/32B [18], MedGemma-4B/27B [30], and MedGemma-1.5-4B. Detailed architecture designs, training stages, and data sources for each medical MLLM are provided in Supplementary Materials B. Our main experiments consist of three complementary parts to systematically diagnose classification performance in MLLMs: (1) Dataset-specific SFT of each medical MLLM. We apply LoRA [15] to all linear layers with rank 8 and alpha 32, efficiently adapting the model to the target classification task. (2) Feature probing on the original medical MLLMs. For each module—vision tower, connector, and LLM—and for each layer within these modules, we attach a lightweight probing head consisting of a two-layer MLP with ReLU activation and dropout rate 0.1. The probe parameters are optimized while the MLLM backbone remains frozen, ensuring that the probe accuracy reflects the amount of task-relevant information preserved at that specific point in the pipeline. (3) Feature probing on finetuned medical MLLMs. We repeat the same probing procedure to

Metrics

We employ standard classification metrics including accuracy, F1score, precision, recall, and AUC to evaluate model and probe performance. Drawing inspiration from the Sharpe Ratio in financial economics, which evaluates return relative to risk, we propose the feature health score (FHS) to holistically assess how well a module preserves and refines task-relevant information across its layers. Formally, let 𝑉 , 𝐶, and 𝐿 represent the vision tower, connector, and large language model, respectively. For a given module 𝑀 ∈ {𝑉 , 𝐶, 𝐿} with 𝑛 layers, we introduce growth factor (𝐺𝐹𝑀 ) and volatility penalty (𝑉 𝑃𝑀 ), defined as follows: √︁ (1) 𝐺𝐹𝑀 = ln 𝑃 (𝑀,end) / 𝑃 (𝑀,start) 𝑃 (𝑀,min)

𝑉 𝑃𝑀 = exp −𝜆 ·

𝑛−1 ∑︁

! 𝑃 (𝑀,𝑖+1) − 𝑃 (𝑀,𝑖 ) /𝑃 (𝑀,end)

(2)

𝑖=1

where 𝑃 (𝑀,start) , 𝑃 (𝑀,min) , and 𝑃 (𝑀,end) represent the probing performance at the first, worst, and last layer of module 𝑀, respectively. 𝑖 is the layer index. 𝜆 is set to 0.2 to control the strength of penalty. The former measures the net improvement from the module’s early and worst layers to its final output. The latter quantifies the stability of the feature evolution path. High total variation indicates erratic oscillations, sharp peaks, or sudden drops, which are signs of architectural fragility or non-robust features. The FHS of module 𝑀 can be formed as follows: 𝐹𝐻𝑆 𝑀 = 𝑃 (𝑀,end) · (1 + 𝐺𝐹𝑀 ) · 𝑉 𝑃𝑀

(3)

which ensures that a module receives a high FHS only if it simultaneously achieves strong final performance, demonstrates meaningful improvement over its early and worst layers, and maintains a smooth, stable trajectory. Therefore, we can obtain a four-score profile: (I) 𝐹𝐻𝑆𝑉 , (II) 𝐹𝐻𝑆𝐶 , (III) 𝐹𝐻𝑆𝐿 , and (IV) 𝑃 𝑓 𝑖𝑛𝑎𝑙 , the probing performance at the final semantic output of the MLLM. The evolution process of feature healthiness offers a holistic and quantitative view, enabling principled apples-to-apples comparisons across different MLLMs.

EVALUATION RESULTS

This section presents the evaluation analysis of classification performance degradation inside medical MLLMs. We first provide an overview of our layer-wise, module-level probing results, then distill the observed patterns into four failure modes, and finally apply the proposed quantitative metrics to enable principled cross-model comparisons. Figure 3 visualizes the probing accuracy curves for all 14 medical MLLMs across the three datasets before and after task-specific fine-tuning. Additional probing metrics are provided in Supplementary Material C.

Lost in the Hype

ArXiv, Preprint, 2026

BUSI

COVID19-CT

Chest-Xray

MedVLM-R1

HuatuoGPT-Vision-7B

HuatuoGPT-Vision-34B

Lingshu-7B

Lingshu-32B

ShizhenGPT-7B-VL

ShizhenGPT-32B-VL

Hulu-Med-4B

Hulu-Med-7B

Hulu-Med-14B

Hulu-Med-32B

MedGemma-4B

MedGemma-27B

MedGemma-1.5-4B

Before SFT

After SFT

Connector Layers

MLLM Response

Figure 3: Layer-wise probing accuracy curves for 14 medical MLLMs across three datasets.

ArXiv, Preprint, 2026

4.1

Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, and Ji Wu

Initial Observations

To minimize cognitive interference from the diverse pre-training data, model architectures, and parameter scales across different medical MLLMs, we first focus on within-module pattern analysis. Specifically, we examine the 84 probing accuracy curves and summarize the recurring behaviors observed in the vision tower, connector, LLM, and final semantic output, respectively. Within the vision tower, the probing accuracy curves exhibit considerable variation across datasets and models. On the BUSI and COVID-19 CT datasets, the most common pattern is a rise-andfluctuate trend: the accuracy rises steadily from early layers, reaches a peak in mid-to-deep layers, and then fluctuates within a narrow range. The curves before and after fine-tuning show similar shapes, indicating that task-specific adaptation does not fundamentally alter how the encoder organizes visual information. However, the behavior diverges markedly on the Chest-Xray dataset. On the one hand, the probing accuracy may exhibit a sustained decline as layers deepen. On the other hand, the curves before and after fine-tuning differ in shape. This counterintuitive observation suggests that dataset-specific fine-tuning may disrupt rather than enhance the discriminative capacity of the vision tower, leading to cumulative degradation with depth. Probing accuracy within the connector exhibits a distinct pattern: it largely preserves the discriminative capability of the vision tower output layer, with only negligible upward or downward fluctuations. Such limited variation suggests that the connector primarily performs dimensional alignment between the vision tower’s output space and the LLM’s input embedding space, rather than functioning as a sophisticated feature refiner. In other words, the connector does not substantially enhance or impair the discriminative information relevant to classification. The LLM exhibits the most diverse behaviors among the three modules. First, probing accuracy may drop sharply at the first LLM layer then recovers steadily, indicating that the LLM initially struggles to decode visual information but gradually regains discriminability. Second, the curve remains largely unchanged, suggesting that the LLM merely inherits the representation quality without further refinement. Third, accuracy gradually decreases with depth. This occurs when upstream modules already suffer from degraded representations, and the LLM fails to salvage them. We observe that the final semantic output accuracy deviates from the probing accuracy at the last LLM layer: sometimes lower, occasionally higher, but rarely equal. Discriminative representations do not guarantee a correct generative semantic output, suggesting that the generative decoding process may introduce an additional gap independent of representation quality.

4.2

Failure Modes

Synthesizing the observations above, we distill the recurring degradation patterns into four critical failure modes that collectively explain why medical MLLMs underperform in image classification. Quality Limitation in Visual Representation. The vision tower fails to encode task-specific discriminative features for fine-grained medical patterns, producing generic representations that lack diagnostic specificity. The consequence is an irrecoverable information

deficit at the pipeline’s inception: downstream modules cannot reconstruct critical visual cues (e.g., lesion boundaries, tissue textures) from impoverished inputs, establishing a performance ceiling that no subsequent processing can breach. Fidelity Loss in Connector Projection. While designed to bridge modality gaps, the connector often operates as a lossy compression channel, diluting fine-grained spatial and textural details essential for medical diagnosis in favor of coarse semantic alignment. Critical diagnostic evidence is silently eroded during projection, leaving the LLM with ambiguous or incomplete visual evidence despite technically aligned representations. Comprehension Deficit in LLM Reasoning. This failure mode arises from the causal attention mechanism inherent to autoregressive LLMs. Specifically, the LLM’s intermediate representations of image tokens evolve solely through self-attention within the visual prefix, without ever being contextualized by the task-relevant language that follows. Consequently, the LLM fails to interpret visual evidence in a medically meaningful way, yielding little discriminative gain, or even incurring degradation. Misalignment of Semantic Mapping. A fundamental incompatibility exists between the generative text decoding objective and the discriminative nature of classification. The LLM is trained to produce fluent, open-ended text, not to map latent representations to a fixed set of categorical outputs with high fidelity. This misalignment manifests as a persistent gap between representation quality and output correctness.

4.3

Quantitative Comparisons

Table 1 presents the four-score profiles (𝐹𝐻𝑆𝑉 , 𝐹𝐻𝑆𝐶 , 𝐹 𝐻𝑆𝐿 , 𝑃 𝑓 𝑖𝑛𝑎𝑙 ) for 14 medical MLLMs across three datasets. This standardized representation transforms diverse architectural complexities into comparable trajectories of feature healthiness, enabling progressive comparisons from the perspective of modules, models, and datasets. Module-level Analysis. By examining each score column vertically, we can identify which models excel or falter at specific stages of the pipeline. The connector score 𝐹𝐻𝑆𝐶 shows the least variation across models: the best and worst values are closer, indicating that the connector’s impact is secondary. In contrast, the vision tower score 𝐹𝐻𝑆𝑉 and LLM score 𝐹𝐻𝑆𝐿 range widely between the best and worst models, highlighting that the quality limitation in visual representation and the comprehension deficit in LLM reasoning are the dominant bottlenecks. Model-level Analysis. Achieving a high 𝑃final necessitates robust health across all preceding modules, with no single stage exhibiting catastrophic failure. Conversely, a poor 𝑃final invariably correlates with at least one severely degraded module. Furthermore, the four-score profile of each model challenges the assumption that parameter scaling alone ensures superior outcomes. Within model series, increasing model size does not consistently improve all 𝐹 𝐻𝑆 metrics, underscoring that larger models do not inherently resolve the identified failure modes. Dataset-level Analysis. Examining the four-score profiles across datasets reveals clear hierarchical trends. Averaged over all models,

Lost in the Hype

ArXiv, Preprint, 2026

Table 1: Feature health scores (𝑭 𝑯 𝑺 𝑽 → 𝑭 𝑯 𝑺 𝑪 → 𝑭 𝑯 𝑺 𝑳 → 𝑷final ) for 14 medical MLLMs across three classification datasets. Results in bold denote the dataset best, while results with underlines indicate the dataset worst. Medical MLLMs

BUSI

Chest-Xray

93.46 → 84.24 → 86.11 → 88.18 92.21 → 82.66 → 74.73 → 78.82 96.34 → 81.92 → 77.58 → 80.30 103.75 → 83.25 → 76.81 → 81.28 84.52 → 79.22 → 66.71 → 80.79 90.02 → 80.10 → 78.06 → 81.28 99.61 → 85.10 → 79.00 → 83.74 84.92 → 81.98 → 81.36 → 75.86 80.78 → 77.98 → 80.69 → 78.82 84.00 → 81.28 → 81.41 → 82.76 78.46 → 82.27 → 67.76 → 78.33 89.83 → 80.79 → 86.08 → 80.79 101.94 → 86.27 → 79.80 → 85.22 91.73 → 81.28 → 70.44 → 83.74 90.83 82.02 77.61 81.42

71.58 → 75.29 → 76.44 → 75.64 78.10 → 78.03 → 81.35 → 78.53 69.02 → 73.81 → 74.15 → 73.56 66.30 → 73.78 → 80.00 → 73.56 65.90 → 73.59 → 75.61 → 76.12 66.39 → 74.17 → 73.80 → 73.72 78.55 → 78.46 → 75.81 → 76.60 78.66 → 74.84 → 78.26 → 77.40 81.37 → 78.17 → 76.42 → 77.08 79.95 → 78.65 → 81.89 → 82.69 83.56 → 83.41 → 77.91 → 83.49 59.91 → 68.43 → 68.96 → 70.51 68.76 → 69.87 → 65.01 → 71.96 64.13 → 70.19 → 63.12 → 70.03 72.30 75.05 74.91 75.78

+3.15%

+1.49%

+1.22% -0.09%

+0.79%

-3.03%

+1.43%

+2.92%

+0.33%

+4.83%

-0.09%

+9.88%

-1.56%

-1.79%

-3.03%

MedVLM-R1-2B 91.42 → 88.41 → 81.21 → 81.33 Huatuogpt-Vision-7B 101.88 → 85.87 → 83.25 → 84.00 Huatuogpt-Vision-34B 109.59 → 88.95 → 79.56 → 86.00 Lingshu-7B 97.69 → 77.74 → 83.34 → 85.33 Lingshu-32B 95.03 → 85.33 → 81.95 → 85.33 ShizhenGPT-7B-VL 102.14 → 87.87 → 87.94 → 86.00 ShizhenGPT-32B-VL 105.04 → 87.07 → 82.18 → 89.33 Hulu-Med-4B 107.01 → 76.83 → 88.58 → 83.33 Hulu-Med-7B 106.55 → 82.87 → 86.94 → 88.00 Hulu-Med-14B 105.23 → 81.54 → 90.29 → 87.33 Hulu-Med-32B 103.12 → 82.67 → 86.20 → 90.67 MedGemma-4B 117.15 → 86.00 → 100.87 → 84.00 MedGemma-27B 115.58 → 90.67 → 83.91 → 86.67 MedGemma-1.5-4B 109.54 → 88.00 → 90.14 → 89.33 Average 104.78 84.99 86.17 86.19

COVID19-CT

-1.92%

+3.94%

+3.71%

-4.17%

ShizhenGP T-7B-VL +2.41%

-1.61%

+2.68%

Lingshu-7B

-3.52%

-1.20%

-3.08%

HuatuoGPT-Vision-7B -3.24%

Qwen2.5-VL-7B

+2.68%

60 Gemma-3-4B

MedGemma -4B

MedGemma -1.5-4B

Figure 4: Marginal gains of medical adaptation over general-domain MLLMs: HuatuoGPT-Vision-7B/Lingshu7B/ShizhenGPT-7B-VL vs. Qwen2.5-VL-7B; MedGemma4B/MedGemma-1.5-4B vs. Gemma-3-4B.

BUSI consistently achieves the highest scores, followed by COVID19 CT, with Chest-Xray lagging substantially behind. This descending trend mirrors the performance gap observed between medical MLLMs and traditional deep learning models. Moreover, the datasetspecific fragility exposes a critical limitation of current generalist medical MLLMs: despite training on diverse imaging modalities and task formats, their capability remains highly dependent on the specific downstream distribution.

INSIGHTFUL DISCUSSIONS

Even when feature evolution within medical MLLMs appears healthy, the final classification accuracy may fall short of clinical expectations and underperform traditional deep learning models. In this section, we dissect three critical barriers that hinder medical MLLMs in image classification.

The Limited Impact of Medical Adaptation. To validate whether extensive medical adaptation for MLLM yields tangible benefits for downstream medical image classification, we conduct two groups of experiments illustrated in Figure 4. Specifically, we compare Qwen2.5-VL-7B and Gemma-3-4B directly fine-tuned on target datasets against the specialized medical MLLMs derived from them, respectively. Across three datasets, the improvements from medical adaptation remain minimal. For the Qwen-VL series, HuatuoGPTVision-7B, Lingshu-7B, and ShizhenGPT-7B-VL achieve average accuracy gains of only 1.37%, 1.27%, and 0.59%, respectively. For the Gemma series, MedGemma-4B actually underperforms its base model by 1.66%, while MedGemma-1.5-4B yields a modest 1.17% improvement. Despite the substantial cost in terms of data and computational resources for domain-specific pre-training and fine-tuning, the resulting medical MLLMs offer only negligible classification benefits over their general-domain counterparts. The Suboptimal State of Vision Tower. A more critical barrier lies in the vision tower itself. As the core module that determines the quality of visual representation, the vision tower is rarely optimized in a way that directly benefits classification. We first examine whether updating the vision tower during task-specific finetuning actually helps. Table 2 compares HuatuoGPT-Vision-7B and MedGemma-4B under two settings: with the vision tower frozen versus fine-tuned via LoRA. The average gaps of -0.38% and 0.27% indicate that unfreezing the vision tower does not reliably improve classification and may even harm it. We further compare three training paradigms using ShizhenGPT7B-VL and MedGemma-1.5-4B: standard MLLM SFT, feature probing on the vision tower (i.e., 𝑃 (𝑉 ,end) ), and vision tower SFT trained via cross-entropy. As shown in Table 3, the vision tower SFT dramatically outperforms both MLLM SFT and feature probing across all datasets. This stark discrepancy exposes the suboptimal state of the vision tower, which has been largely overlooked. When optimized under the autoregressive text-generation objective of MLLMs, the vision tower yields features far less discriminative than those obtained through the simple cross-entropy loss of traditional

ArXiv, Preprint, 2026

Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, and Ji Wu

Table 2: Comparison of frozen vision tower (gray) vs. LoRA-finetuned vision tower on HuatuoGPT-Vision-7B and MedGemma-4B. Models HuatuoGPT-Vision-7B MedGemma-4B

BUSI Acc F1

COVID19-CT Acc F1

Chest-Xray Acc F1

84.00 82.67 84.00 83.33

78.82 78.33 80.79 78.82

78.53 80.29 70.51 74.84

83.19 80.92 83.34 82.51

78.82 78.22 80.98 78.39

85.31 86.32 80.91 83.24

78.82 79.80 81.28

73.72 74.52 88.30

89.33 86.00 93.33

83.74 81.28 89.10

Lingshu-7B

+0.27%

70.03 69.87 85.22

supervised learning, ultimately capping the performance ceiling of medical image classification. The Comprehension-utilization Entanglement in LLM Reasoning. We dissect two internal capabilities of LLM reasoning: comprehension—the quality of visual representations encoded within the LLM layers, measured by probing averaged image token embeddings; utilization—the ability to integrate all information at the decision point, measured by probing the final input token. As illustrated in Figure 5, these two capabilities exhibit a tightly coupled hierarchical dependency. MLLMs are trained via autoregressive next-token prediction, an objective that inherently optimizes the final token’s representational utility rather than the quality of intermediate visual representations. This training bias gives rise to a distinct temporal asynchrony: utilization curves typically originate below comprehension curves in early layers, but gradually catch up in mid-to-deep layers. However, comprehension remains a prerequisite for utilization. The ceiling of utilization (weighted aggregation) is fundamentally constrained by the ceiling of comprehension (averaged aggregation). Consequently, when comprehension is severely limited—as observed on Chest-Xray—utilization struggles to develop even with fine-tuning.

Chest-Xray

-0.38%

ShizhenGPT-7B-VL MedGemma-1.5-4B BUSI COVID19-CT Chest-Xray BUSI COVID19-CT Chest-Xray

MLLM SFT 86.00 Feature Probing 84.67 Vision Tower SFT 89.33

COVID19-CT

Average Gap

Table 3: Accuracy comparison of MLLM SFT, feature probing 𝑷 (𝑽 ,end) , and vision tower SFT on ShizhenGPT-7B-VL and MedGemma-1.5-4B. Training Paradigm

BUSI

CONCLUSIONS

This paper systematically dissects why medical MLLMs underperform in image classification—a foundational task where their promised advantages have consistently fallen short. Through layerwise, module-level probing across 14 models and three datasets, we pinpoint where and how classification signals are distorted along the pipeline. Our findings distill the observed weaknesses into four failure modes: quality limitation in visual representation, fidelity loss in connector projection, comprehension deficit in LLM reasoning, and misalignment of semantic mapping. To enable principled comparisons, we introduce quantitative health scores that capture the evolution of visual representation across modules. Beyond diagnosis, our discussions reveal three critical barriers. First, medical

Lingshu-32B Comprehension (Before SFT)

Comprehension (After SFT)

Utilization (Before SFT)

Utilization (After SFT)

Figure 5: Comprehension and utilization dynamics in LLM reasoning. Layer-wise probing accuracy of Lingshu-7B and Lingshu-32B.

adaptation offers only marginal gains over general-domain counterparts. Second, vision towers trained under autoregressive objectives yield suboptimal classification representations compared to those trained with cross-entropy supervision. Third, comprehension and utilization of visual information in the LLM are tightly entangled, with the former fundamentally limiting the latter. We hope that this work serves as both a caution and a roadmap. The road from hype to clinically reliable MLLMs remains long, and foundational capabilities such as classification deserve deliberate engineering rather than being taken for granted.

REFERENCES [1] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy. 2020. Dataset of breast ultrasound images. Data in brief 28 (2020), 104863. [2] Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 (2016). [3] Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. 2021. Crossvit: Crossattention multi-scale vision transformer for image classification. In Proceedings of the IEEE/CVF international conference on computer vision. 357–366. [4] Junying Chen, Zhenyang Cai, Zhiheng Liu, Yunjin Yang, Rongsheng Wang, Qingying Xiao, Xiangyi Feng, Zhan Su, Jing Guo, Xiang Wan, et al. 2025. Shizhengpt: Towards multimodal llms for traditional chinese medicine. arXiv preprint arXiv:2508.14706 (2025). [5] Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan, et al. 2024. Towards injecting medical visual knowledge into multimodal llms at scale. In Proceedings of the 2024 conference on empirical methods in natural language processing. 7346–7370. [6] Jierun Chen, Shiu-hong Kao, Hao He, Weipeng Zhuo, Song Wen, Chul-Ho Lee, and S-H Gary Chan. 2023. Run, don’t walk: chasing higher FLOPS for faster neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 12021–12031. [7] Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023) 2, 3 (2023), 6. [8] Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2126–2136. [9] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. 2023. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 19358–19369.

Lost in the Hype

[10] George Fisher. 2025. Vision-Language Foundation Models Do Not Transfer to Medical Imaging Classification: A Negative Result on Chest X-ray Diagnosis. medRxiv (2025), 2025–12. [11] Stephanie Fu, Tyler Bonnen, Devin Guillory, and Trevor Darrell. 2025. Hidden in plain sight: Vlms overlook their visual representations. arXiv preprint arXiv:2506.08008 (2025). [12] Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. 2023. Fastervit: Fast vision transformers with hierarchical attention. arXiv preprint arXiv:2306.06189 (2023). [13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778. [14] John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp). 2733–2743. [15] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. arXiv preprint arXiv:2106.09685 (2021). [16] Xiangzuo Huo, Gang Sun, Shengwei Tian, Yan Wang, Long Yu, Jun Long, Wendong Zhang, and Aolun Li. 2024. HiFuse: Hierarchical multi-scale feature fusion network for medical image classification. Biomedical signal processing and control 87 (2024), 105534. [17] Daniel P Jeong, Saurabh Garg, Zachary Chase Lipton, and Michael Oberst. 2024. Medical adaptation of large language and vision-language models: Are we making progress?. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 12143–12170. [18] Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, et al. 2025. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668 (2025). [19] Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally L Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. 2018. Identifying medical diagnoses and treatable diseases by image-based deep learning. cell 172, 5 (2018), 1122–1131. [20] Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems 36 (2023), 28541–28564. [21] Jiachen Li, Ali Hassani, Steven Walton, and Humphrey Shi. 2023. Convmlp: Hierarchical convolutional mlps for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 6307–6316. [22] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. PMLR, 19730–19742. [23] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems 36 (2023), 34892–34916. [24] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. 2024. A visual-language foundation model for computational pathology. Nature medicine 30, 3 (2024), 863–874. [25] Yoojin Nam, Dong Yeong Kim, Sunggu Kyung, Jinyoung Seo, Jeong Min Song, Jimin Kwon, Jihyun Kim, Wooyoung Jo, Hyungbin Park, Jimin Sung, et al. 2025. Multimodal large language models in medical imaging: current state and future directions. Korean Journal of Radiology 26, 10 (2025), 900. [26] Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. 2025. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. In International Conference on Medical Image Computing and ComputerAssisted Intervention. Springer, 337–347. [27] Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. 2021. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems 34 (2021), 12116– 12128. [28] Meet Raval, Tejul Pandit, and Dhvani Upadhyay. 2026. LLM is Not All You Need: A Systematic Evaluation of ML vs. Foundation Models for text and image based Medical Classification. arXiv preprint arXiv:2601.16549 (2026). [29] Zitong Ren, Shiwei Liu, Liejun Wang, and Zhiqing Guo. 2025. Conv-SdMLPMixer: A hybrid medical image classification network based on multi-branch CNN and multi-scale multi-dimensional MLP. Information Fusion 118 (2025), 102937. [30] Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. 2025. Medgemma technical report. arXiv preprint arXiv:2507.05201 (2025). [31] Pooja Singh, Siddhant Ujjain, Tapan Kumar Gandhi, and Sandeep Kumar. 2025. CrossMed: A Multimodal Cross-Task Benchmark for Compositional Generalization in Medical Imaging. In 2025 IEEE EMBS International Conference on Biomedical and Health Informatics (BHI). IEEE, 1–7.

ArXiv, Preprint, 2026

[32] Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. PMLR, 6105–6114. [33] Omkar Chakradhar Thawakar, Abdelrahman M Shaker, Sahal Shaji Mullappilly, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, Jorma Laaksonen, and Fahad Khan. 2024. Xraygpt: Chest radiographs summarization using large medical vision-language models. In Proceedings of the 23rd workshop on biomedical natural language processing. 440–448. [34] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021. Mlp-mixer: An all-mlp architecture for vision. Advances in neural information processing systems 34 (2021), 24261–24272. [35] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023). [36] Betty Van Aken, Benjamin Winter, Alexander Löser, and Felix A Gers. 2019. How does bert answer questions? a layer-wise analysis of transformer representations. In Proceedings of the 28th ACM international conference on information and knowledge management. 1823–1832. [37] Gaurav Verma, Minje Choi, Kartik Sharma, Jamelle Watson-Daniels, Sejoon Oh, and Srijan Kumar. 2024. Cross-modal projection in multimodal llms doesn’t really project visual attributes to textual space. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 657–664. [38] Li Wang, Xi Chen, XiangWen Deng, HuaHui Yi, ZeKun Jiang, Kang Li, and Jian Li. 2026. Evaluating the Diagnostic Classification Ability of Multimodal Large Language Models: Insights from the Osteoarthritis Initiative. arXiv preprint arXiv:2601.02443 (2026). [39] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191 (2024). [40] Tianshi Wang, Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, and Heng Tao Shen. 2025. Cross-modal retrieval: a systematic review of methods and future directions. Proc. IEEE 112, 11 (2025), 1716–1754. [41] Yinuo Wang, Kai Chen, Yue Zeng, Cai Meng, Chao Pan, and Zhouping Tang. 2025. Zero-Shot Multi-modal Large Language Models vs Supervised Deep Learning: A Comparative Analysis on CT-Based Intracranial Hemorrhage Subtyping. Brain Hemorrhages (2025). [42] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. 2022. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 3876–3887. [43] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Hui Hui, Yanfeng Wang, and Weidi Xie. 2025. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. Nature Communications 16, 1 (2025), 7866. [44] Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. 2025. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044 (2025). [45] Xingyi Yang, Xuehai He, Jinyu Zhao, Yichen Zhang, Shanghang Zhang, and Pengtao Xie. 2020. Covid-ct-dataset: a ct scan dataset about covid-19. arXiv preprint arXiv:2003.13865 (2020). [46] Jiarui Ye and Hao Tang. 2025. Multimodal large language models for medicine: A comprehensive survey. arXiv preprint arXiv:2504.21051 (2025). [47] Tan Yu, Xu Li, Yunfeng Cai, Mingming Sun, and Ping Li. 2022. S2-mlp: Spatialshift mlp architecture for vision. In Proceedings of the IEEE/CVF winter conference on applications of computer vision. 297–306. [48] Weihao Yu, Pan Zhou, Shuicheng Yan, and Xinchao Wang. 2024. Inceptionnext: When inception meets convnext. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition. 5672–5683. [49] Yuhui Zhang, Alyssa Unell, Xiaohan Wang, Dhruba Ghosh, Yuchang Su, Ludwig Schmidt, and Serena Yeung-Levy. 2024. Why are visually-grounded language models bad at image classification? Advances in Neural Information Processing Systems 37 (2024), 51727–51753. [50] Xun Zhu, Ying Hu, Fanbin Mo, Miao Li, and Ji Wu. 2024. Uni-med: a unified medical generalist foundation model for multi-task learning via connector-MoE. Advances in Neural Information Processing Systems 37 (2024), 81225–81256. [51] Xun Zhu, Fanbin Mo, Zheng Zhang, Jiaxi Wang, Yiming Shi, Ming Wu, Chuang Zhang, Miao Li, and Ji Wu. 2025. Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Data. In Proceedings of the 33rd ACM International Conference on Multimedia. 2693–2702. [52] Xun Zhu, Zheng Zhang, Xi Chen, Yiming Shi, Miao Li, and Ji Wu. 2025. ConnectorS: a survey of connectors in multi-modal large language models. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence. 10836– 10844.

ArXiv, Preprint, 2026

Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, and Ji Wu

Supplementary Materials: Lost in the Hype: Revealing and Dissecting the Performance Degradation of Medical Multimodal Large Language Models in Image Classification A

INSTRUCTION TEMPLATE

COVID19-CT

We provide one illustrative sample for each diagnostic category in Figures S1, S2, and S3, covering the BUSI, COVID19-CT, and Chest-Xray datasets, respectively.

Instruction: <image> Carefully examine the image to determine if there is COVID present. Please answer ’Yes’ or ’No’.

BUSI Instruction: <image> Carefully examine the image to determine the diagnostic result for breast cancer. Please answer ’Normal’, ’Benign’, or ’Malignant’.

Ground truth: Yes.

Ground truth: No.

Figure S2: The instruction template and class-wise samples of the COVID19-CT dataset. Chest-Xray Ground truth: Normal.

Instruction: <image> Carefully examine the image to determine if there is pneumonia present. Please answer ’Yes’ or ’No’.

Ground truth: Yes.

Ground truth: No.

Ground truth: Benign. Figure S3: The instruction template and class-wise samples of the Chest-Xray dataset.

MEDICAL MLLM

Tabel S1 provides detailed information for each medical MLLM, including the model name, parameter count, the base MLLM it initializes from, vision tower type, connector type, LLM type, model description, and training strategy. Ground truth: Malignant.

Figure S1: The instruction template and class-wise samples of the BUSI dataset.

ADDITIONAL PROBING METRICS

We provide comprehensive layer-wise feature probing results for all MLLMs in Figures S4–S17, comparing model performance before and after SFT. Reported metrics include F1 score, precision, recall, accuracy, and AUC.

Lost in the Hype

ArXiv, Preprint, 2026

Table S1: Detailed information for each medical MLLM. Medical MLLMs

MedVLM-R1

Params

2.21B

Base MLLM

Qwen2-VL

Vision Tower

ViT trained from scratch

Connector

MLP

LLM

Qwen2-2B

HuatuoGPT-Vision-7B

8.29B

Qwen2.5-VL

Redesigned ViT trained from scratch

MLP

Qwen2.5-7B

HuatuoGPT-Vision-34B

34.75B

CLIP ViT-L/14@336

MLP

Yi-1.5-34B

Lingshu-7B

8.29B

Qwen2.5-VL

Lingshu-32B

33.45B

Qwen2.5-VL

ShizhenGPT-7B-VL

8.29B

Qwen2.5-VL

ShizhenGPT-32B-VL

33.45B

Qwen2.5-VL

Redesigned ViT trained from scratch

MLP

Qwen2.5-7B

MLP

Qwen2.5-32B

MLP

Qwen2.5-7B

MLP

Qwen2.5-32B

Hulu-Med-4B

4.83B

SigLIP with 2D RoPE

MLP

Qwen3-VL-4B

Hulu-Med-7B

8.04B

SigLIP with 2D RoPE

MLP

Qwen2.5-7B

Hulu-Med-14B

15.21B

SigLIP with 2D RoPE

MLP

Qwen3-14B

Hulu-Med-32B

33.21B

SigLIP with 2D RoPE

MLP

Qwen2.5-32B

MedGemma-4B

4.30B

Gemma-3

MedSigLIP

MLP

Gemma-3-4B

MedGemma-27B

27.43B

Gemma-3

MedSigLIP

MLP

Gemma-3-27B

MedGemma-1.5-4B

4.30B

Gemma-3

MedSigLIP

MLP

Gemma-3-4B

Description

Training Stages

A lightweight medical VLM trained with reinforcement learning using the GRPO algorithm, achieving strong generalization and high efficiency with only 600 training samples.

Stage 1: Uses GRPO on 600 MRI VQA samples (only final answers, no reasoning labels) to optimize format and accuracy rewards, encouraging emergent reasoning without explicit supervision. No separate SFT or multi-stage alignment.

A medical MLLM trained on PubMedVision (1.3M medical VQA entries), featuring bilingual capabilities and enhanced medical knowledge from HuatuoGPT-II.

Stage 1: Visual-language alignment using 558K general data (LLaVA) and 647K medical data (PubMedVision Alignment VQA), training only the two-layer MLP projector while keeping the CLIP vision encoder and LLM frozen following the LLaVA-1.5 settings. Stage 2: Instruction tuning using 658K general data (LLaVA) and 647K medical data (PubMedVision Instruction-Tuning VQA), updating both the MLP projector and the LLM to enhance instruction-following capabilities following the LLaVA-1.5 settings. Additional: A further 348K bilingual data from a Chinese medical VQA dataset (translated from PubMedVision) and the medical text corpus from HuatuoGPT-II (exact data size not specified) are incorporated to enhance bilingual capability and medical knowledge.

A generalist medical foundation model that progressively infuses extensive medical knowledge and enhances reasoning capabilities through a meticulously curated multi-modal dataset and a multi-stage training paradigm.

Stage 1: Fine-tune the vision encoder and projector (LLM frozen) on ~927K medical image–caption pairs from PMC-OA and ROCO. Stage 2: Perform end-to-end training of the vision encoder, projector, and LLM on ~4.1M image–text pairs, including medical caption datasets (ROCOv2, LLaVA-Med, Quilt-LLaVA, PubMedVision, MIMICCXR, FairVLMed, MedICaT, MedPix-2.0, synthesized long captions) and general caption datasets (LLaVA-1.5 Caption, PixMo). Stage 3: Train the vision encoder, projector, and LLM on ~7.1M instruction samples consisting of medical multimodal instruction datasets (PathVQA, PMC-VQA, SLAKE, VQA-Med-2019, VQA-RAD, CheXpert Plus, IU-Xray, etc.), synthesized QA/OCR/CoT data, medical text instruction datasets (MedQA, MedQuAD, MedReason, MedThoughts-8K, ApolloCorpus, HealthCareMagic-100k, etc.), and general instruction data (LLaVA-1.5 instruct, ALLaVA, OpenHermes-2.5). Stage 4: Apply GRPO-based RL with verifiable rewards on ~100K curated medical QA samples derived from datasets such as MIMIC-ExtMIMIC-CXR-VQA, PMC-VQA, PathVQA, SLAKE, VQA-Med-2019, and VQA-RAD to improve medical reasoning ability.

The first multimodal LLM tailored for Traditional Chinese Medicine (TCM), capable of understanding images, sounds, smells, and pulse signals. Supports TCM’s "Four Diagnostic Methods" (looking, listening/smelling, questioning, and pulse-taking).

Stage 1: Text-only pre-training on 11.9B tokens (6.3B TCM tokens from TCM Web Corpus and TCM Book Corpus + 5.6B general tokens from FineWeb-edu corpus) using full-parameter training of the LLM to inject TCM knowledge while preserving general capabilities. Stage 2: Multimodal pre-training on 3.84B tokens (1.17B TCM imagetext tokens, 0.69B general image-text tokens from ShareGPT-4V, 0.03B TCM audio-text tokens, 0.04B general audio tokens, and 1.75B resampled text tokens) jointly training the LLM, vision adapter, and signal adapter for multimodal alignment. Stage 3: Instruction tuning on multimodal datasets (83,629 TCM text instructions, 65,033 vision instructions with 70,638 images, and signal instructions: 99,195 ECG, 57,957 speech, 4,101 pulse, 672 smell, 456 cough, 189 heartbeat) using full-parameter tuning to obtain the final ShizhenGPT.

A transparent, generalist medical VLM designed to unify language-only, 2D/3D vision-language, and video understanding within a single architecture.

Stage 1: Perform vision-language alignment by training the vision encoder and multimodal projector on 1.4M 2D medical image-caption pairs to align visual features with the LLM embedding space through short caption generation, while keeping the LLM frozen. Stage 2: Perform medical multimodal pretraining by full-model training on 4.9M samples (public medical data and synthetic long captions) to inject medical knowledge and enhance visual understanding via long caption generation and open-ended QA. Stage 3: Perform mixed-modality instruction tuning by continuing training all modules on 10.5M instruction samples (text and multimodal) to develop instruction-following and multi-task capabilities across text, 2D images, 3D volumes, and videos.

A multimodal medical model built on Gemma-3 with the MedSigLIP vision encoder, designed to process medical images and text jointly for clinical reasoning tasks.

Stage 1: Perform vision encoder enhancement by training the vision tower on approximately 33M medical image–text pairs (including 32.5M internal histopathology patches, 231K MIMIC-CXR, 199K EyePACS, ~60K CT-US1, ~48K MRI-US1, 41K PMC medical images, and dermatology datasets) to obtain MedSigLIP. Stage 2: Perform multimodal pretraining using Gemma-3 general multimodal data mixed with medical image–text data (~33M, with medical data upweighted) while training the LLM and the multimodal connector. Stage 3: Conduct post-training with medical QA datasets (~400K samples) including MedMCQA, MedQA, PubMedQA, HealthSearchQA, LiveQA, AfriMed-QA, and synthetic QA, optimizing the LLM and connector. Additional: The training methodology for the 27B variant was the same as for the 4B variant, with the addition of two training datasets: EHRQA, to improve the model’s inherent EHR understanding, and ChestImaGenome, to enable anatomy localization on chest X-ray images.

MedGemma variant with expanded support for high-dimensional medical imaging, WSI, longitudinal medical imaging, anatomical localization, medical document understanding, and EHR understanding, plus improved accuracy on medical text reasoning and modest gains on standard 2D image interpretation over MedGemma 1 4B.

Relative to MedGemma-4B, it incorporates new training datasets, including CT-RATE, CT Dataset 1, MRI Dataset 1, WSI-Path, pathology datasets, MS-CXR-T, Mendeley Clinical Laboratory Test Reports, EHR datasets, and EHRNoteQA.

Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 i Vis on we Laye 2 Tow r L r 4 i o n Vis To e aye r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L i Vis on Tower ayer 14 L i o w Vis n To er ayer 16 L Vision Tower ayer 18 L Vision Tower ayer 20 L i Vis on Tower ayer 22 ion we Lay 24 e r T Co owe Lay r 26 nn r L er ec ay 28 tor er LLM Lay 30 e LLM Lay r 0 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 La er 24 ye r2 6

Metric Value Metric Value

Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 i o w Vis n e Laye 2 r T Vision Toowe Layer 4 r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L i o w Vis n To er ayer 14 L Vision Tower ayer 16 L Vision Tower ayer 18 L i Vis on Tower ayer 20 L i o w Vis n To er ayer 22 ion we Lay 24 e r T o Co we Lay r 26 nn r L er ec ay 28 tor er LLM Lay 30 e LLM Lay r 0 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 La er 24 ye r2 6

Layer Name

Precision

Precision Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 i Vis on we Laye 2 Tow r L r 4 i o n Vis To e aye r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L i Vis on Tower ayer 14 L i o w Vis n To er ayer 16 L Vision Tower ayer 18 L Vision Tower ayer 20 L i Vis on Tower ayer 22 ion we Lay 24 e r T Co owe Lay r 26 nn r L er ec ay 28 to er LLMr Lay 30 e LLM Lay r 0 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 La er 24 ye r2 6

Layer Name

Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 Vis ion we Laye 2 Tow r L r 4 i o n Vis To e aye r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L Vision Tower ayer 14 L i o w Vis n To er ayer 16 L Vision Tower ayer 18 L Vision Tower ayer 20 L i Vis on Tower ayer 22 ion we Lay 24 e r T Co owe Lay r 26 nn r L er ec ay 28 to er LLMr Lay 30 e LLM Lay r 0 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 La er 24 ye r2 6

Metric Value

ArXiv, Preprint, 2026 Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, and Ji Wu

0.8

0.6

0.4

0.9

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

Recall

Accuracy

AUC

Layer Name

Figure S4: Additional feature probing metrics of MedVLM-R1.

0.8

0.6

0.4

0.85 0.80 0.75 0.70 0.65

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

Layer Name

Figure S5: Additional probing metrics of HuatuoGPT-Vision-7B.

Metric Value

Layer Name

0.9

Layer Name F1

F1 Precision

Precision Vis Vision Vision Tow Vision Tower L V io To e a Visisionn Towerr Layyer 0 Vision Tower Layer 2 Vision Towwer Layer 4 Vision Tower L Layer 6 Vision Tower Layeer 8 Vision Tower Layer 10 Vision Tower Layer 12 ion Tower ay r 1 Co Tower LLayeer 146 nn er ay r 1 ec La er 8 t LLoMr Lyaer 220 LLM Layer 2 LLM Layer 0 LLM Layer 0 L y 2 LLMLM LLayeer 4 LLM Laayer 6 LLM Layer r 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 LLM Layer 30 LLM Layer 32 LLM Layer 34 LLM Layer 36 LLM Layer 38 LLM Layer 40 LLM Layer 42 LLM Layer 44 LLM Layer 46 LLM Layer 48 LLM Layer 50 LLM Layer 52 Layer 54 ye 56 r5 8

Vis Vision Vision Tow Vision Tower L V io To e a Visisionn Towerr Layyer 0 Vision Tower Layer 2 Vision Towwer Layer 4 Vision Tower L Layer 6 Vision Tower Layeer 8 Vision Tower Layer 10 Vision Tower Layer 12 ion Tower ay r 1 Co Tower LLayeer 146 nn er ay r 1 ec La er 8 t LLoMr Lyaer 220 LLM Layer 2 LLM Layer 0 LLM Layer 0 L y 2 LLMLM LLayeer 4 LLM Laayer 6 LLM Layer r 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 LLM Layer 30 LLM Layer 32 LLM Layer 34 LLM Layer 36 LLM Layer 38 LLM Layer 40 LLM Layer 42 LLM Layer 44 LLM Layer 46 LLM Layer 48 LLM Layer 50 LLM Layer 52 Layer 54 ye 56 r5 8

Metric Value

Metric Value 1.0

Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 i Vis on we Laye 2 Tow r L r 4 i o n Vis To e aye r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L i Vis on Tower ayer 14 L i o w Vis n To er ayer 16 L Vision Tower ayer 18 L Vision Tower ayer 20 L i Vis on Tower ayer 22 ion we Lay 24 e r T Co owe Lay r 26 nn r L er ec ay 28 to er LLMr Lay 30 e LLM Lay r 0 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 La er 24 ye r2 6

Metric Value

Lost in the Hype ArXiv, Preprint, 2026

0.8

0.6

0.4

0.9

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

0.6

Recall

Recall Accuracy

Accuracy AUC

Layer Name

Figure S6: Additional probing metrics of HuatuoGPT-Vision-34B.

0.8

0.6

0.4

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

0.6

AUC

Figure S7: Additional probing metrics of Lingshu-7B.

Layer Name

Metric Value

Metric Value 0.9

Layer Name

0.9

Layer Name F1

F1 Precision

Precision Vis Vision Vision Tow Vision Tower i VVisisioon TTooweer LLaye Visionn Tower Layer 0 Vision Towwer Layer 2 Vision Tower r Layer 4 Vision Tower Layayer 6 Vision Tower Layer r 8 Vision Tower Layer 10 Vision Tower Layer 12 Vision Tower Layer 14 Vision Tower Layer 16 Vision Tower Layer 18 ion To er La er 20 Co Towwer Layyer 22 nn e La er 24 ec r L ye 26 LLtoMr aLyerr 28 LLM Laye 30 LLM Layer 0 LLM Layer 0 a r LLLMLM Layyer 24 LLM L Layer 6 LLM Layeer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 LLM Layer 30 LLM Layer 32 LLM Layer 34 LLM Layer 36 LLM Layer 38 LLM Layer 40 LLM Layer 42 LLM Layer 44 LLM Layer 46 LLM Layer 48 LLM Layer 50 LLM Layer 52 LLM Layer 54 LLM Layer 56 Laayerr 58 ye 60 r6 2

Vis Vision Vision Tow Vision Tower i VVisisioon TTooweer LLaye Visionn Tower Layer 0 Vision Towwer Layer 2 Vision Tower r Layer 4 Vision Tower Layayer 6 Vision Tower Layer r 8 Vision Tower Layer 10 Vision Tower Layer 12 Vision Tower Layer 14 Vision Tower Layer 16 Vision Tower Layer 18 ion To er La er 20 Co Towwer Layyer 22 nn e La er 24 ec r L ye 26 LLtoMr aLyerr 28 LLM Laye 30 LLM Layer 0 LLM Layer 0 a r LLLMLM Layyer 24 LLM L Layer 6 LLM Layeer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 LLM Layer 30 LLM Layer 32 LLM Layer 34 LLM Layer 36 LLM Layer 38 LLM Layer 40 LLM Layer 42 LLM Layer 44 LLM Layer 46 LLM Layer 48 LLM Layer 50 LLM Layer 52 LLM Layer 54 LLM Layer 56 Laayerr 58 ye 60 r6 2

Metric Value 1.0

Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 i Vis on we Laye 2 Tow r L r 4 i o n Vis To e aye r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L i Vis on Tower ayer 14 L i o w Vis n To er ayer 16 L Vision Tower ayer 18 L Vision Tower ayer 20 L i Vis on Tower ayer 22 ion we Lay 24 e r T Co owe Lay r 26 nn r L er ec ay 28 to er LLMr Lay 30 e LLM Lay r 0 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 La er 24 ye r2 6

Metric Value

ArXiv, Preprint, 2026 Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, and Ji Wu

0.8

0.6

0.4

0.8

0.7

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

Recall

Recall Accuracy

Accuracy AUC

AUC

Layer Name

Figure S8: Additional probing metrics of Lingshu-32B.

0.8

0.6

0.4

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

Layer Name

Figure S9: Additional probing metrics of ShizhenGPT-7B-VL.

Vis Vision T o Vision T wer o Vision T wer Laye o i o Vis n T wer Layer 0 Vis ion owe Lay r 2 T i o Vis n T ower Layer 4 o Vision T wer r Layer 6 o Vision T wer Layeer 8 o i o Vis n T wer Layer 10 o Vision T wer Layer 12 o i Vis on T wer Layer 14 o i o Vis n T wer Layer 16 o Vision T wer Layer 18 ion ow La r 2 e y 0 Co Tower Layer 22 nn r L er ec ay 24 to e LLMr Layr 26 LLM Layer 1 LLM Layer 1 LLM Layer 3 LL La er 5 LLMM Layer 7 LLM Layyer 9 LLM Layer 11 LLM Layer 13 LLM Layer 15 LLM Layer 17 LLM Layer 19 LLM Layer 21 LLM Layer 23 LLM Layer 25 LLM Layer 27 LLM Layer 29 LLM Layer 31 La er 3 ye 3 r3 5 Vis Vision T o Vision T wer o Vision T wer Laye o i Vi on we Lay r 0 Vis sion Tower Layer 2 T i Vis on T ower Layer 4 o Vision T wer r Layer 6 o Vision T wer Layeer 8 o i Vis on T wer Layer 10 o i o Vis n T wer Layer 12 o Vision T wer Layer 14 o i o Vis n T wer Layer 16 o i o Vis n T wer Layer 18 ion ow La r 2 e y 0 Co Tower Layer 22 nn r L er ec ay 24 to e LLMr Layr 26 LLM Layer 1 LLM Layer 1 LLM Layer 3 LL La er 5 LLMM Layer 7 LLM Layyer 9 LLM Layer 11 LLM Layer 13 LLM Layer 15 LLM Layer 17 LLM Layer 19 LLM Layer 21 LLM Layer 23 LLM Layer 25 LLM Layer 27 LLM Layer 29 LLM Layer 31 La er 3 ye 3 r3 5

Metric Value

Metric Value Metric Value

Metric Value 1.0

Layer Name

1.0

Layer Name F1

F1 Precision

Lost in the Hype ArXiv, Preprint, 2026

0.8

0.6

0.4

0.9

0.8

0.7

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

0.95 0.90 0.85 0.80 0.75 0.70

Chest-Xray - Before SFT Chest-Xray - After SFT

Recall

Recall Accuracy

Accuracy AUC

Layer Name

Figure S10: Additional probing metrics of ShizhenGPT-32B-VL.

0.8

0.6

0.4

0.2

0.9

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.8

0.6

0.4

AUC

Figure S11: Additional probing metrics of Hulu-Med-4B.

Layer Name

Vis Vision T Vision Tower Vision Tower Lay V ion owe La er 0 Visision Tower Layer 2 Vision TTower Layer 4 Vision Tower r Layer 6 Vision Tower Layyer 8 Vision Tower Layer 10 Vision Tower Layer 12 Vision Tower Layer 14 Vision Tower Layer 16 Vision Tower Layer 18 ion ow La er 2 e Co Tow r Layer 20 nn er ye 2 ec Lay r 2 tor er 4 LLM Lay 26 LLM Layer 1 LLM Layer 1 LLM Layer 3 L La er LLMLM L yer 5 a LLM Layyer 7 LLM Layer 19 LLM Layer 11 LLM Layer 13 LLM Layer 15 LLM Layer 17 LLM Layer 29 LLM Layer 21 LLM Layer 23 LLM Layer 25 LLM Layer 27 LLM Layer 39 LLM Layer 31 LLM Layer 33 La er 35 ye 7 r3 9 Vis Vision T Vision Tower Vision Tower Lay V ion owe La er 0 Visision Tower Layer 2 Vision TTower Layer 4 Vision Tower r Layer 6 Vision Tower Layyer 8 Vision Tower Layer 10 Vision Tower Layer 12 Vision Tower Layer 14 Vision Tower Layer 16 Vision Tower Layer 18 ion ow La er 2 e Co Tow r Layer 20 nn er ye 2 ec Lay r 2 to e 4 LLMr Layr 26 LLM Layer 1 LLM Layer 1 LLM Layer 3 L La er LLMLM L yer 5 a LLM Layyer 7 LLM Layer 19 LLM Layer 11 LLM Layer 13 LLM Layer 15 LLM Layer 17 LLM Layer 29 LLM Layer 21 LLM Layer 23 LLM Layer 25 LLM Layer 27 LLM Layer 39 LLM Layer 31 LLM Layer 33 La er 35 ye 7 r3 9

Metric Value

Vis i Vis on To i w Vis on To er L i w a Vis on To er L yer 0 i w a Vis on To er L yer 2 Vis ion T wer ayer ion ow La 4 Vis To er yer i w L Vis on To er L ayer 6 i w a Vis on To er L yer 18 i w a Vis on To er L yer 10 i w a Vis on To er L yer 12 i w a Vis on To er L yer 14 i w a Vis on To er L yer 16 i w a Vis on To er L yer 28 ion we ay 0 T r er Co owe Laye 22 nn r L r 2 ec ay 4 tor er LLM Lay 26 e LLM Lay r 1 e LLM Lay r 1 e LLM Lay r 3 e LLM Lay r 5 LLM La er 7 y LLM Lay er 9 e LLM Lay r 11 e LLM Lay r 13 e LLM Lay r 15 e LLM Lay r 17 e LLM Lay r 19 e LLM Lay r 21 e LLM Lay r 23 La er 25 ye r2 7 Vis i Vis on To i w Vis on To er L i w a Vis on To er L yer 0 i w a Vis on To er L yer 2 Vis ion T wer ayer i o L Vis on To wer ayer 4 i w L Vis on To er L ayer 6 i w a Vis on To er L yer 18 i w a Vis on To er L yer 10 i w a Vis on To er L yer 12 i w a Vis on To er L yer 14 i w a Vis on To er L yer 16 i w a Vis on To er L yer 28 ion we ay 0 T r er Co owe Laye 22 nn r L r 2 ec ay 4 tor er LLM Lay 26 e LLM Lay r 1 e LLM Lay r 1 e LLM Lay r 3 e LLM Lay r 5 LLM La er 7 y LLM Lay er 9 e LLM Lay r 11 e LLM Lay r 13 e LLM Lay r 15 e LLM Lay r 17 e LLM Lay r 19 e LLM Lay r 21 e LLM Lay r 23 La er 25 ye r2 7

Metric Value

ArXiv, Preprint, 2026

1.0

Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, and Ji Wu

0.8

0.6

0.4

0.9

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

0.6

Layer Name

Layer Name F1

F1 Precision

Precision Recall

Recall Accuracy

Accuracy AUC

Layer Name

Figure S12: Additional probing metrics of Hulu-Med-7B.

0.8

0.6

0.4

0.9

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

0.6

0.5

AUC

Figure S13: Additional probing metrics of Hulu-Med-14B.

Layer Name

Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 i o w Vis n e Laye 2 r T Vision Toowe Layer 4 r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L i o w Vis n To er ayer 14 L Vision Tower ayer 16 L Vision Tower ayer 18 ion we Lay 20 Tow r L er 2 er ayer 2 LLMLaye 24 r LLM Lay 26 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 La er 30 ye r3 2

Metric Value

Vis Vision Vision Tow Vision Tower i T L VVisisioon Tooweer Laayer Visionn Tower Layer 0 Vision Towwer Layer 2 Vision Tower r Layer 4 Vision Tower Layyer 6 Vision Tower Layer 18 Vision Tower Layer 10 Vision Tower Layer 12 Vision Tower Layer 14 ion To er Layer 6 Co Towwer Layer 128 nn er La er 0 ec Laye 22 t LLoMr Lyaerr 24 LLM Laye 26 LLM Layer 1 LLM Layer 1 r LLLMLM Layyer 35 LLM LaLayer 7 LLM Layeer 9 LLM Layer 11 LLM Layer 13 LLM Layer 15 LLM Layer 17 LLM Layer 19 LLM Layer 21 LLM Layer 23 LLM Layer 25 LLM Layer 27 LLM Layer 29 LLM Layer 31 LLM Layer 33 LLM Layer 35 LLM Layer 37 LLM Layer 39 LLM Layer 41 LLM Layer 43 LLM Layer 45 LLM Layer 47 LLM Layer 49 LLM Layer 51 LLM Layer 53 LLM Layer 55 LLM Layer 57 Layerr 59 ye 61 r6 3 Vis Vision Vision Tow Vision Tower i T L VVisisioon Tooweer Laayer Visionn Tower Layer 0 Vision Towwer Layer 2 Vision Tower r Layer 4 Vision Tower Layyer 6 Vision Tower Layer 18 Vision Tower Layer 10 Vision Tower Layer 12 Vision Tower Layer 14 ion To er Layer 6 Co Towwer Layer 128 nn er La er 0 ec Laye 22 t LLoMr Lyaerr 24 LLM Laye 26 LLM Layer 1 LLM Layer 1 r LLLMLM Layyer 35 LLM LaLayer 7 LLM Layeer 9 LLM Layer 11 LLM Layer 13 LLM Layer 15 LLM Layer 17 LLM Layer 19 LLM Layer 21 LLM Layer 23 LLM Layer 25 LLM Layer 27 LLM Layer 29 LLM Layer 31 LLM Layer 33 LLM Layer 35 LLM Layer 37 LLM Layer 39 LLM Layer 41 LLM Layer 43 LLM Layer 45 LLM Layer 47 LLM Layer 49 LLM Layer 51 LLM Layer 53 LLM Layer 55 LLM Layer 57 Layerr 59 ye 61 r6 3

Metric Value

Metric Value 1.0

Layer Name

1.0

Layer Name F1

F1 Precision

Precision

Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 i Vis on we Laye 2 Tow r L r 4 i o n Vis To e aye r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L i Vis on Tower ayer 14 L i o w Vis n To er ayer 16 L Vision Tower ayer 18 ion we Lay 20 Tow r L er 2 er ayer 2 LLMLaye 24 r LLM Lay 26 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 La er 30 ye r3 2

Metric Value

Lost in the Hype ArXiv, Preprint, 2026

0.8

0.6

0.4

0.2

0.9

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

Recall

Recall Accuracy

Accuracy AUC

Layer Name

Figure S14: Additional probing metrics of Hulu-Med-32B.

0.8

0.6

0.4

0.2

0.8

0.6

0.4

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

0.6

0.5

AUC

Figure S15: Additional probing metrics of MedGemma-4B.

Layer Name

Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 i o w Vis n e Laye 2 r T Vision Toowe Layer 4 r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L i o w Vis n To er ayer 14 L Vision Tower ayer 16 L Vision Tower ayer 18 ion we Lay 20 Tow r L er 2 er ayer 2 LLMLaye 24 r LLM Lay 26 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 La er 30 ye r3 2

Metric Value

Metric Value 1.0

Layer Name

1.0

Layer Name F1

F1 Precision

Precision Vis Vision Vision Tow Vision Tower io T e La VVisision Toowerr Layer Visionn Tower Layer 0 Vision Towwer Layer 2 Vision Tower Layer 4 Vision Tower Layeyer 6 Vision Tower Layer 18 Vision Tower Layer 10 Vision Tower Layer 12 Vision Tower Layer 14 ion Tower Lay r 16 Tower Layer 28 e La er 0 LLrMLayyer 22 LLM Laer 24 LLM Layer26 LLM Layer 0 y LLLMLM Layeer 24 LLM LaLayer 6 LLM Layer r 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 LLM Layer 30 LLM Layer 32 LLM Layer 34 LLM Layer 36 LLM Layer 38 LLM Layer 40 LLM Layer 42 LLM Layer 44 LLM Layer 46 LLM Layer 48 LLM Layer 50 LLM Layer 52 LLM Layer 54 Layer 56 ye 58 r6 0

Vis Vision Vision Tow Vision Tower io T e La VVisision Toowerr Layer Visionn Tower Layer 0 Vision Towwer Layer 2 Vision Tower Layer 4 Vision Tower Layeyer 6 Vision Tower Layer 18 Vision Tower Layer 10 Vision Tower Layer 12 Vision Tower Layer 14 ion Tower Lay r 16 Tower Layer 28 e La er 0 LLrMLayyer 22 LLM Laer 24 LLM Layer26 LLM Layer 0 y LLLMLM Layeer 24 LLM LaLayer 6 LLM Layer r 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 LLM Layer 30 LLM Layer 32 LLM Layer 34 LLM Layer 36 LLM Layer 38 LLM Layer 40 LLM Layer 42 LLM Layer 44 LLM Layer 46 LLM Layer 48 LLM Layer 50 LLM Layer 52 LLM Layer 54 Layer 56 ye 58 r6 0

Metric Value

Metric Value 1.0

Vis Vision To Vision Tower L Vision Tower ayer L Vision Tower ayer 0 i Vis on we Laye 2 Tow r L r 4 i o n Vis To e aye r Vision Tower Layer 6 L Vision Tower ayer r 8 L Vision Tower ayer 10 L Vision Tower ayer 12 L i Vis on Tower ayer 14 L i o w Vis n To er ayer 16 L Vision Tower ayer 18 ion we Lay 20 Tow r L er 2 er ayer 2 LLMLaye 24 r LLM Lay 26 LLM Layer 0 LLM Layer 2 LL La er 4 LLMM Layer 6 LLM Layyer 8 LLM Layer 10 LLM Layer 12 LLM Layer 14 LLM Layer 16 LLM Layer 18 LLM Layer 20 LLM Layer 22 LLM Layer 24 LLM Layer 26 LLM Layer 28 La er 30 ye r3 2

Metric Value

ArXiv, Preprint, 2026 Xun Zhu, Fanbin Mo, Xi Chen, Kaili Zheng, Shaoshuai Yang, Yiming Shi, Jian Gao, Miao Li, and Ji Wu

0.8

0.6

0.4

0.2

0.9

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.8

0.6

0.4

Recall

Recall Accuracy

Accuracy AUC

AUC

Layer Name

Figure S16: Additional probing metrics of MedGemma-27B.

0.8

0.6

0.4

0.2

0.9

0.8

0.7

0.6

BUSI - Before SFT BUSI - After SFT

COVID19-CT - Before SFT COVID19-CT - After SFT

Chest-Xray - Before SFT Chest-Xray - After SFT

0.9

0.8

0.7

0.6

0.5

Layer Name

Figure S17: Additional probing metrics of MedGemma-1.5-4B.

Record · ID 2607 · SHA-256 240bd710adc79979
Conceptio Open Knowledge Archive — every document is proof-bundled with source, license, and retrieval metadata.