A Vision-language Framework for Comparative Reasoning in Radiology

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

A Vision-language Framework for Comparative Reasoning in Radiology

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

machine learning, deep learning, neural networks

A Vision–language Framework for Comparative Reasoning in Radiology Tengfei Zhang1,3,* , Ziheng Zhao2,3,* , Lisong Dai4,5 , Xiaoman Zhang6 , Pengcheng Qiu2,3 , Ya Zhang2,3 , Yanfeng Wang2,3,† and Weidi Xie2,3,† 1

University of Science and Technology of China School of Artificial Intelligence, Shanghai Jiao Tong University 3 Shanghai Artificial Intelligence Laboratory 4 Department of Radiology, Renmin Hospital of Wuhan University 5 Shanghai Sixth People’s Hospital Affiliated to Shanghai Jiao Tong University 6 Department of Biomedical Informatics, Harvard Medical School

arXiv:2606.06407v1 [cs.CV] 4 Jun 2026

Abstract. Medical imaging artificial intelligence has achieved strong performance in isolated image interpretation, but remains poorly aligned with radiological practice, where diagnosis and follow-up rely on comparison across prior studies and analogous reference cases. Here we formulate radiological comparison as an entity-aware cross-image reasoning problem and introduce a framework that supports both reference-case retrieval and temporal comparative interpretation. We construct MedReCo-DB, a large-scale comparative imaging resource derived from routine image–report pairs, comprising more than 690,000 images from over 160,000 patients across eight institutions, four countries and seven imaging modalities. Reports are decomposed into anatomical structures, abnormal findings and pathological conditions to provide supervision for entity-conditioned retrieval and comparative visual question answering. Using this resource, we develop MedReCo, an entity-aware visual encoder for controllable retrieval of clinically analogous cases, and MedReCo-VLM, a vision–language extension for generative interpretation of interval change. Across internal, external and cross-center evaluations, MedReCo achieved the highest Recall@1 in all 12 internal retrieval settings and improved external retrieval by a mean of 6.0 percentage points. In clinically confusable differential groups, it consistently outperformed the strongest baselines. MedReCo-VLM achieved the best performance across all comparative generation evaluations and improved longitudinal follow-up accuracy by 14.5-46.5 percentage points on chest radiographs and 13.0–27.9 percentage points on CT. These findings suggest that entity-aware comparative reasoning can be learned from routine clinical data at scale and may provide a more clinically aligned foundation for medical imaging AI.

1 Introduction Artificial intelligence has achieved strong performance across a wide range of medical image-analysis tasks, including radiology report generation [24, 27], image segmentation [21, 33, 55], disease classification [20, 39] and visual question answering [48, 32, 23]. Yet most medical imaging AI systems still operate under a single-image paradigm: they interpret one examination at a time, often without explicit access to prior studies or analogous reference cases. This differs fundamentally from radiological practice, where images are rarely interpreted in isolation. Instead, radiologists ask comparative questions throughout the diagnostic process: What does this finding resemble? Has it appeared before? Has it grown, resolved or changed in distribution? The ability to compare images across patients and across time is therefore not a peripheral function, but a central component of clinical image interpretation [12, 42, 46] (Fig. 1a). We frame comparative reasoning in radiology around two complementary settings that recur across clinical workflows. The first is reference comparison, in which a current finding is interpreted by comparison with similar historical, canonical or previously diagnosed cases [1, 9, 35]. This process supports differential diagnosis, helps distinguish visually confusable entities and reflects the way radiologists draw on accumulated experience, institutional archives and the literature. The second is temporal comparison, in which a current examination is compared with prior studies from the same patient to assess disease progression, treatment response, stability, or resolution [12, 42]. These two forms of comparison serve different clinical purposes, but they share a common computational requirement: given a target clinical entity, the model must identify the corresponding or analogous entity in another image and reason about its similarity, difference or interval change. † Corresponding Authors.

Q: This is my chest CT scan. How has my lung changed since the last visit?

b MedReCo

History Visits

Last Visit

This Visit

A: There's interval improvement in your lung fields with decreased opacities.

Query image

MedReCo -VLM

MoE ViT

Q: For this brain MRI, retrieve similar cases for the mass and differentiate them.

···

··· 0.34

Crossattention

···

MedReCo -VLM

MedReCo

❄

0.93

A: Compared with the reference case, this mass appears more circumscribed and shows less surrounding edema.

Image pair (prior, current)

MedReCo -VLM

Retrieval results

Shared visual representation

Case Dataset

Patient

0.57

Entity encoder

[Lung]

MoE ViT

LLM

Generation results

Question

d Temporal comparison Q: How has lung opacity changed?

Prior

Current

A: Bilateral peripheral lower-lung opacities have decreased.

Anatomical Structures

Reference comparison

Partial Match

Low match

Similarity: 0.92

Similarity: 0.71

Similarity: 0.23

Bilateral peripheral patchy lung opaci1es, lowerlung predominant.

Diﬀuse bilateral patchy lung opaci1es

Clear lungs but non-lungdominant abnormality

Query image (current)

Abnormal Findings

Entity: lung

Pathological Condition

Reference gallery

Matched

Figure 1 | Overview of this study. a, Radiological comparison is formulated around temporal follow-up and reference-case reasoning. b, MedReCo uses an entity-conditioned visual representation to support both retrieval and vision–language generation. c, MedReCo-DB integrates multi-modal imaging data with entity-level clinical annotations. d, The benchmark includes both prior–current temporal comparisons and cross-case reference comparisons. e, Representative retrieval results evaluate entityconditioned reference matching. f, Representative VQA results evaluate entity-specific comparative reasoning.

These two comparison settings are instantiated by the representative temporal and reference-comparison examples shown in Fig. 1d. Clinical comparison is inherently entity-specific: the same pair of studies may be similar with respect to one entity but different with respect to another. For example, two chest CT scans may have globally similar thoracic anatomy while differing in pulmonary arterial enlargement, hilar lymphadenopathy or ground-glass opacity. Conversely, two longitudinal examinations may show the same disease category but differ in lesion size, distribution or internal composition. A clinically useful comparative model must therefore move beyond global image-level matching. It must represent fine-grained anatomical structures, abnormal findings and pathological conditions; align these entities across images; and express clinically meaningful similarities, differences and temporal changes in language. Current medical AI systems only partially address this need. Image retrieval models can identify visually similar cases, but most rely on holistic image embeddings and are not controllable by a specific clinical entity [30, 47, 50, 15]. Medical vision–language models can generate reports or answer questions, but they are commonly trained on single-image image–text alignment and often lack explicit supervision for cross-image, entity-level comparison [19, 4, 48, 49, 23, 43]. Pairwise or longitudinal report-generation models have begun to address temporal interpretation, but they are usually developed for narrow modalities or tasks and are not integrated with reference-case retrieval [10, 56, 45]. Progress has also been limited by the absence of large-scale datasets that provide structured supervision for comparative reasoning across modalities, institutions, and clinical entities [18, 38, 13, 29]. Here, we formulate radiological comparison as an entity-aware cross-image reasoning problem and introduce a framework designed to support two clinically central workflows: controllable reference-case retrieval and generative temporal comparative interpretation (Fig. 1b). The framework is built around a shared visual representation that is conditioned on clinical entities, including anatomical structures, abnormal findings and pathological conditions. This design allows the model to compare images with respect to a specified entity rather than only according to global visual appearance. The retrieval component, MedReCo, ranks candidate studies according to entity-specific similarity, enabling reference comparison for differential diagnosis. The generative component, MedReCo-VLM, adapts the same entity-aware visual representation to a large language model, enabling natural-language descriptions of similarities, differences and interval changes between paired studies. To develop and evaluate this framework, we constructed MedReCo-DB, a large-scale comparative medical imaging resource derived from routine image–report pairs. MedReCo-DB comprises more than 690,000 images from over 160,000 patients across eight institutions, four countries and seven imaging modalities, including chest radiography, chest CT, abdominal CT, brain MRI, and breast, thyroid and liver ultrasound (Fig. 1c). The dataset was built on the premise that routine radiology reports contain rich entity-level information that can be converted into scalable supervision for comparative reasoning. We decomposed reports into structured descriptions of anatomical structures, abnormal findings and pathological conditions. These entity-specific descriptions were then used to construct two complementary forms of supervision: ranking triplets for entity-conditioned retrieval and comparative visual question-answering examples for generative interpretation (Fig. 1d). The resulting ontology covers 42 anatomical structures, 69 abnormal findings and 28 pathological conditions across the seven imaging modalities. MedReCo uses this supervision to learn fine-grained, entity-aware visual representations through text-guided contrastive ranking. A modality-aware vision encoder accommodates the heterogeneity of radiography, CT, MRI and ultrasound, while entity-conditioned attention helps isolate the visual evidence relevant to the queried clinical concept. MedReCo-VLM then connects the pretrained visual encoder to a large language model through instruction tuning, allowing the system to generate comparative interpretations for image pairs. By sharing the same visual foundation across retrieval and generation, the framework links reference comparison and temporal comparison as two manifestations of a common entity-level reasoning problem. Representative entity-level retrieval and comparative VQA results are highlighted in Fig. 1e,f, with comprehensive evaluations reported in subsequent result sections. We evaluated MedReCo and MedReCo-VLM across progressively more demanding settings. For reference comparison, we assessed internal retrieval, external validation on held-out institutions, cross-center retrieval and clinically confusable differential-diagnosis groups.

MedReCo achieved the highest Recall@1 in all 12 internal retrieval settings, improved external Recall@1 by a mean of 6.0 percentage points and consistently outperformed the strongest baselines in clinically confusable differentials. For temporal comparison, we evaluated broad pairwise comparative interpretation, independent public paired-image VQA benchmarks and same-patient longitudinal follow-up studies. MedReCo-VLM achieved the best performance across all 24 comparative interpretation evaluations, reached 87.1% accuracy on public paired-image VQA benchmarks and improved longitudinal follow-up accuracy by 14.5-46.5 percentage points on chest radiographs and by 13.0–27.9 percentage points on CT. Together, these results suggest that entity-aware comparative reasoning can be learned from routine clinical data at scale and may provide a more clinically aligned foundation for medical imaging AI.

2 Results We evaluated MedReCo and MedReCo-VLM on entity-conditioned tasks designed to instantiate comparative radiological reasoning. For reference comparison, the controllable image retrieval task takes a query image and a target clinical entity, and ranks candidate images according to entity-specific similarity to retrieve clinically analogous reference cases. For temporal comparison, the generative comparative interpretation task takes an image pair and a target clinical entity, and generates a clinically meaningful description of similarities, differences, or interval changes. Both tasks were evaluated at three levels of clinical granularity: anatomical structures, abnormal findings, and pathological conditions. Evaluation datasets. MedReCo-DB aggregates report-paired images from seven public datasets and one in-house collection, spanning four countries (the USA, Turkey, Spain, and China) and seven imaging modalities: brain MRI, chest X-ray, chest CT, abdominal CT, and breast, thyroid and liver ultrasound. For internal validation, we used four public datasets (MIMIC-CXR [25], CT-RATE [15], AMOS-MM [22], and CURG [28]), following their official train-test splits, along with an in-house brain MRI dataset [26] split 9:1 for training and testing. For external validation, we evaluated the model on held-out datasets, including IU-Xray [7], CheXpert Plus [5], and BIMCV-R [6]. To further assess comparative interpretation, we additionally evaluated on Medical-Diff-VQA [17] and MMXU [34], two independent public paired-image VQA benchmarks focused on chest radiographs. More dataset information are provided in Section 4.2, 4.3 and 4.4, while the benchmark composition is detailed in Table S5 S6 S7.

2.1 Reference comparison with controllable image retrieval We first evaluated the reference-comparison capability of MedReCo, using controllable image retrieval across diverse medical image archives. For each query comprising an image and a specified clinical entity, MedReCo ranked candidate images by entity-specific similarity. This setting requires the retrieval of cases that match the queried entity, rather than merely sharing modality, anatomical region, or overall visual appearance. Performance was measured using Recall@k (k ∈ {1, 3, 5}) under an entity-level relevance criterion (Section 4.4). We compared MedReCo with five baseline models: CT-CLIP [15], MedCLIP [47], PMC-CLIP [30], BiomedCLIP [50], and HLIP [52]. Each baseline was evaluated only on the imaging modalities for which it was designed. As these baselines do not support entity-conditioned retrieval, we evaluate them using similarity scores and rankings derived from holistic image whole-image features. Results are reported across four settings of increasing stringency: internal validation, external validation, cross-center retrieval, and clinically confusable differentials. Internal validation. We first evaluated retrieval in an internal setting, where both queries and candidate studies were drawn from the same institution and imaging modality, and all cases were represented in the training distribution. MedReCo achieved the highest Recall@1 in all 12 internal retrieval settings across four imaging modalities and three entity levels (Figure 2a; Supplementary Table S12). Across all imaging modalities, MedReCo improved Recall@1 over the strongest applicable baseline by 5.8, 3.5, and 6.1 percentage points for anatomical structures, abnormal findings, and pathological conditions, respectively. We further stratified performance by individual abnormality categories in CT and brain MRI (Figure 2b; Supplementary Table S13). Across 10 CT disease entities, MedReCo outperformed CT-CLIP by a mean of 6.3 percentage points in Recall@1. In brain MRI, MedReCo improved mean Recall@1 from 31.5% with HLIP to 38.0%. These gains were observed across most abnormality categories, indicating that

Chest X-Ray

100

80 ***

Abnormality

Anatomy

Disease

40 20

+3.6

lm Pu

lit ole

itis

n Bro

c Out-of-domain Chest X-Ray

nia

eu Pn

hy mp

a sem

cic s ora si Thndylo o Sp

ILD ic

rt Ao

100 ***

***

Abnormality

lH ata

e Av

Disease CT-CLIP

+5.4 +6.3

AnatomyAbnormality Disease MedCLIP

+6.4

+4.4 +4.3

+5.0

+4.3

d Cross-dataset Chest X-Ray

***

HLIP

+8.6

+1.6

e e e g e assns age n ssuens micns tion ag ssu ns ens ci nin ns hag n-mlesio orrhtusio oft-tilesio ischleesio infarc Aver t-tilesio perd fo softelesio morr f o o m n n s l hy ral he er ch. . he co h. s ral IC bra IC reb ch. IV/ reb Otharen ench & renc ere IV/ en Ce Ce a C r p r P a P Pa

rag

*** ***

Anatomy

ia ern

ury

e An

100

Disease

+1.0

a on

Anatomy

+1.9

+5.9

Recall@1 (%)

+6.3

s asi

Abnormality

+3.5

TN yH

AbnormalityAnatomy Disease

Brain MR (Abnormality)

+14.1

n.s.

+18.1

+15.0

***

+2.1

+10.3

CT (Disease) +5.5

AbnormalityAnatomy Disease

***

100 ***

***

Recall@1 (%)

***

Brain MR

100

Recall@1 (%)

***

100

BiomedCLIP

80 70 60 50 40 30 20 10 0

***

Anatomy PMC-CLIP

Abnormality MedReCo

CT ***

Disease

80 70 60 50 40 30 20 10 0

***

*** *

AnatomyAbnormality Disease

95% CI (panels a, c, d)

Figure 2 | Entity-conditioned reference-case retrieval performance. a, Internal-validation results across four imaging modalities and three entity levels: anatomical structures, abnormal findings and pathological conditions. Recall@1 is reported. Baselines are shown only for their intended modalities. Statistical significance is denoted as *P <0.05, **P <0.01 and ***P <0.001; n.s., not significant. b, Abnormal-finding subcategory analysis in CT and brain MR. Star markers denote MedReCo and circular markers denote baseline methods. All comparisons in this panel meet the significance threshold P <0.001. Abbreviations: ILD, interstitial lung disease; CHD, coronary heart disease; HTN, hypertension; Parench., parenchymal; IV/IC, intraventricular/intracisternal. c, External validation results on held-out chest-X-ray and CT cohorts. d, Cross-center retrieval from external reference archive.

the improvement was broadly distributed rather than driven by a small subset of findings. External validation within held-out centers. To assess robustness to institutional shift, we next evaluated MedReCo on held-out datasets from previously unseen clinical centers (Figure 2c; Supplementary Table S14). In this setting, both query and candidate studies were drawn from the same held-out institution. MedReCo achieved 71.5% Recall@1 for anatomical-structure retrieval on external chest radiographs, exceeding PMC-CLIP by 7.2 percentage points, and 83.9% on external CT, exceeding CT-CLIP by 6.8 percentage points. These gains were observed across all six external chest-radiograph and CT settings, indicating robustness to differences in acquisition protocols, reporting conventions and patient populations. Cross-center retrieval from external institutions. We then evaluated a stricter cross-center setting in

Figure 3 | Controllable image retrieval on clinically confusable differentials. a, Recall@1 on six visually confusable CT and chest-X-ray differential groups, where candidates share the same modality and anatomical region. Random indicates chance performance; CT-CLIP and PMC-CLIP are the strongest applicable CT and chest-X-ray baselines. b, Representative retrieval examples for pulmonary-artery enlargement versus hilar lymphadenopathy on CT studies.

which queries and candidate galleries were drawn from non-overlapping institutions (Figure 2d; Supplementary Table S14). Specifically, MIMIC-CXR test images were queried against galleries from CheXpert Plus and IU-Xray, and CT-RATE test volumes were queried against BIMCV-R. This setting tests whether clinically relevant reference cases can be retrieved from entirely new institutional archives. MedReCo achieved the highest Recall@1 across all six modality-granularity combinations, exceeding PMC-CLIP by 11.4 percentage points on chest radiographs and CT-CLIP by 7.8 percentage points on CT. The largest gain was observed for disease-based chest-radiograph retrieval (+14.6 percentage points). These findings suggest that MedReCo can retrieve clinically relevant reference cases across institutions with reduced reliance on center-specific visual or reporting patterns. Retrieval for clinically confusable differentials. To test performance in more clinically demanding settings, we curated six groups of confusable differential diagnoses from the external cohorts, each containing two or three findings arising in the same modality and anatomical context but requiring different diagnostic interpretations (Section 4.4; Supplementary Table S15). In these scenarios, visually similar findings may correspond to distinct disease processes or management pathways, making accurate reference retrieval particularly informative. MedReCo improved Recall@1 over the strongest baseline in all six groups, with a macro-average gain of 10.9 percentage points (Figure 3a; Supplementary Table S15). The largest gains were observed for hydronephrosis versus parapelvic/renal sinus cyst (+17.2 percentage points) and complete collapse versus extensive consolidation versus massive effusion (+12.5 percentage points). The remaining groups, including mediastinal lymphadenopathy versus vascular structure/aneurysm, hepatic differentials, and consolidation versus mass versus pleural effusion on chest X-ray, also showed consistent gains, ranging from +4.8 to +10.9 percentage points. These results indicate that MedReCo captures subtle but clinically consequential distinctions relevant to differential diagnosis.

()

(

()

(

()

) .

(

) .

(

) .

(

) .

(

) .

(

) .

(

()

) .

(

()

) .

(

()

) .

(

) .

(

) .

()

) .

(

---

( ( (

- -- --

(

( (

)

(

Figure 4 | Generative comparative interpretation across cross-patient image pairs. a, Internal validation result of generative comparative interpretation across four imaging modalities and three entity levels. Accuracy and RaTEScore are reported for close-ended and open-ended question respectively. Statistical significance is denoted as *P<0.05, **P <0.01 and ***P <0.001; n.s., not significant. b, Results of external validation and independent public benchmarks. The dashed line denotes the 25% random reference.

Representative retrieval examples. Representative examples across clinically confusable CT differentials illustrate the difference between global visual similarity and entity-matched retrieval (Figure 3b). For a query targeting left ureterohydronephrosis, the baseline retrieved a visually plausible case of bilateral renal sinus cysts and small cortical cysts, whereas MedReCo retrieved a case with matched severe left ureterohydronephrosis and residual collecting-system dilatation. For a query targeting ascending aortic ectasia, the baseline retrieved a case with small mediastinal and aortopulmonary-window lymph nodes rather than vascular dilatation, whereas MedReCo retrieved a case with matched ascending aortic ectasia of similar calibre. For a query targeting mediastinal lymphadenopathy, the baseline retrieved a case of aortic aneurysm without reported hilar or mediastinal lymphadenopathy, whereas MedReCo retrieved a case characterized by prevascular, paratracheal, hilar and subcarinal lymphadenopathy. Across these examples, the baseline retrieved CT studies with broadly similar appearance or anatomical context, but failed to identify clinically relevant cases with respect to the specific, localized clinical entity of interest.

b CASE 1

(Prior CT)

CASE 2

(Current CT)

(Prior CT)

(Current CT)

Ques%on

(Resolved) What has happened to the ground-glass opaci4es previously located around consolida4on areas?

(Worsened) How has the anatomical loca4on of the ground-glass opaci4es changed between the two CT scans?

Options

A: GGO is no longer present. (✓) B: GGO increased in extent. C: GGO migrated to diﬀerent lung segments.. D: GGO developed into solid nodules..

A: New bilateral upper-lobe ground-glass opaci4es. (✓) B: Resolu4on of previously seen ground-glass opaci4es. C: Migration of opacities from lower to upper lobes. D: Stable ground-glass opaci4es in the same loca4ons.

✓ Ours: A — The previous scan shows bilateral predominantly subpleural

✓ Ours: A — The prior scan shows no inﬁltra2ve or consolida2ve

✗ HuluMed: B — The ground-glass opacities have increased in extent

✗ HuluMed: D — The ground-glass opaci%es remain stable in the same

Answer and Reasoning

consolida2on areas with surrounding ground-glass opacity, while the current scan shows no mass, nodule, or inﬁltra2on in either lung.

around the consolidation areas, suggesting interval worsening.

abnormality in the lung parenchyma, while the current scan reveals new centriacinar ground-glass densi%es in both lungs, especially in the upper lobes.

loca2ons across both scans, indica%ng no anatomical change.

Figure 5 | Generative comparative interpretation in longitudinal follow-up. a, Accuracy on same-patient prior–current generative comparative interpretation, analysed by abnormal finding and by radiologist-prioritized follow-up attribute. Statistical significance is denoted as *P <0.05, **P <0.01 and ***P <0.001. GGO, ground-glass opacity; Abd./Pelvic, abdominal/pelvic. b, Two representative longitudinal CT cases. The left case shows resolution of ground-glass opacity (GGO) previously seen around bilateral predominantly subpleural consolidation, whereas the right case shows newly developed bilateral centriacinar GGO in both lungs, especially in the upper lobes.

2.2 Temporal comparison with generative comparative interpretation We next evaluated MedReCo-VLM on generative comparative interpretation, which requires the model to compare two studies with respect to a specified clinical entity and generate an entity-specific description of similarity, difference or interval change. Because same-patient longitudinal data are relatively limited, we first evaluated the model on a larger pool of cross-patient image pairs as a broad-scale proxy for temporal comparison. We then evaluated on same-patient longitudinal follow-up studies, in which the query consists of prior and current examinations together with a target finding or follow-up attribute, and the model must describe the interval change. All benchmark questions underwent multi-stage filtering, with final review by a board-certified radiologist (Methods, Section 4.4). Example questions are presented in Figure 7. Performance on closed-ended questions was measured using accuracy, and performance on open-ended generation was measured using BLEU [37], BERTScore [51], METEOR [3], RaTEScore [53], and RadGraph F1 [11], with emphasis on clinically grounded evaluation metrics. We compared MedReCo-VLM with five medical vision–language models: MedGemma-27B [43], RadFM-13B [48], CheXagent-8B [8], Hulu-Med32B [23], and Lingshu-7B [49]. Of these, only Hulu-Med-32B and RadFM-13B support 3D volumetric input. Results are presented across four settings: internal validation, external validation, independent public |8

benchmarks, and same-patient longitudinal follow-up. Internal validation. Under internal validation, MedReCo-VLM ranked first in all 24 evaluations across four imaging modalities and three entity levels (Figure 4a; Supplementary Table S16), with consistent gains on both closed-ended accuracy and open-ended comparative interpretation as measured by RaTEScore. Notably, MedReCo-VLM outperformed the strongest baseline by 25.5 percentage points in accuracy for CT anatomy questions and by 17.1 percentage points in RaTEScore for brain-MRI abnormality comparisons. Validation on external cohorts and independent public benchmarks. We next evaluated MedReCoVLM on held-out external cohorts and independently curated public benchmarks to assess robustness to distribution shift across institutions and benchmark construction procedures (Figure 4b; Supplementary Table S17). On held-out external cohorts, MedReCo-VLM consistently outperformed baseline models, achieving 64.7% accuracy and 60.3% RaTEScore on chest radiographs, and 60.6% accuracy and 57.9% RaTEScore on CT. On two independent public paired-image benchmarks, Medical-Diff-VQA and MMXU, MedReCo-VLM achieved 87.1% accuracy and 44.2% RaTEScore, with a RaTEScore gain of +13.2% relative to the second-best model. These results indicate that the entity-aware comparative representation generalizes beyond the reporting conventions and case composition of the training institutions. Same-patient longitudinal follow-up studies. We then evaluated MedReCo-VLM in the clinically important setting of same-patient longitudinal follow-up, in which the key task is to determine whether a finding has progressed, resolved or remained stable. The test set comprised clinically prioritized findings and associated attributes that are routinely monitored in follow-up imaging, including, for example, anatomical distribution and lesion size for pulmonary consolidation (Methods, Section 4.4). As shown in Figure 5a (Supplementary Table S18), MedReCo-VLM achieved consistent gains across both modalities. On chest radiographs, MedReCo-VLM outperformed the modality-level comparator across all reported findings and follow-up attributes, with gains ranging from (+14.5%) for pulmonary ground-glass opacity to (+46.51%) for anatomical location. For CT, MedReCo-VLM also improved over Hulu-Med-32B across all reported finding and attribute subgroups. At the finding level, gains ranged from (+13.0%) for abdominal/pelvic lymph nodes to (+27.9%) for liver soft-tissue density. At the attribute level, the largest improvement was observed for internal composition ((+44.7%), followed by spatial distribution ((+22.5%) percentage points) and relative signal, density or intensity ((+22.0%). By contrast, lesion size remained more challenging, with a smaller gain of (+2.0%) on CT, indicating that precise quantitative assessment of temporal change remains an important direction for future work. Representative cases. Two longitudinal CT examples, each from a different patient, further illustrate the clinical relevance of correct interval-change interpretation (Figure 5b). In the first case, the prior CT showed bilateral predominantly subpleural consolidation areas with surrounding ground-glass opacity, whereas the current CT showed no mass, nodule, or infiltration in either lung. MedReCo-VLM correctly identified resolution of the ground-glass opacity, whereas the baseline incorrectly reported increased extent around the consolidation areas and interpreted the interval change as worsening. In the second case, the prior CT showed no infiltrative or consolidative abnormality in the lung parenchyma, whereas the current CT showed new centriacinar ground-glass densities in both lungs, especially in the upper lobes. MedReCo-VLM correctly identified new bilateral upper-lobe ground-glass opacities, while the baseline incorrectly reported stable opacities in the same locations. These results show that MedReCo-VLM can generate entity-specific comparative descriptions of clinically meaningful interval change, including resolution, new abnormality development and changes in anatomical distribution.

2.3 Ablation studies To identify the architectural and training components that support both controllable image retrieval and generative comparative interpretation, we focused on two key design elements (Section 4.5): modality-specific routing in the vision encoder, which addresses heterogeneity across imaging modalities, and entity-conditioned representation learning, which organizes representations around clinically meaningful entities. Each ablation modified one component while keeping all other training and evaluation settings fixed, and was conducted on the internal validation sets across all imaging modalities. Modality-specific routing supports controllable image retrieval. To test whether modality-aware

Recall@1 (%)

(a)

Chest X-Ray

p < 0.001 ***

70 60 50 40 30

(b) 80

Anatomy Abnormal Disease

Chest X-Ray

p < 0.001 ***

Accuracy (%)

75 70 65 60 55 50

Anatomy Abnormal Disease

75 70 65 60 55 50 45 40

p < 0.001 ***

Brain MR 80

p < 0.001 ***

Anatomy Abnormal Disease

Shared-ViT (modality-shared MLPs)

MoE-ViT (modality-specific experts)

p < 0.001 ***

InfoNCE-only (Stage 1 warm-up)

Ultrasound

Anatomy Abnormal Disease

Brain MR 70

p < 0.001 ***

Ultrasound 90

Anatomy Abnormal Disease

Full (warm-up + triplet)

Figure 6 | Ablation studies. a, Comparison between MoE-ViT and Shared-ViT for controllable image retrieval. Results are reported as Recall@1 (%) on the internal validation sets across all imaging modalities and three entity granularities. b, Comparison between the Full model and the InfoNCE-only variant for generative comparative interpretation. Results are reported as closed-ended comparative-interpretation accuracy (%) using the same modality and entity-granularity layout. Higher values indicate better performance. Error bars, 95% CI. Top-right value, dataset-level McNemar p (pooled across the three granularities). ***p < 0.001; **p < 0.01; *p < 0.05.

specialization improves controllable retrieval in MedReCo we compared the default MoE-ViT architecture (Figure 8a; Supplementary Table S19 ) with a parameter-matched ViT variant in which all modalities shared the same MLP layers. MoE-ViT improved retrieval Recall@1 by a mean of 5.2 percentage points across 12 modality-entity settings and outperformed the shared-backbone ViT in 12 of 12 settings (Figure 6a). The largest gains were observed at the abnormal-finding level, suggesting that modality-specific routing is particularly beneficial for capturing fine-grained localized clinical variation across heterogeneous medical imaging modalities. Entity-conditioned representation learning improves generative comparative interpretation. To evaluate whether entity-level contrastive ranking enhances the vision encoder’s capacity to capture fine-grained clinical entities, we compared MedReCo-VLM with a baseline variant in which the MoE-ViT is directly adapted to the LLM after initial warm-up image-report alignment (Figure 8b–c). MedReCo-VLM improved closed-ended accuracy by an average of 2.3 percentage points and outperformed the InfoNCE-only baseline in 11 of 12 settings (Figure 6b Supplementary Table S19 ). These results indicate that entity-conditioned learning not only enables entity-level controllable retrieval, but also strengthens the fidelity of fine-grained visual representations required for comparative generation.

3 Discussion Existing medical AI predominantly relies on a single-image paradigm, inherently limiting its capacity for comparative radiological reasoning. Consequently, tasks requiring comparative analysis, such as reference comparison for differential diagnosis and temporal comparison for longitudinal tracking, have remained underexplored. Here, we demonstrate that a unified, entity-aware framework can overcome these limitations, enabling both controllable image retrieval and generative comparative interpretation. Entity-conditioned representation learning. Conventional medical image retrieval generally relies either on global image embeddings, which do not readily isolate specific entities. By contrast, MedReCo learns an entity-conditioned latent space aligned to anatomical structures, abnormal findings and pathological

|10

conditions. This allows the model to distinguish among co-occurring entities within the same study and to retrieve reference cases on the basis of the target clinical concept rather than overall visual similarity. The performance on clinically confusable differential diagnoses is consistent with this interpretation, suggesting that the model captures fine-grained, entity-specific visual features instead of relying primarily on global appearance or dataset-specific correlations. Capturing temporal deltas in longitudinal data. Longitudinal assessment requires not only recognizing whether a finding is present, but also determining how it has changed between studies. In our framework, this is formulated as comparative interpretation over paired images conditioned on a target entity. The model can therefore represent interval change with respect to specific findings and generate descriptions indicating whether those findings have progressed, resolved or remained stable. The gains observed on paired-image comparison and same-patient follow-up tasks suggest that this formulation captures aspects of temporal change that are not accessible to conventional single-image models. More broadly, these results indicate that comparative supervision can extend medical vision-language modelling beyond static image description towards clinically relevant cross-study interpretation. Robustness across modalities and distributions. Trained on a large multi-institutional dataset, MedReCo generalized across chest radiography, CT, MRI and ultrasound, and maintained strong performance on held-out external cohorts, cross-institutional retrieval tasks and independent public benchmarks. This suggests that the model is not simply learning institution-specific visual or linguistic regularities, but is capturing comparative features that transfer across acquisition protocols, patient populations and reporting styles. Such robustness is particularly important for comparative tasks, in which clinically meaningful similarities and differences must be identified despite substantial heterogeneity in image appearance and documentation practices. A scalable pipeline for comparative supervision. Training models for comparative reasoning has historically been limited by the lack of paired, annotated data. MedReCo-DB addresses this by introducing an automated, scalable pipeline that transforms standard clinical reports into entity-conditioned similarity rankings and comparative VQA pairs. By leveraging this pipeline, we curated a dataset of unprecedented scale, comprising over 690,000 images from more than 160,000 patients. This methodology drastically reduces the reliance on costly manual annotations while providing the dense, structured supervision necessary to train advanced comparative vision-language models. We release the benchmark and code to the community to catalyze further research into cross-study radiological reasoning. Limitations and future directions. Although MedReCo-VLM performed well on interval-change interpretation, it remains limited in tasks requiring precise quantitative assessment, such as exact lesion measurement or volumetric change, which may require integration with dedicated localization or segmentation methods. Overall, our results suggest that comparative reasoning is a viable and important direction for medical foundation models. Rather than treating each image as an isolated input, models that compare findings across cases and across time may better reflect the way medical imaging is interpreted in practice. With broader validation, entity-conditioned comparative modelling could provide a foundation for AI systems that support reference-based reasoning, longitudinal assessment and more clinically grounded image interpretation.

4 Method 4.1 Task formulation Controllable medical image retrieval and generative comparative interpretation differ in their output modalities: the former produces an ordered list of relevant reference cases, whereas the latter generates natural-language descriptions of entity-specific similarities and differences. Here, we formulate the two tasks and demonstrate that they can be jointly addressed using shared visual representations, with entity-level supervisory signals. Controllable image retrieval. Given a query image Iq ∈ RD×H×W and a collection of candidate images G = {I1 , . . . , IN } from the same modality, the retrieval task aims to identify reference cases that match the query image on a specified clinical entity Q (an anatomical structure, abnormal finding, or pathological condition). The retrieval model Φret (·) encodes both the query and candidate images, computes entityconditioned similarity score between query and each candidate, with respect to Q, and returns a ranked list of

|11

candidates in descending order of these scores. I ∗ = Φret (Iq , G | Q). When Q is omitted, the model performs standard whole-image retrieval based on global visual similarity. Generative comparative interpretation. Given a pair of medical images Ii , Ij ∈ RD×H×W from the same modality and an instruction Q specifying a clinical entity of interest, the generative model Φgen (·) produces a natural-language description T on the entity-specific similarities and differences between the two images: T = Φgen (Ii , Ij | Q). A shared vision encoder. Both tasks measure the entity-conditioned similarity between two images under the same instruction Q: retrieval returns it as a ranking over candidates, generative comparative interpretation describes it in natural language. Therefore, building a vision encoder that captures fine-grained, entity-specific visual features across multiple imaging modalities can provide the foundational representation for both comparative reasoning tasks. A shared supervisory source. Clinical reports already contain entity-level descriptions of the paired image. By parsing these reports into entity-specific content, we can derive complementary supervision for both tasks at scale, using the same parsed content: for a given entity, the agreement between the reports provides ranking labels for controllable image retrieval, while the contrast between them provides question-answer pairs for generative comparison interpretation. In this way, a report-mining pipeline can produce training data for both Φgen (·) and Φret (·), minimizing the need for manual annotation.

4.2 Data sources To ensure geographic, demographic and modality diversity, we assembled report-paired medical images from seven publicly available datasets and one in-house dataset, covering eight regions and seven imaging modalities. Table S1 summarises the data sources. Chest radiography. We included three chest radiography datasets. MIMIC-CXR [25] comprises 377,110 images from 227,835 studies of 65,379 patients at Beth Israel Deaconess Medical Center, Boston, USA, with paired free-text radiology reports and 14 structured pathology labels. IU-Xray [7] contains 3,955 chest radiographs with paired reports from Indiana University, USA. CheXpert Plus [5] provides 223,228 report– image pairs from 64,725 patients at Stanford Hospital, USA, acquired between 2002 and 2017, and includes a radiology text corpus of approximately 36 million tokens. Chest and abdominal computed tomography. We included three computed tomography datasets. CTRATE [15] contains 50,188 non-contrast chest CT volumes from 21,304 patients at Istanbul Medipol University Mega Hospital, Turkey, with 18 multi-label abnormality annotations. AMOS-MM [22] comprises 2,300 abdominal CT scans with bilingual Chinese–English clinical reports and 19,562 visual question-answering pairs. BIMCV-R [6] contains 8,069 CT volumes, corresponding to more than 2 million slices, from the Valencian Region Medical Image Bank, Spain, covering 96 disease categories annotated by more than 20 medical professionals. Ultrasound. We used the CURG dataset [28], which contains ultrasound images with paired reports from three organ systems: breast, thyroid and liver. The dataset includes 3,521 breast, 2,474 thyroid and 1,395 liver ultrasound cases collected at PLA General Hospital, China. Brain magnetic resonance imaging. We used an in-house brain MRI dataset [26] comprising 24,770 image–report pairs. The dataset includes axial brain MRI examinations and covers more than ten common neurological conditions, including acute and chronic cerebral infarction, brain metastases, haemorrhage, meningioma, glioma, hydrocephalus and brain atrophy.

|12

Figure 7 | MedReCo dataset and benchmark overview. a, Data composition and benchmark splits across seven imaging modalities and eight source datasets spanning clinical centres in the USA, Turkey, Spain, and China. MedReCo comprises over 690,000 images from more than 160,000 patients; source datasets are assigned to internal validation or held-out external validation cohorts to evaluate cross-centre generalisability. b, Comparative data construction pipeline. Image-paired clinical reports are decomposed into structured, entity-specific descriptions at multiple granularities. For each image pair, entity-aware similarity is quantified using three clinical NLP metrics, and comparative VQAs are generated and carefully filtered to ensure clinical relevance and consistency. c, Representative entity-level coverage across modalities at three clinical granularities: anatomical structures, abnormal findings, and pathological conditions, curated per modality with radiologist review. d, Representative comparative VQA examples across modalities. Each image pair is annotated with both closed-ended multiple-choice and open-ended questions targeting modality-specific clinical entities, with key clinical terms highlighted by entity type.

4.3 Clinical entity definition To reflect real-world radiological workflows, we structured the comparative analysis at three hierarchical levels of clinical granularity: anatomical structures, abnormal findings and pathological conditions. This hierarchy follows the typical process of image interpretation, progressing from anatomical localisation to the identification of imaging abnormalities and, ultimately, to diagnostic synthesis. Because clinically relevant features vary across modalities, we curated modality-specific entity sets. Anatomical structures. Anatomical structures provide the spatial context for subsequent comparisons. We initialised the entity set using 42 anatomical categories from an established whole-body computed tomography taxonomy [54]. We then adapted this baseline to each imaging modality by retaining broadly identifiable structures and adding modality-specific anatomy, such as the nipple-areolar complex for breast ultrasound. |13

Figure 8 | Training framework of MedReCo (a) Image–text representation alignment. The vision encoder is first aligned with the text encoder using image–report pairs. (b) Text-guided contrastive pre-training. The aligned vision encoder is further optimized with text-guided contrastive learning to support controllable image retrieval. (c) Instruction tuning for comparative reasoning. Finally, the pre-trained vision encoder is adapted to the LLM through instruction tuning, enabling comparative interpretation with explanatory language.

Abnormal findings. Abnormal findings were defined as observable and localisable deviations from normal anatomy. Starting from 404 findings in the same taxonomy [54], we applied modality-specific filtering to retain abnormalities relevant to each imaging modality. Pathological conditions. Pathological conditions were defined as synthesised clinical diagnoses derived from one or more imaging findings. We first extracted disease entities from radiology reports across all data sources using RaTE-NER [53]. We then used Claude 4.5 Sonnet [2] to cluster synonymous terms, such as “pulmonary embolism” and “PE”. The clustered entities were manually reviewed and curated to produce a standardised disease lexicon. This curation yielded an ontology comprising 42 anatomical structures, 69 abnormal findings and 28 pathological conditions across the seven imaging modalities in MedReCo-DB. A board-certified radiologist reviewed and validated all entity definitions. Modality-specific entity lists are provided in Supplementary S2. In addition to these entity-level comparisons, we defined a whole-image comparison task, in which the model evaluates the complete visual content without conditioning on predefined clinical entities.

4.4 Data construction MedReCo was constructed in three stages: report decomposition, entity-level similarity quantification and comparative visual question answering (VQA) generation. The full pipeline is shown in Figure 7b. Report decomposition. We decomposed each radiology report into entity-specific descriptions using DeepSeek-V3.1 [31]. For each predefined clinical entity, the model identified whether the entity was mentioned and extracted the corresponding descriptive text. For abnormal findings and pathological conditions, we retained both positive mentions, indicating that the entity was present, and explicit negative mentions, indicating that the entity was absent. Negative mentions were retained because absence of disease or abnormality is often clinically informative in comparative interpretation. The prompts used for report decomposition are provided in Supplementary Note S3.1. Similarity quantification. We estimated entity-level visual similarity using the semantic similarity of the corresponding entity-specific report descriptions as a proxy. For each image pair and clinical entity, we computed three complementary text-based similarity metrics: BioLord [40], which captures biomedical semantic similarity using concept embeddings; RaTEScore [53], which assesses factual consistency through entity-level matching; and RadGraph-XL [11], which measures similarity between clinical entity–relation graphs. Together, these metrics provided report-derived supervision for entity-conditioned retrieval training and for benchmarking controllable image retrieval. During both training and evaluation, an image pair was |14

treated as a positive match only when all three entity-specific similarity metrics exceeded 0.9, whereas negative pairs used for contrastive training were defined by all three metrics being below 0.3. Comparative VQA generation. We generated a broad comparative VQA benchmark from the extracted entity-specific report descriptions. For each pair of studies from the same imaging modality, we first selected a target clinical entity from the modality-specific ontology defined in Section 4.3 and retrieved its corresponding descriptions from both reports. We then used DeepSeek-V3.1 to compare the paired descriptions and convert clinically meaningful similarities or differences into VQA samples. Image pairs were sampled from two sources: cross-patient pairs, which supported reference-based and differential comparison, and same-patient prior-current pairs, when available, which supported temporal comparison. For each selected image pair and entity, we generated one four-option closed-ended VQA sample with clinically plausible distractors and one corresponding open-ended VQA sample. To ensure clinical validity and analytical difficulty, we applied a multi-stage filtering pipeline: • Image-dependency filtering. We prompted the language model to remove questions answerable from a single image alone, retaining only questions that required comparative reasoning across both images. • Language-prior filtering. To reduce reliance on language priors, we partitioned the VQA data and trained a text-only LLaMA-3.1-8B model [14] on each half. Each model was evaluated on the held-out split with randomised answer-option orders and queried four times per question. Questions answered correctly in at least three of four attempts were discarded. • Bias mitigation. To reduce positional bias, we randomly shuffled the multiple-choice options and swapped the presentation order of the image pairs, Ii ↔ Ij . • Radiologist review. A board-certified radiologist manually reviewed the test set and removed questions that were not clinically meaningful, such as questions about compositional differences between fracture types without diagnostic relevance. The broad comparative VQA benchmark comprised 102,052 closed-ended and 102,052 open-ended questions across dataset–entity groups. Detailed counts by dataset, modality and entity group are provided in Supplementary Table S6. Longitudinal follow-up VQAs. Apart from the broad comparative VQA benchmark, we constructed a dedicated longitudinal follow-up benchmark from temporally ordered same-patient prior-current studies. This benchmark was designed to assess clinically realistic interval-change reasoning rather than general cross-case comparison. It differs from the broad comparative VQA set in two ways. First, all image pairs are temporally ordered studies from the same patient. Second, the target findings and attributes are restricted to radiologist-prioritised abnormalities and follow-up attributes that are routinely monitored in longitudinal assessment, rather than being sampled from the full entity ontology. This design focuses the benchmark on clinically meaningful interval changes rather than arbitrary report differences. For chest radiography and computed tomography, radiologists defined a checklist comprising four anatomical regions, seven key findings and follow-up attributes including size, number, margin, density and spatial distribution. For each prior-current pair, we identified candidate findings and attributes for comparison, extracted report-grounded attribute values and assigned one interval-change label: new, resolved, increased, decreased, stable, qualitatively changed or unable to compare. Uncertain, clinically trivial or incomparable cases were excluded. Each retained finding-attribute change was then converted into a closed-ended VQA sample using DeepSeek-V3.1. The resulting longitudinal benchmark comprised 930 VQA samples. Its composition is reported in Supplementary Table S7.

4.5 Model architecture The proposed framework comprises two sequential components. First, we develop MedReCo for controllable medical image retrieval through text-guided image contrastive learning. This stage also serves as pre-training for the vision encoder, encouraging it to learn fine-grained visual representations of clinically meaningful entities (Section 4.5). Second, we adapt the resulting vision encoder for generative comparative interpretation |15

by connecting it to a large language model through instruction tuning, yielding MedReCo-VLM (Section 4.5). The overall architecture and training procedure are illustrated in Figure 8. Stage 1: Text-guided contrastive learning for controllable retrieval The core of MedReCo is a vision encoder, ϕvis (·), that learns fine-grained visual representations of clinically meaningful entities. A text encoder, ϕtext (·), encodes the entity of interest, and a cross-attention module, ϕattn (·), produces entity-conditioned image embeddings for similarity estimation and retrieval. A lightweight reranker, ϕrerank , is then applied to the top-ranked candidates to refine the final ordering. We describe each component and the progressive training procedure below. Vision encoder. For an input image Ik ∈ RD×H×W , where D = 1 for two-dimensional images and D > 1 for volumetric inputs, the vision encoder produces a sequence of visual tokens: Vk = ϕvis (Ik ),

Vk ∈ RN ×d ,

(1)

where N is the number of visual tokens and d is the embedding dimension. We implement ϕvis (·) as a mixture-of-experts Vision Transformer variant [41], in which images are routed to modality-specific expert multilayer perceptrons in each transformer block. This design accommodates the heterogeneity of medical imaging modalities, for example the textural appearance of ultrasound and the attenuation-based contrast of computed tomography. To process both two-dimensional images, including radiographs and ultrasound images, and three-dimensional volumes, including CT and MRI, we decompose the transformer into two stages. The first stage comprises eight per-slice spatial self-attention layers over the in-plane dimensions. The second stage comprises four per-location self-attention layers along the depth dimension and is used only for volumetric inputs. We use a patch size of 1 × 20 × 20 for two-dimensional images and 10 × 20 × 20 for three-dimensional volumes, and project all patches to a shared embedding dimension of 512. Text encoder. Given an entity prompt Q, the text encoder outputs an entity embedding: eQ = ϕtext (Q),

eQ ∈ Rd .

(2)

We implement ϕtext (·) as a BERT-style encoder initialized with pretrained BioLORD weights. The encoder takes an optional text prompt specifying the clinical entity of interest and is jointly fine-tuned during contrastive training to align entity semantics with visual representations. Warm-up alignment. We first align images and reports using the raw paired data and a symmetric InfoNCE loss [36] (Figure 8a). This warm-up step initializes a shared multimodal embedding space from image–report pairs and stabilizes subsequent entity-level contrastive learning. Entity-conditioned representation learning. The goal of entity-conditioned representation learning is to learn a query-dependent view of each image, rather than assigning a single fixed representation to the whole study. Clinically, the same image may contain multiple co-occurring entities, and two images may be similar with respect to one entity but different with respect to another. For example, in a chest CT containing both pulmonary artery enlargement and hilar lymphadenopathy, the query “pulmonary artery enlargement” should guide the model to focus on the pulmonary artery region and vascular caliber, whereas the query “hilar lymphadenopathy” should emphasize the hilar and mediastinal nodal regions. We therefore expect the module to learn entity-specific visual evidence by attending to image tokens that are relevant to the queried clinical concept while suppressing visual content that is less relevant to that entity. To obtain entity-aware visual representation, we use a four-layer cross-attention module, ϕattn (·), that conditions the visual tokens on the entity query (Figure 8b): fk = ϕattn (Vk , eQ ),

k ∈ {i, 1, . . . , J},

(3)

where Vk serves as the key and value, eQ serves as the query, J denotes the number of candidate images

|16

compared against the anchor, i.e., the query image, in the retrieval batch, and fk ∈ Rd is the resulting entity-aware image feature. When Q is omitted, we replace eQ with a learnable special token ⟨CLS⟩, recovering the whole-image (entity-agnostic) setting. The entity-conditioned similarity between an anchor image Ii and a candidate image Ij is then computed as (4)

Simg (Ii , Ij | Q) = fi · fj .

Since fi and fj are L2-normalized, this dot product is equivalent to cosine similarity in the entity-conditioned representation space. Training objective. Rather than regressing absolute similarity scores, we enforce the correct relative ordering of candidate images with respect to the anchor. Specifically, the supervision signal is derived from the report-based similarity, as stated in Section 4.4: I(Simg (Ii , Ij | Q)) = I(Srpt (Ri , Rj | Q)),

j ∈ {1, . . . , J}

(5)

where I is the ranking function, and Srpt denotes the semantic similarity between entity-specific contents in reports Ri and Rj , quantified by the averaged ranking from BioLord, RaTEScore, and RadGraph-XL. This objective is implemented with a triplet loss [16]: Ltriplet =

max (0, Simg (Ii , In | Q) − Simg (Ii , Ip | Q) + α)

(6)

(i,p,n)

where (Ii , Ip , In ) denotes an anchor–positive–negative triplet, defined in Section 4.4, and α is the margin, set to 0.2. To ensure robust fine-grained representation learning, we employ a hard negative mining strategy within each mini-batch. Coarse-to-fine re-ranking. The coarse retriever ranks the gallery efficiently using the dot product between the entity-aware image features in Eq. 4. However, representing each image with a single vector may discard localized visual evidence that is important for clinically confusable entities. We therefore apply a lightweight point-wise reranker only to the top candidates returned by coarse retrieval, enabling fine-grained token-level matching with limited computational cost. For each image Ik in a query–candidate pair, where k ∈ {i, j}, we reuse the final-layer cross-attention map from Eq. 3 to identify the visual patches most relevant to the queried entity Q. Specifically, let Ak ∈ RN denote the final-layer cross-attention weights from the entity query to the N visual tokens, averaged over attention heads. We select the indices of the top-K attended visual tokens as Pk = TopK(Ak ),

|Pk | = K.

(7)

The compact entity-conditioned token sequence is then constructed as Ṽk = [fk ; Vk,Pk ] ,

Ṽk ∈ R(K+1)×d ,

(8)

where Vk, Pk ∈ RK×d denotes the subset of visual tokens in Vk indexed by Pk . We set K=64 in all experiments. The reranker takes the compact token sequences from the query and candidate images, along with a learnable ⟨CLS⟩ token, as input: h i Zi,j = ⟨CLS⟩; Ṽi ; Ṽj .

(9)

The sequence Zi,j is processed by a shallow modality-aware MoE self-attention encoder, denoted as ϕrerank , where self-attention models fine-grained interactions between the query and candidate visual tokens, and modality-specific experts adapt the feed-forward layers to different imaging modalities. Let Hi,j = ϕrerank (Zi,j ),

(10)

|17

where Hi,j denotes the output token sequence of the reranker encoder. A linear classification head hcls is then applied to the output ⟨CLS⟩ token to produce the refined relevance score, CLS srerank = σ hcls Hi,j i,j

(11)

CLS where Hi,j denotes the output representation corresponding to the ⟨CLS⟩ token, and σ(·) denotes the sigmoid function.

Stage 2: Instruction tuning for comparative interpretation As the pre-trained vision encoder already captures fine-grained visual representations of clinically meaningful entities, we leverage these representations to build Φgen (·) for generating detailed textual comparisons. Architecture. As shown in Figure 8c, we bridge ϕvis (·) to a large language model ϕLLM (·) through a projection layer. The projection layer consists of a two-layer MLP with a GELU activation function, mapping the visual features into the hidden space of LLM. The comparative interpretation is formulated as: T = ϕLLM Q, Π(ϕvis (Ii )), Π(ϕvis (Ij )) , (12) where Π(·) denotes the projection module, T and Q are the generated comparison and the instruction from our VQA pairs, respectively. We adopt Qwen2.5-7B-Instruct [44] as ϕLLM (·), with all parameters trainable during instruction tuning. Training objective. The model is trained with standard negative log-likelihood loss for auto-regressive next-token prediction: LNLL = −

|T | X

log PϕLLM (Tt | T<t , Q, Π(ϕvis (Ii )), Π(ϕvis (Ij ))) .

(13)

t=1

where Tt denotes the t-th output token and T<t denotes the previously generated tokens. During Stage 2, the vision encoder ϕvis is frozen to preserve the entity-aware representations learned in Stage 1, while the projection module Π(·) and the LLM ϕLLM are optimized for comparative generation.

Training details Stage 1 is optimized with AdamW using linear warm-up and cosine decay, with per-GPU batch sizes of 14 for 3D inputs and 50 for 2D inputs, for 10 epochs. Stage 2 also uses AdamW, with learning rates of 2 × 10−5 for the LLM and 1 × 10−4 for the projection module, a global batch size of 64, and one epoch of instruction tuning. All experiments are conducted on 8×A100 (80 GB) GPUs. Full hyperparameters are provided in Supplementary Table S10.

5 Data Availability All public datasets used in this study are available from their original data providers. Official access links, dataset roles, and access requirements for the public datasets and independent paired-image VQA benchmarks are provided in Supplementary Table S1. These datasets should be accessed and used in accordance with the licenses, data-use agreements, and approval requirements of the original providers. The in-house SSPH brain MRI dataset is not publicly available because it contains institutionally collected clinical imaging data subject to patient-privacy and data-governance restrictions. Derived annotations, benchmark splits, and benchmark-construction scripts will be released with the project repository upon publication, where permitted by the licenses and data-use agreements of the original datasets.

|18

6 Code Availability The code and benchmark will be released at https://github.com/time-seed/MedReCo.

7 Acknowledgements This work is supported by the Innovative Drug Research and Development National Science and Technology Major Project (No. 2025ZD1803101), the Scientific Research Innovation Capability Support Project for Young Faculty (ZYGXQNJSKYCXNLZCXM-I22), the National Natural Science Foundation of China (No. 24Z031503678), and the Science and Technology Innovation Action Plan of Shanghai Municipality (No. 24QA2703800).

References [1] Ceyhun Burak Akgül, Daniel L Rubin, Sandy Napel, Christopher F Beaulieu, Hayit Greenspan, and Burak Acar. Content-based image retrieval in radiology: current status and future directions. Journal of digital imaging, 24(2):208–222, 2011. [2] Anthropic. Introducing claude 4, 2025. [3] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72, 2005. [4] Shruthi Bannur, Kenza Bouzid, Daniel C. Castro, Anton Schwaighofer, Anja Thieme, Sam Bond-Taylor, Maximilian Ilse, Fernando Pérez-García, Valentina Salvatelli, Harshita Sharma, et al. Maira-2: Grounded radiology report generation. arXiv preprint arXiv:2406.04449, 2024. [5] Pierre Chambon, Jean-Benoit Delbrouck, Thomas Sounack, Shih-Cheng Huang, Zhihong Chen, Maya Varma, Steven QH Truong, Chu The Chuong, and Curtis P Langlotz. Chexpert plus: Augmenting a large chest x-ray dataset with text radiology reports, patient demographics and additional image formats. arXiv preprint arXiv:2405.19538, 2024. [6] Yinda Chen, Che Liu, Xiaoyu Liu, Rossella Arcucci, and Zhiwei Xiong. Bimcv-r: A landmark dataset for 3d ct text-image retrieval. In International conference on medical image computing and computer-assisted intervention, pages 124–134. Springer, 2024. [7] Zhihong Chen, Yan Song, Tsung-Hui Chang, and Xiang Wan. Generating radiology reports via memorydriven transformer. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, Nov. 2020. [8] Zhihong Chen, Maya Varma, Jean-Benoit Delbrouck, Magdalini Paschali, Louis Blankemeier, Dave Van Veen, Jeya Maria Jose Valanarasu, Alaa Youssef, Joseph Paul Cohen, Eduardo Pontes Reis, et al. Chexagent: Towards a foundation model for chest x-ray interpretation. arXiv preprint arXiv:2401.12208, 2024. [9] Jooae Choe, Hye Jeon Hwang, Joon Beom Seo, Sang Min Lee, Jihye Yun, Min-Ju Kim, Jewon Jeong, Youngsoo Lee, Kiok Jin, Rohee Park, Jihoon Kim, Howook Jeon, Namkug Kim, Jaeyoun Yi, Donghoon Yu, and Byeongsoo Kim. Content-based image retrieval by using deep learning for interstitial lung disease diagnosis with chest ct. Radiology, 302(1):187–197, 2022. [10] Francesco Dalla Serra, Chaoyang Wang, Fani Deligianni, Jeff Dalton, and Alison O’Neil. Controllable chest x-ray report generation from longitudinal representations. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 4891–4904, 2023. [11] Jean-Benoit Delbrouck, Pierre Chambon, Zhihong Chen, Maya Varma, Andrew Johnston, Louis Blankemeier, Dave Van Veen, Tan Bui, Steven Truong, and Curtis Langlotz. Radgraph-xl: A large-scale expert-annotated dataset for entity and relation extraction from radiology reports. In Findings of the Association for Computational Linguistics, pages 12902–12915, 2024. [12] Elizabeth A. Eisenhauer, Patrick Therasse, Jan Bogaerts, Lawrence H. Schwartz, Daniel Sargent, Robert Ford, Janet Dancey, Susan Arbuck, Steve Gwyther, Margaret Mooney, Larry Rubinstein, Lalitha Shankar, |19

Lori Dodd, Robert Kaplan, Denis Lacombe, and Jaap Verweij. New response evaluation criteria in solid tumours: Revised recist guideline (version 1.1). European Journal of Cancer, 45(2):228–247, 2009. [13] Xiaotang Gai, Jiaxiang Liu, Yichen Li, Zijie Meng, Jian Wu, and Zuozhu Liu. 3d-rad: A comprehensive 3d radiology med-vqa dataset with multi-temporal analysis and diverse diagnostic tasks. Advances in Neural Information Processing Systems, 38, 2026. [14] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [15] Ibrahim Ethem Hamamci, Sezgin Er, Chenyu Wang, Furkan Almas, Ayse Gulnihan Simsek, Sevval Nil Esirgun, Irem Dogan, Omer Faruk Durugol, Benjamin Hou, Suprosanna Shit, et al. Generalist foundation models from a multimodal dataset for 3d computed tomography. Nature Biomedical Engineering, pages 1–19, 2026. [16] Elad Hoffer and Nir Ailon. Deep metric learning using triplet network. In Similarity-based pattern recognition: third international workshop, SIMBAD 2015, Copenhagen, Denmark, October 12-14, 2015. Proceedings 3, pages 84–92, 2015. [17] Xinyue Hu, Lin Gu, Qiyuan An, Mengliang Zhang, liangchen liu, Kazuma Kobayashi, Tatsuya Harada, Ronald Summers, and Yingying Zhu. Medical-Diff-VQA: A Large-Scale Medical Dataset for Difference Visual Question Answering on Chest X-Ray Images. PhysioNet, Feb. 2025. Version 1.0.1. [18] Xinyue Hu, Lin Gu, Qiyuan An, Mengliang Zhang, Liangchen Liu, Kazuma Kobayashi, Tatsuya Harada, Ronald M Summers, and Yingying Zhu. Expert knowledge-aware image difference graph representation learning for difference-aware medical visual question answering. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 4156–4165, 2023. [19] Shih-Cheng Huang, Liyue Shen, Matthew P. Lungren, and Serena Yeung. Gloria: A multimodal globallocal representation learning framework for label-efficient medical image recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3942–3951, 2021. [20] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590–597, 2019. [21] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2):203–211, 2021. [22] Yuanfeng Ji, Haotian Bai, Chongjian Ge, Jie Yang, Ye Zhu, Ruimao Zhang, Zhen Li, Lingyan Zhanng, Wanling Ma, Xiang Wan, et al. Amos: A large-scale abdominal multi-organ benchmark for versatile medical image segmentation. Advances in Neural Information Processing Systems, 35:36722–36732, 2022. [23] Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, et al. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668, 2025. [24] Baoyu Jing, Pengtao Xie, and Eric Xing. On the automatic generation of medical imaging reports. In Proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2577–2586, 2018. [25] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019. [26] Jiayu Lei, Lisong Dai, Haoyun Jiang, Chaoyi Wu, Xiaoman Zhang, Yao Zhang, Jiangchao Yao, Weidi Xie, Yanyong Zhang, Yuehua Li, Ya Zhang, and Yanfeng Wang. Unibrain: Universal brain mri diagnosis with hierarchical knowledge-enhanced pre-training. Computerized Medical Imaging and Graphics, 122:102516, 2025. [27] Christy Y Li, Xiaodan Liang, Zhiting Hu, and Eric P Xing. Knowledge-driven encode, retrieve, paraphrase for medical image report generation. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 6666–6673, 2019. |20

[28] Jun Li, Tongkun Su, Baoliang Zhao, Faqin Lv, Qiong Wang, Nassir Navab, Ying Hu, and Zhongliang Jiang. Ultrasound report generation with cross-modality feature alignment via unsupervised guidance. IEEE Transactions on Medical Imaging, 2024. [29] Wenxuan Li, Pedro RAS Bassi, Xinze Zhou, Jakob Wasserthal, Alan L Yuille, and Zongwei Zhou. Radthinking: A dataset for longitudinal clinical reasoning in radiology. arXiv preprint arXiv:2605.10761, 2026. [30] Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmcclip: Contrastive language-image pre-training using biomedical documents. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 525–536. Springer, 2023. [31] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024. [32] Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. arXiv preprint arXiv:2308.09442, 2023. [33] Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. Nature communications, 15(1):654, 2024. [34] Linjie Mu, Zhongzhen Huang, Shengqian Qin, Yakun Zhu, Shaoting Zhang, and Xiaofan Zhang. Mmxu: A multi-modal and multi-x-ray understanding dataset for disease progression. In Findings of the Association for Computational Linguistics: ACL 2025, pages 9785–9803, 2025. [35] Henning Müller, Nicolas Michoux, David Bandon, and Antoine Geissbuhler. A review of content-based image retrieval systems in medical applications—clinical benefits and future directions. International Journal of Medical Informatics, 73(1):1–23, 2004. [36] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. [37] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Association for Computational Linguistics, pages 311–318, 2002. [38] Eva Prakash, Yunhe Gao, Chong Wang, Justin Xu, Neal Prakash, Arne Michalson, Seena Dehkharghani, Eun Kyoung Hong, Julie Bauml, Roger Boodoo, Jean-Benoit Delbrouck, Sophie Ostmeier, and Curtis Langlotz. Chextemporal: A dataset for temporally-grounded reasoning in chest radiography. arXiv preprint arXiv:2605.11304, 2026. [39] Pranav Rajpurkar, Jeremy Irvin, Kaylie Zhu, Brandon Yang, Hershel Mehta, Tony Duan, Daisy Ding, Aarti Bagul, Curtis Langlotz, Katie Shpanskaya, et al. Chexnet: Radiologist-level pneumonia detection on chest x-rays with deep learning. arXiv preprint arXiv:1711.05225, 2017. [40] François Remy, Kris Demuynck, and Thomas Demeester. Biolord: Learning ontological representations from definitions for biomedical concepts and their textual descriptions. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 1454–1465, 2022. [41] Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583–8595, 2021. [42] Anna A. J. Roelofs, Nico Karssemeijer, Nicola Wedekind, Christoph Beck, Sabine van Woudenberg, Peter R. Snoeren, Jan H. C. L. Hendriks, Marco Rosselli Del Turco, Nils Bjurstam, Horst Junkermann, David Beijerinck, Bruno Seradour, Cees J. G. Evertsz, Linda van Erning, and Mireille J. M. Broeders. Importance of comparison of current and prior mammograms in breast cancer screening. Radiology, 242(1):70–77, 2007. [43] Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. Medgemma technical report. arXiv preprint arXiv:2507.05201, 2025. [44] Qwen Team. Qwen2.5: A party of foundation models, September 2024.

|21

[45] Fuying Wang, Shenghui Du, and Lequan Yu. Hergen: Elevating radiology report generation with longitudinal data. In Computer Vision – ECCV 2024, pages 183–200. Springer, 2024. [46] Yan-Ran Joyce Wang and Akshay S Chaudhari. Ai-driven smart patient retrieval for precision oncology. Nature Reviews Cancer, pages 1–3, 2026. [47] Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, page 3876, 2022. [48] Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Hui Hui, Yanfeng Wang, and Weidi Xie. Towards generalist foundation model for radiology by leveraging web-scale 2d&3d medical data. Nature Communications, 16(1):7866, 2025. [49] Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, et al. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044, 2025. [50] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI, 2(1):AIoa2400640, 2025. [51] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In Proceedings of the International Conference on Learning Representations. [52] Chenhui Zhao, Yiwei Lyu, Asadur Chowdury, Edward Harake, Akhil Kondepudi, Akshay Rao, Xinhai Hou, Honglak Lee, and Todd Hollon. Towards scalable language-image pre-training for 3d medical imaging. arXiv preprint arXiv:2505.21862, 2025. [53] Weike Zhao, Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Ratescore: A metric for radiology report generation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 15004–15019, 2024. [54] Ziheng Zhao, Lisong Dai, Ya Zhang, Weidi Xie, and Yanfeng Wang. Rethinking whole-body ct image interpretation: An abnormality-centric approach. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5179–5189, 2026. [55] Ziheng Zhao, Yao Zhang, Chaoyi Wu, Xiaoman Zhang, Xiao Zhou, Ya Zhang, Yanfeng Wang, and Weidi Xie. Large-vocabulary segmentation for medical images with text prompts. NPJ Digital Medicine, 8(1):566, 2025. [56] Qingqing Zhu, Tejas Sudharshan Mathai, Pritam Mukherjee, Yifan Peng, Ronald M Summers, and Zhiyong Lu. Utilizing longitudinal chest x-rays and reports to pre-fill radiology reports. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 189–198. Springer, 2023.

|22

S1 Supplementary Dataset and Construction Details S1.1 Ethics Approval. The collection and use of the in-house brain MRI dataset was approved by the Institutional Review Board of Shanghai Sixth People’s Hospital (IRB code: 2023-KY-082(K)). Informed consent was waived owing to the retrospective and non-interventional nature of the study and the use of fully de-identified data. All other datasets used in this study are publicly available and were used in accordance with their respective data use agreements. All data were de-identified before being transferred to study investigators.

S1.2 Dataset sources and roles MedReCo draws on eight image–report datasets spanning four imaging modality groups: chest X-ray (CXR), computed tomography (CT), brain magnetic resonance imaging (BRAIN), and ultrasound (US). Two independent paired-image VQA benchmarks are used solely for evaluation. Table S1 summarizes the source, modality, scale, access route, and role of each dataset. Table S1 | Dataset sources used in MedReCo. † In-house datasets are not publicly available. Dataset

Country / centre

Chest X-ray (CXR) MIMIC-CXR USA / Beth Israel Deaconess Medical Center

Modality #Patients #Studies #Images / volumes

Access

CXR

PhysioNet

Internal

Kaggle

External

65,379

227,835 377,110 images

IU-Xray

USA / Indiana University

CXR

3,955

CheXpert Plus

USA / Stanford Hospital

CXR

64,725

187,711 223,228 images

Stanford AIMI External

Computed tomography (CT) CT-RATE Turkey / Istanbul Medipol Chest CT Univ. Mega Hospital

21,304

25,692 50,188 volumes

Hugging Face

Internal

AMOS-MM

China / Longgang District Abdominal Central & People’s CT Hospital

2,300

2,300 2,300 scans

Zenodo

Internal

BIMCV-R

Spain / Valencian Region Medical Image Bank

Wholebody CT

8,069

8,069 8,069 vol. (>2M slices)

Hugging Face

External

Brain MRI

24,770

24,770 24,770 pairs (4 seq.)

Not publicly available

Internal

7,390

7,390 Paired report–image

GitHub

Internal

PhysioNet

Benchmark

GitHub

Benchmark

Brain MRI (BRAIN) SSPH† China / Shanghai Sixth People’s Hospital Ultrasound (US) CURG China / PLA General Hospital

Independent paired-image VQA benchmarks (CXR) Medical-DiffDerived from MIMIC-CXR CXR VQA MMXU

Derived from MIMIC-CXR CXR

— 1,201

3,955 7,470 radiographs

Role

164,324 164,324 QA (diff. pairs subset) 2,469 3K test + 118K dev QA

Note. SSPH contains brain MRI–report pairs across four axial sequences: T1WI, T2WI, T2FLAIR, and DWI. CURG contains organ-specific ultrasound subsets for breast (3,521), thyroid (2,474), and liver (1,395) imaging.

SSPH contains 24,770 brain MRI–report pairs across four sequences (T1WI, T2WI, T2FLAIR, and DWI), covering 13 disease categories with a 9:1 patient-level train/test split. CURG comprises three organ-specific ultrasound sub-datasets: breast (3,521 patients), thyroid (2,474 patients), and liver (1,395 patients), each with paired Chinese-language diagnostic reports. Medical-Diff-VQA provides 700,703 QA pairs in seven categories; we use the difference subset (164,324 QA; train/validation/test = 131,563/16,372/16,389). MMXU evaluates region-specific disease changes via multiple-choice questions.

S1.3 Split construction and leakage prevention All data splits are constructed at the patient level whenever patient identifiers are available. Derived samples—including image pairs, anchor–positive–negative triplets, closed-ended questions, and open-ended questions—are generated only after split assignment. This prevents train–test contamination across both retrieval and generative-comparison tasks. Table S2 | Leakage-control rules used during construction of derived comparative samples.

Derived sample type

Construction rule

Leakage-control rule

Used for

Entity-level report content

Extracted independently from each report for predefined entities.

Extraction is performed after dataset access control; no cross-split context is used.

Retrieval and VQA supervision

Cross-patient image pair

Two cases from the same modality group are paired to compare a target entity.

Both cases must come from the same split.

Broad comparative VQA

Same-patient longitudinal pair

Prior and current studies are ordered temporally within the same patient.

The patient cannot appear in more than one split.

Longitudinal follow-up VQA

Retrieval triplet

Anchor, positive, and negative are selected according to entity-level report similarity.

Anchor, positive, and negative must come from the same split during training/evaluation construction.

Stage-1 contrastive ranking

External retrieval gallery

Query and gallery are formed from held-out centres.

External cohorts are excluded from training and checkpoint selection.

External and cross-centre validation

S1.4 Report-mining and comparative-supervision pipeline MedReCo is constructed from paired images and radiology reports through a shared report-mining pipeline. The same parsed entity-level report content supports both reference comparison and temporal comparison: entity-level agreement between reports provides ranking supervision for controllable retrieval, whereas entitylevel contrast between reports provides instruction–response supervision for comparative interpretation. Report decomposition. For each report, we extract entity-specific content at three levels: anatomy, abnormal finding, and disease. Anatomy prompts extract all report content related to a specific structure. Abnormality prompts extract the presence, location, size, morphology, or characteristics of a target abnormality within a target anatomy. Disease prompts extract content directly related to a target disease or condition. Explicitly negated statements are retained because the absence of an entity can be clinically meaningful when comparing studies. Comparative VQA generation. For each pair of reports, we generate questions that compare Case A and Case B with respect to a specific target abnormality or entity. The questions focus on dimensions such as presence/absence, location, size, morphology, margins, internal characteristics, associated findings, and potential etiology. VQA filtering and review. Generated questions are filtered to remove items that do not require two images, are answerable from language priors alone, contain biased or duplicate answer patterns, or lack clinical specificity. The filtering stages are summarized in Table S4.

Table S3 | Overview of the report-mining and comparative-supervision pipeline.

Step

Module

Input

Output

Entity-specific report decomposition

Radiology report and a predefined entity from the modality-specific vocabulary.

Entity-specific text span or null output; explicit positive and negative statements are retained.

Mention-status assignment

Extracted text span for anatomy, abnormality, or disease.

Mention status: positive, explicitly negative, uncertain, or not mentioned.

Disease-term normalization

Disease mentions extracted from reports.

Standardized disease terms after synonym clustering and manual curation.

Entity-level similarity quantification

Entity-specific report content from two studies.

Report-derived similarity scores used to supervise entity-conditioned visual similarity.

Retrieval triplet construction

Anchor study, candidate studies, entity condition, and report-derived similarity.

Anchor–positive–negative triplets for text-guided contrastive ranking.

Comparative VQA generation

Two reports, target anatomy, target abnormality or disease, and entity-specific descriptions.

Multiple-choice comparative questions with answer and rationale.

Open-ended QA conversion

Multiple-choice question, options, answer, and rationale.

Short-answer comparative question–answer pair.

VQA filtering and review

Generated closed-ended and open-ended questions.

Final VQA samples requiring genuine paired-image reasoning.

Table S4 | Filtering stages for comparative VQA construction.

Filtering stage

Retain

Remove

Output

Image-dependency filter

Questions that strictly require examining both Case A and Case B.

Single-image identification, isolated anatomy description, or non-comparative questions.

Paired-imagedependent VQA

Language-prior filter

Questions whose answer cannot be inferred from the wording alone.

Questions with deterministic linguistic cues, trivial option structure, or obvious answer priors.

Visually grounded VQA

Bias and duplicate filter

Balanced answer distribution and non-duplicated question forms.

Repeated stems, near-duplicate options, and systematic answer-position bias.

De-duplicated VQA

Clinical review

Clinically meaningful, factual, and unambiguous comparisons.

Clinically irrelevant, ambiguous, contradictory, or insufficiently grounded samples.

Final comparative VQA

S1.5 Benchmark composition We further summarize the sample-level composition of the benchmarks used for entity-conditioned retrieval and generative comparative VQA. For retrieval, counts are reported as retained query–entity sample rows after filtering. Internal samples are split into training and held-out test sets, whereas external samples are drawn only from held-out datasets. For VQA, counts are reported by source dataset and entity level. Table S5 | Benchmark composition for entity-conditioned retrieval. Counts indicate retained query–entity sample rows after filtering. Modality group

Entity level

Internal Internal test train

External test

External source(s)

BRAIN

Abnormality

37,296

4,242

—

BRAIN

Anatomy

20,540

2,281

—

BRAIN

Disease

63,223

7,090

—

Abnormality

97,102

6,376

3,770

Anatomy

115,547

7,511

5,174

BIMCV-R

Disease

77,752

6,026

4,120

BIMCV-R

CXR

Abnormality

294,034

5,672

5,846

CheXpert Plus (2,094); IU-Xray (3,752)

CXR

Anatomy

557,330

10,784

9,240

CheXpert Plus (3,736); IU-Xray (5,504)

CXR

Disease

122,026

5,070

4,988

Abnormality

13,290

1,426

—

Anatomy

15,242

1,640

—

Disease

2,062

262

—

Total

All

1,293,418

58,380

33,138

—

BIMCV-R

CheXpert Plus (1,624); IU-Xray (3,364) —

CheXpert Plus, IU-Xray, and BIMCV-R

Note. Internal retrieval samples are grouped by modality family. External test samples are from held-out datasets only: CheXpert Plus and IU-Xray for CXR, and BIMCV-R for CT. No external BRAIN or US retrieval cohort was included in the current benchmark.

Table S6 | Test-set composition for generative comparative VQA. Counts indicate the number of VQA samples by source dataset and entity level. Dataset

Abnormality

Anatomy

Disease

Per-version total

Overall total

AMOS-MM

7,212

8,794

14,430

15,218

30,436

BIMCV-R

4,798

972

600

3,185

6,370

Brain MRI

15,606

4,996

11,684

16,143

32,286

CT-RATE

24,932

8,496

12,726

23,077

46,154

CheXpert Plus

1,746

3,424

1,820

3,495

6,990

MIMIC-CXR

21,614

1,982

7,522

15,559

31,118

CURG / US

36,798

10,622

3,330

25,375

50,750

112,706

39,286

52,112

102,052

204,104

Total

Note. The source statistics contain two matched VQA test variants, denoted as “1113” and “free”. The entity-level columns and the overall total sum both variants; the per-version total reports the size of either variant. Independent public CXR VQA benchmarks are reported separately in Table S1: Medical-Diff-VQA uses the difference-subset test split with 16,389 QA, and MMXU contains 3K test QA.

S2 Supplementary Clinical Entity Groups We organize clinical entities into three granularities—anatomical structures, abnormal findings, and pathological conditions—and group them by four modality families: CXR, CT, BRAIN, and US. The modality-level coverage is summarized in Table S9; the complete entity vocabulary is listed below. |4

Table S7 | Composition of the longitudinal follow-up VQA benchmark by modality and clinical target. Modality

Source cohort

Anatomical group

Finding group

Samples

Percentage

CXR

CheXpert Plus

Pulmonary parenchyma

CXR

CheXpert Plus

Pulmonary parenchyma

Ground-glass opacity

343

36.88%

Lung consolidation

195

20.97%

CXR subtotal

—

538

57.85%

CT CT

CT-RATE

Pulmonary parenchyma

Ground-glass opacity

126

13.55%

CT-RATE

Pulmonary parenchyma

Lung consolidation

122

13.12%

CT-RATE

Liver / hepatic parenchyma

Soft-tissue density lesion

8.71%

CT-RATE

Liver / hepatic parenchyma

Decreased hepatic parenchymal density

3.87%

CT-RATE

Abdominopelvic cavity

Peritoneal findings

2.90%

CT subtotal

—

392

42.15%

Total

—

930

100.00%

Percentages were calculated over the final filtered longitudinal follow-up VQA benchmark of 930 samples. Samples labelled as unable to compare or uncompare were excluded before tabulation. Categories with fewer than five samples after pooling CXR and CT were merged into broader clinically related categories and were not reported as separate strata. Table S8 | Composition of the longitudinal follow-up VQA benchmark by follow-up attribute and interval-change label. Category type