A report-grounded vision–language foundation model for colonoscopy from 280,000 routine reports Jia Yu1,2,4,8† , Yan Zhu2,5† , Yili He1,2 , Zilong Wang8 , Xinyang Jiang8 , Peiyao Fu2,5 , Ruijie Yang4 , Tianyi Chen2 , Siyuan Li2 , Zhihua Wang3,4 , Fei Wu3,4 , Quanlin Li2,5*, Xian Yang6,7*, Pinghong Zhou2,5*, Shuo Wang1,2,7*
arXiv:2607.28466v1 [cs.AI] 30 Jul 2026
1
Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China. 2 Shanghai Collaborative Innovation Center of Endoscopy, Shanghai, China. 3 Zhejiang University, Hangzhou, China. 4 Shanghai Institute for Advanced Study of Zhejiang University, Shanghai, China. 5 Endoscopy Centre and Endoscopy Research Institute, Zhongshan Hospital, Fudan University, Shanghai, China. 6 Alliance Manchester Business School, The University of Manchester, Manchester, UK. 7 Data Science Institute, Imperial College London, London, UK. 8 Microsoft Research Asia, Shanghai, China.
*Corresponding author(s). E-mail(s): [email protected]; [email protected]; [email protected]; [email protected]; † These authors contributed equally to this work. Abstract Vision–language models remain underused in colonoscopy despite the rich expert descriptions recorded in routine reports. These reports document lesion appearance, size and location but summarise entire procedures rather than caption individual frames, leaving clinical findings only weakly linked to the corresponding images. Here we develop EndoCLIP, a colonoscopy vision–language
1
foundation model trained on 125,756 lesion-level image–text pairs progressively recovered from 280,476 routine colonoscopy records. Across lesion-level image–text retrieval, structured report generation and six multi-centre clinical classification tasks, EndoCLIP outperforms general-purpose and biomedical vision–language encoders in both zero-shot and linear-probe settings. On benignversus-malignant classification, its linear probe approaches the performance of expert readers in a blinded study involving 12 endoscopists. These results suggest that recovering finding-to-frame correspondence can transform routine documentation into scalable supervision, enabling clinical targets to be specified in language rather than separately annotated for each task. Keywords: colonoscopy, medical foundation model, vision–language contrastive learning
1 Introduction Colonoscopy is central to the prevention of colorectal cancer, a leading cause of cancer-related mortality worldwide [1–4]. AI-assisted systems have improved adenoma detection and supported the automated detection and classification of colorectal lesions [5–8]. However, most such systems rely on task-specific models trained with densely annotated and carefully curated images, which is difficult to scale across the diverse findings encountered in routine colonoscopy. Their robustness and generalisability across patient populations, imaging systems and clinical centres also remain concerns [9–11]. Self-supervised visual pretraining can reduce this dependence on manual annotation [12]. Visual foundation models such as GastroNet [10] and EndoFM [13] learn transferable visual features from large collections of unlabelled endoscopy frames and videos. These features are not linked to clinical language and therefore do not directly support text-guided retrieval or prompt-based recognition. Vision–language pretraining instead aligns images with text, allowing clinical concepts to be specified in natural language [14–16]. In radiology and pathology, this approach has been successful in zero-shot and label-efficient recognition [17–22], while in colonoscopy it remains less explored [23]. Routine colonoscopy reports describe lesion morphology, surface appearance, size and anatomical location but summarise the examination rather than caption individual frames. A procedure may contain dozens of still frames, many without a clear lesion view, and one report may describe several lesions without linking findings to frames. The report and visual record are therefore only coarsely aligned, especially in multi-lesion procedures (Fig. 1). Here we introduce EndoCLIP, a vision–language foundation model that recovers finding-to-frame correspondence through case-level evidence localisation, single-lesion anchor selection and multi-lesion disambiguation. Applied to 280,476 routine reports, this approach yields 125,756 lesion-level image–text pairs for contrastive pretraining (Fig. 1). We also introduce EndoReport100 [24], a clinician-curated benchmark of 100 multi-lesion cases and 7,002 frame-level annotations for evaluating lesion-level retrieval. EndoCLIP achieves a bidirectional global Recall@1 of 14.3%, compared with 2
2.8% for the strongest comparator. Furthermore, we validated the performance of EndoCLIP on six clinical classification tasks and structured report generation in a multi-centre setting [25]. Its frozen features achieve the highest zero-shot and labelefficient classification performance among the compared vision–language encoders and yield the most accurate structured outputs. These results suggest that recovering finding-to-frame correspondence can turn routine reports into scalable supervision for clinically relevant endoscopy tasks with little or no task-specific annotation. a
b
e
c
d
f
g
Fig. 1: Weak report–image alignment and analysis of the pretraining dataset. a, Routine reports correspond to multi-frame, often multi-lesion cases. b, Dataset curation funnel, from 280.5k de-identified reports to 123.8k polyp-positive reports and a curated training set of 104.5k cases (37.3%) yielding 125.8k image–text pairs. c, Anatomical-site coverage across the 123.8k polyp reports. d, Frequency of morphology and surface descriptors. e, Lesion-size distribution. f, Report and findingsentence length in words. g, Frames per case (n = 123.8k cases, 8.54M frames; median 65, mean 69). In e–g, bars mark distribution percentiles (P25–P99).
3
Stage I: Case - Level Evidence Localisation image - text pairs II (63.6k)
image - text pairs I (123.8k) “Polyp present”
per polyppositive patient (123.8k)
text
N frames
Contrastive Learning
similarity by EndoCLIP-I
text
select top p frames
image - text pairs (156.7k)
Contrastive Learning
image - text pairs (156.7k) EndoCLIP-I
EndoCLIP-II
random
“No Finding”
per polypnegative patient (156.7k)
“Polyp present”
text
“No Finding” random frames
M frames
text
Stage II: Single-Lesion Anchor Selection image - text pairs III (69.6k)
“Polyp present”
corresponding single lesion description
Contrastive Learning
...
similarity by EndoCLIP-II
choose top-1 frame ...
per single-polyp patient (69.6k)
P frames
EndoCLIP-II
“A polyp, reddish in color, with a smooth and rounded surface” selected lesion frame
P frames with a single lesion
corresponding single lesion description
EndoCLIP-III
Stage III: Multi-Lesion Correspondence Disambiguation Lesion description 1 descriptions corresponding to L lesions
Q frames with L lesions
similarity by EndoCLIP-III
multi-polyp instance matching
Contrastive Learning
Lesion description 1
...
Lesion description 2 ... Lesion description L
...
per multi-polyp patient (54.2k)
Q frames
Lesion description L
EndoCLIP-III
image - text pairs IV from multi-polyp patients (56.2k)
...
des crip tion 1 des crip tion 2 des crip tion 3 des crip tion L
Lesion description 2 ...
EndoCLIP
image - text pairs III from single-polyp patients (69.6k)
Evaluation and Application Lesion-level alignment Image to text and text to image retrieval Candidate pool
Text to Image Lesion description
Image to Text
Query
Zero Shot similarity by EndoCLIP
... 0.01
Query
0.54
0.07
Top matched 0.10
Lesion description 1
0.08
Lesion description 2
0.66 Top matched
Lesion description 3 ...
Transferable diagnosis Morphology and pathology classification
Lesion Image
Positive description
Positive lesion
Negative description
Negative lesion
Structured clinical reporting Lesion-level structured reporting
EndoCLIP Features Lesion Image Instruction
Projection
Linear Probe Positive lesion
EndoCLIP Features
Candidate pool
Negative lesion
Lesion Image Linear Layer
Large Language Model "Diameter(cm)":"0.4","ParisType":"0-Isp", "Description":"Congestion,Smooth, Narrow-base,Mucus attachment"
Fig. 2: Overview of EndoCLIP. Routine reports describe multi-frame, sometimes multi-lesion procedures rather than individual images. EndoCLIP derives lesion-level image–text pairs from this case-level signal through case-level evidence localisation, single-lesion anchor selection and multi-lesion disambiguation. The resulting encoder is evaluated on retrieval, classification and structured report generation.
4
2 Results 2.1 Overview of EndoCLIP EndoCLIP learns from routine free-text reports generated during colonoscopy. Because each report documents an entire examination rather than individual frames, its finding sentences are not explicitly linked to the frames that show the corresponding lesions (Fig. 1). To recover this correspondence, we used a three-stage progressive training procedure that localises case-level visual evidence, selects anchors from singlelesion cases and disambiguates finding-to-frame matches in multi-lesion reports (Fig. 2; Methods, Sec. 4.4.2). Applied to 280,476 reports and their still frames, this procedure yields 125,756 lesion-level image–text pairs for contrastive pretraining (Supplementary Table S1). Contrastive training with coarse case-level prompts (“polyp present” or “no finding”) produces EndoCLIP-I, which localises polyp-containing frames. Retraining on the retained high-scoring frames yields EndoCLIP-II, which filters uninformative background. Matching finding sentences to frames in single-lesion cases with EndoCLIP-II produces EndoCLIP-III, the first checkpoint to encode descriptive lesion semantics. Disambiguating multi-lesion cases with EndoCLIP-III and combining all recovered pairs then produces EndoCLIP, the completed model evaluated below; the intermediate checkpoints are used only in stage-wise analyses. We benchmark EndoCLIP against general-purpose and biomedical vision–language encoders across lesion-level retrieval (Fig. 3), six clinical classification tasks spanning a multi-centre public benchmark and an independent pathology cohort (Fig. 4 and Table 1), embedding-space organisation (Fig. 5) and structured report generation (Fig. 6).
2.2 Report-grounded lesion retrieval Lesion-level retrieval tests whether EndoCLIP encodes lesion-specific clinical semantics in its shared representation. We use EndoReport100, a clinician-curated benchmark of 100 multi-lesion cases and 217 lesions that is fully independent of the pretraining corpus. Experienced endoscopists marked polyp presence in each frame and linked every lesion to a representative frame and its finding sentence (Fig. 3a). The global protocol compares each query with all candidates, whereas the intra-case protocol restricts candidates to lesions documented in the same case (mean, 2.2 per case). General-purpose and biomedical encoders show limited lesion-level retrieval. Under the global protocol, where chance Recall@1 is 0.46%, PMC-CLIP [26] reaches a mean Recall@1 of 2.8% (95% CI 1.2–4.7), and the general-purpose CLIP variants perform similarly to the randomly initialised network. EndoCLIP reaches a mean Recall@1 of 14.3% (95% CI 10.1–18.4) across the two retrieval directions and reduces the median matched-item rank to 8–9, compared with 53–76 for the baselines. The paired difference from PMC-CLIP is 11.5 percentage points (p < 0.001, case-level bootstrap). The same ordering is observed under the intra-case protocol, which has a 46% chance level. PMC-CLIP reaches 53.3% (95% CI 48.7–58.1), and EndoCLIP reaches 81.2% (95% CI 75.9–86.0; p < 0.001; Fig. 3b–d; full results in Supplementary Table S3 and Supplementary Fig. S1).
5
a
b
c
d
e
Fig. 3: Lesion-level image–text retrieval. a, Global and intra-case protocols. b,c, Image-to-text (b) and text-to-image (c) retrieval, each showing Recall@K curves, rank statistics, mean reciprocal rank (MRR) and intra-case Recall@1. d, Per-case intra-case Recall@1 across all multi-lesion cases. e, Performance across successive EndoCLIP training stages.
6
Successive checkpoints reveal when descriptive retrieval emerges. EndoCLIP-I and EndoCLIP-II, trained only on the Stage I “polyp present” and “no finding” prompts, detect polyp-containing frames but do not support descriptive lesion retrieval. They reach frame-level AUROCs of 0.843 and 0.889 on the 7,002 labelled EndoReport100 frames (Supplementary Fig. S4 and Supplementary Tables S16–S17). Their global mean Recall@1 values on descriptive retrieval are 1.2% and 0.9%, respectively, and their intra-case Recall@1 is 46.3% (Fig. 3e). EndoCLIP-III, the first checkpoint trained on selected single-lesion sentence–frame pairs, reaches a global mean Recall@1 of 9.2% (95% CI 5.7–13.1) and an intra-case Recall@1 of 75.3% (95% CI 70.0–80.6). Relative to EndoCLIP-III, the completed EndoCLIP increases global and intra-case Recall@1 by 5.1 percentage points (p = 0.02) and 5.8 percentage points (p = 0.003), respectively. Full values are given in Supplementary Table S4 and Supplementary Fig. S2; the case-representative protocol shows the same ordering (Supplementary Table S19). A control encoder trained on the same corpus with naive case-level pairing, in which each case contributes one randomly matched frame–sentence pair, reaches a global mean Recall@1 of 6.5% (95% CI 3.9–9.5) and an intra-case Recall@1 of 70.2% (95% CI 63.5–76.7), below the completed model by 7.8 and 10.9 percentage points, respectively (both p < 0.001; Supplementary Table S21).
2.3 Multi-centre zero-shot and label-efficient recognition We evaluate EndoCLIP across six clinical tasks that combine a multi-centre external benchmark with an independent pathology cohort. Five tasks come from EndoVL [25], which spans nine public source datasets: four-class Paris morphology, mucosal congestion, lobulated surface, villous surface and polyp size. The sixth is benignversus-malignant classification on a Zhongshan Hospital cohort. These targets carry direct clinical weight: Paris type and surface pattern inform invasion risk and resection planning [27, 28], lesion size at the 10 mm threshold sets the surveillance interval [29], and the benign–malignant distinction separates endoscopic from surgical management. All six tasks are assessed under three regimes: prompt-based zero-shot scoring, linear probing on frozen encoders and end-to-end training, with the two supervised regimes using 5–100% of the training partition (Fig. 4; Table 1; Supplementary Tables S2 and S13–S14). Without task-specific fitting, EndoCLIP achieves the highest AUC-ROC among the compared encoders on every classification task. Across the external morphology and surface tasks, AUC-ROC is 0.766–0.851, exceeding PMC-CLIP [26] by 0.176– 0.230; on in-house pathology, it is 0.775, a margin of 0.138 (Fig. 4a; Supplementary Fig. S3 and Supplementary Table S7). For external polyp-size classification at the 10 mm threshold, EndoCLIP reaches 0.670, compared with 0.567 for the strongest comparator (Table 1a). On lobulated surface, BiomedCLIP [30] attains a marginally higher accuracy than EndoCLIP (0.843 versus 0.832) with a lower sensitivity and F1 (0.015 and 0.029 versus 0.746 and 0.581). Class-imbalanced predictions also occur for BiomedCLIP on mucosal congestion (sensitivity 0.972, specificity 0.052) and for PMC-CLIP on polyp size (sensitivity 0.002, specificity 0.998). Across the external binary tasks, EndoCLIP
7
a
b
c
Fig. 4: Multi-centre zero-shot and label-efficient clinical classification. Five tasks are shown: four EndoVL morphology and surface tasks [25] and pathology from Zhongshan Hospital; polyp size is reported in Table 1. a, Prompt-based zero-shot performance, with colour encoding fold change over the random baseline. b,c, AUCROC, F1 and accuracy across five training-data fractions under linear probing and end-to-end training. Lines and shaded bands show the mean and s.d. over three seeds.
8
has a sensitivity between 0.623 and 0.746 and a specificity between 0.827 and 0.863 (Supplementary Table S7). Linear probes on frozen EndoCLIP features have the highest mean AUC-ROC at every label budget. With 5% of labels, AUC-ROC reaches 0.860 on pathology and 0.908 on polyp size; with the full training partitions, it reaches 0.878–0.941 across morphology, surface and pathology and 0.946 on polyp size (Fig. 4b,c; Table 1b; Supplementary Tables S8–S10 and S14). The margin is widest where labels are scarcest. On polyp size, the EndoCLIP probe fitted to 5% of labels already exceeds every comparator probe fitted to the full labelled set (0.908 versus at most 0.845), and on pathology and Paris morphology the 5%-label probe likewise exceeds the CLIPOpenAI probe fitted to all labels (0.860 versus 0.767 and 0.780 versus 0.725). The ordering is already present in the frozen features themselves: EndoCLIP has the highest k-nearest-neighbour leave-one-out accuracy on every task, from 0.702 to 0.906
Table 1: Label-efficient polyp-size classification on the multi-centre EndoVL benchmark. Polyps are dichotomised at the clinically established 10 mm (1 cm) threshold that separates small from large lesions [29]. The biomedical comparators are BiomedCLIP [30] and PMC-CLIP [26]. (a) Zero-shot prediction from clinical prompts. (b) Label-efficient transfer at five training-data fractions under two regimes: a linear probe on the frozen image encoder and end-to-end training with all weights updated; accuracy and AUC-ROC are reported as the mean±s.d. over three seeds. End-to-end training comprises full fine-tuning of EndoCLIP and pretrained ViT-L/14 and training ViT-L/14 from scratch. Within each regime the best entry per column is shown in bold and the second best is underlined. (a) Zero-shot (text prompts, no training labels) Method
Acc.
AUROC
F1
Sens.
Spec.
Prec.
BiomedCLIP PMC-CLIP CLIP-OpenAI EndoCLIP
0.556 0.565 0.556 0.624
0.567 0.378 0.531 0.670
0.533 0.004 0.377 0.631
0.585 0.002 0.310 0.739
0.534 0.998 0.745 0.536
0.491 0.500 0.483 0.550
50%
100%
(b) Label-efficient transfer (accuracy / AUC-ROC, mean±s.d.) Method
Metric
5%
10%
20%
Linear probe (frozen image encoder) Acc. 0.692±0.021 0.726±0.029 0.760±0.016 0.758±0.017 0.769±0.017 BiomedCLIP AUC 0.783±0.009 0.792±0.016 0.827±0.006 0.833±0.003 0.844±0.006 Acc. 0.694±0.015 0.697±0.045 0.754±0.010 0.743±0.018 0.763±0.007 PMC-CLIP AUC 0.764±0.031 0.761±0.043 0.820±0.008 0.831±0.006 0.845±0.008 Acc. 0.608±0.050 0.673±0.037 0.716±0.002 0.732±0.015 0.757±0.021 CLIP-OpenAI AUC 0.716±0.008 0.735±0.042 0.801±0.013 0.818±0.006 0.841±0.014 Acc. 0.832±0.013 0.847±0.022 0.869±0.025 0.878±0.014 0.882±0.026 EndoCLIP AUC 0.908±0.020 0.922±0.028 0.934±0.023 0.941±0.020 0.946±0.019 End-to-end training (all weights updated) Acc. 0.606±0.030 0.625±0.023 0.656±0.035 0.718±0.015 0.742±0.008 ViT-L/14 (scratch) AUC 0.627±0.021 0.638±0.028 0.717±0.038 0.777±0.009 0.813±0.014 Acc. 0.761±0.013 0.816±0.011 0.875±0.019 0.903±0.011 0.914±0.022 ViT-L/14 (pretrained) AUC 0.822±0.004 0.885±0.009 0.929±0.013 0.953±0.010 0.960±0.012 Acc. 0.851±0.014 0.865±0.018 0.904±0.013 0.908±0.013 0.913±0.019 EndoCLIP AUC 0.924±0.016 0.933±0.018 0.958±0.009 0.955±0.010 0.959±0.009
9
Fig. 5: Clinical organisation of the embedding space. Rows show t-SNE projections of frozen image embeddings for pathology, Paris morphology, villous surface, lobulated surface and congestion; columns compare four encoders. Points are coloured by ground-truth class, and panels report kNN leave-one-out accuracy. The morphology and surface tasks use EndoVL; pathology is in-house.
10
against 0.646 to 0.898 for PMC-CLIP, the two models being closest on lobulated and villous surface, where the gap is 0.008 (Fig. 5; Supplementary Table S12). Frozen comparator features also struggle to separate the minority classes. On villous and lobulated surface, probes on BiomedCLIP and CLIP-OpenAI features stay at the majority-class solution at every label fraction, with an F1 of at most 0.011, and PMC-CLIP reaches only 0.314 and 0.488 with the full labelled set. EndoCLIP probes are the only ones to produce a non-trivial minority-class F1 at the smallest budgets, reaching 0.069 and 0.168 with 5% of labels and 0.515 and 0.655 with all labels (Supplementary Tables S9 and S10). The largest margins occur on villous and lobulated surface, the two rarest descriptors in the pretraining corpus, present in 0.7% and 2.4% of polyp reports (Fig. 1d). Both are closely associated with advanced histology [29]. Under end-to-end training, fully fine-tuned EndoCLIP leads the morphology, surface and pathology comparisons in mean AUC-ROC at every label fraction. Its AUC-ROC ranges from 0.754 to 0.888 at 5% of labels and from 0.860 to 0.930 at 100% (Fig. 4b,c). On polyp size, fully fine-tuned EndoCLIP performs best through the 50% label fraction and is similar to the pretrained ViT-L/14 baseline with all labels (0.959 versus 0.960). To relate model performance to clinical readers, we compare EndoCLIP with 12 endoscopists from three experience tiers who were blinded to the histopathological reference (Supplementary Table S15). Reader accuracy ranges from 0.716±0.014 for novices to 0.846±0.014 for experts. The frozen EndoCLIP probe reaches an accuracy of 0.852±0.010, an AUC-ROC of 0.941±0.002 and a specificity of 0.896, compared with a mean reader specificity of 0.704. Exploratory McNemar tests give p = 0.86 against the expert vote and p < 0.05 against the novice-tier vote and each external encoder.
2.4 Structured report generation from frozen visual features Structured report generation tests whether frozen representations retain clinically reportable information. Each image encoder is connected through a trainable multilayer projector to the same frozen Qwen3-14B decoder (Fig. 6a). Given a fixed instruction, the decoder produces JSON containing lesion diameter, Paris type and descriptors from a 19-term vocabulary. All comparisons use the same projector design and training protocol, with separate projector weights fitted to accommodate different encoder dimensions. The EndoVL structured-report set comprises 1,375 training, 197 validation and 394 held-out test images. We compare CLIP-OpenAI, BiomedCLIP [30], PMC-CLIP [26] and EndoCLIP on the test set. EndoCLIP produces the most accurate structured outputs among the encoders compared here (Fig. 6b–d; Supplementary Tables S5 and S6). Its descriptor-set F1 is 0.764, with a Jaccard index of 0.683 and an exact-set-match rate of 0.442; the corresponding values are 0.729, 0.647 and 0.401 for PMC-CLIP and 0.716, 0.631 and 0.401 for BiomedCLIP. EndoCLIP has the highest F1 on six of the eight most frequent descriptors, and its largest gains fall on the two findings that the comparators recover least well: villous surface (0.607, against 0.562 for BiomedCLIP and 0.463 for PMC-CLIP) and lobulated surface (0.660, against 0.593 for PMC-CLIP). PMCCLIP remains ahead on well-defined margins (0.794 versus 0.766) and is level on flat morphology (0.759 versus 0.758). 11
Paris typing follows the same pattern. Overall accuracy is 0.731 for EndoCLIP, 0.685 for PMC-CLIP, 0.650 for BiomedCLIP and 0.579 for CLIP-OpenAI, and EndoCLIP is the most accurate encoder on three of the four classes, with per-class accuracy between 0.718 and 0.747 and PMC-CLIP ahead only on 0-IIa (0.746 versus 0.718; Fig. 6c). Among the misclassified images, 91% are assigned to an adjacent category on a
b
c
d
f
e
g
h
Fig. 6: Structured report generation from frozen features. a, A frozen vision encoder is connected by a trainable multilayer projector to a frozen Qwen3-14B decoder. b, Summary radar over descriptor-set agreement, Paris typing and size tolerance. Each spoke is scaled to the best-performing encoder and labelled with that best native value; only EndoCLIP is shaded. c, Per-class Paris typing accuracy. d, Per-descriptor F1 for eight findings. e, Report text-quality scores (ROUGE-L and CIDEr). f, Fraction of size estimates within increasing relative-error tolerances. g, Row-normalised EndoCLIP confusion matrix for Paris typing; rows are ground truth and the diagonal reproduces the per-class accuracies in c. h, Lesion-size mean absolute error (MAE, cm) and mean absolute percentage error (MAPE). The vision encoder and decoder are frozen; a separate projector with the same architecture and training protocol is fitted for each encoder. Held-out EndoVL test set, n = 394.
12
the protrusion continuum (Fig. 6g). Flat-elevated lesions are typed as sessile in 25% of cases and sessile lesions as flat-elevated in 11%, pedunculated lesions are typed as sub-pedunculated in 21%, and sub-pedunculated lesions divide between their sessile and pedunculated neighbours (15% and 9%). EndoCLIP also gives the closest lesion-size estimates. Its mean absolute error is 0.173 cm, against 0.214 cm for PMC-CLIP, 0.246 cm for BiomedCLIP and 0.267 cm for CLIP-OpenAI, and its mean relative error is 0.211, against 0.269, 0.337 and 0.380 (Fig. 6h). Predicted size falls within 10%, 20% and 50% of the reference for 40.4%, 65.0% and 92.1% of lesions, compared with 35.5%, 58.1% and 89.1% for PMC-CLIP, and EndoCLIP leads at every tolerance (Fig. 6f). Text-similarity scores give the same ranking, with CIDEr, ROUGE-L and Levenshtein similarity of 4.573, 0.873 and 0.956 against 4.182, 0.861 and 0.951 for PMC-CLIP (Fig. 6e).
3 Discussion Vision–language pretraining remains uncommon in gastrointestinal endoscopy. Most endoscopy foundation models learn from images or videos and are evaluated on predefined visual tasks [10, 13]. In radiology and pathology, by contrast, paired clinical text supports retrieval, zero-shot recognition and language generation [21, 22]. EndoCLIP extends this paradigm to colonoscopy by learning jointly from routine reports and procedure frames. The resulting representation supports retrieval from clinical descriptions, prompt-defined recognition, adaptation to local labels and structured reporting. Clinical language can therefore provide a common interface across tasks that otherwise need separate annotations and output heads. Routine reports, however, do not provide the image–caption pairs used in standard CLIP training [14, 26, 30]. The unit of documentation is the procedure, whereas the desired unit of cross-modal learning is a lesion-level frame–sentence pair. A report can summarise many frames and several lesions, so direct case-level pairing risks associating a finding sentence with another lesion or with normal mucosa. EndoCLIP narrows this many-to-many ambiguity progressively, beginning with case-level prompts that localise relevant evidence. Single-lesion reports then provide low-ambiguity descriptive anchors for sentence-to-frame disambiguation in multi-lesion cases. This procedure converts case-level weak supervision into finding-to-frame correspondences for cross-modal training. The checkpoint sequence separates coarse polyp detection from report-grounded lesion retrieval. EndoCLIP-I and EndoCLIP-II reach frame-level AUROCs of 0.843 and 0.889. Despite this, their global descriptive Recall@1 remains 1.2% and 0.9%, and their intra-case Recall@1 remains at the 46.3% chance level. Descriptive training on single-lesion anchors raises these values to 9.2% globally and 75.3% within cases for EndoCLIP-III. Adding disambiguated multi-lesion pairs further raises them to 14.3% and 81.2%, gains of 5.1 and 5.8 percentage points, respectively. Stage-wise classification shows the largest later-stage gains on morphology-rich tasks when 5% of labels are available (Supplementary Table S11). These results separate coarse frame-level lesion detection from descriptive lesion retrieval, which emerges after training on lesion-level sentence–frame pairs.
13
The shared image–text space also changes how a downstream task is specified. An image-only encoder generally requires labelled examples and a task-specific classifier for each new target. EndoCLIP can instead score clinical prompts directly, while retaining the option of a linear probe or end-to-end adaptation when local labels are available. It achieves the highest zero-shot AUC-ROC among the compared encoders on all six classification tasks and exceeds the strongest comparator by 0.176–0.230 across the four external morphology and surface tasks. These targets include Paris morphology, surface appearance and lesion size, which inform invasion risk, resection strategy and surveillance [27–29]. This capability may be relevant to rare or newly defined phenotypes for which large labelled cohorts are difficult to assemble [21, 22]. Structured report generation poses a stricter test because a clinically useful report must reflect the image rather than merely sound plausible. Previous multi-centre studies have developed image-based structured reporting and prospectively evaluated domain-specific multimodal models for upper-gastrointestinal diagnosis and reporting [31, 32]. GI-Bench shows that readable outputs from general-purpose multimodal language models may still contain localisation errors, factual inaccuracies and hallucinated visual findings [33]. With the decoder frozen, EndoCLIP yields the most accurate descriptor sets, Paris types and lesion-size estimates among the encoders compared. These results highlight the value of endoscopy-specific image–text data and visually grounded encoders beyond language-model fluency alone, and support structured drafts for endoscopist review as a clinically relevant use of endoscopy-specific vision–language models. The potential clinical value of these capabilities lies in assisting endoscopists across several parts of the workflow. Language-based retrieval could support case review and teaching by locating lesions from textual descriptions, extending the imagequery paradigm used in EndoFinder [34]. Prompt-based recognition and lightweight local adaptation could support quality assurance when annotation is limited, while structured outputs could reduce repetitive documentation. EndoReport100 provides a benchmark for developing frame-level detection and report-grounded retrieval in multi-lesion procedures. Evaluation on EndoVL and the independent Zhongshan Hospital pathology cohort provides evidence of transfer across heterogeneous data sources beyond the pretraining corpus [25]. On pathology, the frozen probe reaches an accuracy close to the expert-tier point estimate in the reader study. Prospective workflow studies can now determine whether these capabilities improve clinical efficiency or decision-making. Several features of the current evidence constrain the scope of these conclusions. Lesion sizes in the pretraining corpus are parsed from report text rather than measured directly, and EndoReport100, although independent of pretraining, comes from the same institutional archive. Patient- or video-level separation cannot be verified for the EndoVL report-generation partitions because those identifiers are unavailable. Source identity also remains locally predictable in the embeddings (Supplementary Fig. S5 and Supplementary Table S18). The next priority is prospective, patient-grouped evaluation across institutions, report styles and acquisition systems. Such studies should assess calibration, uncertainty, report correctness and effects on clinical workflow [9, 32, 33]. They should
14
also test sensitivity to frame-selection and matching thresholds and include sufficient examples of rare findings. Similar many-frame, many-finding records occur in upper gastrointestinal endoscopy and other procedure-based imaging workflows. Extending correspondence recovery to these settings would test whether routine documentation can provide scalable vision–language supervision beyond colonoscopy.
4 Methods 4.1 Ethics and data source This retrospective study was approved by the Ethics Committee of Zhongshan Hospital, Fudan University (approval number B2025-145R) and was conducted in accordance with the Declaration of Helsinki. The committee granted a waiver of informed consent for the retrospective use of these routinely collected records, which were de-identified before analysis.
4.2 Problem formulation Each colonoscopy case c consists of a set of still frames Xc = {x1 , . . . , xmc } and a free-text report that we segment into finding sentences Tc = {t1 , . . . , tnc }. Because the report is written for the case as a whole, the correspondence between a sentence and the frame that depicts it is unobserved. The initial supervision is therefore the caselevel pairing (Xc , Tc ). Our aim is to learn an image encoder f and a text encoder g that map frames and sentences into a shared space in which a frame lies close to the sentence describing the same lesion. All embeddings are ℓ2 -normalised, and image– text similarity is defined as s(x, t) = ⟨f (x), g (t)⟩. We progressively re-pair frames and sentences, train f and g on the recovered pairs with a contrastive objective, and apply the frozen or fine-tuned image encoder to downstream lesion-level tasks.
4.3 Report corpus and curation The pretraining corpus comprised 280,476 de-identified colonoscopy reports and the still frames captured during the corresponding procedures. Of the reports, 44.2% documented at least one polyp; the remainder described normal examinations. Each report was segmented into individual finding sentences. We parsed the anatomical site, lesion size in centimetres and a controlled vocabulary of nineteen mucosal-surface and polyp-morphology descriptors. Among the polyp-positive reports, 69.6k (56.2%) documented a single polyp and 54.2k (43.8%) documented two or more. A lesion size was stated in 115,245 polyp-positive reports (93%); when present, the median was 0.6 cm (interquartile range, 0.4–0.8 cm). The polyp-positive cases contained 8.54 million frames (median, 65; mean, 69 per case), only some of which depicted a reported lesion. Curation removed low-quality entries and records without usable frames. The final corpus contained 104,542 cases and 125,756 image–text pairs, each linking one frame to one finding sentence.
15
4.4 EndoCLIP architecture and pre-training 4.4.1 Dual-encoder architecture EndoCLIP is a CLIP-style dual-encoder model with an image encoder f (·; θf ) and a text encoder g (·; θg ). The image encoder comprises a Vision Transformer backbone [35], fbackbone (·), followed by a linear projection head fproj (·). The projection head maps the backbone output into a shared d-dimensional embedding space, and θf contains the parameters of both modules. The Transformer text encoder [36] maps a finding sentence into the same space through its own projection head. Both encoders are implemented in the open clip framework [16].
4.4.2 Progressive correspondence recovery Pairing a report with every frame from the corresponding case yields a one-to-many or many-to-many signal in which most frame–sentence pairs are mismatched. We progressively recovered lesion-level report-to-evidence correspondence in three stages. The intermediate checkpoints are denoted EndoCLIP-I, EndoCLIP-II and EndoCLIP-III. In the first stage (case-level evidence localisation), we sampled one frame xi ∈ Xc at random from each case. We paired it with the generic prompt “polyp present” for polyp-positive cases or “no finding” for normal cases. Each case therefore contributed one weak pair, and no descriptive finding sentence was used. Training on this coarse signal produced EndoCLIP-I. We then used EndoCLIP-I to score every frame by its similarity to the case prompt and discarded low-scoring background or otherwise uninformative views. Training on the retained frame set X̃c produced EndoCLIP-II. In the second stage (single-lesion anchor selection), we restricted training to the 69.6k cases documenting one polyp. In each case, one finding sentence t describes the lesion and therefore provides a low-ambiguity anchor. We used EndoCLIP-II to select the best-matching retained frame, x⋆ = arg maxx∈X̃c s(x, t), and formed the pair (x⋆ , t). Training on these lesion-level pairs produced EndoCLIP-III. In the third stage (multi-lesion correspondence disambiguation), we applied EndoCLIP-III to the 54.2k multi-polyp cases. Because each report contributes several finding sentences, we matched sentences to frames. For every finding sentence tj , we retrieved the most similar retained frame, x⋆j = arg maxx∈X̃c s(x, tj ). We retained (x⋆j , tj ) only when s(x⋆j , tj ) exceeded a fixed threshold of 0.28, selected on a separate held-out validation set of multi-lesion cases (Supplementary Fig. S6 and Supplementary Table S22). Sentences without a confident frame match were discarded. This stage produced approximately 56,200 additional lesion-level image–text pairs, which we combined with the single-lesion pairs to train EndoCLIP.
4.4.3 Contrastive pre-training objective Given a batch of N paired examples {(xk , tk )}N k=1 produced by the correspondencerecovery procedure above, let Skl = s(xk , tl ) denote the cosine similarity between the k -th frame and the l-th sentence. The two encoders were trained jointly with the
16
symmetric image–text contrastive (InfoNCE) objective [14, 37]
" # N 1 X exp(Skk /τ ) exp(Skk /τ ) L=− log PN + log PN , 2N l=1 exp(Skl /τ ) l=1 exp(Slk /τ )
(1)
k=1
where τ is a temperature parameter. The first term in equation (1) aligns each frame with its matched sentence relative to the other sentences in the batch. The second aligns each sentence with its matched frame. This objective draws matched pairs together and pushes mismatched pairs apart. We used the same objective at every stage, applied to the image–text pairs available at that stage.
4.4.4 Pre-training configuration At every stage, both encoders were re-initialised and trained with the open clip pipeline. Following the pipeline defaults, we used AdamW [38] with a cosine learningrate schedule and a 50-step linear warmup. Each contrastive stage was trained for 20 epochs in single precision (fp32) with a learning rate of 1 × 10−5 and weight decay of 0.1. The per-GPU batch size was 16 across two GPUs, giving a global batch size of 32. Training minimised the symmetric InfoNCE loss in equation (1) with the standard learnable CLIP logit-scale temperature τ , initialised at 0.07. All contrastive stages were trained on two NVIDIA A100 80 GB GPUs. The primary retrieval, classification and report-generation comparisons used the EndoCLIP image encoder, either frozen or fully fine-tuned as specified below.
4.5 Downstream evaluation 4.5.1 Baseline encoders We benchmarked EndoCLIP against general-purpose and biomedical vision–language encoders under the same downstream protocol (optimisation settings in Supplementary Table S20). The general-purpose baselines were CLIP-OpenAI [14] and a randomly initialised CLIP model (CLIP-Scratch). The biomedical baselines were BiomedCLIP [30], which uses a PubMedBERT text encoder [39], and PMC-CLIP [26].
4.5.2 Multi-centre evaluation cohorts Downstream evaluation comprised retrieval on EndoReport100, multi-centre classification and structured report generation on EndoVL. EndoReport100 was an independent in-house retrieval benchmark of 100 multi-lesion cases and 217 lesions. The classification evaluation combined five EndoVL tasks with benign-versus-malignant pathology on an independent Zhongshan Hospital cohort. EndoVL [25] comprised nine public source datasets: CVC-ClinicDB [40], CVC-300 [41], Kvasir-SEG [42] and six centrespecific subsets of the multi-centre PolypGen collection [43]. The images were acquired using different endoscopes and from different patient populations. From the pooled polyp images, we organised four attribute-classification tasks: four-class Paris morphology (2,106 images), mucosal congestion (2,062), lobulated surface (2,139) and villous surface (2,099). We used the expert size annotations to define a binary polyp-size 17
task at the 10 mm threshold (2,177 images). The expert structured-report annotations provided the ground truth for the report-generation benchmark.
4.5.3 Classification transfer protocols All six classification tasks were evaluated under zero-shot, linear-probe and end-to-end training protocols. Zero-shot prediction used task-specific clinical prompts without updating model weights (Supplementary Table S2). Linear probing froze each image encoder and trained a single linear classifier on 5%, 10%, 20%, 50% or 100% of the training partition. End-to-end experiments used the same label fractions and optimisation protocol for three ViT-L/14 models: EndoCLIP and a pretrained ViT-L/14 were fully fine-tuned, whereas a randomly initialised ViT-L/14 was trained from scratch (Supplementary Table S20).
4.5.4 Structured report generation protocol The EndoVL structured-report dataset comprised 1,375 training, 197 validation and 394 test images. Each frozen image encoder supplied its full sequence of visual tokens to an encoder-specific trainable projector. The projector consisted of a linear layer, GELU activation, a second linear layer and layer normalisation, and mapped the visual tokens to a frozen Qwen3-14B decoder. Given a fixed instruction, the decoder generated JSON fields for lesion diameter, Paris type and a subset of nineteen descriptors. Projectors were trained for 20 epochs with target-only causal language-model loss in bfloat16, using a batch size of 4 and four-step gradient accumulation. The maximum learning rate was 1 × 10−4 , weight decay was 0.1, gradient clipping was 1.0. The checkpoint with the lowest validation loss was selected, and test outputs were decoded deterministically with one beam, no sampling and at most 128 new tokens.
4.5.5 Statistical analysis Unless otherwise stated, label-efficient results are reported as the mean and standard deviation over three random seeds (42, 123 and 456). In the zero-shot classification heatmap, each metric is expressed as its fold change relative to the random-chance baseline for that task. The t-SNE projections [44] were computed on frozen image embeddings for visualisation. We quantified embedding structure with a k-nearestneighbour leave-one-out classifier on the same features. For EndoReport100, we quantified uncertainty with a case-level cluster bootstrap of 2,000 resamples. Whole cases were resampled to preserve within-case correlation and to obtain 95% confidence intervals for each metric. Differences between EndoCLIP and each comparator were assessed with a two-sided paired bootstrap test over shared resamples, with p < 0.05 treated as significant. For the benign–malignant reader study, we used exploratory two-sided exact McNemar tests on the shared seed-42 test images.
5 Data availability The de-identified EndoReport100 benchmark, including its frames, clinician-curated frame-level polyp labels and lesion–sentence links, is publicly available on Figshare at 18
doi.org/10.6084/m9.figshare.33085637 [24]. De-identified restricted data may be made available by the corresponding authors for non-commercial research upon reasonable request, subject to institutional and ethical approval and an appropriate data-use agreement. The external images used to construct EndoVL originate from the public CVC-ClinicDB [40], CVC-300 [41], Kvasir-SEG [42] and PolypGen [43] datasets and remain available under their original access terms. The derived EndoVL expert annotations and split manifests are publicly available on Figshare [25].
6 Code availability Code to reproduce the experiments reported in this study is available at github.com/Jia7878/EndoCLIP.
References [1] Bray, F., Laversanne, M., Sung, H., Ferlay, J., Siegel, R.L., Soerjomataram, I., Jemal, A.: Global cancer statistics 2022: GLOBOCAN estimates of incidence and mortality worldwide for 36 cancers in 185 countries. CA: A Cancer Journal for Clinicians 74(3), 229–263 (2024) https://doi.org/10.3322/caac.21834 [2] Winawer, S.J., Zauber, A.G., Ho, M.N., O’Brien, M.J., Gottlieb, L.S., Sternberg, S.S., Waye, J.D., Schapiro, M., Bond, J.H., Panish, J.F., et al.: Prevention of colorectal cancer by colonoscopic polypectomy. New England Journal of Medicine 329(27), 1977–1981 (1993) [3] Zauber, A.G., Winawer, S.J., O’Brien, M.J., Lansdorp-Vogelaar, I., Ballegooijen, M., Hankey, B.F., Shi, W., Bond, J.H., Schapiro, M., Panish, J.F., et al.: Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths. New England Journal of Medicine 366(8), 687–696 (2012) [4] Bretthauer, M., Løberg, M., Wieszczy, P., Kalager, M., Emilsson, L., Garborg, K., Rupinski, M., Dekker, E., Spaander, M., Bugajski, M., et al.: Effect of colonoscopy screening on risks of colorectal cancer and related death. New England Journal of Medicine 387(17), 1547–1556 (2022) [5] Urban, G., Tripathi, P., Alkayali, T., Mittal, M., Jalali, F., Karnes, W., Baldi, P.: Deep learning localizes and identifies polyps in real time with 96% accuracy in screening colonoscopy. Gastroenterology 155(4), 1069–1078 (2018) [6] Ahmad, O.F., Soares, A.S., Mazomenos, E., Brandao, P., Vega, R., Seward, E., Stoyanov, D., Chand, M., Lovat, L.B.: Artificial intelligence and computeraided diagnosis in colonoscopy: current evidence and future directions. The lancet Gastroenterology & hepatology 4(1), 71–80 (2019) [7] Wang, P., Berzin, T.M., Glissen Brown, J.R., Bharadwaj, S., Becq, A., Xiao, X., Liu, P., Li, L., Song, Y., Zhang, D., et al.: Real-time automatic detection
19
system increases colonoscopic polyp and adenoma detection rates: a prospective randomised controlled study. Gut 68(10), 1813–1819 (2019) [8] Repici, A., Badalamenti, M., Maselli, R., Correale, L., Radaelli, F., Rondonotti, E., Ferrara, E., Spadaccini, M., Alkandari, A., Fugazza, A., et al.: Efficacy of real-time computer-aided detection of colorectal neoplasia in a randomized trial. Gastroenterology 159(2), 512–520 (2020) [9] Parasa, S., Repici, A., Berzin, T., Leggett, C., Gross, S.A., Sharma, P.: Framework and metrics for the clinical use and implementation of artificial intelligence algorithms into endoscopy practice: recommendations from the american society for gastrointestinal endoscopy artificial intelligence task force. Gastrointestinal Endoscopy 97(5), 815–8241 (2023) https://doi.org/10.1016/j.gie.2022.10.016 [10] Boers, T.G.W., Fockens, K.N., Putten, J.A., Jaspers, T.J.M., Kusters, C.H.J., Jukema, J.B., Jong, M.R., Struyvenberg, M.R., Groof, J., Bergman, J.J., With, P.H.N., Sommen, F.: Foundation models in gastrointestinal endoscopic ai: Impact of architecture, pre-training approach and data efficiency. Medical Image Analysis 98, 103298 (2024) https://doi.org/10.1016/j.media.2024.103298 [11] Yu, J., Zhu, Y., Fu, P., Chen, T., Huang, J., Li, Q., Zhou, P., Wang, Z., Wu, F., Wang, S., Yang, X.: Robust polyp detection and diagnosis through compositional prompt-guided diffusion models. IEEE Transactions on Medical Imaging 44(12), 5245–5257 (2025) [12] Zhou, Y., Chia, M.A., Wagner, S.K., Ayhan, M.S., Williamson, D.J., Struyven, R.R., Liu, T., Xu, M., Lozano, M.G., Woodward-Court, P., et al.: A foundation model for generalizable disease detection from retinal images. Nature 622(7981), 156–163 (2023) https://doi.org/10.1038/s41586-023-06555-x [13] Wang, Z., Liu, C., Zhang, S., Dou, Q.: Foundation model for endoscopy video analysis via large-scale self-supervised pre-train. In: International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), pp. 101–111 (2023). https://doi.org/10.1007/978-3-031-43996-4 10 . Springer [14] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 8748–8763. PMLR, Virtual (2021). https://proceedings.mlr.press/v139/radford21a.html [15] Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q.V., Sung, Y.-H., Li, Z., Duerig, T.: Scaling up visual and visionlanguage representation learning with noisy text supervision. In: Proceedings of the 38th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 139, pp. 4904–4916. PMLR, Virtual (2021). 20
https://proceedings.mlr.press/v139/jia21b.html [16] Cherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C., Schuhmann, C., Schmidt, L., Jitsev, J.: Reproducible scaling laws for contrastive language-image learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818–2829 (2023). https://doi.org/10.1109/CVPR52729.2023.00276 [17] Zhang, Y., Jiang, H., Miura, Y., Manning, C.D., Langlotz, C.P.: Contrastive learning of medical visual representations from paired images and text. In: Machine Learning for Healthcare Conference (MLHC) (2022). PMLR [18] Huang, S.-C., Shen, L., Lungren, M.P., Yeung, S.: GLoRIA: A multimodal global-local representation learning framework for label-efficient medical image recognition. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3942–3951 (2021) [19] Tiu, E., Talius, E., Patel, P., Langlotz, C.P., Ng, A.Y., Rajpurkar, P.: Expert-level detection of pathologies from unannotated chest X-ray images via self-supervised learning. Nature Biomedical Engineering 6(12), 1399–1406 (2022) [20] Huang, Z., Bianchi, F., Yuksekgonul, M., Montine, T.J., Zou, J.: A visual– language foundation model for pathology image analysis using medical Twitter. Nature Medicine 29(9), 2307–2316 (2023) [21] Lu, M.Y., Chen, B., Williamson, D.F.K., Chen, R.J., Liang, I., Ding, T., Jaume, G., Odintsov, I., Le, L.P., Gerber, G., et al.: A visual-language foundation model for computational pathology. Nature Medicine 30(3), 863–874 (2024) [22] Hamamci, I.E., Er, S., Wang, C., Almas, F., Simsek, A.G., Esirgun, S.N., Dogan, I., Durugol, O.F., Hou, B., Shit, S., et al.: Generalist foundation models from a multimodal dataset for 3D computed tomography. Nature Biomedical Engineering (2026) https://doi.org/10.1038/s41551-025-01599-y [23] Wang, S., Zhu, Y., Yang, Z., Luo, X., Zhang, Y., Fu, P., Wang, H., Wang, M., Song, Z., Li, Q., Zhou, P., Guo, Y.: Leveraging large language and vision models for knowledge extraction from large-scale image–text colonoscopy records. Nature Biomedical Engineering (2025) https://doi.org/10.1038/s41551-025-01500-x [24] Li, S.: EndoReport100. Figshare. Dataset (2026). https://doi.org/10.6084/m9. figshare.33085637 . https://doi.org/10.6084/m9.figshare.33085637 [25] Fu, P., Li, S., Zhu, Y., Wang, S., Li, X., Liu, X., He, M., Li, Q., Zhou, P.H.: EndoVL: An expert annotated dataset of public gastrointestinal endoscopic images with uncertainty labels. Figshare. Dataset (2026). https://doi.org/10. 6084/m9.figshare.32320998 . https://doi.org/10.6084/m9.figshare.32320998
21
[26] Lin, W., Zhao, Z., Zhang, X., Wu, C., Zhang, Y., Wang, Y., Xie, W.: Pmc-clip: Contrastive language-image pre-training using biomedical documents. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2023, pp. 525– 536. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-43993-3 51 [27] Participants in the Paris Workshop: The Paris endoscopic classification of superficial neoplastic lesions: esophagus, stomach, and colon. Gastrointestinal Endoscopy 58(6), 3–43 (2003) https://doi.org/10.1016/S0016-5107(03)02159-X [28] Hayashi, N., Tanaka, S., Hewett, D.G., Kaltenbach, T.R., Sano, Y., Ponchon, T., Saunders, B.P., Rex, D.K., Soetikno, R.M.: Endoscopic prediction of deep submucosal invasive carcinoma: validation of the narrow-band imaging international colorectal endoscopic (NICE) classification. Gastrointestinal Endoscopy 78(4), 625–632 (2013) https://doi.org/10.1016/j.gie.2013.04.185 [29] Gupta, S., Lieberman, D., Anderson, J.C., Burke, C.A., Dominitz, J.A., Kaltenbach, T., Robertson, D.J., Shaukat, A., Syngal, S., Rex, D.K.: Recommendations for follow-up after colonoscopy and polypectomy: a consensus update by the US multi-society task force on colorectal cancer. Gastroenterology 158(4), 1131–1153 (2020) https://doi.org/10.1053/j.gastro.2019.10.026 [30] Zhang, S., Xu, Y., Usuyama, N., Xu, H., Bagga, J., Tinn, R., Preston, S., Rao, R., Wei, M., et al.: A multimodal biomedical foundation model trained from fifteen million image–text pairs. NEJM AI 2(1) (2025) https://doi.org/10.1056/ AIoa2400640 [31] Qu, J.-Y., Li, Z., Su, J.-R., Ma, M.-J., Xu, C.-Q., Zhang, A.-J., Liu, C.-X., Yuan, H.-P., Chu, Y.-L., Lang, C.-C., Huang, L.-Y., Lu, L., Li, Y.-Q., Zuo, X.-L.: Development and validation of an automatic image-recognition endoscopic report generation system: A multicenter study. Clinical and Translational Gastroenterology 12(1), 00282 (2020) https://doi.org/10.14309/ctg.0000000000000282 [32] Jiang, R., Chen, B., Dong, Z., Zeng, X., You, H., Li, Y., Deng, Y., Mu, G., Wang, J., Huang, L., Li, J., Cheng, D., Zhou, W., Yu, H.: Domain specific multimodal large language model for automated endoscopy reporting with multicenter prospective validation. npj Digital Medicine 9(1), 394 (2026) https: //doi.org/10.1038/s41746-026-02569-7 [33] Zhu, Y., Luo, T., Fu, P.-Y., Zhang, Z., Wang, Z.-L., Qu, Y.-F., Geng, Z.-H., Xu, J.-Q., Yao, L., Ma, L.-Y., Su, W., Chen, W.-F., Li, Q.-L., Wang, S., Zhou, P.-H.: GI-Bench: A panoramic benchmark revealing the knowledge-experience dissociation of multimodal large language models in gastrointestinal endoscopy against clinical standards. arXiv preprint arXiv:2601.08183 (2026) [34] Yang, R., Zhu, Y., Fu, P., Zhang, Y., Wang, Z., Li, Q., Zhou, P., Yang, X., Wang, S.: EndoFinder: Online Image Retrieval for Explainable Colorectal Polyp Diagnosis . In: Medical Image Computing and Computer Assisted Intervention 22
– MICCAI 2024. Lecture Notes in Computer Science, vol. 15010, pp. 251–262. Springer, Cham (2024). https://doi.org/10.1007/978-3-031-72117-5 24 [35] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021) [36] Vaswani, A.: Attention is all you need. Advances in Neural Information Processing Systems (2017) [37] Oord, A., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018) [38] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: International Conference on Learning Representations (ICLR) (2019) [39] Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., Naumann, T., Gao, J., Poon, H.: Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare 3(1), 1–23 (2021) [40] Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., Gil, D., Rodrı́guez, C., Vilariño, F.: WM-DOVA maps for accurate polyp highlighting in colonoscopy: Validation vs. saliency maps from physicians. Computerized Medical Imaging and Graphics 43, 99–111 (2015) [41] Vázquez, D., Bernal, J., Sánchez, F.J., Fernández-Esparrach, G., López, A.M., Romero, A., Drozdzal, M., Courville, A.: A benchmark for endoluminal scene segmentation of colonoscopy images. Journal of Healthcare Engineering 2017, 4037190 (2017) [42] Jha, D., Smedsrud, P.H., Riegler, M.A., Halvorsen, P., Lange, T., Johansen, D., Johansen, H.D.: Kvasir-SEG: A segmented polyp dataset. In: International Conference on Multimedia Modeling, pp. 451–462 (2020). Springer [43] Ali, S., Jha, D., Ghatwary, N., Realdon, S., Cannizzaro, R., Salem, O.E., Lamarque, D., Daul, C., Riegler, M.A., Anonsen, K.V., et al.: A multi-centre polyp detection and segmentation dataset for generalisability assessment. Scientific Data 10(1), 75 (2023) [44] Maaten, L., Hinton, G.: Visualizing data using t-SNE. Journal of Machine Learning Research 9, 2579–2605 (2008)
23