Beyond Visual Cues: CoT-Enhanced Reasoning for Semi-supervised Medical Image Segmentation Yuming Chen1,2 , Yuxin Xie1,2 , Tao Zhou3 , and Yi Zhou1,2(B)
arXiv:2606.17958v1 [cs.CV] 16 Jun 2026
1
School of Computer Science and Engineering, Southeast University, Nanjing, China 2 Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications, Ministry of Education, Nanjing, China 3 Nanjing University of Science and Technology, Nanjing, China {[email protected], [email protected]}
Abstract. Semi-supervised medical image segmentation has emerged as a dominant research problem in medical image analysis, mitigating annotation scarcity by leveraging consistency regularization on unlabeled data. However, existing approaches operate predominantly via visual pattern matching, relying heavily on pixel-level similarities. This visualcentric dependency often falters in clinical scenarios characterized by the visual-semantic mismatch, where visually similar lesions warrant distinct diagnostic conclusions, thus failing to capture the underlying diagnostic logic used by experts. To address this, we move beyond visual cues and propose CERS (CoT-Enhanced Reasoning Segmentation), a framework that integrates Chain-of-Thought (CoT) reasoning to distinguish pathologically distinct cases. Specifically, we construct a knowledge pool enriched with linguistic reasoning descriptions generated by large language models (LLMs). A semantic-aware reference selection strategy is introduced to identify historical evidence, filtering candidates first by morphology, and then refining them via CoT consistency to eliminate hard negatives. Furthermore, a multi-scale coordinate attention module (MCAM) is designed to effectively fuse this reasoning-derived context into the decoding process. Extensive experiments demonstrate the superiority of CERS against state-of-the-art approaches, particularly in resolving boundary ambiguities and semantic inconsistencies. The code is available at https://github.com/cymasuna/CERS. Keywords: Semi-supervised Medical Image Segmentation · Beyond Visual Cues · Reasoning-derived Context.
1
Introduction
Medical image segmentation is a cornerstone of computer-aided diagnosis, enabling precise delineation of organs and lesions [9,21]. However, the efficacy of deep learning models heavily relies on large-scale, high-quality annotations, which are difficult and laborious to obtain [7]. To address this, Semi-Supervised
2
Chen et al.
Learning (SSL) has emerged as a widely adopted paradigm, leveraging limited labeled data alongside abundant unlabeled data [7,18]. Although existing SSL approaches, ranging from consistency regularization [19,1,24,10] to pseudo label supervision [16,28], have achieved promising results, they predominantly focus on pixel-level visual features. A fundamental limitation of current SSL methods is their reliance on visual pattern matching. Although effective for standard cases, this approach often fails when faced with ambiguous boundaries, low-contrast lesions, imaging artifacts caused by respiration, or anatomically similar structures that mimic pathological appearance, leading to false positives or missed detections [27]. In clinical practice, physicians overcome such challenges by recalling similar historical cases and applying diagnostic reasoning to infer lesion boundaries beyond purely visual evidence [23]. Following this observation, recent studies have suggested that introducing external reference cues can significantly enhance model performance [29]. However, a critical challenge remains that clinical scenarios often exhibit a visualsemantic mismatch, where images with high visual similarity warrant distinct radiological reading conclusions [5]. Consequently, existing text-driven segmentation methods, which rely solely on superficial and incomplete medical text descriptions, often fail to capture the actual radiological reading logic [10,26,25]. This highlights the necessity to incorporate high-level reasoning capabilities to distinguish pathologically distinct cases that appear visually similar. To address this challenge, we propose CERS (CoT-Enhanced Reasoning Segmentation), a framework that integrates Chain-of-Thought (CoT) [22] reasoning into the semi-supervised segmentation task. Instead of relying on unreliable visual matching, CERS generates linguistic reasoning descriptions to capture the semantic context of lesions. Using these CoT descriptions, our method performs a semantic-aware retrieval that bridges the gap between visual appearance and diagnostic logic. This ensures that the utilized historical references are not just visually similar, but logically consistent with the target case. These reasoningguided cues are then effectively injected into the segmentation network to resolve ambiguities in unlabeled data. The main contributions are summarized as follows. (1) We propose CERS, a CoT-enhanced framework that generates segmentation-aware reasoning descriptions to provide high-level semantic guidance for semi-supervised segmentation. (2) We design a Multi-scale Coordinate Attention Module (MCAM) with a dual-decoder consistency scheme to effectively fuse reasoning-derived context into the decoder, significantly improving segmentation accuracy in challenging scenarios. (3) We conduct extensive experiments on multiple public datasets, showing that CoT-enhanced reasoning moves segmentation beyond purely visual cues and significantly improves semi-supervised performance.
2
Method
We study semi-supervised medical image segmentation with a labeled set Sl = H×W ×C l l is a medical image, yil ∈ {0, 1}H×W is {(xli , yil , tli )}N i=1 , where xi ∈ R
Title Suppressed Due to Excessive Length First Stage
3
Second Stage
Labeled Pool
Dec. with
l
Unlabeled Pool Text
MCAM
R
Enc.
sup
Pred
Unilateral pulmonary infection, one infected areas, left lung and middle...
Mask
Dec. Unilateral pulmonary infection, one infected areas, left lung and middle right lung.
S
Enc.
Write
u
CoT
Knowledge Pool Image Emb.
clip
f0
Retrieval
Texture: ... Boundary: ... Context: ... Summary: ...
Dec.
Enc.
S
GPT-5.2
Proj.
CoT Texture: Predominantly heterogeneous ground-glass... Boundary: Largely ill-defined, hazy margins with low contrast...
img
Bert
EMA Update
CoT Emb.
cons
Dec. with
Bert
CLS
MCAM
*
{v i }( v , z ) i
*
R
i
Fig. 1. Overview of CERS. A frozen LLM generates CoTs, and labeled CoTs form a knowledge pool for imageCoT hybrid retrieval. Retrieved context is fused with image features via MCAM and used by a retrieval-aware decoder to enable reasoning-guided semi-supervised segmentation beyond visual cues.
the corresponding mask and tli is the associated text, and an unlabeled set u Su = {(xuj , tuj )}N j=1 . For every sample, we generate a Chain-of-Thought (CoT) via a frozen LLM, denoted cli for labeled and cuj for unlabeled samples. Only labeled CoTs are used to populate the knowledge pool K, avoiding leakage from unlabeled data. As illustrated in Fig. 1, our multi-modal CERS model is fθ = {E, DS , DR }, where E is a shared encoder, DS a standard segmentation decoder and DR a retrieval-aware decoder that fuses retrieved context. CERS employs an imageCoT hybrid retrieval in two stages: a preliminary screening retrieves visually similar candidates to ensure morphological consistency, and a fine-grained re-ranking uses CoT similarity to remove hard negatives that are visually similar but semantically distinct. Retrieved features are fused with the the current features via the Multi-scale Coordinate Attention Module to yield a retrieval-aware enhanced representation, which is consumed by DR . 2.1
Segmentation-Aware Reasoning Knowledge Pool Construction
CoT Generation. We generate Chain-of-Thought (CoT) descriptions for each sample by prompting GPT-5.2 [17]. For labeled samples we provide the image xli , its mask yil (as a translucent overlay), and text tli ; for unlabeled samples we provide only the image xuj and text tuj . The resulting CoTs for labeled and unlabeled sets are denoted by C l and C u respectively. Prompts elicit four diagnostic aspectstexture, boundary, context, and a concise summarythereby encouraging the LLM to produce semantically discriminative cues for segmentation. We omit explicit spatial coordinates because free-text locations are often ambiguous; morphology-aware image embeddings already supply spatial priors. To verify the reliability of the generated prompts, we randomly sampled 100 cases for expert manual review, which confirmed the high clinical accuracy (more than 95%) of the LLM-generated CoT.
4
Chen et al.
Warm-up to Build Knowledge Pool. To initialize the model and ensure stable retrieval in the subsequent SSL phase, we first conduct a supervised warm-up (Eq. 2) using only the labeled set Sl . During this phase the multi-modal model fθ = {E, DS , DR } is trained on labeled samples to learn reliable segmentation and cross-modal embeddings. The supervised segmentation loss is defined as: Lsup =
1 |Sl |
X
h
i Lseg DS (E(x, t)), y + Lseg DR (E(x, t)), y ,
(1)
(x,y,t)∈Sl
where Lseg combines standard cross-entropy loss and Dice loss. We obtain an image embedding vi = E(xli , tli ) and a CoT embedding zi = B(cli ) using a texture encoder B and apply contrastive objective [14] Lclip to align these embeddings. The overall warm-up loss is defined as: Lwarm = Lseg + λclip Lclip ,
(2)
where λclip balances segmentation supervision and cross-modal alignment. After warm-up convergence, the trained image encoder E and texture encoder B are fixed and used to encode all labeled samples. These paired representations are l then stored in the knowledge pool K = {(vi , zi )}N i=1 , which serves as the retrieval database in the subsequent semi-supervised learning stage. 2.2
ImageCoT Joint Cues
In the second training stage, we continue optimization based on the warm-up model and leverage the constructed knowledge pool K to retrieve reliable cues for both labeled and unlabeled samples. Given an input sample x and the associated text with its generated CoT c, we first extract the image embedding v = E(x, t) and the CoT embedding z = B(c). CERS performs retrieval in a coarse-to-fine manner: (1) Image-based coarse filtering: To ensure visual consistency, we conduct an image-based coarse retrieval over the knowledge pool using cosine similarity, simg = cos(v, vi ), where (vi , zi ) ∈ K. The top-K samples i with the highest simg are selected to form a candidate set Kimg , which contains i morphologically similar samples. (2) CoT-based fine re-ranking: Among the candidates in Kimg , we further compute CoT similarity to refine retrieval results, scot = cos(z, zi ), where (vi , zi ) ∈ Kimg . The candidates are re-ranked i according to scot i , and the top-M samples are retained as the final retrieved evidence set K∗ . This fine-grained re-ranking effectively filters out hard negatives that are visually similar but diagnostically inconsistent with the query. 2.3
Multi-scale Coordinate Attention Module
Using the features f 0 = E(x, t) from the encoder as queries and the retrieved features {vi }(vi ,zi )∈K∗ as keys and values enables the model to selectively retrieve and integrate relevant external knowledge guided by the current visual
Title Suppressed Due to Excessive Length
f0
Upsample, Proj.
X Avg Pool
Conv1×1, Sigmoid
Conv1×1
C L −1
f1
s
L − 1
·
C
Hadamard Product
×
+ Conv7×7, LN, Conv1×1, GELU
C Concatenation
Conv1×1
+
C
+ Element-wise Addition
fv1
Multiply
Proj.
C
Conv5×5, LN, Conv1×1, GELU (shared)
×
k, v
i
Conv1×1, Sigmoid
+
*
Y Avg Pool
+
i
Upsample, Proj.
Conv7×7, LN, Conv1×1, GELU
MCAM
vi ( v , z )
f
split Conv1×1, BN, GELU
Multi-head Cross Attention
f0 q
·
+
Multi-scale Coordinate Attention Module (MCAM) 0 v
5
Fig. 2. Detailed structure of Multi-scale Coordinate Attention Module.
context, thereby enhancing feature discriminability and contextual consistency. It is formulated via multi-head cross attention [20] as: fv0 = MHCA(f 0 , {vi }, {vi }), (vi , zi ) ∈ K∗ .
(3)
As depicted in Fig. 2, Multi-scale Coordinate Attention Module (MCAM) is employed in the decoder to fuse the feature f 0 with fv0 in a spatially aware manner. The resulting representation f 1 combined with skip features sl−1 of l − 1th encoder layer and fv1 is subsequently fed into next MCAM in DR . The decoding and fusion for each stage can be expressed as follows: (f i+1 , fvi+1 ) = MCAM(Concat(f i , sL−i ), fvi ), i ∈ [1, L − 1],
(4)
where L is the number of main stages in the encoder. Inspired by coordinate attention [8], MCAM enhances spatial sensitivity by modeling feature responses along horizontal and vertical directions, which is beneficial for preserving boundary structures during decoding. In addition, multi-scale feature processing allows the module to capture both local details and global context, improving the effectiveness of reference feature integration. 2.4
Knowledge-Guided Learning Objective
Knowledge Update. The knowledge pool K stores a visual prototype viK and a CoT prototype zK i for each labeled sample i. The visual prototype are updated during training via exponential moving average (EMA). Concretely, given the current pooled image feature f̄i = E(xli ), the pool entries are updated as: viK ← α viK + (1 − α) f̄i ,
(5)
where α ∈ [0, 1) is the EMA momentum. Updates are performed only for labeled entries, and stored prototypes are detached from the gradient graph to stabilize retrieval and allow inexpensive persistence. Total Loss Function. For unlabeled samples we enforce cross-branch consistency between the two decoder outputs. The consistency loss is written as: Lcons =
1 X DS (E(x, t)) − DR E(x, t), {vi }(vi ,zi )∈K∗ |Su | x∈Su
2 . 2
(6)
6
Chen et al.
Overall training objective is by weighting supervised and consistency terms: Ltotal = Lsup + λcons Lcons .
(7)
Notably, retrieval affects only the output of DR , so the consistency term encourages the base decoder DS to absorb useful retrieval-induced cues without directly exposing DS to retrieved features.
3
Experiments
3.1
Datasets and Experimental Details
Datasets. We conducted experiments on three public datasets, covering different modalities. (1) MosMedData+ [13] consists of 2 729 chest CT scans with COVID-19 infection annotations. (2) QaTa-COV19 [4] contains 9 258 chest X-ray images with COVID-19 infection segmentation masks. (3) BRISC 2025 [6] consists of 4 793 brain MRIs with annotated tumor regions and lesion masks. We split each dataset into train/val/test with an 8:1:1 ratio. The textual annotations for MosMedData+ and QaTa-COV19 follow LViT [10], while BRISC 2025 annotations were auto-generated from metadata and graphical features. Implementation Details. All experiments including both training and evaluation phases were conducted on a single NVIDIA RTX 4090 GPU. We set λclip = 0.5 and α = 0.99. The λcons was initialized as 0.003 and increased via a standard sigmoid ramp-up. For fair comparison, all methods adopted ConvNeXt [11] as the backbone and CXR-BERT [2] as CoT encoder whenever applicable. Each model was re-implemented and trained for 200 epochs under the same training protocol. The batch size was set to 16, and optimization was performed using SGD with an initial learning rate of 1e−2 and a weight decay of 1e−4 . Performance was evaluated using Dice and IoU. 3.2
Comparison with State-of-the-Art Methods
We compare against six SOTA SSL baselines: BCP [1], DuCiSC [24], LeFeD [28], LViT [10], MT [19] and UCMT [16]. We evaluate semi-supervised performance under 50% and 25% labeled ratios across three public datasets. The quantitative comparison results are presented in Table 1. Compared with existing methods, CERS consistently outperforms SOTA methods in most metrics and datasets. Fig. 3 provides qualitative examples where CERS better localizes lesions and preserves boundaries under ambiguous, low-contrast, artifact-corrupted, or visually confusing casesdemonstrating the value of moving beyond purely visual cues. Unlike pseudo-labeling or teacher-updating methods that suffer from confirmation bias, CERS mitigates noisy supervision by leveraging imageCoT joint cues. Our method remains effective even under fully supervised learning, significantly outperforming the U-Net [15] model that employ the same backbone.
Title Suppressed Due to Excessive Length
7
Table 1. Quantitative comparison in terms of Dice (%) and IoU (%) on MosMedData+, QaTa-COV19 and BRISC 2025 with 50% and 25% labeled ratio. Bold and underlined entries indicate the best and second-best results respectively. Method
Text Backbone Labeled
U-Net [15] Ours
ConvNeXt 100% ✓ ConvNeXt 100%
MosMedData+ QaTa-COV19 BRISC 2025 Dice
IoU
Dice
IoU
Dice
IoU
76.99 79.21
66.03 68.53
75.70 64.90 82.57 74.21 81.40 71.60 84.40 76.28
BCP [1] ConvNeXt DuCiSC [24] ConvNeXt LeFeD [28] V-Net [12] LViT [10] ✓ LViT MT [19] ConvNeXt UCMT [16] ConvNeXt Ours ✓ ConvNeXt
50% 50% 50% 50% 50% 50% 50%
72.57 60.79 72.59 60.92 76.66 67.42 74.24 63.08 71.51 59.84 78.85 70.67 74.77 63.38 74.64 63.74 79.63 72.13 75.47 63.27 79.08 69.65 83.85 75.88 73.44 61.97 73.39 62.22 76.38 67.88 75.95 64.65 74.42 63.33 80.04 71.70 78.26 66.87 81.23 71.42 84.03 76.03
BCP [1] ConvNeXt ConvNeXt DuCiSC [24] LeFeD [28] V-Net [12] LViT [10] ✓ LViT MT [19] ConvNeXt ConvNeXt UCMT [16] Ours ✓ ConvNeXt
25% 25% 25% 25% 25% 25% 25%
71.31 59.37 72.44 60.81 72.46 62.43 71.67 60.30 70.20 58.22 76.38 68.04 70.45 58.95 74.31 63.61 78.64 71.01 71.16 58.13 77.48 67.30 80.51 72.98 70.08 58.34 72.86 61.48 71.12 62.42 72.00 60.47 72.15 60.69 74.32 66.19 74.73 63.39 80.52 70.52 80.34 71.80
Image
Ground Truth
CERS (Ours)
BCP
DuCiSC
LeFed
LViT
MT
UCMT
Fig. 3. Exemplar qualitative results of different approaches on MosMedData+ (row 1), QaTa-COV19 (row 2) and BRISC 2025 (row 3).
8
Chen et al.
Table 2. Ablation study on different retrieval strategies of CERS. Latency refers to the time overhead incurred per training step due to the retrieval computation. Image MosMedData+ QaTa-COV19 Latency Text CoT Retrieval Retrieval Retrieval per Step Dice IoU Dice IoU ✓ ✓
3.3
✓ ✓
✓ ✓ ✓
73.51 62.36 74.07 62.86 76.68 65.54 79.43 69.13 77.22 66.07 80.86 71.05 73.60 62.14 80.66 70.66 77.07 65.90 80.20 70.16 78.26 66.31 81.23 71.42
0.0 +0.048s +0.055s +0.047s +0.099s +0.103s
Ablation Study
Impact of Different Retrieval. Table 2 presents the results of different search strategies on the MosMedData+ and QaTa-COV19 datasets with a 50% labeled ratio. The retrieval strategy provides a reference for the model to perform segmentation. Image retrieval defines a coarse boundary for the search space, while CoT further bridges the gap between visual appearance and diagnostic logic. Impact of Top-K on Different Labeled Data Ratios. The performance of different numbers of CoT retrieval cues under varying labeled ratios is illustrated in Fig. 4. Utilizing too few cues fails to provide a robust reference for the model, whereas an excessive number increases the risk of introducing noise. For the MosMedData+ dataset, K = 5 proves to be a reasonable choice. Impact of MCAM. We replace the MCAM with a standard U-Net [15] decoder and a Swin U-Net [3] decoder, with the results presented in Table 3. As demonstrated in Fig. 3, MCAM effectively preserves boundary structures and generates smoother, more precise segmentation masks compared to other decoders. To further validate the internal design of MCAM, we conduct ablation studies by replacing its sub-components with 1Œ1 convolutions. Specifically,