Toward Robust In-Context Segmentation via Concept Guidance Zhigang Chen1 , Xiawu Zheng1 , Rongrong Ji1,B
arXiv:2606.28149v1 [cs.CV] 26 Jun 2026
1
Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University, 361005, P.R. China B Corresponding Author: Rongrong Ji
ABSTRACT In-context segmentation (ICS) requires a model to segment target regions in a query image using only a few reference images and their corresponding masks, without updating any parameters. Despite recent progress, prior ICS studies have largely overlooked a critical aspect: system robustness, i.e., , whether the model can produce stable segmentation results for the same query under different references. In this work, we revisit ICS from the robustness perspective and introduce a novel paradigm, Concept-Guided In-Context Segmentation (CG-ICS), which performs segmentation by extracting high-level semantic concepts from references rather than relying solely on low-level visual matching. Specifically, CG-ICS introduces a concept reasoning module that uses an MLLM to propose candidates and a SAM3-driven scoring function with tree-search refinement to select reliable textual concepts, together with a parallel visual exemplar route that provides query-side spatial grounding via a simple context construction. Both the textual concept and the visual exemplar are then used to activate the segmentation capability of a frozen SAM3 backbone. Extensive experiments on standard ICS benchmarks demonstrate that CG-ICS not only achieves state-of-the-art accuracy but also substantially improves robustness, yielding a more reliable ICS system with significantly reduced variance across diverse reference choices. Email: [email protected]
1
Introduction
Image segmentation stands as a fundamental task in computer vision. However, traditional fully supervised approaches are restricted by their heavy reliance on costly annotations and limited generalization to novel categories. Recently, inspired by the success of In-context Learning in Large Language Models (LLMs) [6], In-Context Segmentation (ICS) [5, 35, 36] has emerged to address these challenges. ICS aims to segment the target region in a query image using only a few reference examples, without updating any model parameters. It offers a flexible and efficient solution for open-world segmentation. While recent ICS methods have made notable progress in segmentation accuracy [22–24, 36, 40, 41], they largely overlook an equally critical aspect: system robustness, i.e., whether the model can produce stable segmentation results for the same query under different references. Due to the few-shot nature of ICS, where predictions are guided by only a handful of in-context examples, they are highly sensitive to the choice of reference instances. To make this issue concrete, we probe a state-of-the-art training-free method, GF-SAM [40], by varying the in-context references while keeping the query fixed. As shown in Fig. 1, GF-SAM exhibits pronounced variance across different references and can even fail catastrophically when paired with unfavorable reference instances. Recently, several works [30, 38, 42] have also exposed this sensitivity, but they mainly focus on selecting the best prompt from an available pool of candidate references. However, such a candidate pool is often absent in real deployments, where users provide arbitrary references on the fly, and the system has no opportunity to search for a better alternative. UNICL-SAM [28] also recognizes the robustness 1
MAC-AutoML
Reference
GF-SAM
CG-ICS
Figure 1. We present the qualitative results of a query sample with multiple distinct reference examples and report the IoU achieved by GF-SAM [40] and our CG-ICS.
issue and attempts to incorporate the uncertainty of reference samples into the training process. However, it only focuses on the model’s performance when the reference samples are corrupted, without considering references that are inherently low quality. Therefore, a fundamental requirement emerges: an ICS system should remain accurate and stable under any reasonable reference, rather than performing well only when paired with a fortunate one. In this paper, we take robustness as a first-class objective for ICS, and aim to reduce the prediction variance induced by reference choice while preserving strong segmentation quality. Our methodology is directly inspired by the rapid evolution of segmentation foundation models [7, 19, 26]. In particular, SAM3 [7] introduces a Promptable Concept Segmentation (PCS) paradigm, where the model can effectively interpret a textual concept prompt and produce accurate segmentation masks accordingly. We posit that such a concept-guided formulation offers a promising direction for advancing ICS, as textual concepts provide a higher-level and more invariant representation of the target than brittle low-level visual correspondences. Consequently, concept-based guidance is expected to yield more robust and stable predictions under varying references. However, adapting PCS to the ICS task presents a unique challenge: the standard ICS setting strictly provides only a reference image and a binary mask, lacking any prior category names or textual descriptions. In that case, our primary objective is to autonomously derive these concepts. The most straightforward solution is to leverage a model that can understand images and output textual descriptions. Therefore, we adopt a Multimodal Large Language Model (MLLM) [2–4, 32, 34], as the concept generator, which not only meets the above requirement but also exhibits strong open-world generalization capabilities to resolve visual ambiguities and improve robustness. Since the MLLM and SAM3 are decoupled, a key challenge lies in selecting the optimal concept that can best prompt SAM3 for accurate segmentation. Therefore, as illustrated in Fig. 2, we design a tree-search-based Concept Reasoning procedure. Specifically, the MLLM-driven Concept Generation module expands tree branches by producing multiple candidate concepts as tree nodes. For each node, a SAM3-based Concept Scoring module evaluates the candidate by jointly considering the reference and the query. The resulting scores determine whether the current concept should be expanded (to generate refined child nodes) or pruned. This iterative expand–score–prune process continues until a predefined stopping criterion is met, yielding the best concept for prompting SAM3. However, relying solely on linguistic concepts remains insufficient in challenging cases. SAM3 also highlights that providing visual exemplars (bounding boxes of the target) can help specify uncommon categories [7]. Therefore, we introduce a parallel Visual Exemplars Extraction route that extracts query-side visual exemplars from the concatenated reference-query image. Finally, CG-ICS fuses the selected textual concept with the visual exemplars in a single SAM3 inference 2
MAC-AutoML
to produce the query mask. We term this framework Concept-Guided In-Context Segmentation (CGICS). Overall, CG-ICS effectively recasts ICS as a PCS problem, leading to not only improved segmentation accuracy but also substantially enhanced robustness to reference selection, as evidenced in Fig. 1. Our contributions and the main findings are summarized as follows: • We tackle a critical yet underexplored issue in ICS, namely the pronounced sensitivity to in-context references. To address this problem, we make the first attempt to build a robust ICS system that maintains stable performance under diverse and low-quality references, rather than focusing on selecting an optimal reference as in prior work. • The proposed CG-ICS introduces a concept reasoning module to extract the most suitable textual concept, together with a parallel visual exemplar route that provides query-side spatial grounding. With this design, we, for the first time, successfully bring the powerful SAM3 into the ICS. • Through extensive experiments on four ICS benchmarks under both standard and robust evaluation protocols, we demonstrate that CG-ICS achieves strong segmentation performance while substantially reducing variance across diverse reference selections and quality, offering practical insights for building more reliable ICS systems.
2
Related work
In-Context Segmentation. In-context segmentation (ICS) has emerged as a practical variant of visual incontext learning, where a model segments the target region in a query image conditioned on a few reference image–mask pairs without updating parameters. Since Bar et al. [5] first introduced the in-context learning concept into vision, this paradigm has rapidly attracted extensive attention and led to a series of generalist segmentation systems. Existing ICS methods can be broadly categorized into training-based and training-free paradigms. Training-based methods explicitly learn the ICS paradigm by jointly feeding the in-context reference and the query into a unified model during training. Painter [35] and SegGPT [36] formulate ICS via masked image modeling [17] on the concatenated image of reference and query. Another line of works, including SegIC [24], SINE [22], and Iris [13], freeze a powerful vision foundation model [8, 25] to extract dense correspondences and train a lightweight decoder to predict masks. Diffusion-based [18, 27] approaches such as DiffewS [43] and LDIS [33] explore generative modeling as a vehicle for in-context segmentation. In addition, some methods adapt promptable segmenters by fine-tuning the SAM [19, 26] family for ICS, such as VRP-SAM [29] and SANSA [10]. Despite strong performance, training-based ICS typically requires substantial in-domain training data and may exhibit limited generalization when faced with novel domains, corrupted annotations, or distribution shifts. Training-free methods avoid parameter updates by extracting visual correspondences between reference and query and converting them into prompts for a frozen promptable segmenter (e.g., SAM). PerSAM [41], Matcher [23], and GF-SAM [40] follow this paradigm by deriving point or region cues from reference-query matching to drive mask prediction. While these approaches primarily focus on improving average segmentation accuracy, the robustness of ICS systems under diverse and imperfect reference choices remains underexplored. A few studies [30, 42] have noticed the sensitivity of ICS to references, but they mainly aim to retrieve the best one from an available pool rather than ensuring stable behavior under arbitrary references. In contrast, our work targets high performance and low variance across diverse reference examples, which better matches real-world usage where reference quality and choice cannot be guaranteed. Robust Segmentation. Robust segmentation [1, 9, 11, 14, 37, 39] studies examine how dense prediction models behave under distribution shifts and real-world degradations. More recently, robustness has been explored in few-shot segmentation and in-context segmentation settings, where the prediction can be highly sensitive to the choice and quality of reference examples. Tang et al. [31] estimate the contribution of each reference image and then enhance or prune low-utility references to improve few-shot segmentation performance. UNICL-SAM [28] reduces the performance drop under corrupted references by injecting uncertainty signals during training, which makes the model less brittle to degraded references. However, it does not explicitly evaluate stability across different references, and its training-based paradigm also incurs higher data and computation requirements. In contrast, our method adopts a training-free design that leverages advanced 3
MAC-AutoML
foundation models to perform ICS, offering a more flexible and lightweight solution while directly targeting robustness to reference variation.
3
Method
We present Concept-Guided In-Context Segmentation (CG-ICS), a framework that performs ICS without updating any model parameters. Built on MLLM and SAM3, CG-ICS derives textual concepts and visual exemplars and combines them as prompts for final mask prediction. We first introduce the ICS problem setup and the two foundation models in preliminaries, and then present CG-ICS in detail. The pipeline is described in a one-shot setting for clarity, and then extended to multiple reference examples.
3.1
Preliminaries
ICS. Given a single user-provided reference example (Ir , Mr ) and a query image Iq , ICS aims to produce a mask Mq that captures the content in the query image. Only the reference image and mask are provided, and no category label or predefined class information is assumed. MLLM. A MLLM is a generative model that jointly processes visual inputs and natural language, enabling unified perception and language reasoning. MLLMs are known to be robust in handling diverse scenes and open-vocabulary descriptions, and can generalize well under distribution shifts and noisy visual conditions. Formally, given an image (or a set of images) and a textual instruction prompt, an MLLM produces a textual response as T = MMLLM (I, P ) , (1) where I denotes the visual input, P denotes the input prompt, and T is the generated text output. SAM3. SAM3 is a foundation model that performs promptable concept segmentation. The target can be specified by a short noun phrase (Textual Concept), a visual exemplar bounding box, or both. Given a concept prompt, SAM3 predicts semantic and instance-level masks for all instances matching the concept. The model also includes a presence head to indicate whether the queried concept is present in the input image. Formally, we write SAM3 inference as M ins , M sem , S pres = MSAM3 (I, T, V ) , (2) where I is the image, T is a textual concept prompt, V is the visual exemplar boxes, and the outputs are instance masks M ins , a semantic mask M sem , and a presence score S pres .
3.2
Concept Reasoning
We propose a tree-search-based reasoning strategy to automatically extract the most suitable concept for SAM3 from the in-context references. The problem is formulated as searching over a concept tree, where each node corresponds to a candidate textual concept and each edge denotes a refinement from a parent concept to a more specific one. Concept Generation. We extract candidate textual concepts from the reference image–mask pair using a frozen MLLM, which acts as the branch expansion operator in our concept tree. To preserve global scene context while highlighting target details, we provide the MLLM with a two-view visual input. The first view is the original reference image Ir . The second view is a highlighted image IrMr , obtained by overlaying the masked region in red on Ir . This two-view design retains full-image context from Ir while enforcing fine-grained understanding of the target object, producing more discriminative branches for subsequent search. The MLLM was then queried with an instruction prompt Pgen that follows three rules. (1) It requests a short noun phrase to match the textual concept input required by SAM3. (2) It explicitly asks the model to focus on the red-highlighted masked region. (3) If a parent-node concept is provided, it instructs the model to propose either synonymous alternatives or more fine-grained refinements conditioned on that concept, enabling progressive expansion along the tree. Formally, we generate the candidates set as Mr {Ti }N (3) i=1 = MMLLM Ir , Ir , Pgen , where {Ti }N i=1 denotes the resulting concept candidates. 4
MAC-AutoML CG-ICS Pipeline
Concept Reasoning You are a expert in...
Dog 1.Dog 2.Teddy 3.Animal
MLLM Concept Reasoning
SAM3
N Candidates
Visual Exemplar Extraction
Concept generation
Hound
Dog 0.75
Concept Scoring
Teddy 0.32
0.68
0.81
0.71
Pet Dog
Hound
Puppy
Animal 0.62
......
SAM3
0.55
0.54
0.42
Pet
Creature
Beast
Visual Exemplar Extraction SAM3
stitch
Figure 2. Overview of the proposed CG-ICS framework. Given a reference image and mask (Ir , Mr ) together with a query image Iq , CG-ICS performs concept reasoning and visual exemplar extraction in parallel. The concept reasoning formulates concept selection as a score-guided tree search, where the MLLM performs concept generation to expand branches and SAM3 serves as the scoring function to evaluate nodes, yielding the best-scoring concept for the target. Meanwhile, the visual exemplar extraction uses SAM3 to obtain a visual prior from the reference–query pair. Finally, SAM3 fuses the selected concept with the visual exemplars to predict the target mask.
Concept Scoring. Given a candidate concept Ti , we design a SAM3-based scoring function that serves as the node evaluation module in our concept tree. This score quantifies how well Ti (i) explains the annotated reference target and (ii) transfers to the query image. We first define Reference Fidelity (RF) score to measure whether the concept accurately describes the reference target indicated by Mr . For each Ti , we prompt it to sem SAM3 on the reference image Ir and obtain the semantic mask Mr,i : sem Mr,i = MSAM3 (Ir , Ti , ∅)
sem
.
(4)
It was then compared with the ground-truth reference mask Mr by computing their IoU. We use the IoU as the RF score: sem SiRF = IoU Mr,i , Mr . (5) Besides, the concept should also match the to-be-segmented part in the query image. We prompt SAM3 with the same Ti on the query image Iq and use the presence score as the Query Matchability (QM) score: SiQM = MSAM3 (Iq , Ti , ∅)
pres
.
(6)
Finally, we combine the two criteria with a multiplicative rule to obtain the concept score: Si = SiRF · SiQM .
(7)
Tree Search for Concept Refinement. Based on Concept Generation and Concept Scoring, we perform a score-guided tree search to refine textual concepts. We maintain a frontier F (k) at round k, and initialize the 5
MAC-AutoML
search by expanding the root node with Concept Generation to obtain N first-layer candidates. For each node concept Tik ∈ F (k) , we compute its score Si using Concept Scoring. We prune nodes with Sik < τpruned , and only expand the remaining ones. Each surviving concept node will be passed into Concept Generation again to produce N children concepts, including near-synonyms and finer-grained refinements. We maintain a concept buffer B that stores all visited concepts to prevent duplicate generation during expansion. The resulting children form the next frontier F (k+1) , and the expand–score–prune loop repeats up to K rounds. The looping will be early stopped if any node reaches Sik ≥ τstop . If all nodes in a round are fully pruned, the expansion will restart by returning to the parent node to generate totally different concepts. Finally, we select the best concept with the highest score in B by T ∗ = arg max Si . B
3.3
(8)
Visual Exemplar Extraction
Relying on textual concepts alone can be unreliable, since MLLMs may hallucinate or output over-specific phrases that do not correspond to the true target. Moreover, SAM3 highlights that providing visual exemplars (bounding boxes of the target) can help specify uncommon categories. However, the standard SAM3 exemplar prompt is defined within a single image and does not directly support cross-image exemplar transfer. Inspired by SegGPT [36], which stitches the reference and query in a single input, we propose a simple yet effective way to derive a visual exemplar of the query image. We first form a stitched image by concatenating the reference and query images horizontally: Irq = Stitch(Ir , Iq ).
(9)
The reference exemplar boxes can be extracted from the reference mask. Vr = BBox(Mr ).
(10)
Next, we use Vr as a visual exemplars prompt to invoke SAM3 on the stitched image. ins Mrq = MSAM3 (Irq , ∅, Vr )ins .
(11)
Since SAM3 can segment all matched instances under a prompt, it can return masks on both the reference side ins and the query side within Irq . We take the instance mask on the query half, denoted as Mqins = RightHalf(Mrq ), and convert it into coarse target exemplar boxes: V ∗ = Vq = BBox Mqins . (12) The resulting V ∗ serves as the visual exemplars, which can be subsequently used together with the textual concept to stabilize the final segmentation.
3.4
Segmentation with Joint Prompts
After selecting the best textual concept T ∗ and deriving the target exemplar boxes V ∗ , we feed them jointly to SAM3 for the final segmentation. The textual concept provides category-level semantics, while the visual exemplars constrain the spatial reference on the query image. Formally, we perform the final inference on the query image as Mqins , Mqsem , Sqpres = MSAM3 (Iq , T ∗ , V ∗ ) . (13) We take the semantic mask Mqsem as the final prediction Mq .
3.5
Multiple-Reference Setting
When multiple reference examples {(Irn , Mrn )}m n=1 are available, we extend CG-ICS to the multiple-shot setting. For textual concepts, due to the MLLM’s limited ability to reliably reason over many images at once, we perform concept reasoning on each reference independently and obtain one final concept per reference, denoted as {T ∗ }m n=1 . We then compare these m concepts and select the best one by re-scoring each of them on all reference image-mask pairs to measure cross-reference consistency. For visual exemplar extraction, we stitch each reference with the query separately to obtain query-side bboxes from each reference, and then collect all such bboxes as visual exemplars. 6
MAC-AutoML
Table 1. Comparison with state-of-the-art methods under the Standard ICS protocol. Gray indicates methods trained on in-domain datasets that include the test categories. Bold denotes the best result, and underline denotes the second best.
Methods
Pascal-5i
COCO-20i
LVIS-92i
FSS-1000
1-shot 5-shot 1-shot 5-shot 1-shot 5-shot 1-shot 5-shot
SegGPT [36] SINE [22] DiffwS [43] UNICL-SAM [28] SANSA [10]
83.0 85.4 88.3 -
89.8 86.2 87.8 -
Training-based 56.1 67.9 64.5 66.1 71.3 72.2 77.8 78.7 78.6 75.6
18.6 31.2 31.4 34.1 50.3
25.4 35.5 35.4 37.1 59.0
85.6 – 87.8 84.0 90.0
89.3 – 88.0 86.3 91.0
Matcher [23] GF-SAM [40] CG-ICS (ours)
68.1 72.1 89.3
74.0 82.6 89.6
Training-free 52.7 60.7 58.7 66.8 72.3 72.6
33.0 35.2 55.4
40.0 44.2 56.2
87.0 88.0 90.2
89.6 88.9 89.3
4
Experiments
4.1
Datasets and Protocol
Datasets. To demonstrate semantic ICS capability, we evaluate CG-ICS on four representative ICS benchmarks, including Pascal-5i , COCO-20i , LVIS-92i , and FSS-1000. Pascal-5i is built upon Pascal VOC 2012 [12] and SDS [16]. It contains 20 object categories that are split into 4 folds, with 5 classes per fold. COCO-20i is derived from MS COCO [21] and includes 80 categories in total. The 80 classes are partitioned into 4 folds, with 20 classes per fold. LVIS-92i [23] is built based on LVIS [15] which is more challenging. It selects 920 categories with more than two images per class and divides them into 10 folds. FSS-1000 [20] contains 1000 categories that are split into training, validation, and test containing 520, 240, and 240 categories, respectively. Protocol. Three types of experiments are conducted. (1) Standard ICS evaluates overall segmentation performance as in prior ICS work, and thus we follow the setting of Matcher [23] for fair comparison. For Pascal-5i and COCO-20i , we sample 1,000 queries per fold from the validation split, for LVIS-92i we sample 2,300 queries per fold, and for FSS-1000 we use all queries in the official test split. We evaluate both 1-shot and 5-shot settings on all datasets and report the mean Intersection-over-Union (mIoU) as the metric. (2) Robustness to Reference Selection. For each fold on Pascal-5i , COCO-20i , and LVIS-92i , we sample 500 query images and pair each query with 50 different reference examples from the same class. In addition to the Std mean IoU, the Standard Deviation (Std) and the Coefficient of Variation CV = mIoU across these references are also reported. The lower values of Std and CV indicate better relative stability. (3) Robustness to Corrupted References. Following UNICL-SAM [28], we apply diverse corruptions to the reference examples and report the mIoU with clean data as well as the performance drop under corruptions. This evaluation is conducted on 1500 sampled instances from COCO-20i .
4.2
Implementation Details.
We adopt Qwen3-VL-4B [3] as the MLLM backbone for concept generation. For scoring and mask prediction, the official SAM3 [7] release is adopted as the promptable segmentation backbone. All model parameters are frozen. For the tree search, each successful node expands to N =5 refined concept candidates generated by the MLLM. A node is pruned if its score is smaller than τpruned =0.5. We enable early stopping when any node reaches τstop =0.8 to reduce computation. The search runs for at most K=3 refinement rounds.
4.3
Comparison with state-of-the-art.
Standard ICS. We compare CG-ICS with state-of-the-art training-free and training-based generalist ICS methods in Table 1. Among training-free approaches, CG-ICS delivers a clear and consistent lead across all benchmarks, e.g., +17.2 mIoU on Pascal-5i , +13.6 on COCO-20i , and +20.2 on LVIS-92i in 1-shot 7
MAC-AutoML
Table 2. Comparison results under the Robustness to Reference Selection protocol. Gray indicates methods trained on in-domain datasets that include the test categories. Bold denotes the best result, and underline denotes the second best.
Pascal-5i
Methods
COCO-20i
mIoU
Std
CV
SegGPT [36] SINE [22] SANSA [10]
75.1 81.7 83.9
17.3 15.8 9.3
Matcher [23] GF-SAM [40] CG-ICS(ours)
65.3 70.6 88.4
20.6 19.3 6.9
mIoU
Std
LVIS-92i CV
mIoU
Std
CV
Training-based 23.0% 45.7 22.7 19.3% 68.3 16.1 11.1% 74.3 13.1
49.7% 23.6% 17.6%
19.4 30.9 50.8
13.5 20.3 18.2
69.6% 65.6% 35.8%
Training-free 31.5% 53.9 18.7 27.3% 58.3 16.4 7.8% 72.1 9.3
34.7% 28.1% 12.9%
38.8 42.2 57.1
19.2 18.7 17.2
49.4% 44.3% 30.1%
Table 3. Comparison results under the robustness to Corrupted References protocol. Gray indicates methods trained on in-domain datasets that include the test categories. Bold denotes the best result, and underline denotes the second best.
Methods
Clean
Color
Blurriness
Compression
Space
Domain
Deformation
Mean
Ratio
SegGPT [36] SINE [22] SANSA [10] UNICL-SAM [28]
62.1 70.0 70.5 79.8
-2.7 -0.7 -1.2 -1.0
-3.0 -1.7 -4.0 -1.8
Training-based -2.7 -5.9 -11.1 -3.0
-7.8 -2.8 -5.8 -3.3
-6.2 -6.5 -12.9 -4.8
-8.8 -12.4 -1.4 -5.0
-5.2 -5.0 -6.1 -3.1
-8.4% -7.1% -8.6% -3.9%
-2.5 -0.8 -0.3
Training-free -6.5 -5.3 -4.2
-3.8 -1.9 -3.4
-5.3 -3.8 -5.2
-6.2 -3.0 -0.5
-4.1 -2.5 -2.3
-9.9% -4.1% -3.1%
Matcher [23] GF-SAM [40] CG-ICS(ours)
41.6 61.5 74.6
-0.5 -0.1 -0.1
setting over the strongest training-free baseline GF-SAM [40]. Notably, despite being training-free, CG-ICS remains competitive with training-based methods trained with in-domain data that includes the test categories. CG-ICS still achieves state-of-the-art results on Pascal-5i (89.3), LVIS-92i (55.4), and FSS-1000 (90.2) in 1-shot setting. In the remaining settings, CG-ICS only slightly trails the best training-based competitors, e.g., 0.2 mIoU on Pascal-5i 5-shot and 3.3/6.0 mIoU on COCO-20i 1-shot/5-shot. Overall, strong performance across diverse datasets indicates that our CG-ICS is a powerful paradigm. Robustness to Reference Selection. Table 2 reports robustness under varying in-context references. Across all three benchmarks, CG-ICS obtains the lowest CV , indicating the most stable segmentation relative to its mean performance. On Pascal-5i and COCO-20i , CG-ICS achieves the lowest CV (7.8% and 12.9%), markedly improving over GF-SAM (27.3% and 28.1%) and remaining more stable than training-based baselines such as SANSA (11.1% and 17.6%). On LVIS-92i , CG-ICS still yields the best robustness with 30.1% CV , outperforming SANSA (35.8%) and largely reducing the instability of SegGPT/SINE (69.6%/65.6%). Overall, the consistently smallest CV indicates that CG-ICS maintains strong accuracy while being the least sensitive to reference selection. Robustness to Corrupted References. Table 3 compares robustness under six corruptions by reporting the mIoU on clean data and the mIoU drops. CG-ICS attains the best decline Ratio across all compared generalists, reaching only −3.1%, which is better than the strongest training-based baseline UNICL-SAM (−3.9%). This advantage is not achieved by sacrificing accuracy, as CG-ICS also delivers a high clean mIoU of 74.6. In contrast, other training-free methods suffer noticeably larger relative drops, e.g., GF-SAM and Matcher 8
MAC-AutoML
Table 4. Ablation study of the each component on COCO-20i . The first row represents directly feeding a single concept prompt generated by the MLLM into SAM3 for segmentation. We then incrementally add candidate sampling, RF/QM scoring, tree-search, and the visual branch.
Concept Reasoning Candidates ✓ ✓ ✓ ✓ ✓ ✓
(a) Performance
RF-Score
QM-Score
Search
✓ ✓ ✓ ✓
✓ ✓ ✓ ✓
✓ ✓
(b) Robustness
✓
(a) Performance
Figure 3. Effect of the number of search nodes N on performance and robustness.
COCO-20i
Visual
mIoU
Std
CV
67.6 66.5 69.3 62.8 70.1 71.2 72.1
15.1 15.3 13.9 18.2 11.3 10.5 9.3
22.3% 23.0% 20.5% 29.0% 16.1% 14.7% 12.9%
(b) Robustness
Figure 4. Effect of the number of search rounds K on performance and robustness.
degrade to −4.1% and −9.9%, respectively, while training-based SegGPT/SINE/SANSA are even more sensitive, with Ratio ranging from −7.1% to −8.6%. Across individual corruptions, CG-ICS remains nearly invariant to color and blur (both ≤ 0.3 drop) and is particularly resilient to deformation (−0.5), which jointly contributes to its lowest overall Ratio. Overall, the consistently smallest Ratio demonstrates that CG-ICS provides state-of-the-art robustness against distribution shifts, even outperforming training-based generalists.
4.4
Ablation Studies.
We conduct ablation studies under the 1-shot setting of Robustness to Reference Selection on COCO-20i , which are the most widely used ICS benchmarks. Ablation study of key components. As shown in Table 4, directly using a single MLLM-generated concept prompt yields limited performance and robustness, with 67.6 mIoU and 22.3% CV . Introducing multiple candidates alone brings marginal changes, while adding the proposed RF-Score and QM-Score consistently improves both accuracy and stability. In particular, combining RF and QM reduces the relative variation to 16.1% CV and boosts performance to 70.1 mIoU , indicating that concept selection is critical under varying references. The search-based selection further strengthens robustness (14.7% CV ), and enabling the visual exemplar branch achieves the best overall trade-off, reaching 72.1 mIoU with the lowest dispersion (9.3 Std, 12.9% CV ). Overall, each component contributes incrementally, and the full design is necessary to obtain both high accuracy and reliable reference-insensitive behavior. Ablation on the search nodes and rounds. On COCO-20i , increasing the number of search nodes (concept candidates) N steadily improves the average segmentation accuracy and enhances robustness across different 9
MAC-AutoML
Worst Reference
Best Reference
Reference
GT
GF-SAM
CG-ICS
Reference
GT
GF-SAM
CG-ICS
Figure 5. Qualitative comparison of GF-SAM and CG-ICS under varying in-context references ranked from worst to best according to GF-SAM performance.
references. As shown in Fig. 3a, mIoU increases from 67.6 at N =1 to 71.9 at N =4, and then quickly saturates around 72.1–72.4 for larger N (with 72.10 at N =5). Meanwhile, Fig. 3b shows that Std drops from 15.1 at N =1 to 10.5 at N =4, reaches 9.30 at N =5, and stays nearly unchanged afterwards. Overall, both curves become almost flat once N > 5, suggesting diminishing returns from generating more candidates. Therefore, we set N =5 in all experiments to balance performance, robustness, and inference cost. For the search rounds K, Fig. 4 exhibits the same trend, where both mIoU and Std change marginally once K > 3.
4.5
Qualitative Results.
We visualize qualitative comparisons between GF-SAM and CG-ICS under varying in-context references in Fig. 5. GF-SAM exhibits strong sensitivity to the reference choice, and its predictions can change noticeably as the reference varies from “worst” to “best.” When the reference viewpoint shifts significantly (the first column of the top panel), it incorrectly segments the airplane tail instead of the whole target, indicating a tendency to match salient subparts rather than preserving the full object extent. When the object is partially occluded (the first column in the bottom panel), it is easily distracted by occluders and co-occurring instances, and thus tends to segment the glass rather than the bottle. Moreover, even for moderately better references, GF-SAM may still produce incomplete masks or leak to nearby regions with similar textures, suggesting inconsistent 10
MAC-AutoML
correspondence under subtle appearance changes. This also implies that GF-SAM often requires carefully selected “good” references to avoid failure cases, limiting its practicality in real-world usage. Such errors reflect the limitation of pure visual matching: correspondence is largely driven by local appearance similarity, which becomes unreliable under viewpoint changes and occlusions, leading to semantic drift and unstable boundaries. In contrast, CG-ICS remains notably stable across the same reference range and consistently outputs high-quality masks, demonstrating stronger robustness to imperfect in-context references.
5
Conclusion
In this work, we study ICS with robustness as a first-class objective and establish evaluation protocols that measure both average performance and reference-induced variance. We introduce Concept-Guided In-Context Segmentation (CG-ICS), a training-free framework that performs concept selection through a score-guided tree search, where the MLLM performs concept generation to expand branches and SAM3 serves as the scoring function to evaluate nodes to identify a reliable textual prompt, complemented by a parallel visual exemplar route for query-side grounding. Across standard ICS benchmarks and robust evaluation protocols, CG-ICS achieves strong accuracy while substantially reducing variance, moving ICS toward more reliable real-world deployment.
Acknowledgements This work was supported by the New Generation Artificial Intelligence-National Science and Technology Major Project (No. 2025ZD0122701), the National Natural Science Foundation of China ( No. 62576299), and Fundamental Research Funds for the Central Universities and Xiaomi Young Talents Program.
11
MAC-AutoML
References [1] Aakanksha and A. N. Rajagopalan. Improving robustness of semantic segmentation to motion-blur using classcentric augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10470–10479, June 2023. [2] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023. [3] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and Ke Zhu. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. [4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923, 2025. [5] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances in Neural Information Processing Systems, 35:25005–25017, 2022. [6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. [7] Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang, Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung-Han Wu, Yu Zhou, Liliane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan Zhang, and Christoph Feichtenhofer. Sam 3: Segment anything with concepts, 2025. [8] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9650–9660, October 2021. [9] Wei-Ting Chen, Yu-Jiet Vong, Sy-Yen Kuo, Sizhou Ma, and Jian Wang. Robustsam: Segment anything robustly on degraded images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4081–4091, June 2024. [10] Claudia Cuttano, Gabriele Trivigno, Giuseppe Averta, and Carlo Masone. Sansa: Unleashing the hidden semantics in sam2 for few-shot segmentation, 2025. [11] Kazuki Endo, Masayuki Tanaka, and Masatoshi Okutomi. Semantic segmentation of degraded images using layer-wise feature adjustor. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3205–3213, January 2023. [12] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303–338, 2010. [13] Yunhe Gao, Di Liu, Zhuowei Li, Yunsheng Li, Dongdong Chen, Mu Zhou, and Dimitris N. Metaxas. Show and segment: Universal medical image segmentation via in-context learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20830–20840, June 2025. [14] Dazhou Guo, Yanting Pei, Kang Zheng, Hongkai Yu, Yuhang Lu, and Song Wang. Degraded image semantic segmentation with dense-gram networks. IEEE Transactions on Image Processing, 29:782–795, 2020.
12
MAC-AutoML
[15] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5356–5364, June 2019. [16] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 991–998, 2011. [17] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16000–16009, June 2022. [18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020. [19] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollar, and Ross Girshick. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015–4026, October 2023. [20] Xiang Li, Tianhan Wei, Yau Pun Chen, Yu-Wing Tai, and Chi-Keung Tang. Fss-1000: A 1000-class dataset for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2869–2878, June 2020. [21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. [22] Yang Liu, Chenchen Jing, Hengtao Li, Muzhi Zhu, Hao Chen, Xinlong Wang, and Chunhua Shen. A simple image segmentation framework via in-context examples. Advances in Neural Information Processing Systems, 37: 25095–25119, 2024. [23] Yang Liu, Muzhi Zhu, Hengtao Li, Hao Chen, Xinlong Wang, and Chunhua Shen. Matcher: Segment anything with one shot using all-purpose feature matching. In International Conference on Learning Representations, volume 2024, pages 17904–17925, 2024. [24] Lingchen Meng, Shiyi Lan, Hengduo Li, Jose M Alvarez, Zuxuan Wu, and Yu-Gang Jiang. Segic: Unleashing the emergent correspondence for in-context segmentation. In European Conference on Computer Vision, pages 203–220. Springer, 2024. [25] Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang-Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023. [26] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. In International Conference on Learning Representations, volume 2025, pages 28085–28128, 2025. [27] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684–10695, June 2022. [28] Dianmo Sheng, Dongdong Chen, Zhentao Tan, Qiankun Liu, Qi Chu, Tao Gong, Bin Liu, Jing Han, Wenbin Tu, Shengwei Xu, and Nenghai Yu. Unicl-sam: Uncertainty-driven in-context segmentation with part prototype discovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20201–20211, June 2025. [29] Yanpeng Sun, Jiahui Chen, Shan Zhang, Xinyu Zhang, Qiang Chen, Gang Zhang, Errui Ding, Jingdong Wang, and Zechao Li. Vrp-sam: Sam with visual reference prompt. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23565–23574, June 2024.
13
MAC-AutoML
[30] Wei Suo, Lanqing Lai, Mengyang Sun, Hanwang Zhang, Peng Wang, and Yanning Zhang. Rethinking and improving visual prompt selection for in-context learning segmentation. In European Conference on Computer Vision, pages 18–35. Springer, 2024. [31] Wailing Tang, Biqi Yang, Pheng-Ann Heng, Yun-Hui Liu, and Chi-Wing Fu. Overcoming support dilution for robust few-shot semantic segmentation. arXiv preprint arXiv:2501.13529, 2025. [32] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. [33] Chaoyang Wang, Xiangtai Li, Henghui Ding, Lu Qi, Jiangning Zhang, Yunhai Tong, Chen Change Loy, and Shuicheng Yan. Explore in-context segmentation via latent diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 7545–7553, 2025. [34] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. [35] Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6830–6839, June 2023. [36] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Towards segmenting everything in context. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1130–1140, October 2023. [37] Weihao Xia, Zhanglin Cheng, Yujiu Yang, and Jing-Hao Xue. Cooperative semantic segmentation and image restoration in adverse environmental conditions. arXiv preprint arXiv:1911.00679, 2019. [38] Chengming Xu, Chen Liu, Yikai Wang, Yuan Yao, and Yanwei Fu. Towards global optimal visual in-context learning prompt selection. Advances in Neural Information Processing Systems, 37:74945–74965, 2024. [39] Zhou Yang, Weisheng Dong, Xin Li, Jinjian Wu, Leida Li, and Guangming Shi. Self-feature distillation with uncertainty modeling for degraded image recognition. In European Conference on Computer Vision, pages 552–569. Springer, 2022. [40] Anqi Zhang, Guangyu Gao, Jianbo Jiao, Chi Liu, and Yunchao Wei. Bridge the points: Graph-based few-shot segment anything semantically. Advances in Neural Information Processing Systems, 37:33232–33261, 2024. [41] Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Yu Qiao, Gao Peng, and Hongsheng Li. Personalize segment anything model with one shot. In International Conference on Learning Representations, volume 2024, pages 18250–18279, 2024. [42] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. What makes good examples for visual in-context learning? Advances in Neural Information Processing Systems, 36:17773–17794, 2023. [43] Muzhi Zhu, Yang Liu, Zekai Luo, Chenchen Jing, Hao Chen, Guangkai Xu, Xinlong Wang, and Chunhua Shen. Unleashing the potential of the diffusion model in few-shot semantic segmentation. Advances in Neural Information Processing Systems, 37:42672–42695, 2024.
14