Conceptio › Archive › arXiv CS

arXiv CSopen access

SIEVES: Selective Prediction Generalizes through Visual Evidence Scoring

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

knowledge-representation reasoning

artificial intelligence, reasoning, knowledge representation

SIEVES: Selective Prediction Generalizes through Visual Evidence Scoring Hector G. Rodriguez and Marcus Rohrbach

arXiv:2604.25855v1 [cs.CV] 28 Apr 2026

TU Darmstadt & hessian.AI, Germany Abstract. Multimodal large language models (MLLMs) achieve everstronger performance on visual-language tasks. Even as traditional visual question answering benchmarks approach saturation, reliable deployment requires satisfying low error tolerances in real-world out-ofdistribution (OOD) scenarios. Precisely, selective prediction aims to improve coverage, i.e. the share of inputs the system answers, while adhering to a user-defined risk level. This is typically achieved by assigning a confidence score to each answer and abstaining on those that fall below a certain threshold. To enable reliable generalization, we require reasoner models to produce localized visual evidence while answering, and design a selector that explicitly learns to estimate the quality of the localization provided by the reasoner. We show that SIEVES (Selective Prediction through Visual Evidence Scoring) improves coverage by up to three times on challenging OOD benchmarks (V* Bench, HRBench-8k, MME-RealWorld-Lite, VizWiz, and AdVQA), compared to non-grounding baselines. Beyond better generalization to OOD tasks, the design of the SIEVES selector enables transfer to proprietary reasoners without access to their weights or logits, such as o3 and Gemini3-Pro, providing coverage boosts beyond those attributable to accuracy alone. We highlight that SIEVES generalizes across all five tested OOD datasets and reasoner models (Pixel-Reasoner, o3, and Gemini-3-Pro), without benchmark- or reasoner-specific training or adaptation. What is the color of the guard’s glove?

SIEVES: SelectIvE Prediction through Visual Evidence Scoring Answering w. localization Visual Evidence Scoring

Let’s zoom in

How much can I trust the answer?

Answer Correctness 0.34/1 Localization 0.13/1

GT Answer white

+ The cropped image provides a clearer view of the guard. Answer: black.

Reject! 0.28/1

Crop-Answer Coherence 0.41/1

Fig. 1: Selective prediction through Visual Evidence Scoring. Given a visual question, a tool-augmented reasoner provides visual evidence by zooming in on relevant regions before answering. The selector scores the answer along three axes: correctness (is the final answer accurate?), localization (did the model look at the right part of the image?), and coherence (does the visual evidence support the answer?). These scores are combined, and for poorly localized or incoherent answers the model abstains, avoiding incorrect answers from reaching the user in high-stakes deployment scenarios.

H. G. Rodriguez and M. Rohrbach

Introduction

Recent generations of large language models (LLMs) [1, 21] and vision-language models (VLMs) [3, 33] are capable of performing an increasing number of economically valuable tasks autonomously, from document understanding [29, 42] to assistive visual answering [13, 17]. Often in these tasks, mistakes can be costly, both in human and financial terms. However, deciding which tasks are beyond the reach of these models, and hence the model should abstain or defer to a human expert, is challenging. Selective prediction, the ability to know when to answer and when to abstain at a given risk tolerance, is far from being a solved problem [44]. Visual Question Answering (VQA) has been a good testbed for VLMs. Nonetheless, even as traditional VQA benchmarks [2, 18] become saturated from an accuracy perspective [36], selective prediction remains a challenge [11]. There are several reasons why selective prediction continues to be difficult. Most importantly, machine learning (ML) systems are predominantly designed to answer as many questions correctly as possible, with little or no training to abstain when in doubt [25]. Since the user’s risk tolerance is unknown in advance, a truly binary abstain/answer decision is too rigid. Selective prediction, therefore, assigns each answer a numerical confidence score, which in turn makes the task harder: the selector must now rank the full distribution of answer-question pairs. Existing selective prediction methods are often model-specific [11, 31, 45] and cannot be applied to frontier models, since they rely on model internals like log-probabilities or hidden activations, which are not available in proprietary model APIs1 . Other approaches [26, 38] make selective prediction much more costly than the original answering task. Here, we aim to solve both limitations. We propose Selective Prediction through Visual Evidence Scoring (SIEVES) for generalizable selective prediction. A strong reasoner that can zoom-in on the high-resolution image produces multimodal chain-of-thought reasoning with zoomed-in crops before providing a final answer. This exposes rich evidence that makes confidence prediction easier. We train the selector to take as input the full conversation, i.e. question, image, the multimodal chain-of-thought (MMCoT) with crops, and the final answer, and output a scalar confidence that the answer is correct. Crucially, the SIEVES selector does not only predict a single confidence score. Since the MM-CoT exposes where the model looked at before answering, the selector also explicitly assesses the quality of the localization shown in the MM-CoT. Concretely, the selector predicts three signals (Fig. 1): correctness (is the final answer accurate?), localization (did the model look at the right part of the image?), and coherence (does the visual evidence support the answer?). We show that training the selector to explicitly estimate the quality of the localization, instead of only predicting correctness, is essential to reliably use the localization evidence provided by the reasoner. 1

OpenAI: https://community.openai.com/t/logprobs-deprecated-for-gpt-5models/1355427; Anthropic: https://docs.anthropic.com/en/api/openai-sdk# detailed-openai-compatible-api-support.

SIEVES: Selective Prediction via Visual Evidence Scoring

We evaluate on a wide range of VQA benchmarks, which are out-of-distribution for the reasoner and selector, requiring varying degrees of generalization. These benchmarks range from high-resolution natural image benchmarks (V*-bench, HR-Bench-8k), diverse high resolution tasks including diagrams, tables, and remote sensing (MME-RealWorld-Lite), to real-world questions from blind users (VizWiz) and Human-Adversarial problems (AdVQA). We demonstrate that SIEVES improves the coverage at relevant risk tolerances by up to three times compared to non-grounding baselines, as shown in Tab. 2. Our main contributions can be summarized as follows: 1. We study selective prediction in challenging and diverse VQA settings, which require confidence prediction methods to generalize. 2. We propose requiring the reasoner model to provide visual evidence of its final answer via a zoom-in tool, which eases inferring the quality of the given answer. Zoom-in tools have been shown to improve answer accuracy in highresolution VQA tasks, but leveraging such localization as answer evidence remains, to the best of our knowledge, unexplored. 3. We develop a framework that trains a selector to explicitly estimate the quality of the visual evidence provided by the answering model. This significantly outperforms predicting answer correctness alone, and is key to realizing the benefits of the reasoner providing the visual evidence in OOD scenarios. 4. We design a selector that relies only on observable outputs (question, image, multimodal chain-of-thought, final answer) without using model internals like log-probabilities or hidden activations. This makes the selector model agnostic and allows it to generalize from weaker open source reasoners to proprietary frontier models. Concretely, training the SIEVES selector only on traces from finetuned Qwen2.5-VL-7B (i.e. Pixel-Reasoner [39]), SIEVES achieves even higher coverage when filtering o3 and Gemini-3-Pro answers without training on any o3 and Gemini traces/answers.

Related work

Reasoning with tools. Tool usage has become central to the recent explosion in reasoning capabilities of multimodal large language models (MLLMs). For VQA tasks, providing VLMs with the ability to use zoom-in tools has significantly improved the ability to answer questions about high-resolution images, through a form of multimodal chain-of-thought reasoning [14, 39, 54]. Frontier VLMs can benefit from access to these tools without explicit training [14, 34], whereas smaller models are trained specifically to elicit such zooming in capability [39, 54]. However, this zooming-in ability has only been, to the best of our knowledge, studied as a means of improving accuracy. In this work, we propose leveraging the multimodal chain-of-thought that results from zooming in to assess how well grounded the final answer is. Therefore, we do not focus on improving the abilities of these models to zoom-in, but rather use these existing abilities to make answer correctness prediction easier. Unlike existing methods, which only show the benefits of zooming-in to improve accuracy in high-resolution VQA

H. G. Rodriguez and M. Rohrbach

tasks, we employ the MM-CoT as a source of answer grounding evidence. Using this MM-CoT, the selector model has a richer context to output a confidence score for each answer. Intuitively, the ability to analyze where the model looked before answering eases confidence score prediction. Test-time scaling via verification. Our work has connections to the use of external models to rank different model generations to the same question [9, 28], namely, using verifiers for test-time scaling. However, we focus on the selective prediction task, which requires ranking over the joint space of questions and answers, whereas verification effectively factors the problem by conditioning on the question. This makes the problem strictly harder, as scores must be comparable across different questions. Furthermore, verifiers are usually models of the same size and inference budget (i.e., chain-of-thought length) as the reasoners [37]. This implies that verification is as hard as answering the question itself. Here, we demonstrate that by having a reasoner provide visual evidence for the answer, and explicitly learning to estimate the quality of such evidence, confidence score prediction is made easier, and a smaller discriminative selector can be used. Selective prediction. Selective prediction formalizes reliable deployment by maximizing the portion of answered queries (coverage) under a user-specified error rate (risk), typically summarized with risk-coverage curves or AURC. Early approaches rely on intrinsic model uncertainty, e.g., maximum softmax probability or logit margins, to decide when to abstain [8, 10, 15, 47]. Such approaches are not directly applicable to generative models and suffer from the gap between the accuracy-maximizing training objective and the selective prediction task. Whitehead et al. [45] demonstrate that it is effective to learn a dedicated lightweight selector to predict confidence in the final answer given both model’s internal representations and question-image input. Dancette et al. [11] further refine this approach for out-of-distribution settings, and Mushtaq et al. [31] adapt the selector design to abstain when using generative VLMs as answerers. However, hidden activations are entirely model dependent, tying the selector to a specific model architecture. Furthermore, such model internals are not available for frontier models queried through proprietary APIs, which limits the applicability of such selectors. In our work, we design a selector that relies only on observable outputs, and hence is model-agnostic. Given the limitations of trained selectors that require model internals, recent works have developed alternative methods for abstaining with generative LLMs or VLMs. Khan and Fu [26] propose consistency as a proxy for certainty and use it to abstain. Srinivasan et al. [38] utilize an elaborate prompting scheme to use zero-shot selectors. Both of these methods make selective prediction much more expensive than answering, can only provide coarse confidence scores, and do not demonstrate significant coverage at demanding risk levels. In contrast, by using the multimodal reasoning process as a source of evidence for the selector, we avoid duplicating the compute devoted to solving the task, and can provide better confidence estimates with a smaller selector.

SIEVES: Selective Prediction via Visual Evidence Scoring Implicit confidence estimation Answering w/o localization

Selector

must find the answer again or guess

Confidence estimation

Question What is the color of the guard’s glove?

Training

I can see the guard wearing black gloves. Answer: black.

Correctness

SIEVES (ours) Answering w. localization

Let’s zoom in

Selector

black

white

+ The cropped image provides a clearer view of the guard. Answer: black.

≠

white

leverages localization evidence

Visual Evidence Scoring Correctness 0.34/1 Localization 0.13/1

GT Answer

Training black

≠

white

IoGT=0

incoherent Coherence 0.41/1

[…] view of the guard. Answer: whiteblack.

Fig. 2: A selective prediction framework must output an answer and a confidence score to a visual question. Top: Standard implicit confidence estimation performs language-only reasoning before answering. The selector model is trained to output a scalar confidence score, given the entire conversation {Question, Image, Reasoning, Answer}. Without evidence, the selector must solve the visual task entirely again or infer the confidence from the language-only reasoning. Bottom: We propose SIEVES (SelectiveIvE prediction through Visual Evidence Scoring). A reasoner with access to zoom-in tools localizes the visual evidence for its answer. The SIEVES selector is trained to predict correctness (whether the answer matches ground truth), localization (whether the visual evidence is localized), and coherence (whether the final textual reasoning and answer are coherent with the provided visual evidence). SIEVES filters out answers when the combined confidence score is low.

Selective Prediction through Visual Evidence Scoring

We study selective prediction for visual question answering by combining a strong reasoner with the ability to zoom-in on the image and a compact selector that outputs a confidence score for the answer provided by the reasoner. 3.1

Providing visual evidence via zooming-in tool

The reasoner model f is tasked with providing an answer A to the question Q about the image I. In the process, since it benefits answer accuracy, it produces multimodal chain-of-thought reasoning, denoted as R. Therefore, after the reasoner answers {R, A} = f (Q, I),

(1)

we have a conversation tuple {Q, I, R, A}. We use reasoners that can use a zoomin tool, a common capability of recent VLMs [34, 39]. 3.2

Visual Evidence Scoring

Fine-grained confidence scoring using visual evidence. Standard selective prediction learns to predict confidence scores that infer whether a final answer

H. G. Rodriguez and M. Rohrbach

matches the ground-truth label [11, 31, 45]. However, this signal alone does not capture why an answer is correct or incorrect, making it prone to overfitting to in-distribution patterns like question domain or reasoner patterns, and degrading under distribution shift. Intuitively, if the reasoner provides visual evidence for its answer, and the quality of that evidence can be evaluated, the confidence prediction task becomes easier and more robust. For VQA, in addition to the standard answer correctness confidence (ccorr ), two natural ways of scoring visual evidence signals arise: whether the model localized the relevant region (cloc ), and whether its final answer is coherent with the visual evidence it provides (ccoh ). As shown in Fig. 2, the selector consumes the question, image, MM-CoT and final answer tuple, and outputs three scalar confidence values: ccorr that the answer is correct, cloc that the relevant visual information was localized, and ccoh that the final answer is coherent with the provided evidence. Formally, {ccorr , cloc , ccoh } = s(Q, I, R, A) ∈ [0, 1]3 .

(2)

Finally, these individual confidence scores are aggregated to obtain a single confidence score csel . This confidence score, paired with a threshold, can enable abstention to maintain a desired error rate on the answered questions. Learning localization confidence. The predicted localization confidence score cloc aims to convey whether the model gathers the relevant visual information required to answer correctly. To learn this confidence score, the model’s predicted crops are matched to the ground-truth bounding boxes. Concretely, the groundtruth localization signal gloc is computed as the intersection-over-ground-truth (IoGT) of the predicted crop(s) and the ground truth bounding box(es). For the j i , and the j-th predicted crop Bcrop i-th ground-truth bounding box Bgt IoGTi,j =

i j Bgt ∩ Bcrop . i Bgt

(3)

If more than one ground truth bounding box is provided, we average the IoGT over all N ground truth boxes, allowing for one predicted crop to match multiple ground truth boxes. Explicitly, we define the mean IoGT, mIoGT , as N

mIoGT =

1 X max IoGTi,j . N i=1 j

(4)

Empirically, we find that discretizing this signal, namely using a binary spatial recall [30], performs better than regressing the continuous value. More concretely, gloc = 𝟙[mIoGT ≥ 0.75] ∈ {0, 1}.

(5)

Although IoGT is less common than IoU in the object detection literature, it is more suitable for our purpose, as the reasoners are not trained to provide tight bounding boxes, but rather to zoom in enough to "see" the relevant objects. Learning crop-answer coherence. The coherence confidence score tries to model whether the answering model, after correctly localizing the relevant region,

SIEVES: Selective Prediction via Visual Evidence Scoring

has an answer that is coherent with the relevant crop. Formally, the groundtruth coherence signal gcoh is computed as the maximum similarity between any j of the image crops Rcrop and the text of the last message, which includes the final textual reasoning and answer. Formally, j gcoh = gloc · max sim(Rcrop , [Rlast , A]) ∈ {0, 1}. j

(6)

Empirically, we find it best to determine similarity by asking an external VLM (Qwen 2.5 VL 7B [3]) for a binary coherence judgment gcoh ∈ {0, 1}, given the zoomed-in crops and last message. Rather than requiring manual annotation, gcoh offers a practical and scalable training signal, which we show empirically improves OOD selective prediction. As a sanity check, we run a small human evaluation on 200 randomly chosen training samples and find that human annotations match the same gcoh label in 92% of cases. Notably, if the localization is not correct, the coherence score is 0. The inclusion of the coherence signal, gcoh , is motivated by the observation that, even after zooming in to the relevant region, the reasoner would in some scenarios provide an answer that was not correlated with the provided crop. Fig. 2 shows an example of these labels. In Sec. 5, Tab. 3b analyzes the effect of model size on coherence labeling. Training the selector. For training the selector, all objectives are learned using binary cross-entropy (BCE) loss. The final loss becomes a weighted sum of the losses per objective, where L = λcorr · BCE(ccorr , y) + λloc · BCE(cloc , gloc ) + λcoh · BCE(ccoh , gcoh ). (7) λcorr , λloc , and λcoh represent the scalar weights for the correctness, localization, and coherence objectives. In Sec. 5, Tab. 3a shows a sensitivity analysis for the weights, demonstrating that including all objectives is better than excluding any, and showing our method is robust to variations in the weight values. Final confidence prediction. At inference, we use the predicted visual evidence scores and the answer correctness confidence prediction, to obtain a final confidence value. These three separate scores are then combined using the same weights as for training, to obtain the final confidence scalar score csel , i.e., csel = λcorr · ccorr + λloc · cloc + λcoh · ccoh .

Experimental setup

4.1

Training setting

(8)

Here we overview the training setting for the selector, details in Sec. B. Selector training data. A selector is tasked with mapping a conversation tuple {Q, I, R, A} into a confidence scalar value csel ∈ [0, 1]. Therefore, we first collect training data by generating answers using Pixel-Reasoner [39] as the reasoner model. Pixel-Reasoner [39] has been finetuned to use a zoom-in tool to improve high-resolution VQA accuracy. As the source of these questions, we use the

H. G. Rodriguez and M. Rohrbach

Benchmark

#Q Format

V* Bench [46] 191 HR-Bench-8k [43] 800 MME-RW-L [52] 1,782 VizWiz [22] 4,319 AdVQA [35] 10,000

MC MC MC OE OE

Avg. res.

Domain

2246×1582 Natural images: Attribute recognition, relative position 7680×4320 Attribute recognition, OCR, map & chart analysis, spatial reasoning 2000×1500 Remote sensing, autonomous driving, monitoring, diagrams & tables, OCR 1224×1224 Blind-user mobile photos: yes/no, counting, other (33% unanswerable) 640×480 Adversarially crafted: counting, OCR, rare concepts, reasoning

Table 1: Out-Of-Distribution (OOD) Benchmarks. We evaluate SIEVES on benchmarks of varying sizes (#Q), using Multiple-Choice (MC) and Open-Ended (OE) formats, high and lower resolution images, and very diverse domains.

Thyme [51] and TAT-DQA [55] datasets. These datasets comprise two different domains: high-resolution natural images and financial documents. We generate 5 responses from the reasoner model to each question. To improve the share of answers with localization, we modify the reasoner’s prompt to force it to zoom-in at least once. Since the questions in the training data are open-ended, when there is no string match between the ground truth and predicted answer, we prompt Qwen3-8B [48] using the ground truth and predicted answer to evaluate whether the predicted answer is correct. Selector model. We adopt a compact multimodal instruction-tuned model, specifically Gemma-3-4b-it [40], as the selector. Being smaller than the reasoner(s), this allows us to demonstrate that by using a reasoner that provides visual evidence, the selector can be made smaller and hence more efficient. We add three separate value heads, one for each of the confidence scores ccorr , cloc , ccoh . These value heads project the final hidden state of the last sequence element to a scalar value. Crucially, this means that predicting the additional confidence scores adds no noticeable overhead to the standard single confidence score selector. We finetune all linear layers of the backbone using LoRA [23] with rank 512. We train with 10% label smoothing. We select the best checkpoint for each risk level using a multiple-choice held-out set of 10% of Thyme [51]. 4.2

OOD Benchmarks

We evaluate on five complementary benchmarks spanning high-resolution natural images, diverse real-world tasks including diagrams, tables and remote sensing, real-world photos from blind photographers, and human-adversarial visual QA. This diversity stresses different failure modes and provides varied operating points for selective prediction, unlike traditionally used VQA benchmarks which tend to be in-distribution for recent models. Given the small size of V*Bench (191 samples), we generate five answers per question for all reasoners. For additional statistical soundness, we do the same for all other benchmarks with Pixel-Reasoner, except for AdVQA which is almost an order of magnitude larger than the rest. Due to cost constraints, when answering with o3, we only repeat this on HR-Bench-8k and VizWiz. For VizWiz and AdVQA, we use the validation splits, since the test splits are not public and the evaluation server for the test set does not support selective prediction metrics. These two benchmarks

SIEVES: Selective Prediction via Visual Evidence Scoring

are open-ended, so we use a hard accuracy metric with the same LLM judge used during training. Tab. 1 summarizes key properties of each benchmark, and Sec. E provides further details. 4.3

Selective prediction metrics

Coverage at risk (C@r). [15, 16] measures, for an answering system with an abstention option, the largest fraction of questions the system can answer while keeping the error rate among answered questions at most r. Intuitively, a better selector can answer more questions at the same accuracy guarantee. We study C@r at risk levels r ∈ {1, 5, 10, 15, 20, 25, 30}%. We report the average C@r across all risk levels. If the answering model has low accuracy, attaining meaningful coverage at low risk is difficult. In addition, low coverages become noisier at low risk levels if the benchmark size is small. For example, in the case of V* Bench (191 samples), C@1 at high accuracy can largely change based on a single sample. Therefore, Tab. 2 reports the lowest level at which at least one method reaches double-digit coverage. Sec. D contains results at all considered risk levels. Area under the risk–coverage curve (AURC). [15] integrates the error rate over all coverage levels, adding a sample at a time based on the confidence score provided by a selection method. Rather than fixing a single risk level, AURC rewards selectors that rank correct answers consistently above incorrect ones across the full operating range. Lower AURC is better. While AURC does not focus on the low-risk regime most relevant in deployment, it provides a useful single-number summary of ranking quality. Further details about both metrics can be found in Sec. A. 4.4

Baselines

In this paper, we explore whether (i) having a reasoner which localizes the relevant visual evidence by using a zooming tool helps reliability, and (ii) having a selector that explicitly evaluates the grounding of the answer improves estimating answer correctness. Therefore, our baselines are chosen to test both of these hypotheses. Firstly, we run our reasoners (Pixel-Reasoner and o3) with and without the zoom-in tool, and show whether the same abstention methods can answer more reliably when provided visual evidence. Secondly, we compare our grounding selector with various alternative abstention methods. Intrinsic reasoner confidence. We extract the log-probability of the final message, i.e. the last reasoning step and the final answer, and use it as the confidence score [6, 50]. We experiment with different sections of the conversation to extract the log-probability from, and find this to perform best in general. Notably, log-probabilities are not available for frontier proprietary models, so we cannot use this baseline for our experiments with o3 or Gemini-3-Pro. Zero-shot selector baselines. Given the strong zero-shot capabilities of recent VLMs across many domains, it is natural to consider whether they can estimate

H. G. Rodriguez and M. Rohrbach V* Bench

Method

HR-Bench-8k

MME-RW-L

VizWiz

AdVQA

C@5 C@r AURC↓ Acc C@1 C@r AURC↓ Acc C@5 C@r AURC↓ Acc C@20 C@r AURC↓ Acc C@10 C@r AURC↓ Acc

Pixel-Reasoner w/o localization Logprobs 0.2 52.7 14.7 0.0 6.2 79.6 0.9 30.8 Selector 2.3 54.6 15.7

30.3 20.3

0.1 8.3 0.0 17.8

35.3 0.0 51.6 30.4 1.6

0.1 3.5

48.8 45.6

0.1 0.0

0.3 7.7

42.5 33.7

Pixel-Reasoner w/ localization Logprobs 0.1 50.2 15.7 0.3 28.9 22.6 0.0 0.7 Zero-shot verb. 0.0 42.9 17.2 0.0 13.3 28.9 0.0 0.0 Zero-shot logit 0.0 47.1 15.6 80.5 0.0 32.8 21.3 68.4 0.0 4.7 Selector 1.4 53.1 14.1 0.0 42.7 17.4 0.0 16.9 SIEVES (ours) 9.7 59.5 12.8 2.3 41.1 18.1 4.0 18.7

43.5 43.5 36.4 31.9 31.0

0.1 0.0 51.3 0.0 0.1 7.6

0.2 0.0 0.0 1.5 6.6

51.3 0.0 50.8 0.0 53.1 38.0 0.0 44.2 0.2 42.3 2.8

0.0 0.0 0.0 6.3 9.3

40.3 40.7 38.6 58.1 33.0 32.2

o3 w/o localization Selector 0.1 21.8

23.4

71.8 0.3 33.3

19.2

72.1 1.1 13.4

29.3

56.9 6.3

3.8

40.9

o3 w/ localization Zero-shot verb. 0.0 64.7 Zero-shot logit 1.7 65.1 Selector 0.0 61.0 SIEVES (ours) 23.1 70.8

9.1 9.9 9.7 8.4

85.9

0.0 54.2 0.0 56.3 1.0 61.7 8.0 67.7

13.5 13.1 10.5 9.3

82.0

Gemini-3-Pro w/o localization Selector 8.8 68.4 8.8 86.4 5.8 74.5

5.4

85.6 5.1 34.7

Gemini-3-Pro w/ localization Zero-shot verb. 88.3 84.0 3.1 0.0 71.4 Zero-shot logit 93.5 84.8 2.9 1.4 79.9 94.3 Selector 40.7 77.6 3.5 4.5 80.1 SIEVES (ours) 95.0 85.6 3.1 14.4 80.8

6.4 4.9 3.5 4.4

90.1

65.6

38.1

45.8 0.0 19.4

0.0 0.0 36.1 0.0 0.0 41.6 0.0 0.0 4.8 35.3 0.0 0.0 39.0 0.0 58.7 46.9 0.1 16.6 28.6 0.0 3.7 35.2 3.8 1.7 17.5 28.4 11.7 12.4 33.3 8.2

0.0 9.3 0.0 19.5 1.2 28.2 12.2 32.3

21.9

25.2

54.5

68.0

27.5 24.9 30.6 21.9 72.4 33.3 20.1 36.5 19.6

60.9 3.8

4.2

41.9

44.1 15.0 55.0

12.8

80.3

28.2 0.0 26.4 0.0 63.2 22.8 0.0 21.5 8.1

0.0 0.0 2.7 8.7

46.2 0.0 42.9 17.0 47.7 0.0 42.9 17.3 43.6 81.4 39.5 32.5 58.9 11.9 37.2 33.3 59.8 11.5

Table 2: Selective prediction across reasoners and OOD benchmarks. C@r: coverage at the lowest risk level were any method obtains >10(%); C@r: average coverage across risk levels {1,5,10,15,20,25,30}%; AURC; Acc: answer accuracy (%), per reasoner, equal across abstention methods. SIEVES is trained on Pixel-Reasoner answers only; and generalizes to o3 and Gemini-3-Pro and these OOD benchmarks. Bold indicates best result per reasoner, underlined second best.

answer correctness without task-specific training. Therefore we present two zeroshot baselines which repurpose an existing VLM to produce confidence scores. In the zero-shot logit selector, the model is asked to think step by step and then output yes or no to express whether it thinks the answer is correct. We extract the log-probability of the yes token (normalized by the sum of the yes and no tokens) and use that as confidence score [4, 5, 41]. Alternatively, we ask the zero-shot verbalized selector to output a numerical confidence from 0 to 100 using tokens. For conciseness, Tab. 2 only reports these baselines for the stronger setting where the answering model is required to provide visual evidence zero-shot. Additional results answering using plain language CoT are provided in Sec. D, where we show it achieves considerably lower coverage and AURC. Selection without localization estimation. Lastly, we compare to a selector that is trained to predict only correctness, i.e. the scalar confidence that the answer is correct. Notably, we run two versions of this baseline. One is trained on answers generated by a reasoner without access to zoom-in tools, where no localization evidence is provided. In this case the selector must rely on the reasoner’s own textual reasoning to infer confidence, or solve the task again to estimate answer correctness. The other version is trained on answers generated by a reasoner with access to zoom-in tools, where localization evidence is provided.

SIEVES: Selective Prediction via Visual Evidence Scoring

C@1

C@5

C@10

C@15 41.4 42.3

C@r (%)

40 30.6

30 21.1

20.1

15.3

16.9

19.4

22.2 22.4 17.6

8.8 10.0

10 0

0.9 0.3 0.0 1.0

3.5

Selector w/o visual evidence

4.8

4.7

Zero-shot logit

Zero-shot verb.

Selector

SIEVES (ours)

Fig. 3: OOD coverage at varying risk levels for frontier proprietary reasoners not used for training the (SIEVES) selector. C@r averaged across OOD benchmarks and o3 and Gemini-3-Pro reasoners, excluding trivial cases with reasoner error < risk. SIEVES consistently achieves the highest coverage, especially at low risk.

Results

Visual evidence facilitates answer selection. Requiring localization for every answer simplifies selective prediction by producing explicit visual evidence. Across our benchmarks, we observe higher coverage and lower AURC in Tab. 2 when Pixel-Reasoner has access to the zoom-in tool and provides visual evidence. Notably, although zooming in has been predominantly used as means to improve answer accuracy [39, 54], we observe here that even when accuracy improvements are modest, the coverage gains obtained by the selector having access to visual evidence are much greater. This is most evident when comparing C@10 for Gemini-3-Pro on AdVQA. Even for the same standard selector, the coverage doubles from 15.0 to 32.5 with a mere 1.1% accuracy improvement. For SIEVES, this is even more pronounced, going from 1.6 C@20 for the correctness-only selector without localization in VizWiz, to 7.6 when using SIEVES, even though the reasoner accuracy actually decreases from 38.1 to 38.0. Importantly, to fully exploit the visual evidence produced by the reasoner, the selector must explicitly estimate the localization quality of each answer. This is evidenced by two correctness-only selectors with identical architectures but different training data: the selector trained on chains from the localized, tool-augmented answerer occasionally performs worse than the one trained on chains from the non-localized answerer, e.g. on MME-RealWorld-Lite and V*Bench (Tab. 2). Adding visual evidence scoring reverts this trend. However, when explicitly looking at the generalization to a stronger reasoner, o3 and Gemini-3-Pro in Tab. 2, we observe that the selector which receives answers with localization transfers more effectively. This further confirms that visual evidence helps generalizing to OOD reasoners, as selectors can overfit textual expressions of confidence which are model-specific. Localization estimation enables generalization. Estimating localization quality alongside answer correctness can maximally exploit the localization evidence provided by the reasoner, further improving the generalization beyond correctness-only selectors. By explicitly requiring that the selector evaluates the

H. G. Rodriguez and M. Rohrbach What is the traffic signal that the ego vehicle should pay attention to? (A) No parking. (B) Turn right. (C) Green light. (D) Go straight. (E) The image does not feature the object.

What is in this can, please?

GT Answer

Unanswerable

GT Answer Green light

Implicit confidence estimation Confidence Answering w/o localization estimation The visible can in the background shows an image of yellow kernels, which strongly suggests canned corn. Given that the foreground can appears to be the same […] Answer: corn

SIEVES (ours) Answering w. localization

+ Let’s zoom in

Answer: Corn (sweet corn kernels or creamed corn)

0.93/1 Accept!

Visual Evidence Scoring

Implicit confidence estimation Confidence estimation

Answering w/o localization […] There is a set of traffic lights facing the direction of the ego vehicle […] illuminated with a circular green light. […] Answer: Green light

SIEVES (ours) Answering w. localization

+ Let’s zoom in

0.07/1 Reject!

Visual Evidence Scoring

Correctness 0.41/1

Correctness 0.83 /1

Localization 0.13/1

Localization 0.93 /1

Coherence 0.17/1 0.31/1 Reject!

The traffic light hanging over the intersection is clearly visible and is displaying a green circular light […] Answer: Green light.

Coherence 0.83 /1 0.86/1 Accept!

Fig. 4: Qualitative examples in high-stakes settings where SIEVES correctly abstains or accepts while an implicit selector fails. Left: On this high-stakes VizWiz question from a blind user, SIEVES correctly assigns low confidence to o3’s answer for which the visual evidence points to a can in the background rather than the foreground, and where image clarity is low. The implicit confidence selector instead assigns very high confidence, even though the question is in fact unanswerable. Right: On a high-stakes autonomous-driving task from MME-RealWorld-Lite, SIEVES assigns high localization and coherence scores and correctly accepts Gemini-3-Pro’s answer because of high quality visual evidence. In contrast, the implicit confidence selector cannot identify the green light in the image and incorrectly rejects the correct answer.

grounding of the answer, the selector can discount overconfident but weakly grounded answers and learn to prefer well-grounded ones at the same expressed reasoner confidence. We find that SIEVES is the most consistent method offering improved coverage and AURC, as shown in Tab. 2. Fig. 4 shows high-stakes qualitative examples where, by explicitly scoring the quality of visual evidence, SIEVES correctly accepts or abstains when the implicit selector fails. Notably, zero-shot selectors perform very poorly in general. We hypothesize that, although these models are strong zero-shot answerers, estimating comparable confidence scores across the entire distribution of question-answer pairs is a challenging task, for which they have not been trained. Generalizing to stronger reasoners. An advantage of our design, which takes only model outputs and not model internals like hidden activations or

SIEVES: Selective Prediction via Visual Evidence Scoring (a) Objective weight ablation.

(b) Crop-answer coherence annotator.

OOD

λc λl λh C@r ↑ AURC↓ C@r ↑ AURC↓ 1.0 0.0 0.0 26.7 25.8 24.1 28.1 0.0 1.0 0.0 3.0 38.4 17.3 33.0 0.0 0.0 1.0 9.1 34.9 21.4 29.9 0.6 0.4 0.0 24.3 27.0 27.0 27.4 21.4 28.0 25.2 27.6 PR 0.6 0.0 0.4 0.6 0.3 0.1 26.6 25.7 27.0 27.3 0.6 0.2 0.2 23.5 27.0 25.6 27.5 0.3 0.3 0.3 19.4 29.6 24.2 28.9 0.2 0.4 0.4 14.1 33.1 21.9 30.2 f

1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.6 0.4 0.0 0.6 0.0 0.4 0.6 0.3 0.1 0.6 0.2 0.2 0.3 0.3 0.3 0.2 0.4 0.4

– – – – – – – – –

35.3 37.5 39.2 38.4 39.8 41.0 37.7 42.6 41.3

20.8 20.7 19.3 20.4 20.4 19.8 21.3 19.0 19.6

1.0 0.0 0.0 0.0 1.0 0.0 0.0 0.0 1.0 0.6 0.4 0.0 G3P 0.6 0.0 0.4 0.6 0.3 0.1 0.6 0.2 0.2 0.3 0.3 0.3 0.2 0.4 0.4

– – – – – – – – –

49.5 36.4 40.6 52.1 53.6 53.4 54.7 54.1 51.6

16.2 22.9 19.8 16.5 16.6 15.5 15.7 15.2 16.3

OOD

C@r ↑ AURC↓ C@r ↑ AURC↓

Model

Qwen 2.5 VL 7B PR Gemma 3 4B Qwen 2.5 VL 72B

26.6 24.4 26.8

25.7 27.1 25.5

27.0 25.5 25.5

27.3 28.1 27.8

Qwen 2.5 VL 7B o3 Gemma 3 4B Qwen 2.5 VL 72B

– – –

41.0 41.2 38.5

19.8 20.3 21.0

Qwen 2.5 VL 7B G3P Gemma 3 4B Qwen 2.5 VL 72B

– – –

53.4 53.6 54.7

15.5 16.2 15.8

OOD

C@r ↑ AURC↓ C@r ↑ AURC↓

BBox annot.

Human Gemini-3-Flash

26.6 24.2

25.7 27.5

27.0 26.3

27.3 27.6

Human Gemini-3-Flash

– –

41.0 40.0

19.8 21.0

G3P

Human Gemini-3-Flash

– –

53.4 52.1

15.5 16.2

Table 3: Ablations. We study modifications to the default SIEVES configuration , reporting AURC and average coverage (C@r). The OOD results are averaged over 5 benchmarks, Thyme MC holdout set is ID. We report the results on 3 reasoners f : Pixel-Reasoner (PR), o3 and Gemini-3-Pro (G3P). Dash (−) indicates ID entries omitted for OOD reasoners (o3, Gemini-3-Pro), as no ID tuning is performed. (a) Shows that excluding any of the objectives when training SIEVES is detrimental, and the setting is robust to modification of the λcorr , λloc , λcoh which weight each objective. (b) Demonstrates that both a larger VLM (Qwen2.5-VL-72B) can be used for obtaining coherence labels gcoh , as well as the same base model. (c) Gemini-3-Flash can be used to obtain bounding box annotations, enabling scalable training with minimal impact to performance. Bold indicates best result per reasoner, underlined second best.

log-probabilities, is that it can be applied to any reasoner. This includes frontier models served through proprietary APIs, which generally do not provide access to such internal signals. Tab. 2 confirms that this selector design can indeed generalize to stronger reasoners, such as o3 and Gemini-3-Pro. Particularly, we perform no adaptation whatsoever for the o3 or Gemini-3-Pro reasoners. They receive the same prompt as the Pixel-Reasoner, and we use the exact same selector checkpoints which are trained and tuned for abstaining on Pixel-Reasoner answers. Not only does the coverage at relevant risk levels not decrease under the change of reasoner, but it significantly increases in most cases. Notably, when selecting among these answers provided by o3 or Gemini-3-Pro with access to tools, the SIEVES selector consistently shows an advantage over the correctness-only selector. Fig. 3 further highlights that at low risk tolerances, SIEVES consistently achieves highest coverage, often by large margins.

H. G. Rodriguez and M. Rohrbach

The accuracy results indicate that both zoom-in tool use and stronger reasoners tend to yield higher answer correctness. It is worth noting that the increase in coverage that our SIEVES selector provides is beyond the increase in accuracy provided by these stronger reasoner settings. This means that SIEVES is indeed able to exploit the improved visual evidence provided by the stronger reasoner with access to zoom-in tools. Explicitly, when analyzing the behaviors of the reasoners with tools on V* Bench, we observe that o3 zooms in more (2.3 crops on average vs 1.1) and better (77% of questions have mIoGT ≥ 0.75 vs 35%). This illustrates yet another generalization dimension, where o3 and Gemini-3-Pro generate significantly higher number of crops on average, since Pixel-Reasoner was explicitly trained to zoom-in just once, and yet our selector trained on Pixel-Reasoner can generalize to o3 and Gemini-3-Pro. Similarly, the correctness-only selector is able to generalize better to o3 and Gemini-3-Pro when observing localization evidence, which was not the case with the in-domain Pixel-Reasoner answers. This can be explained by the fact that, when presented only textual reasoning, the selector often relies on the verbally expressed confidence from the reasoner, which varies across reasoners. In contrast, the visual evidence can be directly assessed by the selector, and better visual evidence (i.e. better localization quality) further helps this selector to achieve higher coverage. Ablating the selector design. Tab. 3a ablates the weights λcorr , λloc , λcoh for correctness, localization, and coherence. Empirically, including all objectives is better than excluding any, and the framework is robust to changes in the weights. Tab. 3b varies the VLM used to annotate crop-image coherence during training. We find that a larger labeler offers marginal gains, whereas a smaller model (in this case the same base model as the selector) can also be used with modest decrease in performance. Tab. 3c shows that SIEVES is robust to the use of fully-automated bounding box annotations. Specifically, we use Gemini-3-Flash to obtain bounding boxes instead of using human annotations for the training set. This shows automatic bounding-box annotation enables scalable SIEVES.

Conclusion

We introduce SIEVES (Selective Prediction through Visual Evidence Scoring) for generalizable selective prediction in challenging visual question answering tasks. By explicitly scoring the quality of visual evidence, SIEVES filters out overconfident but weakly grounded answers. On a diverse set of challenging OOD VQA benchmarks, SIEVES achieves up to three times higher coverage at low risk when compared with non-grounding baselines. Additionally, because the SIEVES selector only consumes model outputs, not internals such as log-probabilities or hidden states, we showed that it can generalize from being trained on a weaker open-source model to evaluated using a proprietary frontier model such as o3 and Gemini-3-Pro. Selective prediction will be increasingly important for safely automating a larger share of end-to-end tasks, and we hope our work can inspire the use of visual evidence to improve reliability in visual tasks.

SIEVES: Selective Prediction via Visual Evidence Scoring

Acknowledgements This research was partially funded by an Alexander von Humboldt Professorship in Multimodal Reliable AI, sponsored by the Federal Ministry of Research, Technology, and Space (BMFTR), by a LOEWE-Spitzen-Professur (LOEWE/4a//519/ 05.00.002(0010)/93), and has benefited from the Excellence Cluster “Reasonable AI” by the German Research Foundation (Deutsche Forschungsgemeinschaft – DFG) under Germany’s Excellence Strategy – EXC-3057. We gratefully acknowledge support from the hessian.AI Service Center (funded by the Federal Ministry of Research, Technology and Space, BMFTR, grant no. 16IS22091) and the hessian.AI Innovation Lab (funded by the Hessian Ministry for Digital Strategy and Innovation, grant no. S-DIW04/0013/003). We thank Tobias Wieczorek and Mohamed Abdelsalam for their feedback on a previous version of this manuscript.

Bibliography

[1] Anthropic: System card: Claude opus 4 & claude sonnet 4. https://wwwcdn.anthropic.com/6d8a8055020700718b0c49369f60816ba2a7c285.pdf (2025) [2] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: Visual question answering. In: ICCV (2015), https://doi.org/ 10.1109/ICCV.2015.279 [3] Bai, S., Chen, K., Liu, X., Wang, J., Ge, W., Song, S., Dang, K., Wang, P., Wang, S., Tang, J., et al.: Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025) [4] Belinkov, Y., Gurnee, W., Nanda, N., Sachan, M., Song, X., Stolfo, A., Wu, B.: Confidence regulation neurons in language models. In: NeurIPS (2024), https://doi.org/10.52202/079017-3970, URL https://doi.org/10. 52202/079017-3970 [5] Bhatt, U., Collins, K., Dooley, S., Goldblum, M., Gruver, N., Kapoor, S., Pal, A., Roberts, M., Weller, A., Wilson, A.: Large language models must be taught to know what they don’t know. In: NeurIPS (2024), https://doi. org/10.52202/079017-2729, URL https://doi.org/10.52202/0790172729 [6] Chen, J., Yoon, J., Ebrahimi, S., Arik, S., Pfister, T., Jha, S.: Adaptation with self-evaluation to improve selective prediction in LLMs. In: Bouamor, H., Pino, J., Bali, K. (eds.) Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5190–5213, Association for Computational Linguistics, Singapore (Dec 2023), https://doi.org/10.18653/ v1/2023.findings-emnlp.345, URL https://aclanthology.org/2023. findings-emnlp.345/ [7] Chen, T., Xu, B., Zhang, C., Guestrin, C.: Training deep nets with sublinear memory cost. arXiv preprint arXiv:1604.06174 (2016), URL https: //arxiv.org/abs/1604.06174 [8] Chow, C.K.: On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory 16(1), 41–46 (1970), https://doi.org/10.1109/TIT.1970. 1054406 [9] Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al.: Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 (2021) [10] Cortes, C., DeSalvo, G., Mohri, M.: Boosting with abstention. In: NeurIPS (2016), URL https://proceedings.neurips.cc/paper_files/paper/ 2016/file/7634ea65a4e6d9041cfd3f7de18e334a-Paper.pdf [11] Dancette, C., Whitehead, S., Maheshwary, R., Vedantam, R., Scherer, S., Chen, X., Cord, M., Rohrbach, M.: Improving selective visual question answering by learning from your peers. arXiv preprint arXiv:2306.08751 (2023), URL https://arxiv.org/abs/2306.08751

SIEVES: Selective Prediction via Visual Evidence Scoring

[12] Dao, T.: Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691 (2023), URL https: //arxiv.org/abs/2307.08691 [13] Eyes, B.M.: Be my ai. https://www.bemyeyes.com/be-my-ai (2026), accessed on 03/05/2026 [14] Fu, X., Hu, Y., Krishna, R., Ostendorf, M., Roth, D., Shi, W., Smith, N., Zettlemoyer, L.: Visual sketchpad: Sketching as a visual chain of thought for multimodal language models. In: NeurIPS (2024), https://doi.org/ 10.52202/079017-4423, URL https://doi.org/10.52202/079017-4423, neurIPS 2024 [15] Geifman, Y., El-Yaniv, R.: Selective classification for deep neural networks. In: NeurIPS (2017) [16] Geifman, Y., El-Yaniv, R.: Selectivenet: A deep neural network with an integrated reject option. In: Chaudhuri, K., Salakhutdinov, R. (eds.) Proceedings of the 36th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 97, pp. 2151–2159, PMLR (09–15 Jun 2019), URL https://proceedings.mlr.press/v97/geifman19a.html [17] Google: Lookout: Assisted vision. https : / / support . google . com / accessibility/android/answer/9031274 (2026), accessed on 03/05/2026 [18] Goyal, Y., Khot, T., Summers-Stay, D., Batra, D., Parikh, D.: Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In: CVPR (2017), https://doi.org/10.1109/CVPR.2017.670 [19] Groot, T., Valdenegro-Toro, M.: Overconfidence is key: Verbalized uncertainty evaluation in large language and vision-language models (2024), URL https://arxiv.org/abs/2405.02917 [20] Gu, S., Lugmayr, A., Danelljan, M., Fritsche, M., Lamour, J., Timofte, R.: Div8k: Diverse 8k resolution image dataset. 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW) pp. 3512–3516 (2019) [21] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al.: Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948 (2025) [22] Gurari, D., et al.: Vizwiz grand challenge: Answering visual questions from blind people. In: CVPR (2018), https://doi.org/10.1109/CVPR.2018. 00380 [23] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 10 (2021) [24] Jang, C., Choi, M., Kim, Y., Lee, H., Lee, J.: Verbalized confidence triggers self-verification : Emergent behavior without explicit reasoning supervision. In: ICML 2025 Workshop on Reliable and Responsible Foundation Models (2025), URL https://openreview.net/forum?id=Ub3eXwQ0uA [25] Kalai, A.T., Nachum, O., Vempala, S.S., Zhang, E.: Why language models hallucinate. arXiv preprint arXiv:2509.04664 (2025) [26] Khan, Z., Fu, Y.: Consistency and uncertainty: Identifying unreliable responses from black-box vision-language models for selective visual question answering. In: CVPR, pp. 10854–10863 (2024), https://doi.org/10. 1109/CVPR52733.2024.01032

H. G. Rodriguez and M. Rohrbach

[27] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015–4026 (2023) [28] Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., Cobbe, K.: Let’s verify step by step. In: The Twelfth International Conference on Learning Representations (2024) [29] Lucarini, D., Pelz-Sharpe, A.: Intelligent document processing market analysis 2025-2028: Idp at the crossroads. Tech. rep., Deep Analysis (2025), URL https://www.deep-analysis.net/intelligent-documentprocessing-market-analysis-2025-2028/ [30] Müller, P., Brummel, M., Braun, A.: Spatial recall index for machine learning algorithms. In: Finlayson, G.D., Triantaphillidou, S. (eds.) London Imaging Meeting 2021: Imaging for Deep Learning, LIM 2021, online, September 20-22, 2021, pp. 58–62, Society for Imaging Science and Technology (2021), https://doi.org/10.2352/ISSN.2694-118X.2021.LIM-58, URL https://doi.org/10.2352/issn.2694-118X.2021.LIM-58 [31] Mushtaq, E., Fabian, Z., Bakman, Y.F., Ramakrishna, A., Soltanolkotabi, M., Avestimehr, S.: Harmony: Hidden activation representations and model output-aware uncertainty estimation for vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) (2025), URL https:// openaccess.thecvf.com/content/CVPR2025W/TMM-OpenWorld/papers/ Mushtaq_HARMONY_Hidden_Activation_Representations_and_Model_ Output-Aware_Uncertainty_Estimation_for_CVPRW_2025_paper.pdf [32] OpenAI: Gpt-5 system card. https://cdn.openai.com/gpt-5-systemcard.pdf (2025) [33] OpenAI: Openai o3 and o4-mini system card. https://cdn.openai. com/pdf/2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-minisystem-card.pdf (2025) [34] OpenAI: Thinking with images. https://openai.com/index/thinkingwith-images/ (2025) [35] Sheng, S., Singh, A., Goswami, V., Magana, J.A.L., Galuba, W., Parikh, D., Kiela, D.: Human-adversarial visual question answering (2021), URL https://arxiv.org/abs/2106.02280 [36] Shrivastava, A., Goyal, Y., Batra, D., Parikh, D., Agrawal, A.: Real openended track leaderboard results, vqa challenge 2021. https://visualqa. org/roe.html (2021) [37] Singhi, N., Bansal, H., Hosseini, A., Grover, A., Chang, K.W., Rohrbach, M., Rohrbach, A.: When to solve, when to verify: Compute-optimal problem solving and generative verification for llm reasoning (2025), URL https: //arxiv.org/abs/2504.01005 [38] Srinivasan, T., Hessel, J., Gupta, T., Lin, B.Y., Choi, Y., Thomason, J., Chandu, K.: Selective “selective prediction”: Reducing unnecessary abstention in vision-language reasoning. In: Findings of the Association for Computational Linguistics: ACL 2024, pp. 12935–12948 (2024), https:

SIEVES: Selective Prediction via Visual Evidence Scoring

/ / doi . org / 10 . 18653 / v1 / 2024 . findings - acl . 767, URL https : / / aclanthology.org/2024.findings-acl.767 [39] Su, A., Wang, H., Ren, W., Lin, F., Chen, W.: Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning (May 2025) [40] Team, G., Kamath, A., Ferret, J., Pathak, S., Vieillard, N., Merhej, R., Perrin, S., Matejovicova, T., Ramé, A., Rivière, M., Rouillard, L., Mesnard, T., Cideron, G., Grill, J.b., Ramos, S., Yvinec, E., Casbon, M., Pot, E., Penchev, I., Liu, G., Visin, F., Kenealy, K., Beyer, L., Zhai, X., Tsitsulin, A., Busa-Fekete, R., Feng, A., Sachdeva, N., Coleman, B., Gao, Y., Mustafa, B., Barr, I., Parisotto, E., Tian, D., Eyal, M., Cherry, C., Peter, J.T., Sinopalnikov, D., Bhupatiraju, S., Agarwal, R., Kazemi, M., Malkin, D., Kumar, R., Vilar, D., et al.: Gemma 3 technical report (2025), URL https://arxiv.org/abs/2503.19786 [41] Varshney, N., Baral, C.: Post-abstention: Towards reliably re-attempting the abstained instances in QA. In: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 967–982, Association for Computational Linguistics, Toronto, Canada (Jul 2023), https://doi.org/10.18653/v1/2023.acl-long.55, URL https: //aclanthology.org/2023.acl-long.55/ [42] Venter, M.: 40 intelligent document processing statistics. https://www. pdfreaderpro.com/blog/document-processing-statistics (2025) [43] Wang, W., Ding, L., Zeng, M., Zhou, X., Shen, L., Luo, Y., Tao, D.: Divide, conquer and combine: A training-free framework for high-resolution image perception in multimodal large language models (2024), URL https:// arxiv.org/abs/2408.15556 [44] Wen, B., Yao, J., Feng, S., Xu, C., Tsvetkov, Y., Howe, B., Wang, L.L.: Know your limits: A survey of abstention in large language models. Transactions of the Association for Computational Linguistics 13, 529–556 (2025), https://doi.org/10.1162/tacl_a_00754, URL https://direct.mit. edu/tacl/article/doi/10.1162/tacl_a_00754 [45] Whitehead, S., Petryk, S., Shakib, V., Gonzalez, J., Darrell, T., Rohrbach, A., Rohrbach, M.: Reliable visual question answering: Abstain rather than answer incorrectly. In: Computer Vision – ECCV 2022 Workshops, pp. 148–166, Springer, Cham (2022), https://doi.org/10.1007/978-3-03120059-5_9 [46] Wu, P., Xie, S.: V*: Guided visual search as a core mechanism in multimodal llms. In: CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024) [47] Xin, J., Tang, R., Yu, Y., Lin, J.: The art of abstention: Selective prediction and error regularization for natural language processing. In: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL) (2021), https://doi.org/10.18653/v1/2021.acl-long.84, URL https://aclanthology.org/2021.acl-long.84

H. G. Rodriguez and M. Rohrbach

[48] Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al.: Qwen3 technical report. arXiv preprint arXiv:2505.09388 (2025) [49] Yang, D., Tsai, Y.H.H., Yamada, M.: On verbalized confidence scores for LLMs. In: ICLR Workshop: Quantify Uncertainty and Hallucination in Foundation Models: The Next Frontier in Reliable AI (2025), URL https://openreview.net/forum?id=CVRdNQvFPE [50] Yoshikawa, H., Okazaki, N.: Selective-LAMA: Selective prediction for confidence-aware evaluation of language models. In: Vlachos, A., Augenstein, I. (eds.) Findings of the Association for Computational Linguistics: EACL 2023, pp. 2017–2028, Association for Computational Linguistics, Dubrovnik, Croatia (May 2023), https://doi.org/10.18653/v1/2023. findings-eacl.150, URL https://aclanthology.org/2023.findingseacl.150/ [51] Zhang, Y.F., Lu, X., Yin, S., Fu, C., Chen, W., Hu, X., Wen, B., Jiang, K., Liu, C., Zhang, T., Fan, H., Chen, K., Chen, J., Ding, H., Tang, K., Zhang, Z., Wang, L., Yang, F., Gao, T., Zhou, G.: Thyme: Think beyond images (2025), URL https://arxiv.org/abs/2508.11630 [52] Zhang, Y.F., Zhang, H., Tian, H., Fu, C., Zhang, S., Wu, J., Li, F., Wang, K., Wen, Q., Zhang, Z., Wang, L., Jin, R., Tan, T.: Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? (2024), URL https://arxiv.org/abs/2408.13257 [53] Zheng, Y., Zhang, R., Zhang, J., Ye, Y., Luo, Z., Feng, Z., Ma, Y.: Llamafactory: Unified efficient fine-tuning of 100+ language models. In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Association for Computational Linguistics, Bangkok, Thailand (2024), URL http://arxiv.org/ abs/2403.13372 [54] Zheng, Z., Yang, M., Hong, J., Zhao, C., Xu, G., Yang, L., Shen, C., Yu, X.: DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning (May 2025) [55] Zhu, F., Lei, W., Feng, F., Wang, C., Zhang, H., Chua, T.S.: Towards complex document understanding by discrete reasoning. In: ACMMM (2022), https://doi.org/10.1145/3503161.3548422, URL https://doi.org/ 10.1145/3503161.3548422

SIEVES: Selective Prediction via Visual Evidence Scoring

Supplementary Material We briefly describe how the appendix is organized. Section A defines the selective-prediction metrics and explains how repeated answers are aggregated for evaluation. Section B exposes further experimental details. Section C extends the ablations from the main text, including the localization-threshold study. Section D provides coverage at all risk levels for all benchmarks, as well as including the zero-shot verbalized and logit baselines for the reasoners without localization. Additional qualitative examples are also displayed. Section E expands on the OOD benchmarks used in our evaluation. Section F presents the prompts used for pseudo-annotation and judging.

Selective prediction metrics

Coverage at risk (C@r). Selective prediction seeks to maximize coverage (fraction of answered, i.e., non-abstained, questions) at a user-chosen risk level (error rate on answered examples) [15, 16]. For sample i, let Ai be the predicted answer and yi the ground-truth answer, both represented in language space. Let cisel ∈ [0, 1] denote the final scalar confidence prediction for answer Ai , obtained from the individual confidence scores {ccorr , cloc , ccoh } as described in Sec. 3, and let τ ∈ [0, 1] be the abstention threshold. The coverage at risk r is C@r ≜ C@R(r) = max C(τ ) τ

s.t.

R(τ ) ≤ r,

where N

1 X i 𝟙[csel ≥ τ ], N i=1 PN 𝟙[Ai ̸= yi ] 𝟙[cisel ≥ τ ] R(τ ) = i=1PN . i i=1 𝟙[csel ≥ τ ]

C(τ ) =

(9) (10)

Area under the risk–coverage curve (AURC). Reporting coverage at a single risk level provides a snapshot of the performance of a confidence prediction method. Alternatively, one can aggregate the error rate (i.e. risk) across all coverage levels, sorting samples by confidence. The continuous version of AURC is given by: Z 1 Z τmin dC(τ ) dτ AURC = R(C) dC = R(τ ) dτ 0 τmax In practice, we compute the AURC by sorting all samples by confidence, c[1] ≥ c[2] ≥ · · · ≥ c[N ] , and then computing risk at each sample (which is equivalent to each coverage level), summing over all samples, starting from the highest confidence ones. We define the risk at the k-th ordered sample as: k

R[k] =

1X 𝟙[A[i] ̸= y[i] ] k i=1

H. G. Rodriguez and M. Rohrbach

from which the discrete AURC follows: N

AURC =

1 X R[k] . N k=1

While AURC is a convenient single-number summary of ranking quality over the entire coverage range, if a desired risk level is known, coverage at risk can be more relevant for deployment. Repeated answers and aggregation. As described in Sec. 4, we frequently sample five answers per question to mitigate the effects of answer variance. This is particularly important for smaller benchmarks, like V*Bench, which contains only 191 samples, and coverage at low risk, where a handful of samples can significantly skew the results. Since answering using these models is a stochastic process, we apply two steps to avoid sensitivity of the results to such sampling noise. Firstly, as mentioned, we sample multiple responses per question. Secondly, we pool all question-answer pairs from all repetitions, and we compute all metrics over that joint set. Tab. 4 illustrates empirically how averaging metrics per repetition can lead to high variance, and how pooling question-answer pairs across repetitions generally obtains similar results, more so in larger benchmarks and at higher achieved coverages. To formalize the difference between aggregating and averaging over repetitions, let q ∈ {1, . . . , M } index questions and t ∈ {1, . . . , T } index sampled q,t answers per question, with T = 5, confidence cq,t sel , prediction A , and groundtruth answer yq . Our default pooled evaluation applies the original coverage definition to the full set of M T question-answer pairs: M

Cpool (τ ) =

1 X X q,t 𝟙[c ≥ τ ]. M T q=1 t=1 sel

(11)

The pooled operating threshold is therefore ⋆ τpool (r) = arg max Cpool (τ ) τ

s.t.

Rpool (τ ) ≤ r,

⋆ and pooled C@r is Cpool (τpool (r)). For comparison, an averaged per-repetition alternative evaluates each repetition t separately: M

Ct (τt ) =

1 X q,t 𝟙[c ≥ τt ]. M q=1 sel

For each repetition t, we choose its own operating threshold τt⋆ (r) = arg max Ct (τ ) τ

s.t.

Rt (τ ) ≤ r,

and the averaged per-repetition summary reported in Tab. 4 is T

1X Ct (τt⋆ (r)), T t=1

(12)

SIEVES: Selective Prediction via Visual Evidence Scoring

Table 4: Pooled vs. averaged per-repetition C@5. To mitigate metric senstitivity in small benchmarks and for low risk metrics, we sample 5 answers per question. “Pooled” evaluates C@5 on the set of question-answer pairs aggregated over all repetitions. “Avg. per-rep.” computes C@5 separately for each of the five repetitions and then reports mean ± sample std. The two estimates are similar, but pooling is more robust because it uses the joint set when selecting the operating point. As expected, variance is larger for smaller benchmarks, e.g. V* Bench (191 samples), than for larger ones, e.g. MMERealWorld-Lite (1,782 samples). Benchmark

Reasoner

Avg. per-rep. Pooled

V* Bench

Pixel-Reasoner 16.3 ± 9.1 o3 41.4 ± 14.0 Gemini-3-Pro 85.9 ± 21.5

9.7 42.9 95.0

HR-Bench-8k

Pixel-Reasoner o3

7.8 ± 2.1 23.4 ± 2.6

7.1 21.6

MME-RealWorld-Lite

Pixel-Reasoner o3

5.3 ± 1.9 4.0 ± 1.7

4.0 1.7

with the table also showing the sample standard deviation across the T repetitions. Pooling is more stable at low risk, especially on small datasets where the accepted set can be determined by only a few examples. Still, Tab. 4 shows that pooled and averaged per-repetition estimates remain close in practice, while the averaged per-repetition estimate exhibits larger variance on smaller benchmarks such as V* Bench.

Experimental details

All selectors are trained for 74880 steps with batch size 8, using LLaMA-Factory [53]. To reduce the memory footprint, we use gradient checkpointing [7] and flash attention 2.0 [12]. We use cosine learning rate scheduling with a warmup period of 211 steps and a maximum learning rate of 1e-6. Thyme is a high-resolution natural image QA, where objects are often small and localized, comprising 8083 question-image pairs. Since no validation or test splits are provided for Thyme, we construct a holdout set by randomly choosing 750 of the questions. TAT-DQA is a document QA dataset based on financial reports, annotated by experts, which contains 16,558 question-answer pairs, out of which 13,251 are included in the training split. To increase the frequency of answers with localization, we modify the reasoner’s prompt to force it to zoom-in at least once, and only accept the final answer if a zoom-in crop is generated or the maximum number of turns reached, whichever comes first. This guarantees that for 94% of the questions in Thyme and 77% in TAT-DQA, a zoom-in crop is generated, with 51% and 31% respectively having a mIoGT ≥ 0.75.

H. G. Rodriguez and M. Rohrbach

Table 5: Spatial recall threshold ablation. We vary the mIoGT threshold t in gloc = 𝟙[mIoGT ≥ t], used to define the ground truth label indicating whether localization is correct or not for a reasoning trace. Reported are average coverage (C@r; ↑) and AURC (↓), on the ID tuning set (Thyme) and averaged across five OOD benchmarks, for Pixel-Reasoner with localization. The row highlighted in gray denotes the final SIEVES setting (t = 0.75) , as in the main text. ID

OOD

C@r ↑ AURC↓ C@r ↑ AURC↓

0.25 0.50 0.75

26.5 24.0 26.6

25.8 27.1 25.7

25.6 26.9 27.0

27.8 27.7 27.3

For the model selection we employ Thyme’s holdout set, cast into a multiplechoice (MC) format for more similarity with MC benchmarks. To construct this MC version, we pad the ground-truth answer with an incorrect response from Pixel-Reasoner (if a wrong one was generated), and GPT-5 [32] generated additional distractor options, until 4 answers per question are obtained. To generate multiple choices for the Thyme hold out set, we draw incorrect answers (hard negatives) from Pixel-Reasoner. If these are not enough to reach 4 choices (including the ground-truth), we generate distractor options with GPT5. The exact prompt templates used for distractor options generation, answering with localization, correctness judging, coherence labeling, and localization annotation are shown in Sec. F.

Ablating threshold for binarizing localization

Here, we also ablate the mIoGT threshold used to binarize the localization target, which then propagates to the coherence target: gloc = 𝟙[mIoGT ≥ t],

(13)

j gcoh = gloc · max sim(Rcrop , [Rlast , A]), j

(14)

where t ∈ {0.25, 0.50, 0.75}. For Pixel-Reasoner with localization, Tab. 5 compares these three choices on the Thyme tuning split and across the five OOD benchmarks. The selected value t = 0.75 is best in-distribution, achieving the highest average coverage and the lowest AURC. This transfers to OOD, where it also gives the highest average coverage and the lowest AURC.

Additional results

Qualitative examples Fig. 5 shows additional examples where SIEVES correctly accepts or abstains when the implicit selector fails.

SIEVES: Selective Prediction via Visual Evidence Scoring

What is the position of the blue backpack relative to the warning sign? (A) To the right (B) To the left (C) On top (D) Below

What color is the non-red motorcycle closest to the road?

GT Answer

(B) To the left

Green

Implicit confidence estimation Confidence estimation

Answering w/o localization

[…] The motorcycles positioned closest to the paved edge are the ones on the far left and one in the center. The other motorcycles (black, white, green, silver) are parked further back. […] Answer: blue

[…] the blue backpack is [...] on the left side […]. The small warning sign (black with a yellow triangle) is located near the base of the wall on the right side. Answer: B

SIEVES (ours) Answering w. localization

+ Let’s zoom in

The motorcycle closest to the road is the blue one on the far left. […] The red motorcycle is further back. Answer: blue

0.92/1 Accept!

Visual Evidence Scoring

SIEVES (ours) Answering w. localization

+ Let’s zoom in

0.02/1 Reject!

Visual Evidence Scoring

Correctness 0.52 /1

Correctness 0.67 /1

Localization 0.48 /1

Localization 0.35 /1

Coherence 0.39 /1 0.49/1 Reject!

we can identify two key objects […] The blue backpack is situated on the left-hand […] The warning sign is situated on the right. Answer: B

Coherence 0.38 /1 0.55/1 Accept!

Fig. 5: Additional qualitative examples where SIEVES correctly accepts or rejects Gemini-3-Pro answers, while an implicit selector fails. Left: On an AdVQA question, SIEVES correctly abstains because the answer is grounded on the wrong motorcycle. The implicit confidence selector wrongly assigns high confidence to the incorrect answer. Right: On an HR-Bench-8k question about relative position of two objects in a high-resolution image, SIEVES correctly accepts the answer after assessing the visual evidence. The implicit confidence selector incorrectly abstains.

Additional baselines and more coverage levels on OOD benchmarks When evaluating on OOD benchmarks, the SIEVES selector consistently shows an advantage over the correctness-only selector and other baselines. The overall trends are generally consistent across risk levels. We also compare to a verbalized zero-shot selector, which is asked to output a confidence score between 0 and 100 in natural language [19, 24, 49]. Tabs. 6 to 10 show the coverage at different risk levels for the OOD benchmarks, using the same visible baseline rows as the main paper while preserving the additional snapshot rows as comments in the source. As for the ablation results, low risk levels can be less informative, e.g. in VizWiz, if the task is challenging for the answerer. Similarly, coverage at high risk levels are saturated for V* Bench. In-distribution results We also report the coverage at different risk levels for the hold-out sets of the training dataset in Tab. 11. The SIEVES selector achieves the best coverage at lower risk, being a close second to correctness-only

H. G. Rodriguez and M. Rohrbach

Table 6: V* Bench baselines. indicates whether the reasoner f explicitly localizes the answer or not. C@r: average coverage across risk levels 1–30%. f

Abstention Method Acc↑

Pixel-Reasoner

Logprobs Zero-shot verb. ✗ Zero-shot logit Selector

0.2 0.0 79.6 0.0 1.9

0.2 0.0 0.0 2.3

52.7 42.0 42.6 54.6

14.7 17.3 16.9 15.7

Logprobs Zero-shot verb. ✓ Zero-shot logit Selector SIEVES (ours)

0.1 0.0 80.5 0.0 1.5 1.9

0.1 9.8 0.0 0.0 0.0 5.5 1.4 8.4 9.7 31.7

41.3 100.0 100.0 100.0 0.0 100.0 100.0 100.0 24.2 100.0 100.0 100.0 60.4 100.0 100.0 100.0 73.4 100.0 100.0 100.0

50.2 42.9 47.1 53.1 59.5

15.7 17.2 15.6 14.1 12.8

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 71.8 0.0 0.0 0.1 0.1

0.0 0.0 0.0 100.0 0.0 23.6 77.3 100.0 0.1 0.6 51.5 100.0

14.3 28.7 21.8

26.2 22.7 23.4

84.4 84.4 84.4 100.0 100.0 84.2 84.2 84.2 100.0 100.0 26.7 100.0 100.0 100.0 100.0 69.3 100.0 100.0 100.0 100.0

64.7 65.1 61.0 70.8

9.1 9.9 9.7 8.4

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 0.0 100.0 100.0 100.0 100.0 86.4 0.0 0.0 0.0 100.0 100.0 100.0 100.0 0.6 8.8 69.1 100.0 100.0 100.0 100.0

57.1 57.1 68.4

12.0 13.8 8.8

Zero-shot verb. Zero-shot logit ✓ Selector SIEVES (ours)

0.0 88.3 100.0 100.0 100.0 100.0 100.0 0.0 93.5 100.0 100.0 100.0 100.0 100.0 94.3 2.3 40.7 100.0 100.0 100.0 100.0 100.0 4.3 95.0 100.0 100.0 100.0 100.0 100.0

84.0 84.8 77.6 85.6

3.1 2.9 3.5 3.1

o3 ✓

Gemini-3-Pro

Coverage at Risk (C@r) ↑ C@r ↑ AURC↓ @1 @5 @10 @15 @20 @25 @30

Zero-shot verb. Zero-shot logit Selector SIEVES (ours)

85.9

0.0 0.0 1.7 1.7 0.0 0.0 3.4 23.1

9.7 61.5 97.5 100.0 100.0 0.0 0.0 93.8 100.0 100.0 0.0 0.0 98.0 100.0 100.0 8.4 73.0 96.3 100.0 100.0

0.0 0.0 0.1

selector for lower-stakes scenarios (i.e. higher risk). However, the differences in performance are not as pronounced as in some OOD cases, again highlighting the generalization capabilities of explicitly estimating localization quality.

Details on the OOD benchmarks

V* Bench. Wu and Xie [46] target fine-grained perception in high-resolution images, focusing on attribute recognition (“What is the material of the glove?”) and spatial relationships (“Is the red suitcase on the left or right side of the red stool?”). Images are 2246 × 1582 on average, sourced from SA-1B [27], and relevant objects represent a small percentage of image area. HR-Bench-8k. Wang et al. [43] extend the V* Bench by increasing both the size and resolution by 4. It contains 800 questions on “Fine-grained Single-instance Perception” (aligns with V* Bench’s attribute recognition), and “Fine-grained Cross-instance Perception” (aligns with V* Bench’s spatial relationship reasoning). We use the 8K resolution version. The domain of the images is broader. Gathered from DIV8K [20], questions pertain not only natural images but various domains like map and chart analysis. This benchmark is more challenging

SIEVES: Selective Prediction via Visual Evidence Scoring

Table 7: HR-Bench-8k baselines. indicates whether the reasoner f explicitly localizes the answer or not. C@r: average coverage across risk levels 1–30%. f

✗ Pixel-Reasoner

Gemini-3-Pro

Abstention Method Acc↑

Coverage at Risk (C@r) ↑ C@r ↑ AURC↓ @1 @5 @10 @15 @20 @25 @30

Logprobs Zero-shot verb. Zero-shot logit Selector

0.0 0.0 0.0 0.9

65.6

0.0 0.0 0.0 1.8

0.3 0.4 3.9 9.1 29.7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 47.1 75.8 2.9 20.2 44.2 64.9 80.7

0.5 0.7 7.0 0.0 0.0 0.0 0.0 2.4 12.1 3.5 19.4 44.5 7.1 19.0 35.0

38.0 0.0 47.3 58.3 55.5

63.3 0.0 73.9 78.4 73.0

6.2 0.0 17.6 30.8

30.3 30.6 25.9 20.3

92.8 93.4 93.9 94.5 95.6

28.9 13.3 32.8 42.7 41.1

22.6 28.9 21.3 17.4 18.1

Logprobs Zero-shot verb. ✓ Zero-shot logit Selector SIEVES (ours)

0.3 0.0 68.4 0.0 1.0 2.3

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 72.1 0.0 0.0 0.3 0.8

0.0 0.0 0.0 84.3 100.0 0.0 0.0 12.6 82.0 100.0 3.3 12.7 30.3 86.0 100.0

26.3 27.8 33.3

24.6 22.4 19.2

Zero-shot verb. Zero-shot logit ✓ Selector SIEVES (ours)

0.0 0.0 0.0 89.7 89.7 100.0 100.0 0.0 0.0 16.5 88.7 88.7 100.0 100.0 82.0 1.0 0.8 37.9 92.2 100.0 100.0 100.0 8.4 22.1 50.5 93.0 100.0 100.0 100.0

54.2 56.3 61.7 67.7

13.5 13.1 10.5 9.3

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 0.0 100.0 100.0 100.0 100.0 85.6 0.0 0.0 80.0 100.0 100.0 100.0 100.0 5.8 31.4 84.6 100.0 100.0 100.0 100.0

57.1 68.6 74.5

10.9 10.1 5.4

Zero-shot verb. Zero-shot logit ✓ Selector SIEVES (ours)

0.0 0.0 100.0 100.0 100.0 100.0 100.0 1.4 58.0 100.0 100.0 100.0 100.0 100.0 90.1 4.5 56.5 100.0 100.0 100.0 100.0 100.0 14.4 50.9 100.0 100.0 100.0 100.0 100.0

71.4 79.9 80.1 80.8

6.4 4.9 3.5 4.4

and less sensitive to noise (due to the size) than V* Bench, while still explicitly stresses tiny-object recognition and reading fine details that only appear when zooming. MME-Realworld-Lite. Zhang et al. [52] focus on real-world tasks that require perceiving small or far-away elements in high-resolution imagery. The Lite split contains 1,782 unique multiple-choice questions sampled from the full set for faster evaluation, while preserving the benchmark’s breadth. The benchmark organizes tasks into five top-level domains, Remote Sensing (RS), Autonomous Driving (AD), Monitoring (MO), Diagram and Table (DT), and Optical Character Recognition (OCR), and up to 43 sub-tasks beneath them. Images are 2,000 × 1,500 pixels on average, with relevant objects frequently being small. However, the difficulty of this benchmark (top model in the public leaderboard barely reaches 60% accuracy) can be attributed mostly to the nature of the questions, which are much more out of distribution for models compared to the other common VQA benchmarks. VizWiz. Gurari et al. [22] aim to evaluate models on a real-world setting they are often not trained for: answering questions pertaining images taken by blind users, relating to their daily life. These images are often blurry or even don’t con-

H. G. Rodriguez and M. Rohrbach

Table 8: MME-RealWorld-Lite baselines. indicates whether the reasoner f explicitly localizes the answer or not. C@r: average coverage across risk levels 1–30%. Abstention Method Acc↑

Coverage at Risk (C@r) ↑ C@r ↑ AURC↓ @1 @5 @10 @15 @20 @25 @30

Logprobs Zero-shot verb. Zero-shot logit Selector

0.1 0.0 0.0 0.0

1.1 4.5 7.0 16.7 28.4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 27.7 0.0 18.1 27.1 34.8 44.8

8.3 0.0 4.0 17.8

35.3 37.0 37.7 30.4

Logprobs Zero-shot verb. ✓ Zero-shot logit Selector SIEVES (ours)

0.0 0.0 51.3 0.0 0.0 0.0

0.0 0.0 0.0 0.0 4.0

0.2 0.2 0.2 1.4 3.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.0 28.7 4.0 13.5 25.9 33.8 41.0 9.9 17.9 25.9 32.4 41.1

0.7 0.0 4.7 16.9 18.7

43.5 43.5 36.4 31.9 31.0

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 0.0 0.0 0.0 0.0 0.0 56.9 0.0 0.0 0.0 0.0 4.5 16.6 32.6 0.0 1.1 2.2 5.3 8.1 31.1 45.9

0.0 7.7 13.4

36.8 34.1 29.3

Zero-shot verb. Zero-shot logit ✓ Selector SIEVES (ours)

0.0 0.0 58.7 1.2 0.7

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 10.8 23.1 9.6 14.6 16.7 26.4 47.9 7.7 14.4 21.7 30.9 45.2

0.0 4.8 16.6 17.5

36.1 35.3 28.6 28.4

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 0.0 0.0 0.0 0.0 51.2 60.9 0.0 0.0 0.0 12.7 25.8 43.5 57.9 2.4 5.1 24.2 37.4 47.2 56.8 70.1

7.3 20.0 34.7

29.5 26.9 21.9

Zero-shot verb. Zero-shot logit ✓ Selector SIEVES (ours)

0.0 0.0 0.0 0.0 0.0 0.0 65.0 0.0 0.0 0.0 0.0 28.6 44.9 63.0 63.2 0.7 1.2 18.2 26.8 37.3 48.5 64.4 3.4 12.2 17.2 31.6 39.6 53.5 68.9

9.3 19.5 28.2 32.3

28.2 26.4 22.8 21.5

✗ Pixel-Reasoner

Gemini-3-Pro

51.6

0.0 0.0 0.1 1.7

tain the object in question (33% of questions in the validation set are marked as non-answerable), and the textual questions are more conversational, as they are transcriptions of enunciated questions by the blind users. Unlike the prior three benchmarks we presented, this benchmark is open-ended. Each visual question has ten crowd answers, and items are labeled answerable vs non-answerable. The validation split contains 4,319 examples, of which 2,934 ( 67%) are answerable. We evaluate on validation rather than test, since test labels are not public and the official server does not support selective-prediction metrics. Notably, we do not use train or validation splits for model development. For evaluation to be consistent with that of our other benchmarks, we follow the stricter “hard” accuracy: we select the most frequent human answer per item and count a prediction correct only if it matches such ground truth answer, either by exact match or with the same LLM judge we use during training. For questions which are annotated as unanswerable, any response is considered incorrect, as the optimal strategy is to abstain. Although nominally high resolution (1224 × 1224 pixels on average), unlike in the other benchmarks, zooming in is generally not required

SIEVES: Selective Prediction via Visual Evidence Scoring

Table 9: VizWiz baselines. indicates whether the reasoner f explicitly localizes the answer or not. C@r: average coverage across risk levels 1–30%.

Abstention Method Acc↑

Coverage at Risk (C@r) ↑ C@r ↑ AURC↓ @1 @5 @10 @15 @20 @25 @30

Logprobs Zero-shot verb. Zero-shot logit Selector

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

0.0 0.0 0.0 0.1

0.0 0.0 0.0 0.7

0.0 0.0 0.0 1.6

0.1 0.4 0.0 0.0 0.0 0.0 5.4 16.5

0.1 0.0 0.0 3.5

48.8 46.5 48.1 45.6

Logprobs Zero-shot verb. ✓ Zero-shot logit Selector SIEVES (ours)

0.0 0.0 0.0 0.0 38.0 0.0 0.0 0.0 0.0 0.0 0.0

0.0 0.0 0.0 0.1 0.0

0.0 0.0 0.0 0.6 1.1

0.1 0.3 1.1 0.0 0.0 0.0 0.0 0.0 0.0 0.1 0.1 9.3 7.6 14.7 22.5

0.2 0.0 0.0 1.5 6.6

51.3 50.8 53.1 44.2 42.3

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 0.0 0.0 0.0 0.0 0.0 45.8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 6.3 5.4 14.6

0.0 0.0 3.8

43.1 47.1 40.9

Zero-shot verb. Zero-shot logit ✓ Selector SIEVES (ours)

0.0 0.0 0.0 0.0 46.9 0.1 0.0 0.2 0.0

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.5 0.0 1.3 19.1 7.2 11.7 27.2 40.5

0.0 0.0 3.7 12.4

41.6 39.0 35.2 33.3

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 0.0 0.0 0.0 0.0 0.0 44.1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.7 3.8 6.5 18.4

0.0 0.0 4.2

43.8 45.0 41.9

Zero-shot verb. Zero-shot logit ✓ Selector SIEVES (ours)

0.0 0.0 0.0 0.0 43.6 0.0 0.1 0.0 0.0

0.0 0.0 2.7 8.7

46.2 47.7 39.5 37.2

✗ Pixel-Reasoner

Gemini-3-Pro

38.1

0.0 0.0 0.2 0.0

0.0 0.0 1.0 0.0

0.0 0.0 4.2 2.0

0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.5 9.4 8.1 21.3 29.2

or expected to be beneficial. Nonetheless, the OOD nature of this benchmark, together with the high-stakes nature of the tasks, makes it a valuable testbed for our grounding-aware selective prediction framework. AdVQA. Sheng et al. [35] introduce an adversarially collected VQA benchmark where crowd workers interact with a strong model and craft valid questions that fool it. This “stress test” distribution is substantially harder than prior benchmarks (e.g. VQAv2 [18]), and highlights brittle reasoning and spurious priors. The validation split (v1.02 ) contains 10,000 questions. Following our setup for VizWiz described above, we use the validation split using a hard accuracy metric. AdVQA is the largest of the benchmarks we use by total question count and is particularly useful for selective prediction because it is intentionally designed to expose failure cases where abstention should be preferred.

https://adversarialvqa.org/download.html

A10

H. G. Rodriguez and M. Rohrbach

Table 10: AdVQA baselines. indicates whether the reasoner f explicitly localizes the answer or not. C@r: average coverage across risk levels 1–30%. f

Pixel-Reasoner

Gemini-3-Pro

Abstention Method Acc↑

Coverage at Risk (C@r) ↑ C@r ↑ AURC↓ @1 @5 @10 @15 @20 @25 @30

Logprobs Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 0.0 0.0 54.5 0.0 0.0 0.0 0.0

0.1 0.0 0.0 0.0

0.1 0.5 1.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 19.0 33.6

0.3 0.0 0.0 7.7

42.5 42.7 42.2 33.7

Logprobs Zero-shot verb. ✓ Zero-shot logit Selector SIEVES (ours)

0.0 0.0 0.0 0.0 58.1 0.0 0.0 0.0 0.0 0.0 1.3

0.0 0.0 0.0 0.2 2.8

0.0 0.0 0.0 0.9 3.5

0.1 0.1 0.1 0.0 0.0 0.0 0.0 0.0 0.0 3.0 11.2 28.8 7.5 15.7 34.1

0.0 0.0 0.0 6.3 9.3

40.3 40.7 38.6 33.0 32.2

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 0.0 0.0 68.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 2.5

0.0 0.0 0.0 0.0 0.0 33.6 7.6 40.8 85.2

0.0 4.8 19.4

31.8 30.2 25.2

Zero-shot verb. Zero-shot logit ✓ Selector SIEVES (ours)

0.0 0.0 0.0 0.0 72.4 0.0 0.1 0.1 0.1

0.0 0.0 0.0 92.4 100.0 0.0 0.0 38.0 75.9 100.0 3.8 15.8 30.1 83.1 100.0 8.2 18.5 43.6 84.9 100.0

27.5 30.6 33.3 36.5

24.9 21.9 20.1 19.6

Zero-shot verb. ✗ Zero-shot logit Selector

0.0 0.0 0.0 0.0 100.0 100.0 100.0 80.3 0.0 0.0 0.0 0.0 100.0 100.0 100.0 0.1 0.1 15.0 69.6 100.0 100.0 100.0

42.9 42.9 55.0

16.6 17.4 12.8

Zero-shot verb. Zero-shot logit ✓ Selector SIEVES (ours)

0.0 0.0 0.0 0.0 100.0 100.0 100.0 0.0 0.0 0.0 0.0 100.0 100.0 100.0 81.4 0.0 0.1 32.5 79.9 100.0 100.0 100.0 0.1 6.1 33.3 78.9 100.0 100.0 100.0

42.9 42.9 58.9 59.8

17.0 17.3 11.9 11.5

Table 11: Thyme-750 (MC) baselines. indicates whether the reasoner f explicitly localizes the answer or not. C@r: average coverage across risk levels 1–30%.

Pixel-Reasoner

Abstention Method Acc↑

Coverage at Risk (C@r) ↑ C@r ↑ AURC↓ @1 @5 @10 @15 @20 @25 @30

Logprobs Zero-shot verb. ✗ Zero-shot logit Selector

0.3 0.3 0.0 0.0 53.7 0.0 0.0 0.2 0.2

0.8 1.1 2.1 4.8 19.3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.4 10.0 17.1 35.0 53.4

4.1 0.0 0.0 16.8

36.3 36.8 39.2 30.1

Logprobs Zero-shot verb. ✓ Zero-shot logit Selector SIEVES (ours)

0.1 0.1 0.0 0.0 54.0 0.0 0.0 0.8 1.1 0.7 2.1

0.1 0.1 0.4 1.1 1.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 8.4 22.3 39.6 51.9 63.0 9.9 24.9 38.7 50.3 59.6

0.4 0.0 0.0 26.7 26.6

42.0 44.6 42.5 25.8 25.7

SIEVES: Selective Prediction via Visual Evidence Scoring

A11

Prompt templates

The exact prompts referenced in the experimental details are collected here for completeness. Thyme multiple-choice distractor generation prompt. We use this prompt to convert Thyme into a multiple-choice setting by generating hard distractors conditioned on the question, the relevant crop, and optionally a wrong model answer. In the main experiments, this chat template is queried with GPT-5. The full message sequence is defined in Fig. 6. Reasoner with localization prompt. To encourage explicit visual evidence, we augment the reasoner prompt with the instruction block in Fig. 7. This tells the model to use the crop tool before giving its first final answer, so the resulting trace contains localization evidence. Correctness judge prompt. For open-ended questions, when normalized string match fails, we run the prompt in Fig. 8 to decide whether the prediction should still count as correct. This makes correctness labels more robust to valid rewordings and semantically equivalent answers. Grounding-coherence labeling prompt. We use the prompt in Fig. 9 to create the crop-answer coherence labels used for selector training. The judge receives the question, the reasoner’s final answer, and one predicted crop, and decides whether the crop contains enough evidence and whether the answer is supported by that crop. Localization labeling prompt. For automatic localization annotation on Thyme, we prompt Gemini-3-Flash with the two-stage pipeline defined in Fig. 10. The first stage extracts the target object phrase from the question, and the second localizes it in the full image.

A12

H. G. Rodriguez and M. Rohrbach

Thyme multiple-choice distractor generation prompt System message: You are an expert at creating plausible but incorrect distractor options for visual question answering tasks. Given a question, image, ground truth answer, and optionally a wrong answer from a model, generate additional plausible distractor options that are wrong but could seem reasonable. User message (same-category example): Category: {category} Question: {question} Ground truth answer: {ground_truth} Here is the cropped region of interest: [cropped ground-truth box image] Generate 2-3 plausible but incorrect distractor options. Assistant message (example distractors): [example distractor 1] [example distractor 2] [example distractor 3] User message (current Thyme example): Category: {category} Question: {question} Ground truth answer: {ground_truth} Wrong answer from a model: {wrong_answer} [included only when available] Here is / Here are the cropped region(s) of interest from the image: [one or more cropped ground-truth box images] Generate exactly {num_distractors_needed} plausible but incorrect distractor options. Make them specific and realistic based on the image. Return only the distractors, one per line, without numbering.

Fig. 6: Thyme multiple-choice distractor generation prompt. We use this prompt to turn Thyme examples into multiple-choice questions by generating plausible wrong options that stay close to the image and question. The template is shown as a four-message chat: a system instruction, one fixed same-category example, its assistant response, and the current Thyme example to complete. The Thyme categories are ocr_recognition, quantity_recognition, object_recognition, attribute_recognition, position_recognition, and chart_understanding; the fewshot example is chosen from the same category as the current question. If an incorrect Pixel-Reasoner answer is available, it is included so the generated distractors can target realistic model mistakes.

SIEVES: Selective Prediction via Visual Evidence Scoring

A13

Prompt for reasoner with localization Guidelines: Understand the given visual information and the user query. Employ the given visual operations (tools) to observe better the visual elements necessary to answer the question. For an image, we can look closer by ‘crop_image_normalized‘. Please always crop into the relevant region before answering the question. You must use the cropping tool first and finish your turn. After the user returns the zoomed-in image, you can reason about what you observe and give your final answer. You must always use the cropping tool in the first turn, end your turn, wait for the cropped image. If you see the relevant object or objects needed to answer the question clearly, you can give the final answer. Otherwise, try to use the ‘crop_image_normalized‘ tool again, until you find the relevant object of objects. You have up to 10 turns. Reason with the visual information step by step, and put your final answer within \boxed{}. It is very important to follow these instructions.

Fig. 7: Prompt for reasoner with localization. We prompt the reasoner with the tool definition and the guidelines presented here, which encourage it to use the zoom-in tool to provide visual evidence for its answer.

A14

H. G. Rodriguez and M. Rohrbach Open-ended correctness-judge prompt Compare the predicted answer to the ground truth answer and determine if they convey the same meaning, and if the model was likely referring to the same object or situation when answering visual questions. You can think step-by-step about whether the predicted answer conveys the same meaning as the ground truth answer, but after that, output only ANSWER: and yes or no after that. Sometimes, the predicted answer will also contain the model’s thinking process and justificatioon. You can take it into account, but focus on the validity of the final answer, which will generally appear at the end. If the ground truth answer is only ’t’, this means the question is not answerable. In that case, you should always mark the question as wrong, because no answer can be correct, not even saying that there is no answer. For example: Question: What is the person holding? Predicted answer: Blue pullover Ground truth answer: Sweater A pullover is a type of sweater. The colour is not mentioned in the ground truth answer, so we can assume it is correct. ANSWER: yes ## Another example ## Question: Which object was put down by the person? Predicted answer: jacket Ground truth answer: The shoe. A jacket is a type of clothing, similar to a shoe, which is also a type of clothing/accessory. However, a jacket does not look similar to a shoe, so it is unlikely that the model confused them, and instead simply answered incorrectly. ANSWER: no - The other options (food, blanket, sandwich) are not related to clothing or accessories. ## Another example ## Question: What verification is this paper for? Predicted answer: freeboard verification Ground truth answer: Freeboard Clearly the model shows in its response it refers to the verification being a freeboard verification, which conveys the same meaning. ANSWER: yes ## Real user request ## Question: {question} Predicted answer: {pred_answer} Ground truth answer: {gt_answer}

Fig. 8: Open-ended judge prompt. For open-ended questions, we use this prompt when normalized exact match fails to decide whether the predicted answer should still count as correct. This makes correctness labels more robust to valid rewordings and semantically equivalent answers that exact match would reject. We prompt Qwen38B with a single user message at temperature 0. If multiple ground-truth answers are available, the judge is queried sequentially and stops at the first positive match.

SIEVES: Selective Prediction via Visual Evidence Scoring

A15

Grounding-coherence labeling prompt for selector training You are an expert evaluator assessing whether a model’s response to a visual question is grounded in the provided image crop. Given: - Question: {question} - Model’s Response (with final answer in \boxed{}): {last_message_with_boxed_answer} - Image: [Provided image crop] Please evaluate two aspects: 1. **Crop Sufficiency**: Is the provided image crop sufficient to support the model’s response? Does it contain all the necessary visual information referenced in the response? If the model explicitly states they use the global view to answer this question, you should consider this as not grounded in the prompt. Note you are not provided this final image, and only the crop, which the model should only use to give the final answer. 2. **Answer Coherence**: Is the model’s response coherent with what is actually visible in the image? Or is the model hallucinating information or obtaining it from elsewhere (not from the image)? Think step by step about both aspects, then provide your final assessment. Output your final decision as \boxed{Yes} if the answer is well-grounded in the image crop (both crop is sufficient AND answer is coherent), or \boxed{No} if there are issues with either aspect. Examples: - If the crop shows a clear view of a red car and the model answers "red car" -> \boxed{Yes} - If the crop shows a partial view that doesn’t contain enough information to answer -> \boxed{No} - If the crop shows a dog but the model answers "cat" -> \boxed{No} - If the crop shows a room but the model mentions specific details not visible in the crop -> \boxed{No} Your response:

Fig. 9: Crop-answer grounding coherence labeling prompt for selector training. We use this prompt to create the crop-answer coherence labels used to train the selector. The judge receives the question, the reasoner’s final answer, and the visual evidence (crop), and decides whether that crop contains enough evidence and whether the answer is supported by what is visible. The parser reads \boxed{Yes} or \boxed{No} as a binary coherence label.

A16

H. G. Rodriguez and M. Rohrbach

Subprompt 1: target object extraction System: You extract the target object from a visual question. Return only a short noun phrase describing the object to locate in the image. If the question uses relational or positional language, keep that context (e.g., ’object on the woman’s left ring finger’). User: {question}

Subprompt 2: full-image grounding System: You are a precise visual grounding assistant. Return only JSON. User: Please return the bounding box coordinates of "{target_object}". Use normalized 0-1000 coordinates in [top, left, bottom, right] order. Return a JSON list like: [{"box_2d": [top, left, bottom, right], "label": "..."}]. Image: [Provided full image]

Fig. 10: Localization annotation prompt for SIEVES training. We use this two-stage pipeline to automatically annotate Thyme with bounding boxes indicating the object referred to in the question. The two subprompts are shown separately. The first call extracts the target object phrase from the question, and the second call takes the full image plus that extracted phrase and returns one or more boxes.

Record · ID 141518 · SHA-256 af5933e2605fc13f
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.