RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models

RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models Canjie Liu1 , Jiawen Kang†1 , Jinbo Wen2 , Zishao Zhong3 1 School of Automation, Guangdong University of Technology, Guangzhou, China 2 Department of Computer Science, City University of Hong Kong, Hong Kong, China 3 The Second Affiliated Hospital of Guangzhou University of Chinese Medicine, Guangzhou, China

2112404359@mail2.gdut.edu.cn, kavinkang@gdut.edu.cn jinbowen2-c@my.cityu.edu.hk, zhongzishao@gzucm.edu.cn

arXiv:2609.02731v1 [cs.CV] 2 Sep 2026

Abstract Large vision-language models have achieved remarkable success in vision-language tasks. However, they remain prone to Visual Hallucinations (VHs), undermining their reliability in real-world applications. Existing solutions typically require curated datasets, additional training, or multi-round decoding, resulting in considerable computational overhead. In this paper, we propose RVSD (Retrieval Vision Sparse Decoding), a training-free and plugand-play decoding framework that, for the first time, unifies token sparsification and SemanticSpace Visual Retrieval (SSVR) within a single decoding pass. Within RVSD, we introduce a semantics-directed token selection strategy that selectively sparsifies redundant tokens while preserving critical visual information. We further propose the SSVR mechanism, which reformulates visual compensation as an on-demand cross-modal retrieval process within a shared semantic space. Extensive experiments demonstrate that RVSD achieves state-of-the-art performance in mitigating VHs while maintaining robust suppression capabilities under long-context generation settings. Our code is available here.1

1

Introduction

Large Vision-Language Models (LVLMs) have recently demonstrated impressive capabilities across a wide range of vision-language tasks (Liu et al., 2024b; Bai et al., 2023; Liu et al., 2024c), such as image captioning, visual question answering, and open-ended visual reasoning. Despite their rapid progress, LVLMs remain highly susceptible to Visual Hallucinations (VHs) (Bai et al., 2024; Liu et al., 2024a), where generated responses contradict or lack grounding in the actual visual content. This fundamental flaw seriously compromises the reliability of LVLMs in real-world applications (Wang † 1

Corresponding author. https://github.com/canjie-liu/RVSD

Vision token

Text token

Pruned token

External corpus

Annotated data

(a) Training-based Alignment/RAG: Low hallucination; External data or retrain Augment Fine-tune

Needs curated data, retraining or an external store

(b) Existing Decoding Intervention (VCD, M3ID): Training-free; Low hallucination; Low efficiency Clean tokens Perturbed tokens Two forward passes at every step

(c) Existing Sparse Decoding (FastV, SparseVLM): Training-free; High hallucination; High efficiency Static prune Pruned tokens

Discard the pruned tokens Pruned evidence can never be recovered

(d) Retrieval Vision Sparse Decoding (Ours): Training-free; Low hallucination; High efficiency Dynamic prune Memory bank Pruned tokens become a retrievable memory bank

Figure 1: Comparison of representative paradigms for mitigating VHs in LVLMs: (a) training-based alignment and RAG (Sun et al., 2024), (b) decoding interventions (Leng et al., 2024; Favero et al., 2024), (c) sparse decoders (Chen et al., 2024; Zhang et al., 2025), and (d) our RVSD, which turns pruned tokens into a retrievable memory bank and restores grounding on demand in a single forward pass.

et al., 2023b), where misinterpretation of visual data can lead to significant consequences. Existing solutions can be categorized into four paradigms, which we compare with RVSD in Fig. 1. Training-based alignment (Sun et al., 2024) mitigates VHs via additional supervision and preference alignment, but requires expensive retraining and annotated data. Retrieval-augmented generation (Lewis et al., 2020) grounds generation using external knowledge bases, yet introduces substantial storage and retrieval overhead. Decoding-side interventions (Liu et al., 2025) adjust token distributions during inference, but often require iterative decoding procedures that incur considerable latency. More recently, sparsification-based decoding (Dang et al., 2026) has emerged as a promising direction that simultaneously accelerates inference and suppresses distracting visual tokens, offering an attractive trade-off between efficiency and VH mitigation for LVLMs.

Hallucination Rate (%)

60

sual grounding information.

CHAIR Sentence CHAIR Instance Log-fit Trend

50

48.0 49.0

• We propose a Semantic-Space Visual Retrieval (SSVR) mechanism that performs ondemand cross-modal retrieval in a shared semantic space. When visual grounding becomes insufficient during generation, SSVR dynamically retrieves relevant visual evidence from a deferred visual memory bank to recalibrate hallucinated output distributions.

40 +145%

30 +172%

20 10 0 20

Saturation (Len ≥ 94)

19.6

7.2

12.8 13.3 5.9

3.0

30

40

50

60

70

80

90

100

Caption Length (Len)

Figure 2: Sparsification-hallucination paradox, exemplified by the representative sparse decoder VASparse (Zhuang et al., 2025) on LLaVA-1.5 using the Caption Hallucination Assessment with Image Relevance (CHAIR) dataset.

However, existing sparse decoding methods remain largely modality-agnostic, often pruning visual tokens that appear statistically redundant yet contain semantically critical information (Zhuang et al., 2025). Once such fine-grained visual tokens are discarded, the resulting information loss becomes irreversible. Our empirical analysis further reveals that the hallucination rates of State-of-theArt (SOTA) sparse decoders (Zhuang et al., 2025) increase substantially as generation length grows, as shown in Fig. 2. We refer to this phenomenon as the sparsification-hallucination paradox, highlighting the fundamental tension between inference efficiency and reliable visual grounding in LVLMs. In this paper, we propose RVSD (Retrieval Vision Sparse Decoding), a training-free and plugand-play decoding framework for mitigating VHs in LVLMs. RVSD addresses the sparsificationhallucination paradox by combining semanticsdirected token sparsification with on-demand visual compensation in a unified single-pass inference process. Our contributions are as follows: • We propose RVSD, a novel training-free and plug-and-play decoding framework that addresses the sparsification-hallucination paradox. RVSD jointly integrates token sparsification and dynamic visual compensation within a single inference pass, achieving an effective balance between inference efficiency and visual faithfulness. • We develop a semantics-directed token selection strategy that selectively distinguishes redundant tokens from visually critical ones. Unlike prior modality-agnostic pruning methods, the proposed strategy performs targeted sparsification while preserving essential vi-

• Extensive experiments across multiple prominent benchmarks demonstrate that RVSD achieves SOTA performance, simultaneously improving both generation quality and decoding efficiency while exhibiting strong robustness under long-context generation settings.

2

Related Work

Large Vision-Language Models. Building upon the rapid advances in large language models (Zhao et al., 2026), recent research has extended them to the multimodal domain, giving rise to LVLMs (Liu et al., 2023, 2024b,c). LVLMs are typically trained via large-scale vision-language pretraining followed by instruction-based fine-tuning, while subsequent studies further align generation behavior with human preferences via Reinforcement Learning from Human Feedback (RLHF) (Sun et al., 2024) and preference optimization techniques. Despite their strong multimodal capabilities, LVLMs remain prone to VHs (Bai et al., 2024; Yin et al., 2024; Wu et al., 2023), severely undermining the reliability of LVLMs in real-world applications. Visual Hallucination Mitigation. Existing efforts mitigate VHs from several perspectives. Trainingcentric approaches (Sun et al., 2024) improve multimodal alignment through additional supervision, RLHF, or preference optimization techniques. Decoding-time methods intervene during inference, where contrastive strategies, such as VCD (Leng et al., 2024) and M3ID (Favero et al., 2024), suppress linguistic priors via perturbed visual inputs, while attention-guided methods, including VTI (Liu et al., 2025), AvisC (Woo et al., 2025), and SHARP (Wu et al., 2025), enhance attention toward visually relevant regions. Adaptive visual reinforcement methods (Zhu et al., 2026) further strengthen visual evidence on demand during generation. Closest to our setting, RVCD (Lee and Song, 2025) performs retrieval-based contrastive

Large Language Models Stage 1 : Semantics-Directed Token Selection K

" Please describe this image in detail. "

Tokenizer

Prompt:

Max Socre Tokens

In addition to the fruits, there is a bottle placed near the bottom of the basket, possibly containing a beverage or a sauce to accompany the fruits. The combination of the fruits and the bottle creates a colorful and inviting scene.

Original Decoding

Q

Text Tokens

Visual Input

𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆𝑆

𝑄𝑄𝐾𝐾 𝑇𝑇

There is a bottle placed near the bottom of the basket, possibly containing a beverage or a sauce to accompany the fruits. The combination of the fruits and the bottle creates a colorful and inviting scene.

𝑑𝑑𝑘𝑘

Vision Encoder

Sparse Decoding Vision Tokens Hidden states Pruned Tokens

Original Tokens

Sparse Tokens

[B,L,D]

The image features a bowl filled with various fruits, including bananas and coconuts......The coconuts are scattered throughout the bowl, with some located near the top, middle, and bottom of the bowl. The arrangement of the fruits creates a visually appealing and colorful display.

[B,D]

[L,D]

Semantic Similarity

Retrieval Tokens

Merged Tokens

RVSD

Stage 2 : Semantic-Space Visual Retrieval

Figure 3: Overview of the proposed RVSD framework, which consists of two stages. Stage 1 performs semantics-directed token selection, partitioning visual tokens into an active sparse set and a deferred memory bank through cross-modal attention. Stage 2 applies the SSVR mechanism to retrieve on-demand visual evidence from the deferred bank and re-anchor decoding within a single forward pass, transforming hallucinated content from the original and sparse decoders into faithful generation.

decoding by referencing an offline database of generated single-concept images at the logit level. In contrast, RVSD requires no external image store, detector, or draft decoding pass, and retrieves the model’s own deferred visual tokens directly in the semantic space of a single forward pass. Despite their different formulations, these methods still rely on dense visual processing throughout decoding, leaving substantial room for improving efficiencyaware visual grounding. Sparse Decoding for Efficient Inference. Token sparsification has been utilized to improve both inference efficiency and visual faithfulness (Wu et al., 2026). General-purpose accelerators, such as SparseVLM (Zhang et al., 2025) and FastV (Chen et al., 2024), prune visual tokens using modality-agnostic criteria primarily designed for acceleration. Recent methods, such as VASparse (Zhuang et al., 2025), LTS-FS (Dang et al., 2026), PruneHal (Sun et al., 2025), MINT (Wang et al., 2025), and EAZY (Che et al., 2025), further explore hallucination-aware sparsification by enabling LVLMs to focus on task-relevant visual tokens. Nevertheless, existing methods still perform irreversible pruning, which may remove fine-grained visual evidence. Unlike prior methods, RVSD reorganizes discarded tokens into a deferred visual memory bank, enabling ondemand restoration of visual grounding within the

same decoding pass without additional training or iterative decoding.

3

Problem Formulation

Given an LVLM MMLLM parameterized by θ, the θ model consists of a text embedding layer, a vision encoder, a vision-language interface module, a decoder with L transformer layers, and an affine output layer ς(·) for next-token prediction. For an image-grounded generation task with textual query x and input image v, the vision encoder first extracts visual features from v, which are then transformed into aligned visual tokens zv via a Multi-Layer Perceptron (MLP) or Q-Former interface. The visual tokens zv are concatenated with the query x and fed into the decoder to autoregressively generate the response y, given by yt ∼ pθ (·|v, x, y<t ) ∝ softmax(fθ (·|v, x, y<t )),

(1)

where yt denotes the token generated at step t, y<t denotes the previously generated prefix, and fθ represents the unnormalized logits from MMLLM . θ When the generated response y contradicts or is unsupported by the input image v, the LVLM is considered to exhibit VHs. Accordingly, the objective of VH mitigation is to suppress unfaithful generations while preserving overall response quality. In the following, we present RVSD, which

improves visual faithfulness during inference with minimal computational overhead.

4

Methodology

As illustrated in Fig. 3, we present the architecture of the proposed RVSD. In the following, we describe its single-pass inference process and two key components: semantics-directed token selection and the SSVR mechanism. 4.1

Single Forward Pass

Let the visual token bank be denoted by Zv = [z1 , . . . , zNv ] ∈ RNv ×D , where Nv is the total number of visual tokens and D is the hidden dimension. The decoder hidden states at decoding step (l) (l) (l) t in layer l are denoted by Ht = [h1 , . . . , ht ], (l) in which each hi ∈ RD denotes the hidden state at textual position i. Thus, the text-to-vision atten(l) tion weight Ai,j from the i-th text token to the j-th visual token at layer l is defined as  √  (l) (l) (l) exp (WQ hi )⊤ (WK zj )/ D  √ , (l) (l) (l) ⊤ v r=1 exp (WQ hi ) (WK zr )/ D

(l) Ai,j = P N

(l)

(2)

(l)

where WQ , WK denote the query and key projection matrices at layer l, respectively. At each decoding step, RVSD maintains two complementary visual token sets: an active sparse set St for standard decoding and a deferred set Pt that forms a deferred visual memory bank, where St ∪Pt = 1, . . . , Nv and St ∩Pt = ∅. Decoding is performed over St , while Pt remains semantically retrievable and provides relevant visual evidence on demand. Based on Eq. (1), RVSD autoregressively predicts the token at generation step t as pRVSD = softmax (fθ (·|Zv [St ], x, y<t ; ∆t )) , θ

(3)

where ∆t denotes an optional SSVR branch activated only when visual grounding becomes insufficient. This design preserves decoding efficiency under normal generation while enabling querydriven visual retrieval during uncertain steps, without requiring multi-round decoding. Algorithm 1 presents the overall single forward pass of RVSD. The dynamic sparsity module for creating St and Pt and the SSVR module realizing ∆t are detailed in Sections 4.2 and 4.3, respectively. 4.2

Dynamic Sparsity

The objective of dynamic sparsity is to retain visual tokens most relevant to the salient textual semantics in the current generation trajectory. Rather

Algorithm 1 Single Forward Pass of RVSD Require: An LVLM Mθ , an image v, and a query x 0 1: Encode visual tokens: Zv ← Menc θ (v); initialize s̃ ← 0 2: for t = 1, 2, . . . do 3: Select token subset Tt via β t using Eq. (4) 4: Update smoothed saliency s̃t from st using Eq. (6) 5: Partition visual tokens into St and Pt using Eq. (7) 6: trigger ← T RUE 7: for l = ls to le do (l) 8: Compute uncertainty score ut using Eq. (8) (l) 9: if trigger and ut > γ then 10: Retrieve deferred tokens from Zv [Pt ] via (l) query qt using Eq. (9) Aggregate retrieved features Z̃v from top-kc 11: candidates using Eq. (10) adpt 12: Construct adaptive adapter weights W1,2 using Eq. (11) 13: Fuse retrieved visual evidence using Eq. (13) 14: trigger ← FALSE 15: end if 16: end for 17: Predict token yt over Zv [St ] using Eq. (3) 18: end for 19: return Response sequence y

than performing modality-agnostic pruning, RVSD explicitly scores visual tokens by aggregating crossmodal attention from salient textual positions. At decoding step t, we first compute textual saliency over generated tokens {1, . . . , t − 1} as (L)

exp(∥hi ∥2 /τ ) βit = Pt−1 , (L) r=1 exp(∥hr ∥2 /τ )

(4)

(L)

where ∥ · ∥2 denotes the ℓ2 norm, hi is the lastlayer hidden state of the i-th token, and τ is a temperature parameter. We then select the major textual token set Tt = TopK(β t , m), where m is the number of major textual tokens retained. Given a layer subset Ls for stable attention aggregation, we define the visual relevance score stj as stj =

X i∈Tt

βit

 1 X  (l) Ai,j . |Ls |

(5)

l∈Ls

To suppress step-wise noise, we apply temporal smoothing, which is expressed as s̃t = ηs̃t−1 + (1 − η)st ,

(6)

where η ∈ [0, 1) is the smoothing coefficient, and s̃t represents the smoothed relevance vector at step t. We then construct the active sparse set as St = TopK(s̃t , k), Pt = {1, . . . , Nv } \ St ,

(7)

where k is the budget that controls the size of the retained sparse set. Therefore, RVSD retains the visual tokens most relevant to the current decoding

context while deferring the remaining tokens for potential later retrieval. With k ≪ Nv , the perlayer attention complexity is reduced from O((t + Nv )2 ) to O((t + k)2 ). 4.3

Dynamic sparsity alone may still discard finegrained visual evidence required at later decoding stages. To address this issue, RVSD reformulates visual compensation as on-demand cross-modal retrieval in a shared semantic space. Instead of directly restoring pruned features, RVSD treats the deferred set Pt as a deferred visual memory bank and uses the current decoder state as a semantic query to retrieve relevant visual evidence whenever grounding becomes insufficient. 4.3.1 Uncertainty-triggered retrieval gate At decoding step t, we scan intermediate layers and compute the normalized predictive uncertainty as (l)

N 1 X (l) (l) p log pθ,n , log N n=1 θ,n

(8)

(l)

where N is the vocabulary size, and pθ,n represents the predicted probability of the n-th vocabulary token derived from the layer-l hidden state. When (l) ut > γ for l ∈ [ls , le ], where γ is a predefined entropy threshold and [ls , le ] denotes the intermediatelayer scanning window, the model triggers a single retrieval call at this step, indicating that the current generation trajectory has insufficient visual grounding and requires re-anchoring. 4.3.2 Cross-modal retrieval Once triggered, we project the current decoder hidden state into the shared visual-text semantic space (l) (l) and use it as a query qt = ht [−1, :], i.e., the hidden state at the last token position of layer l, to query the deferred visual memory bank: (l)

q zj⊤ rj = t√ , j ∈ Pt , w = softmax(r), D

Z̃v = diag(w̃)Zv [It ],

(10)

where diag(w̃) denotes a diagonal matrix whose main diagonal is formed by w̃.

Semantic-Space Visual Retrieval

ut = −

query-conditioned visual representation, which is expressed as

(9)

where rj measures the semantic relevance between the textual query and each deferred visual token, and w defines a normalized retrieval distribution over Pt . With kc = min(K, |Pt |), where K is a predefined upper bound, we select the top-kc tokens (w̃, It ) = TopK(w, kc ), where w̃ denotes the corresponding weights, and It ⊆ Pt denotes their indices in the deferred memory bank. The weights are renormalized and aggregated into a

4.3.3 Lightweight injection To inject the retrieved evidence into the decoding stream without disturbing the model’s intrinsic distribution, we introduce a transient adapter whose weights are scale-matched to the host FeedForward Network (FFN), expressed as W1adpt =

sup sdown ⊤ Z̃v , W2adpt = Z̃v . s̃v s̃v

(11)

Here, W1adpt and W2adpt denote the up- and downprojection weights of the transient adapter, respectively. Wup and Wdown denote the corresponding up- and down-projection matrices of the host FFN, respectively. sup = mean(|Wup |), sdown = mean(|Wdown |), and s̃v = mean(|Z̃v |) denote the mean absolute magnitudes used for amplitude alignment. The adapter performs a low-rank projection that re-grounds the textual representation using the retrieved visual evidence, expressed as G(x) = x(W1adpt )⊤ (W2adpt )⊤ =

sup sdown xZ̃v⊤ Z̃v , (12) s̃2v

where x denotes the input residual-stream hidden state fed into the adapter. After amplitude normalization, the retrieved visual signal â is fused with (l+1) the block-output residual stream ht , i.e., the hidden state after FFN residual addition in layer l that feeds into the attention sublayer of layer l+1: (l+1)

o = (1 − α) · ht

+ α · â,

(13)

where o denotes the fused hidden state passed to layer l+1, and α ∈ [0, 1] denotes the injection ratio that balances the original residual stream and the retrieved visual evidence. After retrieval, the trigger is reset, and the transient adapter is released. This enables parameter-free and on-demand visual retrieval in the semantic space, i.e., visual evidence is retrieved only when grounding fails, and the overhead is removed after each step, ensuring efficiency while mitigating VHs.

5

Experiments

5.1

Experimental Settings

Benchmarks. We evaluate RVSD on five complementary benchmarks. For discriminative hallucination, we use POPE (Li et al., 2023) with random,

Models

Methods

Random

Popular

Adversarial

Average

Acc (%)↑

F1 (%)↑

Acc (%)↑

F1 (%)↑

Acc (%)↑

F1 (%)↑

Acc (%)↑

F1 (%)↑

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) +VTI (Liu et al., 2025) +AvisC (Woo et al., 2025) +RVSD (Ours)

83.8 ↑0.0 85.0 ↑1.2 86.1 ↑2.3 83.0 ↓0.8 82.3 ↓1.5 87.3 ↑3.5

84.2 ↑0.0 84.2 ↑0.0 85.0 ↑0.8 83.6 ↓0.6 83.5 ↓0.7 86.3 ↑2.1

82.0 ↑0.0 82.1 ↑0.1 82.8 ↑0.8 80.4 ↓1.6 78.2 ↓3.8 86.6 ↑4.6

82.6 ↑0.0 83.2 ↑0.6 84.1 ↑1.5 81.8 ↓0.8 80.5 ↓2.1 85.3 ↑2.7

75.8 ↑0.0 76.3 ↑0.5 77.1 ↑1.3 76.0 ↑0.2 74.2 ↓1.6 84.7 ↑8.9

78.1 ↑0.0 78.7 ↑0.6 78.9 ↑0.8 78.8 ↑0.7 77.7 ↓0.4 83.5 ↑5.4

80.5 ↑0.0 81.1 ↑0.6 82.0 ↑1.5 79.8 ↓0.7 78.2 ↓2.3 86.2 ↑5.7

81.6 ↑0.0 82.0 ↑0.4 82.7 ↑1.1 81.4 ↓0.2 80.6 ↓1.0 85.0 ↑3.4

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) LLaVA-NEXT +VTI (Liu et al., 2025) +AvisC (Woo et al., 2025) +RVSD (Ours)

84.4 ↑0.0 86.0 ↑1.6 85.5 ↑1.1 84.8 ↑0.4 85.2 ↑0.8 88.5 ↑4.1

82.3 ↑0.0 84.3 ↑2.0 83.6 ↑1.3 83.1 ↑0.8 82.8 ↑0.5 87.6 ↑5.3

83.2 ↑0.0 84.5 ↑1.3 84.2 ↑1.0 81.8 ↓1.4 83.9 ↑0.7 87.9 ↑4.7

81.5 ↑0.0 82.9 ↑1.4 82.4 ↑0.9 81.6 ↑0.1 81.6 ↑0.1 86.7 ↑5.2

79.5 ↑0.0 80.9 ↑1.4 80.6 ↑1.1 79.0 ↓0.5 81.8 ↑2.3 86.5 ↑7.0

78.1 ↑0.0 79.7 ↑1.6 79.2 ↑1.1 78.1 ↑0.0 80.2 ↑2.1 85.4 ↑7.3

82.4 ↑0.0 83.8 ↑1.4 83.4 ↑1.0 81.9 ↓0.5 83.6 ↑1.2 87.6 ↑5.2

80.6 ↑0.0 82.3 ↑1.7 81.7 ↑1.1 80.9 ↑0.3 81.5 ↑0.9 86.6 ↑6.0

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) +VTI (Liu et al., 2025) +AvisC (Woo et al., 2025) +RVSD (Ours)

84.9 ↑0.0 85.5 ↑0.6 85.3 ↑0.4 85.3 ↑0.4 82.9 ↓2.0 86.2 ↑1.3

82.9 ↑0.0 83.6 ↑0.7 83.4 ↑0.5 83.5 ↑0.6 80.0 ↓2.9 84.7 ↑1.8

84.0 ↑0.0 84.9 ↑0.9 84.2 ↑0.2 83.0 ↓1.0 82.8 ↓1.2 85.3 ↑1.3

81.9 ↑0.0 83.6 ↑1.7 82.7 ↑0.8 82.3 ↑0.4 80.1 ↓1.8 83.3 ↑1.4

82.1 ↑0.0 84.0 ↑1.9 83.2 ↑1.1 83.2 ↑1.1 81.2 ↓0.9 84.5 ↑2.4

80.2 ↑0.0 82.0 ↑1.8 80.8 ↑0.6 81.8 ↑1.6 78.5 ↓1.7 82.6 ↑2.4

83.7 ↑0.0 84.8 ↑1.1 84.2 ↑0.5 83.8 ↑0.1 82.3 ↓1.4 85.3 ↑1.6

81.7 ↑0.0 83.1 ↑1.4 82.3 ↑0.6 82.5 ↑0.8 79.5 ↓2.2 83.5 ↑1.8

LLaVA-1.5

Qwen-VL

Table 1: Performance comparison of RVSD with the baselines on the POPE-MSCOCO discrimination benchmark across the random, popular, and adversarial splits. The average column reports the mean accuracy and F1 scores over all splits. Green (•) and red (•) indicate improvements and degradations relative to the base model, respectively, while (↑/↓) denotes the absolute change. Best results are highlighted in bold.

popular, and adversarial splits, the MME hallucination subset (Fu et al., 2026) covering object-level and attribute-level probes, and the discrimination subset of AMBER (Wang et al., 2023a), reporting accuracy and F1 scores. For generative hallucination, we evaluate on CHAIR (Rohrbach et al., 2018) over 500 MS-COCO images and the AMBER generative subset, and also report AMBER discrimination accuracy and F1 scores for a unified view of caption-level performance. To assess whether VH mitigation affects general capability, we further evaluate on MM-Vet (Yu et al., 2024), which covers six vision-language abilities and uses GPT-4 as the evaluator. Models. We evaluate RVSD on the 7B variants of LLaVA-1.5 (Liu et al., 2024b), LLaVANEXT (Liu et al., 2024c), and Qwen-VL (Bai et al., 2023). LLaVA-1.5 adopts a two-layer MLP for vision-language alignment, and LLaVANEXT further introduces the AnyRes mechanism to support higher-resolution visual understanding, while Qwen-VL instead couples a cross-attentionbased vision-language adapter with a non-LLaMA language backbone. Together, they cover both standard- and high-resolution settings as well as heterogeneous vision-language interfaces, enabling a comprehensive evaluation of RVSD. Moreover,

RVSD is model-agnostic and can be seamlessly integrated into different LVLM architectures. Baselines. We compare RVSD with representative training-free baselines from three categories: (i) contrastive decoders, including VCD (Leng et al., 2024) and M3ID (Favero et al., 2024); (ii) the attention-intervention method AvisC (Woo et al., 2025); and (iii) the latent-space steering method VTI (Liu et al., 2025). All baselines leverage the hyperparameters recommended in their original papers under the same decoding budget, while the vanilla LVLM with sampling decoding serves as the base model. 5.2

Experimental Results

Results on POPE. In Table 1, we see that RVSD achieves the best performance across all POPE splits on the three backbones, obtaining the highest accuracy on every split and the highest F1 scores in almost all cases. On LLaVA-1.5-7B, RVSD improves the average Accuracy/F1 from 80.5/81.6 to 86.2/85.0. On LLaVA-NEXT, the scores increase from 82.4/80.6 to 87.6/86.6, and on Qwen-VL, from 83.7/81.7 to 85.3/83.5. The largest gains are observed on the adversarial split, with improvements of +8.9/+5.4, +7.0/+7.3, and +2.4/+2.4 on the three backbones, respectively. In contrast, VCD

MM-Vet Score

50

LLaVA-1.5 · Vanilla LLaVA-NEXT · Vanilla Qwen-VL · Vanilla

LLaVA-1.5 · RVSD LLaVA-NEXT · RVSD Qwen-VL · RVSD

Rec

Gen

40 30 20 10 0

OCR

Know

Spat

Total

MM-Vet Dimension

Figure 4: Per-dimension MM-Vet scores of Vanilla decoding and RVSD on LLaVA-1.5, LLaVA-NEXT, and Qwen-VL, covering the Rec, OCR, Know, Gen, and Spat capability dimensions as well as the overall total score. RVSD improves every reported dimension on all three backbones.

and M3ID yield only limited improvements, while VTI and AvisC occasionally degrade performance. These results demonstrate the effectiveness of combining semantics-directed sparsification with ondemand retrieval for reliable visual grounding. Results on MM-Vet. To verify that VH mitigation does not compromise general multimodal capability, we evaluate RVSD on MM-Vet and report the Rec, OCR, Know, Gen, and Spat dimensions together with the overall total score in Fig. 4. We observe that RVSD improves the total score over the vanilla model on all three backbones, from 30.0 to 33.1 on LLaVA-1.5, from 42.1 to 42.7 on LLaVA-NEXT, and from 36.7 to 38.4 on Qwen-VL, with gains observed across every reported dimension. The improvements are most pronounced on Know, suggesting that restoring pruned visual evidence also benefits knowledge-intensive reasoning. The results demonstrate that RVSD mitigates VHs while preserving general multimodal competence. Results on MME. As shown in Fig. 5, RVSD obtains the highest total score on the MME hallucination subset across all three backbones, reaching 651.7 on LLaVA-1.5, 668.3 on LLaVA-NEXT, and 593.4 on Qwen-VL, and consistently surpassing the vanilla decoder on the total axis of every panel. On the two LLaVA backbones, the improvements are concentrated on the attribute-level probes, where RVSD achieves the highest scores on both Color and Position, whereas the scores of most baselines decline markedly on Position for LLaVA-NEXT. On Qwen-VL, the gain is instead driven by object Existence, while Count and Position remain below the strongest baselines. These results further demonstrate the effectiveness of RVSD in mitigating fine-grained VHs.

Results on CHAIR and AMBER. As shown in Table 2, we further evaluate caption-level behavior on CHAIR and AMBER, jointly assessing discrimination (Accuracy/F1) and generative hallucination (CHAIRS , CHAIRI , AMBER CHAIR, HalRate, Score). We observe that RVSD achieves the best discrimination performance and the highest AMBER score across all three backbones, and obtains the lowest CHAIRI on the two LLaVA backbones and the lowest CHAIRS on Qwen-VL, indicating effective suppression of object-level hallucinations during long-form generation. In contrast, existing baselines provide only limited or inconsistent improvements. Overall, RVSD consistently improves both faithfulness and caption quality across different LVLM architectures, demonstrating the effectiveness of combining semantics-directed sparsification with on-demand visual retrieval. 5.3

Ablation Studies

Mechanism Contribution. To evaluate the contribution of each component, we conduct an ablation study on the full MME benchmark using LLaVA1.5 by separately removing the semantics-directed sparse selection (w/o Sparse) and the SSVR module (w/o SSVR). As shown in Table 3, removing either component degrades the overall total score, demonstrating that both modules are essential to RVSD. In particular, removing SSVR causes the largest performance drop, reducing Cognition from 371.79 to 348.21 and yielding the greatest decline in the total score, highlighting the importance of on-demand visual retrieval for restoring visual grounding during reasoning-intensive decoding. In contrast, removing sparse selection slightly improves the perception score by retaining all visual tokens, but noticeably reduces the cognition score by 14.29, indicating that dense decoding introduces redundant visual information that weakens effective grounding. Overall, RVSD achieves the best total and cognition scores, demonstrating that semantics-directed sparsification and SSVR complement each other to improve visual grounding. 5.4

Inference Latency

As shown in Table 4, we further benchmark inference efficiency under the same decoding budget. We observe that RVSD is the only method that maintains the same latency as the Vanilla baseline (1.00×) while simultaneously reducing memory usage by 2.2%, Tera Floating-Point Operations Per Second (TFLOPs) by 15.6%, and Time-to-First-

Total

Total

Total

651.7

668.3

593.4

Existence

Count

190.0

Existence

Count

195.0

153.3

Existence

Count

180.0

150.0

136.7 106.7

170.0

175.0

133.3

180.0

143.3

Color

Position

Color

Position

(a) LLaVA-1.5

Color

(b) LLaVA-NEXT

Vanilla

VCD

M3ID

AvisC

Position

(c) Qwen-VL VTI

RVSD (Ours)

Figure 5: Performance comparison of RVSD with the baselines on the MME hallucination subset for (a) LLaVA-1.5, (b) LLaVA-NEXT, and (c) Qwen-VL, covering object-level (Existence, Count) and attribute-level (Position, Color) probes, with Total denoting their sum. Values annotated in red are those of RVSD, and each axis is scaled independently.

Models

CHAIR

Methods

AMBER

CHAIRS ↓

CHAIRI ↓

Recall↑

Length

Acc (%)↑

F1 (%)↑

CHAIR↓

HalRate↓

Score↑

LLaVA-1.5

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) +AvisC (Woo et al., 2025) +VTI (Liu et al., 2025) +RVSD (Ours)

52.8 ↓0.0 51.0 ↓1.8 56.2 ↑3.4 44.0 ↓8.8 36.8 ↓16. 38.4 ↓14.

15.9 ↓0.0 14.9 ↓1.0 17.0 ↑1.1 13.7 ↓2.2 13.6 ↓2.3 10.3 ↓5.6

77.3 ↑0.0 77.2 ↓0.1 79.3 ↑2.0 72.9 ↓4.4 66.6 ↓10. 74.4 ↓2.9

93.4 101.9 97.1 89.8 66.7 90.6

67.0 ↑0.0 67.3 ↑0.3 67.3 ↑0.3 70.7 ↑3.7 66.5 ↓0.5 73.6 ↑6.6

71.0 ↑0.0 71.1 ↑0.1 70.9 ↓0.1 74.2 ↑3.2 70.6 ↓0.4 76.9 ↑5.9

12.0 ↓0.0 10.0 ↓2.0 9.1 ↓2.9 11.8 ↓0.2 6.9 ↓5.1 7.3 ↓4.7

51.0 ↓0.0 43.6 ↓7.4 42.8 ↓8.2 51.2 ↑0.2 27.0 ↓24. 31.2 ↓19.

79.5 ↑0.0 80.6 ↑1.1 80.9 ↑1.4 81.2 ↑1.7 81.9 ↑2.4 84.8 ↑5.3

LLaVA-NEXT

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) +AvisC (Woo et al., 2025) +VTI (Liu et al., 2025) +RVSD (Ours)

35.8 ↓0.0 40.2 ↑4.4 35.2 ↓0.6 40.4 ↑4.6 26.4 ↓9.4 26.2 ↓9.6

12.0 ↓0.0 10.7 ↓1.3 10.3 ↓1.7 12.4 ↑0.4 7.7 ↓4.3 6.7 ↓5.3

59.5 ↑0.0 62.1 ↑2.6 61.4 ↑1.9 60.0 ↑0.5 58.5 ↓1.0 63.7 ↑4.2

179.0 171.2 152.7 183.8 187.6 185.0

72.9 ↑0.0 74.3 ↑1.4 75.0 ↑2.1 77.7 ↑4.8 75.2 ↑2.3 80.0 ↑7.1

78.6 ↑0.0 79.6 ↑1.0 80.0 ↑1.4 82.1 ↑3.5 81.9 ↑3.3 83.8 ↑5.2

12.0 ↓0.0 11.8 ↓0.2 10.2 ↓1.8 14.2 ↑2.2 10.3 ↓1.7 8.9 ↓3.1

59.6 ↓0.0 58.6 ↓1.0 52.8 ↓6.8 65.2 ↑5.6 58.3 ↓1.3 47.9 ↓11.

83.3 ↑0.0 83.9 ↑0.6 84.9 ↑1.6 84.0 ↑0.7 87.1 ↑3.8 87.5 ↑4.2

Qwen-VL

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) +AvisC (Woo et al., 2025) +VTI (Liu et al., 2025) +RVSD (Ours)

2.8 ↓0.0 1.4 ↓1.4 1.7 ↓1.1 1.6 ↓1.2 1.9 ↓0.9 1.2 ↓1.6

3.0 ↓0.0 1.2 ↓1.8 1.3 ↓1.7 1.6 ↓1.4 1.9 ↓1.1 1.6 ↓1.4

31.0 ↑0.0 30.8 ↓0.2 31.8 ↑0.8 32.0 ↑1.0 30.2 ↓0.8 31.9 ↑0.9

5.3 4.0 3.4 4.4 4.5 5.9

82.9 ↑0.0 84.1 ↑1.2 84.0 ↑1.1 84.1 ↑1.2 83.5 ↑0.6 84.3 ↑1.4

86.9 ↑0.0 87.9 ↑1.0 87.1 ↑0.2 87.9 ↑1.0 87.3 ↑0.4 88.6 ↑1.7

4.8 ↓0.0 3.5 ↓1.3 3.7 ↓1.1 4.3 ↓0.5 3.1 ↓1.7 3.4 ↓1.4

9.2 ↓0.0 7.7 ↓1.5 8.2 ↓1.0 8.6 ↓0.6 6.8 ↓2.4 7.3 ↓1.9

91.1 ↑0.0 92.2 ↑1.1 91.7 ↑0.6 91.8 ↑0.7 92.4 ↑1.3 92.6 ↑1.5

Table 2: Performance comparison on the CHAIR benchmark (CHAIRS , CHAIRI , Recall, Length) and the AMBER benchmark for open-ended captioning, where the AMBER block jointly reports discrimination metrics (Accuracy, F1) and generative metrics (CHAIR, HalRate, Score). Best results are highlighted in bold.

Methods

Perception↑ Cognition↑

Total↑

RVSD

1511.40

371.79

1883.19

w/o Sparse

1514.60

357.50

1872.10

w/o SSVR

1520.36

348.21

1868.57

Table 3: Component-level ablation study on the full MME benchmark using LLaVA-1.5, evaluating the contributions of the semantics-directed sparse selection module and the SSVR module. Best results are highlighted in bold.

Token (TTFT) by 20.2%. In contrast, contrastive decoding methods (VCD and M3ID) introduce over 2× latency overhead due to auxiliary forward passes, while AvisC incurs the highest cost from

dual forward inference and additional attention extraction. Although VTI is relatively lightweight, it still increases latency by 1.09×. These results demonstrate that RVSD effectively improves visual grounding without sacrificing inference efficiency, benefiting from semantics-directed sparsification and the uncertainty-triggered retrieval mechanism that is activated only when necessary.

6

Conclusion

In this paper, we have proposed RVSD, a trainingfree and plug-and-play decoding framework that, for the first time, resolves the previously overlooked sparsification-hallucination paradox in existing sparse decoders. By integrating semantics-

Methods Latency ↓ Memory ↓ TFLOPs ↓ TTFT ↓ Vanilla +VCD +M3ID +AvisC +VTI +RVSD

61.9 131.6 132.5 173.6 67.3 62.0

14334 14979 14977 15501 14363 14015

4.36 8.72 8.72 8.45 4.36 3.68

243.8 445.4 477.8 725.2 266.3 194.6

Table 4: Inference efficiency comparison between RVSD and representative decoding-side VH mitigation methods, reporting per-token decoding latency (ms/token), peak GPU memory usage (MB), TFLOPs (T), and TTFT (ms). Best results are highlighted in bold.

directed token selection with the SSVR mechanism within a single decoding pass, RVSD transforms irreversible visual token pruning into reversible and on-demand cross-modal retrieval, enabling visual grounding to be restored only when necessary. Extensive experiments on five representative benchmarks across three LVLM backbones demonstrate that RVSD achieves SOTA performance in VH mitigation while simultaneously improving decoding efficiency and maintaining robustness under long-context generation, where existing sparse and decoding-based methods often deteriorate. For future work, we will extend RVSD to more challenging multimodal settings, such as long-form video understanding, where extended temporal dependencies and richer modality interactions further exacerbate the sparsification-hallucination paradox. We hope this work provides a promising direction toward trustworthy and efficient LVLM inference.

Limitations While RVSD shows promising results, several limitations remain. Firstly, the uncertainty-triggered retrieval gate in SSVR relies on a predefined entropy threshold to determine when cross-modal retrieval should be activated. Although RVSD remains robust across a reasonable range of threshold values, a more adaptive gating strategy that dynamically adjusts the trigger according to input complexity can further reduce unnecessary retrieval calls and improve efficiency on simpler queries. Additionally, the current implementation uses a simple dot-product similarity to retrieve evidence from the deferred visual memory bank. More advanced retrieval mechanisms, such as learned semantic projections or multi-head cross-modal attention, may further improve performance on finegrained tasks. Although our evaluation already

spans two distinct architectural families, i.e., the LLaVA series and Qwen-VL, we also plan to extend RVSD to larger-scale and more heterogeneous LVLMs to further evaluate its generality.

Ethical Considerations This work aims to improve the trustworthiness of LVLMs by mitigating VHs, thereby contributing to the safer deployment of multimodal artificial intelligence systems. RVSD is a training-free and inference-only framework that requires no additional data collection or human annotation. All datasets (i.e., POPE, MME, AMBER, CHAIR, MM-Vet, MSCOCO, A-OKVQA, and GQA) and backbone models (LLaVA-1.5, LLaVA-NEXT, and Qwen-VL) used in this study are publicly available for academic research and are used strictly in accordance with their original licenses. While RVSD substantially reduces VHs, it does not eliminate them. Therefore, we recommend pairing it with downstream verification or human oversight before deployment in safety-critical applications such as medical diagnosis or assistive technologies.

Acknowledgements This work was supported in part by the National Natural Science Foundation of China (NSFC) under Grant No. 62572132, and the Guangdong Basic and Applied Basic Research Foundation under Grant No. 2025A1515010137.

References Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966. Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930. Liwei Che, Tony Qingze Liu, Jing Jia, Weiyi Qin, Ruixiang Tang, and Vladimir Pavlovic. 2025. Hallucinatory image tokens: A training-free eazy approach to detecting and mitigating object hallucinations in lvlms. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV), pages 21635–21644. IEEE. Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. 2024. An image is worth 1/2 tokens after layer 2: Plug-and-play

inference acceleration for large vision-language models. In European Conference on Computer Vision, pages 19–35. Springer. Tiantian Dang, Chao Bi, Shufan Shen, Jinzhe Liu, Qingming Huang, and Shuhui Wang. 2026. Locatethen-sparsify: Attribution guided sparse strategy for visual hallucination mitigation. arXiv preprint arXiv:2603.16284. Alessandro Favero, Luca Zancato, Matthew Trager, Siddharth Choudhary, Pramuditha Perera, Alessandro Achille, Ashwin Swaminathan, and Stefano Soatto. 2024. Multi-modal hallucination control by visual information grounding. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14303–14312. IEEE. Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and 1 others. 2026. Mme: A comprehensive evaluation benchmark for multimodal large language models. Advances in Neural Information Processing Systems, 38. Jihoon Lee and Min Song. 2025. Retrieval visual contrastive decoding to mitigate object hallucinations in large vision-language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8200–8219. Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large visionlanguage models through visual contrastive decoding. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13872– 13882. IEEE. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459– 9474.

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024c. Llavanext: Improved reasoning, ocr, and world knowledge. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892– 34916. Sheng Liu, Haotian Ye, and James Y Zou. 2025. Reducing hallucinations in large vision-language models via latent space steering. In International Conference on Learning Representations, volume 2025, pages 72402–72419. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 4035–4045. Fengyuan Sun, Hui Chen, Xinhao Xu, Dandan Zheng, Jingdong Chen, Jun Zhou, Jungong Han, and Guiguang Ding. 2025. Prunehal: Reducing hallucinations in multi-modal large language models through adaptive kv cache pruning. arXiv preprint arXiv:2510.19183. Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, and 1 others. 2024. Aligning large multimodal models with factually augmented rlhf. In Findings of the association for computational linguistics: ACL 2024, pages 13088–13110. Chao Wang, Jianming Yang, and Yang Zhou. 2025. Mint: Mitigating hallucinations in large visionlanguage models via token reduction. arXiv preprint arXiv:2502.00717. Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, and 1 others. 2023a. Amber: An llmfree multi-dimensional benchmark for mllms hallucination evaluation. arXiv preprint arXiv:2311.07397.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 292– 305.

Sheng Wang, Zihao Zhao, Xi Ouyang, Qian Wang, and Dinggang Shen. 2023b. Chatcad: Interactive computer-aided diagnosis on medical image using large language models. arXiv preprint arXiv:2302.07257.

Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024a. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253.

Sangmin Woo, Donguk Kim, Jaehyuk Jang, Yubin Choi, and Changick Kim. 2025. Don’t miss the forest for the trees: Attentional vision calibration for large vision language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1927–1951.

Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024b. Improved baselines with visual instruction tuning. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26286–26296. IEEE.

Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Philip S Yu. 2023. Multimodal large language models: A survey. In 2023 IEEE international conference on big data (bigdata), pages 2247–2256. IEEE.

Junfei Wu, Yue Ding, Guofan Liu, Tianze Xia, Ziyue Huang, Dianbo Sui, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2025. Sharp: Steering hallucination in lvlms via representation engineering. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14357–14372. Qiong Wu, Wenhao Lin, Yiyi Zhou, Weihao Ye, Zhanpeng Zeng, Xiaoshuai Sun, and Rongrong Ji. 2026. Accelerating multimodal large language models via dynamic visual-token exit and the empirical findings. Advances in Neural Information Processing Systems, 38:168378–168403. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review, 11(12). Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2024. MM-vet: Evaluating large multimodal models for integrated capabilities. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 57730–57754. Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis A Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, and Shanghang Zhang. 2025. SparseVLM: Visual token sparsification for efficient vision-language model inference. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 74840–74857. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Zican Dong, Yupeng Hou, Beichen Zhang, Yingqian Min, Junjie Zhang, Peiyu Liu, and 1 others. 2026. A survey of large language models. Frontiers of Computer Science, 20(12):2012627. Xingyu Zhu, Kesen Zhao, Liang Yi, Shuo Wang, Zhicai Wang, Beier Zhu, Hanwang Zhang, and Xiangnan He. 2026. Look carefully: Adaptive visual reinforcements in multimodal large language models for hallucination mitigation. In International Conference on Learning Representations, volume 2026, pages 81235–81256. Xianwei Zhuang, Zhihong Zhu, Yuxin Xie, Liming Liang, and Yuexian Zou. 2025. Vasparse: Towards efficient visual hallucination mitigation via visualaware token sparsification. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4189–4199. IEEE.

Appendix A

More Experiments and Results

Results on POPE A-OKVQA and GQA. To further evaluate the cross-dataset generalization of RVSD, we conduct additional experiments on the POPE A-OKVQA and GQA subsets, as shown in Tables 5 and 6. Compared with POPE-MSCOCO (refer to Table 1), these datasets contain denser scenes and stronger object co-occurrence biases, making the popular and adversarial splits more challenging. RVSD achieves the best accuracy across all splits on all three backbones and the best F1 scores on the two LLaVA backbones, with the largest improvements observed on the adversarial split. In particular, RVSD improves Accuracy/F1 by 11.6/5.4 on LLaVA-1.5 A-OKVQA and by 10.9/3.8 on LLaVA-1.5 GQA, demonstrating strong robustness against misleading linguistic priors. On Qwen-VL, RVSD still obtains the highest accuracy on every split, improving the adversarial accuracy by 2.4 on A-OKVQA and by 4.4 on GQA, while its F1 scores remain comparable to the strongest contrastive baseline. In contrast, existing baselines provide only limited or inconsistent improvements, while AvisC notably degrades the average performance on both A-OKVQA and GQA. These results demonstrate that the effectiveness of RVSD generalizes consistently across different POPE datasets and is not dependent on a specific data distribution.

B

Detailed Experimental Setup

B.1

Datasets and Benchmarks

We provide additional details for the five benchmarks used in the main paper, all of which are publicly released and standard in the VH literature. POPE (Li et al., 2023) evaluates object-level hallucination through binary “Yes/No” existence questions and reports accuracy and F1 scores under three sampling strategies, i.e., random, popular, and adversarial splits. The popular split samples frequently co-occurring object categories. In contrast, the adversarial split selects objects with strong co-occurrence priors relative to the image, making it particularly challenging for VH mitigation. We evaluate POPE on three data sources, namely MSCOCO (Rohrbach et al., 2018), A-OKVQA, and GQA, each containing 3,000 yes/no queries. MME Hallucination Subset (Fu et al., 2026) evaluates hallucination through four probing tasks, in-

cluding object-level probes (Existence, Count) and attribute-level probes (Position, Color). Each task contains 30 binary questions and is evaluated using Acc/Acc+, where Acc+ requires both yes/no questions associated with the same image to be answered correctly. The total score is computed as the sum of the four subtask scores. AMBER (Wang et al., 2023a) contains both a discrimination subset and a generative subset. The discrimination subset uses binary yes/no questions and is evaluated with accuracy and F1 scores, while the generative subset evaluates open-ended captioning using CHAIR, HalRate, and a holistic Score that jointly measures caption coverage and hallucination penalty. AMBER covers existence-, attribute-, and relation-level hallucinations, with a particular emphasis on long-form caption generation. CHAIR (Rohrbach et al., 2018) evaluates object hallucination in generated captions by extracting mentioned objects and matching them against MSCOCO ground-truth annotations. It reports two metrics, which are given by |{hallucinatedobjects}| , |{allmentionedobjects}|

(14)

|{captionsw/hallucination}| . |{allcaptions}|

(15)

CHAIRI =

CHAIRS =

Following (Liu et al., 2025), we randomly sample 500 MSCOCO Val2014 images and prompt the model with “Please describe this image in detail” to elicit long-form responses. MM-Vet (Yu et al., 2024) consists of 218 evaluation samples designed to probe six vision-language capabilities: Recognition (Rec), OCR, Knowledge (Know), Generation (Gen), Spatial reasoning (Spat), and Math. We follow the official evaluation protocol and use GPT-4 as the judge to assign a continuous score for each response, with the total score computed as the mean across all dimensions. B.2

Baselines

We compare RVSD against four representative training-free decoders spanning three paradigms. All baselines follow the recommended hyperparameters and are evaluated under the same decoding budget as RVSD for a fair comparison. VCD (Visual Contrastive Decoding) (Leng et al., 2024) introduces a perturbed visual input by adding diffusion-style Gaussian noise and performs contrastive decoding between the clean and corrupted

Models

Methods

Random

Popular

Adversarial

Average

Acc (%)↑

F1 (%)↑

Acc (%)↑

F1 (%)↑

Acc (%)↑

F1 (%)↑

Acc (%)↑

F1 (%)↑

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) +AvisC (Woo et al., 2025) +VTI (Liu et al., 2025) +RVSD (Ours)

81.8 ↑0.0 81.2 ↓0.6 82.9 ↑1.1 79.1 ↓2.7 83.0 ↑1.2 87.5 ↑5.7

83.5 ↑0.0 83.2 ↓0.3 84.6 ↑1.1 82.4 ↓1.1 83.6 ↑0.1 86.4 ↑2.9

75.3 ↑0.0 74.7 ↓0.6 75.8 ↑0.5 71.8 ↓3.5 76.1 ↑0.8 85.4 ↑10.

78.7 ↑0.0 78.5 ↓0.2 79.4 ↑0.7 77.2 ↓1.5 79.5 ↑0.8 84.5 ↑5.8

67.4 ↑0.0 68.1 ↑0.7 68.3 ↑0.9 64.4 ↓3.0 68.2 ↑0.8 79.0 ↑11.

73.7 ↑0.0 74.6 ↑0.9 74.6 ↑0.9 73.0 ↓0.7 74.6 ↑0.9 79.1 ↑5.4

74.8 ↑0.0 74.7 ↓0.1 75.7 ↑0.9 71.8 ↓3.0 75.8 ↑1.0 84.0 ↑9.2

78.6 ↑0.0 78.8 ↑0.2 79.5 ↑0.9 77.5 ↓1.1 79.2 ↑0.6 83.3 ↑4.7

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) LLaVA-NEXT +AvisC (Woo et al., 2025) +VTI (Liu et al., 2025) +RVSD (Ours)

83.8 ↑0.0 84.8 ↑1.0 85.3 ↑1.5 85.8 ↑2.0 84.8 ↑1.0 90.1 ↑6.3

83.0 ↑0.0 83.9 ↑0.9 84.3 ↑1.3 84.4 ↑1.4 84.2 ↑1.2 89.6 ↑6.6

81.4 ↑0.0 81.5 ↑0.1 82.2 ↑0.8 83.9 ↑2.5 80.4 ↓1.0 88.6 ↑7.2

80.8 ↑0.0 81.2 ↑0.4 81.6 ↑0.8 82.8 ↑2.0 79.7 ↓1.1 88.2 ↑7.4

73.2 ↑0.0 74.7 ↑1.5 74.6 ↑1.4 75.6 ↑2.4 72.8 ↓0.4 81.8 ↑8.6

74.5 ↑0.0 76.0 ↑1.5 75.7 ↑1.2 76.6 ↑2.1 74.3 ↓0.2 82.4 ↑7.9

79.5 ↑0.0 80.3 ↑0.8 80.7 ↑1.2 81.8 ↑2.3 79.3 ↓0.2 86.8 ↑7.3

79.4 ↑0.0 80.4 ↑1.0 80.5 ↑1.1 81.3 ↑1.9 79.4 ↑0.0 86.7 ↑7.3

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) +AvisC (Woo et al., 2025) +VTI (Liu et al., 2025) +RVSD (Ours)

86.8 ↑0.0 87.4 ↑0.6 87.1 ↑0.3 84.7 ↓2.1 85.0 ↓1.8 87.4 ↑0.6

85.8 ↑0.0 86.6 ↑0.8 85.9 ↑0.1 83.0 ↓2.8 85.3 ↓0.5 86.2 ↑0.4

85.6 ↑0.0 86.3 ↑0.7 85.9 ↑0.3 83.9 ↓1.7 82.5 ↓3.1 87.7 ↑2.1

84.7 ↑0.0 85.1 ↑0.4 84.6 ↓0.1 82.4 ↓2.3 83.3 ↓1.4 84.5 ↓0.2

80.4 ↑0.0 80.7 ↑0.3 80.5 ↑0.1 78.1 ↓2.3 74.5 ↓5.9 82.8 ↑2.4

80.5 ↑0.0 80.8 ↑0.3 80.4 ↓0.1 77.3 ↓3.2 75.0 ↓5.5 81.1 ↑0.6

84.3 ↑0.0 84.8 ↑0.5 84.5 ↑0.2 82.2 ↓2.1 80.7 ↓3.6 86.0 ↑1.7

83.7 ↑0.0 84.2 ↑0.5 83.6 ↓0.1 80.9 ↓2.8 81.2 ↓2.5 83.9 ↑0.2

LLaVA-1.5

Qwen-VL

Table 5: Performance comparison on the POPE discrimination benchmark over the A-OKVQA dataset. The average column reports the mean accuracy and F1 scores across the random, popular, and adversarial splits. Green (•) and red (•) indicate improvements and degradations relative to the base model (sampling baseline), respectively, while (↑/↓) denotes the absolute change. Best results within each model block are highlighted in bold.

visual conditions, thereby suppressing unimodal language priors. M3ID (Multi-Modal Mutual-Information Decoding) (Favero et al., 2024) reweights output logits using a mutual-information-based score between visual and textual modalities, aiming to enhance visual grounding during generation. AvisC (Attentional Vision Calibration) (Woo et al., 2025) identifies high-attention “blind tokens” that dominate cross-modal attention and recalibrates predictions via dual forward passes with and without these tokens. VTI (Visual Token Intervention) (Liu et al., 2025) steers latent representations in the large language model hidden space along a pre-computed direction associated with improved visual grounding. B.3

Models

We instantiate RVSD on three LVLMs drawn from two distinct architectural families. LLaVA-1.57B (Liu et al., 2024b) adopts CLIP-ViT-L/14-336px as the visual encoder, a two-layer MLP as the cross-modal projector, and Vicuna-7B-v1.5 as the language backbone, producing 576 visual tokens per image. LLaVA-NEXT-7B (Liu et al., 2024c) extends LLaVA-1.5 with the AnyRes mechanism, which partitions the input image into up to four

high-resolution sub-images, increasing the number of visual tokens to as many as 2,880 and substantially amplifying the burden on visual grounding. Qwen-VL (Bai et al., 2023) instead couples OpenCLIP-ViT-bigG with a single-layer positionaware cross-attention adapter that compresses the visual feature map into a fixed set of 256 visual tokens, and adopts Qwen-7B as the language backbone, differing from the LLaVA series in both the vision-language interface and the language backbone. Overall, the backbones cover standard and high-resolution settings as well as both MLPprojection and cross-attention resampler interfaces, and demonstrate that RVSD is model-agnostic.

C

Implementation Details

Hyperparameter Settings of RVSD. RVSD is implemented in PyTorch on top of the official LLaVA codebase and introduces no additional trainable parameters. Table 7 summarizes the default hyperparameter configuration used across all experiments unless otherwise specified. We adopt a unified pruning layer set Ls = 2, 6, 15 for all three backbones, where the visual-token bank is progressively pruned at these layers. For LLaVA1.5-7B, the sparse-set budget is set to k = 192, approximately one third of Nv = 576, resulting

Models

Methods

Random

Popular

Adversarial

Average

Acc (%)↑

F1 (%)↑

Acc (%)↑

F1 (%)↑

Acc (%)↑

F1 (%)↑

Acc (%)↑

F1 (%)↑

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) +AvisC (Woo et al., 2025) +VTI (Liu et al., 2025) +RVSD (Ours)

81.6 ↑0.0 82.2 ↑0.6 83.3 ↑1.7 79.0 ↓2.6 79.8 ↓1.8 85.3 ↑3.7

83.5 ↑0.0 84.1 ↑0.6 84.5 ↑1.0 82.2 ↓1.3 82.3 ↓1.2 83.8 ↑0.3

73.1 ↑0.0 71.5 ↓1.6 72.3 ↓0.8 67.4 ↓5.7 73.5 ↑0.4 81.3 ↑8.2

77.5 ↑0.0 76.8 ↓0.7 77.1 ↓0.4 74.8 ↓2.7 78.0 ↑0.5 80.3 ↑2.8

68.0 ↑0.0 67.6 ↓0.4 67.2 ↓0.8 64.1 ↓3.9 68.0 ↑0.0 78.9 ↑11.

74.5 ↑0.0 74.5 ↑0.0 74.0 ↓0.5 72.9 ↓1.6 74.6 ↑0.1 78.3 ↑3.8

74.2 ↑0.0 73.8 ↓0.4 74.3 ↑0.1 70.2 ↓4.0 73.8 ↓0.4 81.8 ↑7.6

78.5 ↑0.0 78.5 ↑0.0 78.5 ↑0.0 76.6 ↓1.9 78.3 ↓0.2 80.8 ↑2.3

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) LLaVA-NEXT +AvisC (Woo et al., 2025) +VTI (Liu et al., 2025) +RVSD (Ours)

83.1 ↑0.0 83.4 ↑0.3 84.3 ↑1.2 84.9 ↑1.8 83.3 ↑0.2 88.2 ↑5.1

82.5 ↑0.0 83.4 ↑0.9 84.3 ↑1.8 83.5 ↑1.0 82.6 ↑0.1 87.2 ↑4.7

78.5 ↑0.0 78.2 ↓0.3 78.8 ↑0.3 80.5 ↑2.0 78.6 ↑0.1 84.9 ↑6.4

78.5 ↑0.0 78.6 ↑0.1 78.9 ↑0.4 79.5 ↑1.0 78.2 ↓0.3 84.3 ↑5.8

73.3 ↑0.0 74.2 ↑0.9 74.2 ↑0.9 73.2 ↓0.1 73.2 ↓0.1 81.6 ↑8.3

74.5 ↑0.0 75.6 ↑1.1 75.4 ↑0.9 74.3 ↓0.2 74.3 ↓0.2 81.4 ↑6.9

78.3 ↑0.0 78.6 ↑0.3 79.1 ↑0.8 79.5 ↑1.2 78.4 ↑0.1 84.9 ↑6.6

78.5 ↑0.0 79.2 ↑0.7 79.5 ↑1.0 79.1 ↑0.6 78.4 ↓0.1 84.3 ↑5.8

Vanilla +VCD (Leng et al., 2024) +M3ID (Favero et al., 2024) +AvisC (Woo et al., 2025) +VTI (Liu et al., 2025) +RVSD (Ours)

81.3 ↑0.0 82.0 ↑0.7 82.4 ↑1.1 80.5 ↓0.8 82.0 ↑0.7 84.3 ↑3.0

79.2 ↑0.0 80.5 ↑1.3 79.7 ↑0.5 77.8 ↓1.4 82.4 ↑3.2 81.7 ↑2.5

75.9 ↑0.0 75.9 ↑0.0 76.8 ↑0.9 74.2 ↓1.7 76.5 ↑0.6 82.7 ↑6.8

74.9 ↑0.0 75.6 ↑0.7 77.0 ↑2.1 72.3 ↓2.6 77.9 ↑3.0 80.3 ↑5.4

75.5 ↑0.0 76.7 ↑1.2 77.1 ↑1.6 75.5 ↑0.0 71.0 ↓4.5 79.9 ↑4.4

74.4 ↑0.0 76.2 ↑1.8 76.7 ↑2.3 73.4 ↓1.0 74.5 ↑0.1 77.8 ↑3.4

77.6 ↑0.0 78.2 ↑0.6 78.8 ↑1.2 76.7 ↓0.9 76.5 ↓1.1 82.3 ↑4.7

76.2 ↑0.0 77.4 ↑1.2 77.8 ↑1.6 74.5 ↓1.7 78.3 ↑2.1 79.9 ↑3.7

LLaVA-1.5

Qwen-VL

Table 6: Performance comparison on the POPE discrimination benchmark over the GQA dataset. The average column reports the mean accuracy and F1 scores across the random, popular, and adversarial splits. Green (•) and red (•) indicate improvements and degradations relative to the base model (sampling baseline), respectively, while (↑/↓) denotes the absolute change. Best results within each model block are highlighted in bold.

Symbols

Descriptions

LLaVA-1.5

LLaVA-NEXT

Qwen-VL

k m τ η Ls γ [ls , le ] K α

Sparse-set budget Major textual tokens Saliency softmax temperature Temporal smoothing coefficient Pruning layer set Entropy trigger threshold SSVR layer scanning window Retrieval top-K bound Injection ratio

192 ⌊|y<t |/2⌋ 1.0 0 {2, 6, 15} 0.75 [6,27] 128 0.2

Scaled by image_shape ⌊|y<t |/2⌋ 1.0 0 {2, 6, 15} 0.5 [5,12] 7 0.09

192 (tier) → [133, 89, 52] ⌊|y<t |/2⌋ 1.0 0 {2, 6, 15} 0.75 [5,16] 57 0.2

Table 7: Default hyperparameter settings of RVSD used across all experiments unless otherwise specified.

in a 67% reduction in visual tokens. For LLaVANEXT-7B, the budget is rescaled according to the image shape to maintain a consistent retention ratio under the AnyRes high-resolution setting. For Qwen-VL, whose cross-attention adapter emits only Nv = 256 visual tokens, the budget starts from k = 192 and is progressively tightened to [133, 89, 52] at the three pruning layers, yielding a final retention ratio comparable to that of LLaVA1.5. The number of major textual tokens is defined as m = ⌊|y< t|/2⌋, which selects the most recent half of the generated prefix for visual-saliency aggregation and ensures a length-adaptive allocation consistent with the Top-K formulation in the main

paper. Temporal smoothing in Eq. (6) is disabled by setting η = 0, i.e., s̃t = st . For SSVR, the entropy threshold γ, the intermediate-layer window [ls , le ], the retrieval bound K, and the injection ratio α are tuned per backbone to accommodate differences in visual-token granularity across LLaVA-1.5 (single 24×24 grid), LLaVA-NEXT (multi-granularity AnyRes grids), and Qwen-VL (256 resampled tokens without an explicit spatial grid). Decoding Protocol. Following (Leng et al., 2024; Liu et al., 2025), all methods, including RVSD and baselines, are evaluated under a unified decoding setup using nucleus sampling with temperature T = 1.0, top-p = 0.9, no repetition penalty, and

Existence

0.0

0.0

0.0

0.0

0.0

0.0

Count

1.7

0.0

0.0

0.0

1.7

-5.0

Position

0.0

0.0

0.0

0.0

0.0

0.0

Color

0.0

0.0

0.0

0.0

0.0

-5.0

10.0

Hyperparameters

Setting

Acc (%) ↑

F1 (%) ↑

Injection ratio α

0.05 0.10 0.20 0.50

84.7 84.7 84.7 84.4

83.4 83.4 83.5 83.2

Retrieval bound K

32 64 128 256

84.6 84.6 84.7 84.6

83.4 83.4 83.5 83.4

Entropy threshold γ

0.3 0.5 0.7

84.7 84.7 84.6

83.4 83.4 83.4

96 192 288 576 (= Nv )

83.0 84.7 85.0 85.0

81.1 83.5 84.1 84.1

7.5

5.0

Posters

-2.0

0.0

0.0

0.0

-1.7

5.4

Celebrity

0.3

0.0

0.0

0.3

0.3

-2.1

Scene

0.0

0.0

0.0

0.0

0.8

2.0

Landmark

0.8

-0.8

0.0

0.0

0.0

-3.8

Artwork

1.0

0.0

0.0

0.0

1.0

-3.5

OCR

7.5

0.0

0.0

0.0

7.5

-10.0

Commonsense

0.0

-0.7

-2.1

0.0

-4.3

-9.3

Numerical

0.0

7.5

7.5

0.0

-2.5

7.5

Translation

7.5

0.0

0.0

0.0

7.5

-5.0

2.5

0.0

Code

-10.0

0.0

0.0

0.0

-10.0

0.0

(0.10, 128)

(0.20, 256)

(0.20, 32)

(0.20, 160)

(0.05, 128)

(0.50, 128)

-2.5

-5.0

-7.5

-10.0

Figure 6: Subtask-level ∆ heatmap on MME under nonbaseline (α, kc ) configurations, where ∆ = score(cfg) − score(base) is computed relative to the baseline setting (α = 0.20, kc = 128).

batch size 1 for a fair comparison. The generation budget, i.e., the maximum number of new tokens, is benchmark-dependent: It is set to 2 for binary yes/no tasks in POPE and the MME hallucination subset, and to 1,024 for open-ended generation benchmarks including AMBER (generative), MMVet, and CHAIR. For the inference latency evaluation in Table 4, we follow the protocol specified in the table caption, setting the maximum number of new tokens to 32 with batch size 1. Implementation Details. We conduct all experiments on a single NVIDIA RTX A6000 GPU (48 GB) with NVIDIA driver 535.183.01 and CUDA 12.2 on Ubuntu 22.04. The software stack includes PyTorch 2.0 and Hugging Face Transformers 4.40.

D

Hyperparameter Sensitivity Analysis

Subtask-Level Sensitivity on MME. We first conduct a one-factor-at-a-time analysis around the baseline (α = 0.20, kc = 128) on MME with LLaVA-1.5, where α denotes the retrieval injection ratio and kc the top-k retrieval size. As shown in Fig. 6, α is the primary driver of subtask-level variations: Small values of α (0.05/0.10) consistently improve OCR and translation by up to +7.5, whereas a larger value (α = 0.50) leads to broad degradations across OCR, commonsense, counting, and color reasoning. In contrast, varying kc within [32, 256] yields negligible changes across all subtasks. This analysis identifies (α = 0.10, kc = 128) as a stable operating point and further reveals that the aggregate MME score can obscure task-specific regressions, particularly in reasoningintensive categories. Hyperparameter Sensitivity. To complement the

Sparse budget k

Table 8: Hyperparameter sensitivity analysis on the POPEMSCOCO Adversarial split with LLaVA-1.5. The best result in each block is highlighted in bold.

subtask-level analysis above, we further investigate the sensitivity of RVSD to its four key hyperparameters on the POPE-MSCOCO adversarial split with LLaVA-1.5, which is the most discriminative setting for hallucination evaluation. We perform a one-factor-at-a-time analysis, varying each hyperparameter while keeping the others fixed at their default values in Table 7, and the results are reported in Table 8. Effect of Injection Ratio α. RVSD is robust to the injection ratio α in the practical range [0.05, 0.20], where Accuracy/F1 remains stable at 84.7/83.4-84.7/83.5. Increasing α to 0.50 overinjects retrieved evidence relative to the residualstream magnitude, leading to a mild degradation to 84.4/83.2. This behavior is consistent with the amplitude-alignment design in Eq. (11), where the retrieval branch is intended to re-anchor rather than overwrite the host representation. Effect of Retrieval Bound K. Accuracy/F1 scores vary by less than 0.1 across the retrieval bound K ∈ {32, 64, 128, 256}, indicating strong robustness of the softmax-weighted aggregation in Eq. (9). Irrelevant entries in the deferred memory bank receive near-zero weights and are effectively suppressed, such that increasing K neither introduces noise nor yields additional benefit. Effect of Entropy Threshold γ. RVSD is stable across γ ∈ 0.3, 0.5, 0.7, with the accuracy score remaining at 84.7 for the two lower thresholds and

decreasing by only 0.1 at γ = 0.7. This suggests that, on the POPE adversarial split, the SSVR gate activates on a well-separated subset of uncertain steps, and moderate variations in γ do not materially affect which steps are re-anchored. Effect of Sparse Budget k. The sparse budget k is the most influential among the four hyperparameters. A small budget (k = 96, approximately one-sixth of Nv ) removes too much finegrained visual evidence and reduces Accuracy/F1 to 83.0/81.1. Increasing k to 288 or 576 slightly improves performance to 85.0/84.1. The default setting (k = 192) achieves competitive Accuracy/F1 (84.7/83.5) while maintaining a substantially lower attention cost than k = 576, and is therefore adopted as the efficiency–faithfulness trade-off point reported in Table 4. This trend empirically supports the claim in Section 4.2 that visual grounding in RVSD is primarily driven by semantics-directed selection rather than the sheer number of retained tokens.

E

Algorithmic Complexity

We analyze the asymptotic overhead introduced by RVSD over vanilla decoding. Let D denote the hidden dimension, Nv the number of visual tokens, t the current decoding step, and k the sparse-set budget. The dominant cost of vanilla decoding at step t arises from per-layer self-attention, with complexity O((t + Nv )2 D). In RVSD, this is reduced to O((t + k)2 D) through the sparse set St , leading to computational savings when k ≪ Nv . The additional overhead of RVSD consists of three components. First, the textual saliency computation in Eq. (4) and the relevance scoring in Eq. (5) incur costs of O(t) and O(mNv |Ls |), respectively. Both are linear and negligible compared with the quadratic attention term. Second, SSVR retrieval in Eq. (9) requires O(|Pt |D) for similarity computation and O(|Pt | log K) for topK selection, which remains dominated by selfattention. Third, the residual-stream adapter in Eq. (11) and the fusion step in Eq. (13) incur O(KD) per triggered step. However, the entropybased gate in Eq. (8) activates retrieval only on a small subset of steps. Overall, the additional perstep cost is bounded by O(Nv D) and is negligible relative to attention. Empirically, Table 4 further shows that RVSD reduces per-token latency, maximum memory, TFLOPs, and TTFT compared with vanilla decoding, a profile not achieved by previous

decoding-side methods for VH mitigation.

F

License of Artifacts

We use all datasets and models strictly for academic research in accordance with their original licenses and terms. MSCOCO is released under the Creative Commons Attribution 4.0 License, and POPE is released under the MIT License. AMBER, MME, and MM-Vet are used under their respective official non-commercial academic research licenses. LLaVA-1.5 and LLaVA-NEXT are released under the Apache 2.0 License, while their LLaMA-2/Vicuna backbones are subject to the Meta LLaMA-2 community license. QwenVL is released under the Tongyi Qianwen License, which allows use for academic research. We do not redistribute any of these resources beyond the permissions granted by their licenses, and we will release our RVSD implementation under a permissive open-source license upon publication.

G

Case Study

To provide an intuitive understanding of how RVSD mitigates VHs during generation, we present qualitative case studies in Figs. 7 and 8. Compared with the vanilla decoder and representative sparse decoding baselines, RVSD produces captions that are more faithfully grounded in the visual content, successfully suppressing hallucinated objects, attributes, and spatial relations that are absent from the input image. As shown in Fig. 7, RVSD consistently mitigates object- and attributelevel hallucinations across diverse scenes, while Fig. 8 further demonstrates its effectiveness under more challenging long-form generation settings with denser visual content and finer-grained semantic dependencies. These cases collectively demonstrate that the semantics-directed token selection effectively preserves visually critical evidence, while the SSVR mechanism re-anchors decoding through on-demand cross-modal retrieval whenever visual grounding becomes insufficient.

H

Usage of AI Assistant

We used GPT-4o exclusively for the language polishing of this manuscript, including grammar correction and sentence-level rephrasing. All generated outputs were manually reviewed and edited to ensure technical accuracy. No part of the research ideation, method design, experimental analysis, or interpretation of the results involved AI assistance.

Question: Is this artwork kupferstichkabinett, gotha?

displayed

in

Question: Is this movie originated from the country or region of usa?

Ground Truth Answer: No

Ground Truth Answer: No

LLaVA-1.5 (Baseline): Yes

LLaVA-1.5 (Baseline): Yes

+VCD: Yes

+VCD: Yes

+AVISC: Yes

+AVISC: Yes

+VTI: Yes

+VTI: Yes

+RVSD (Ours): No

+RVSD (Ours): No

Question: Is this artwork created by wit, jacob de?

Question: Is this picture captured in a place of swamp?

Ground Truth Answer: No

Ground Truth Answer: No

LLaVA-1.5 (Baseline): No

LLaVA-1.5 (Baseline): No

+VCD: Yes

+VCD: No

+AVISC: Yes

+AVISC: No

+VTI: No

+VTI: Yes

+RVSD (Ours): No

+RVSD (Ours) : No

Question: Is this artwork created by witte, emanuel de?

Question: Is this a picture of Kartause Ittingen?

Question: Is this a photo of Huxi Mosque?

Ground Truth Answer: No

Ground Truth Answer: No

LLaVA-1.5 (Baseline): Yes

LLaVA-1.5 (Baseline): Yes

LLaVA-1.5 (Baseline): Yes

+VCD: Yes

+VCD: Yes

+VCD: Yes

+AVISC: Yes

+AVISC: Yes

+AVISC: Yes

+VTI: Yes

+VTI: Yes

+VTI: No

+RVSD (Ours): No

+RVSD (Ours): No

+RVSD (Ours): No

Question: Is there a cow in the image?

Question: Is there a bottle in the image?

Question: Is there an adult in the image?

Ground Truth Answer: No

Ground Truth Answer: No

Ground Truth Answer: No

LLaVA-1.5 (baseline): Yes

LLaVA-1.5 (baseline): Yes

LLaVA-1.5 (baseline): Yes

+VCD: Yes

+VCD: Yes

+VCD: Yes

+AVISC: Yes

+AVISC: Yes

+AVISC: Yes

+VTI: Yes

+VTI: Yes

+VTI: Yes

+RVSD (Ours): No

+RVSD (Ours): No

+RVSD (Ours): No

Ground Truth Answer: No

Figure 7: Qualitative case studies of RVSD compared with the vanilla decoder and baseline methods. Hallucinated content is highlighted in red, while faithful content grounded in the input image is highlighted in green. RVSD consistently produces visually faithful descriptions across diverse scenes.

Textual input: Please describe this image in detail. LLaVA-1.5: The image is a gathering of several children sitting on benches and waiting areas, likely at an airport. There are a bunch of young kids and their luggage, including backpacks and suitcases, scattered throughout the scene. Some suitcases are placed on the floor close to the children, while others are further away. Multiple children and their belongings are visible in the central area of the image, with their luggage close by. There are also several chairs within the area, along with a couple of benches that the kids are seated on. The scene gives a sense of anticipation as the children wait for their transportation, likely for their next flight. VCD: The image portrays a group of people, including children, sitting on multiple benches inside a building, likely a mall or airport. They appear to be waiting or relaxing in the public space. There are several benches scattered throughout the scene, with some individuals occupying them. In addition to the people, there are numerous personal belongings like backpacks, handbags, and a suitcase. Two backpacks are placed on the ground near some of the benches, while another backpack is placed further away. A handbag is situated near a bench, and a suitcase is placed close to one of the people. A cup can also be seen on a surface in the scene.Overall, the atmosphere is lively, with people engaged in conversation, waiting, or simply enjoying their time in the public space. VTI: The image features a very busy and crowded airport terminal with numerous people present. There is a small family consisting of a man, a woman, and two little children seated together on a bench close to a large mirror. The woman is possibly accompanying the group, comforting them as they wait. Various luggage is scattered throughout the scene, including a suitcase close to the center bench, another suitcase towards the top right, and a handbag lying closer to the woman on the large bench. The airport's entrance can be seen in the upper part of the image, and it is filled with people trying to find their flights or looking for their gates. AVISC: The image features a group of people sitting on a bench at a travel place. The bench is occupied by several individuals, including children and adults. A chair is visible towards the right side of the scene. The setting is likely a travel spot, as several backpacks can be seen placed around the people. There are also two handbags in the area, one close to the center of the scene and the other towards the left side. The people seem to be enjoying their time together, waiting for their journey to continue. RVSD: The image features a busy airport scene with several people scattered throughout the location. A group of children can be seen walking around, while other individuals are sitting on multiple benches or chairs. One girl, in particular, is sitting at the edge of a bench, seemingly posing for the photograph. There are multiple backpacks placed in various locations around the children, with some being close to the benches where they're sitting and others further away. The children appear to be enjoying themselves as they navigate through the busy transport area.

Textual input: Please describe this image in detail. LLaVA-1.5: The scene features a basket filled with tropical fruit and vegetables, including a group of bananas, coconuts, a guava, and other vegetables. There are bananas scattered throughout the basket with some placed at the top, center, and sides. In addition to the fruits, there is a bottle placed near the bottom of the basket, possibly containing a beverage or a sauce to accompany the fruits. The combination of the fruits and the bottle creates a colorful and inviting scene. This presentation of fruit and vegetables makes for a vibrant and visually appealing arrangement. VCD: The image features a bowl placed on a table, filled with a variety of fruits and vegetables. There are several bananas in the bowl, some of which are partially eaten or rotting. A few apples are also present in the bowl, adding to the assortment of fruit. In addition to the bananas and apples, the bowl also contains a handful of coconuts, adding a tropical touch to the display. A leaf is situated close to the bananas, further enhancing the natural and organic appearance of the scene. VTI: The image depicts a basket containing ripe, yellow fruits, possibly bananas or cantaloupes. These fruits are spread across the basket and partially overlapping each other. In addition to the fruit, there is a small jar of honey sitting on top of the basket. The combination of fruits and honey creates a natural and appealing arrangement for consumption. AVISC: The image shows a table with a bunch of bananas and two bowls. One bowl contains a variety of bananas and coconuts, while the other bowl appears to have some fruit in it as well. A bottle can be seen placed near the table, and the bananas are placed in a variety of spots on the table. RVSD: The image depicts a basket filled with a variety of fruits including bananas and coconuts. There are six bananas situated in different areas of the basket, ranging from the top left to the bottom right. Close to the bananas, the basket also contains eight coconuts, spread out across the top, middle, and bottom parts of the container. This colorful and diverse fruits and coconuts arrangement creates an appetizing display.

Figure 8: Additional qualitative case studies of RVSD on more challenging long-form generation samples. Hallucinated content is highlighted in red, while faithful content grounded in the input image is highlighted in green. RVSD continues to deliver visually faithful generations even under denser scenes and finer-grained semantic dependencies, where baseline decoders frequently exhibit object-, attribute-, and relation-level hallucinations.

Related documents

• Reducing Hallucination in Vision-Language Models via Stage-wise Preference Optimization under Distribution Shift#200592
• RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models#924867
• RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models#920633
• Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM#819691
• Beyond Language Priors: Diagnosing and Fixing Visual-Origin Hallucinations in Multimodal LLM#809695