arXiv CSopen access

Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificial intelligence, reasoning, knowledge representation

arXiv:2604.25642v1 [cs.CV] 28 Apr 2026

Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models Chengsheng Zhang1 , Chenghao Sun1 , Xinyan Jiang2,3 , Wei Li1 , Xinmei Tian1 *, 1 University of Science and Technology of China 2 Shanghai Advanced Research Institute, Chinese Academy of Sciences, Shanghai, China 3 University of Chinese Academy of Sciences, Beijing, China {zhangcs66, chsun, lwzkd}@mail.ustc.edu.cn, [email protected] LVLM Decoder Layer ×L

(a) Inference-Time Intervention Abstract

... Prefill Token Generation Large Vision-Language Models (LVLMs) have achieved re... Steering Layer markable progress in visual-textual understanding, yet their reliability is critically undermined byIntervene hallucinations, i.e., FFNof factually incorrect or inconsistent rethe generation Visual Hidden State Textual Hidden State sponses. While recent studies usingTextual steering Direction vectors demonstrated promise in reducing hallucinations, notable chal(b) Prefill-Time a Intervention (Ours) MHSA lenge remains: they inadvertently amplify the severity of KV Cache residual hallucinations. We attribute excluPrefillthis Tokento their Generation Q K V ... sive focus on the decoding stage, where errors accumulate autoregressively and progressively Intervene worsen subsequent hallucinatory outputs. To address this, we propose PrefillKV Textual KV Image Text Time Intervention (PTI), a novelVisual steering paradigm that Visual KV Direction Textual KV Direction ... intervenes only once during the prefill stage, enhancing : Visual/Textual KV Cache the initial Key-Value (KV) cache before error accumulation occurs. Specifically, PTI is modality-aware, deriving distinct directions for visual and textual representations. This intervention is decoupled to steer keys toward visually-grounded objects and values to filter background noise, correcting hallucination-prone representations at their source. Extensive experiments demonstrate PTI’s significant performance in mitigating hallucinations and its generalizability across diverse decoding strategies, LVLMs, and benchmarks. Moreover, PTI is orthogonal to existing decoding-stage methods, enabling plug-and-play integration and further boosting performance. Code is available at: https://github.com/huaiyi66/PTI. Hidden Space

1. Introduction Remarkable advances in large language models (LLMs) [2, 9, 14, 39, 53] have driven the emergence of large visionlanguage models (LVLMs) [4, 18, 26, 27, 31, 38], which aim to integrate visual perception with linguistic understanding. Despite their impressive capabilities, LVLMs re* Corresponding author.

LVLM

(a) Decoding-Time Intervention

Decoder Layer ×L

...

Hidden Space

Prefill Token

Steering Layer

...

Generation

...

... Intervene

FFN

Visual Hidden State Textual Direction

(b) Prefill-Time Intervention (Ours)

MHSA

Textual Hidden State

KV Cache Prefill Token

...

Generation Generation

...

... Intervene

Image

...

Text

Visual KV

Visual KV Direction

Textual KV

Textual KV Direction

Figure 1. Comparative analysis. (a): Decoding-Time Intervention methods continuously intervene in the hidden states of the prefill and generated token. (b): Our method applies modal-specific interventions to the KV cache only once in the prefill phase.

main prone to hallucinations [3, 36, 51, 55, 56], generating factually inconsistent outputs that contradict the visual input. Common manifestations include imaginary entities [21, 33], incorrect attributes [35, 57], and nonexistent relationships [46]. These failures in factual grounding significantly undermine user trust and impede the safe deployment of LVLMs in real-world interactive applications [8, 25, 58]. To mitigate these issues, recent approaches adopt the Decoding-Time Intervention (DTI) paradigm [7, 23, 29, 54, 55], demonstrating strong potential for steering LVLM behavior without parameter modification. These methods typically apply uniform steering vectors to model representations (e.g. hidden states) during decoding (Figure 1(a)). However, a notable challenge remains: although DTI methods such as VISTA [23] reduce the frequency of hallucinations, they inadvertently exacerbate the severity of residual hallucinations. This exacerbation manifests as “snowball

ＨａｌｌｕｃｉｎａｔｉｏｎＭｅｔｒｉｃｏｆＣＨＡＩＲＢｅｎｃｈｍａｒｋ（↓）２０３０４０

０

５０

１０

ＳｄＩＶＨＤ

ＣＨＡＩＲ：Ｖａｎｉｌｌａ

ＣＨＡＩＲ：ＶＩＳＴＡＣＨＡＩＲ：ＰＴＩ（Ｏｕｒｓ）

ＣＨＡＲ，

ＰＳＨ：ＶａｎｉｌｌａＰＳＨ：ＶＩＳＴＡ

ＰＳＨ：ＰＴＩ（Ｏｕｒｓ）

ＰＳＨ０

１０

２０

３０４０６０７０５０ＰｒｏｐｏｒｔｉｏｎｏｆＳｎｏｗｂａｌｌＨａｌｌｕｃｉｎａｔｉｏｎ（↓）

８０

Figure 2. Quantitative analysis of LLAVA-1.5 on CHAIR Benchmark [33]. We report CHAIRS /CHAIRI to measure hallucination frequency at sentence/instance levels, and PSH [37] (the proporHallucinations tion of snowball hallucination, i.e., Snowball × 100%) to Overall Hallucinations quantify the severity of cascading hallucinations.

hallucinations” [37, 60]: as Figure 2 illustrates, once an initial error is generated, the continuous intervention struggles to halt its propagation, leading to a cascade of progressively degrading inaccuracies. We attribute this exacerbation to inherent limitations within the DTI paradigm, rooted in how, what, and when it intervenes. First, DTI typically employs a uniform steering vector derived solely from the textual state [23, 29, 55]. This modality-agnostic manner ignores the distinct sensitivity of the text decoder to visual representations, potentially worsening the modality misalignment [29] that seeds the initial error. Furthermore, the intervention targets coarsegrained representation (e.g., high-level hidden states) primarily. This lack of precision makes it ill-equipped to correct the fine-grained visual perception errors [7, 20]. Finally, and most critically, the intervention is applied continuously and reactively during the decoding stage. This timing means that it acts after an initial poorly-grounded representation has already been computed, permitting initial errors to accumulate [6, 22, 43] and snowball [37] autoregressively. The solution, therefore, lies not in the decoding stage, but in specific improvements at the source—during the prefill stage when initial representations are formed. This motivates shifting the focus from continuous decoding-time correction to shaping the model’s initial states, which, in Transformer-based LVLMs, are materialized as the key-value (KV) cache during the prefill stage [40, 47]. Critically, the KV cache is not merely a storage module—it actively shapes every subsequent decoding step by providing contextual information to the attention mechanism [42]. This pivotal role makes it a natural intervention point: recent studies demonstrate that manipulating the KV cache can significantly improve model performance in reasoning [5, 11, 34], inference acceleration [40, 45, 50], and long-context handling [44, 49]. Crucially, these gains stem

from the pivotal role of the KV cache in shaping the entire subsequent generation process. We therefore hypothesize that a targeted intervention on the initial KV cache can effectively mitigate hallucinations in LVLMs. Based on this hypothesis, we propose Prefill-Time Intervention (PTI), illustrated in Figure 1 (b). Specifically, PTI is structured to directly address the inherent limitations of DTI, adhering to three core principles: when to intervene, how to intervene, and what to target. First, to rectify DTI’s reactive timing, PTI is proactive: it intervenes only once during the prefill stage, before error accumulation can begin. Second, to address DTI’s modality-agnostic failure, PTI is modality-aware, treating visual and textual inputs distinctly (Figure 3) to precisely counter the modality misalignment that seeds the initial error. Finally, to solve DTI’s lack of precision, PTI targets the fine-grained KV cache rather than coarse-grained hidden states. This enables direct intervention within the attention mechanism itself [42], which naturally acts as a decoupled control mechanism grounded in the distinct roles of keys (where to attend) and values (what to aggregate). We leverage this property by using object-vs-background contrasts to derive vectors that steer keys toward visually grounded objects and values to filter background noise. The enhanced cache then serves as a well-grounded initial state for decoding. We experimentally validate PTI on three representative and architecture-distinct LVLMs: LLaVA-1.5 [27], QwenVL-Chat [4], and DeepSeek-VL-Chat [31]. PTI achieves superior performance on widely adopted object hallucination and comprehensive benchmarks, outperforming existing decoding-time studies. In addition, PTI is orthogonal to existing decoding-time methods, enabling seamless plugand-play integration to further boost their performance. Our contributions can be summarized as follows. • We propose Prefill-Time Intervention (PTI), a novel, plug-and-play, and multi-modal steering paradigm for mitigating hallucinations in LVLMs. • Distinct from existing DTI methods, PTI derives distinct directions from the multi-modal cache and decouples them to simultaneously enhance object-centric attention and improve the robustness to background noise. • Extensive experiments demonstrate that PTI provides a general and robust solution to the existing dilemma of how, what, and when to intervene.

2. Related Work 2.1. Large Vision-Language Models The predominant architecture for modern LVLMs integrates an open-source LLM backbone (e.g., LLAMA [39]) with a visual encoder using a connector module that projects visual tokens into the LLM’s embedding space. Most representatively, LLaVA [26] utilizes a minimal architecture that

connects a standard CLIP [32] to its LLM via a simple MLP. Recent models introduce more sophisticated visual processing. Qwen-VL-Chat [4] employs a cross-attention resampler to condense variable-length features from its ViT [10] into a fixed-length sequence for the LLM, while DeepSeekVL-Chat [31] instead uses a hybrid vision encoder, fusing features from parallel high- and low-resolution encoders [16, 59] to enhance detail-oriented understanding. Through subsequent modality alignment and instruction tuning, these models exhibit remarkable capabilities on a wide array of visual-language tasks [21, 33, 46]. Despite their remarkable capabilities, the increased complexity and deployment of LVLMs have also exposed them to various security threats and vulnerabilities, motivating the exploration of robust mitigation strategies.

2.2. Mitigating Hallucination in LVLMs Existing studies for mitigating hallucination in LVLMs can be broadly categorized into two types: 1) Visual contrastive decoding [17, 36, 37, 48, 56]. These methods contrast model logits derived from original and distorted visual inputs across decoding time steps. However, the multi-round decoding significantly increases the inference latency. Moreover, recent works [7, 36, 37, 56] have argued that they might discard beneficial language priors and compromise the coherence of generated content. 2) Decodingtime Intervention (DTI) [7, 19, 23, 29]. These methods leverage contrastive input samples to extract steering vectors, which are then applied to guide the model’s behavior toward specific directions. A key limitation is that they use a fixed steering vector across all token positions during the overall generation phase, which may exacerbate initial errors and worsen subsequent hallucinatory output. Although VTI [29] attempts an additional intervention in the visual encoder, its intervention on the linguistically-dominant decoder remains undifferentiated, failing to distinguish between modalities and intervention timing. In contrast, our method directly enhances the initial multi-modal KV cache to improve the fidelity of subsequent decoding. There are two concurrent studies [5, 28] mostly related to ours. [28] trains a differentiable “coprocessor” for pregeneration augmentation, while [5] utilized GPT4o [15] to transfer controllable reasoning styles to small LLMs. In contrast, our method is systematically designed around the intervention’s modalities and positions to mitigate hallucination in LVLMs. Moreover, our method is more general without re-training or powerful auxiliary models.

3. Methodology 3.1. Preliminaries: Inference with KV Cache The standard generative inference process in LVLMs involves two primary stages: 1) prefilling with multi-modal

input and 2) decoding with initial KV cache. Prefilling with Multi-modal Input. During the prefill phase, the model processes the input prompt, which contains sequential visual and textual tokens. Specifically, this input sequence is represented as a sequence of embeddings X ∈ RNx ×D , where Nx is the total sequence length and D is the hidden dimension of the model. To construct the initial KV cache, this input is processed by each decoder layer l ∈ [1, L]. At layer l, the input embeddings X l ∈ RNx ×D are projected via key and value l projection matrices WK , WVl ∈ RD×D as follows: \mathbf {K}^l =X^l \mathbf {W}^l_K, \quad \mathbf {V}^l = X^l \mathbf {W}^l_V.

(1)

The resulting tensors Kl , Vl ∈ RNx ×D are then reshaped to RNh ×Nx ×dh to accommodate Nh attention heads, each with dimension dh . For clarity, we refer to these reshaped tensors as K l and V l in subsequent steps. We denote the structured initial cache as the key set K = {K l }L l=1 and value set V = {V l }L across L decoder layers. This initial l=1 cache stores the layer- and head-level token information, facilitating subsequent token generation. Decoding with Initial KV Cache. In the decoding phase, the model generates the output sequence autoregressively, leveraging and updating the KV cache at each step. At a given time step t in layer l, the new key kt,l txt and t,l value vtxt vectors are computed only for the current input D token embeddings xt,l txt ∈ R . Simultaneously, the query t,l tensor q is computed using the query projection matrix l WQ ∈ RD×D . These new tensors are also reshaped into t,l t,l t,l multi-head format ktxt , vtxt , qtxt ∈ RNh ×1×dh . The cache at layer l is then updated by concatenating the new key and value vectors. Denoting concatenation by [·, ·], the updating process is defined as: K^l \leftarrow [K^l, k^{t,l}_{\text {txt}}], \quad V^l \leftarrow [V^l, v^{t,l}_{\text {txt}}].

(2)

The model then performs self-attention at layer l using the query and the updated layer cache. The output token ot,l txt = Softmax

t,l qtxt (K l )T √ dh

V l ∈ RNh ×1×dh is then re-

shaped back to RD and processed by subsequent sub-layers. Finally, the output of the final decoder layer is then used to predict the next token xtout . Our PTI operates on the initial layer-wise caches (K l , V l ) computed during the prefill stage, before the decoding process begins.

3.2. Object Directions Extraction To extract the distinct visual and textual directions, we construct contrastive visual and textual inputs designed to isolate the object signal from the background context. Since MSCOCO [24] provides high-quality image-caption pairs and object segments, we utilize this resource to derive taskagnostic directions. Notably, the extractions for different modalities in Figure 3 (Stage I) operate independently.

Contrastive Pair Constructions

Stage I: Object Directions Extraction LVLM Decoder ...

Visual Contrast Prompt

Stage II: Downstream Intervention Multi-modal Intervention

Visual Direction Extraction KV Cache

KV Cache

Postive Prefill Token layer: l+1

Postive Image: ��pos

Negative Image: ��neg

... FFN

A cat is sitting on top of a vehicle.

� Postive Caption: �pos

� Negative Caption: ��

Positive Sample

...

AVG

...

... Visual KV Cache

LVLM

KV Cache

e.g., LLAVA-1.5

... layer: l-1

Negative Prefill Token

...

Index Index

Textual KV Cache

Vanilla w/ PTI: The image features a tempting and thick hamburger in between a bun.

Intervene

Index & AP (·)

Postive Prefill Token

A cat is sitting on top of a vehicle.

Negative Sample

Index & AP (·)

Prefill Token

Subtract

Textual Direction Extraction

MHSA Q

...

Negative Prefill Token

layer: l

Task-agnostic Dataset e.g., MSCOCO Textual Contrast Prompt

...

Vanilla: The image features a half of a chili cheese dog sitting on a plate, placed on a dining table.

Subtract

Please help me describe this image in detail.

AVG

Visual KV Direction

Textual KV Direction

Token Index

Figure 3. Pipeline overview of our PTI. PTI consists of two stages. Stage I: we separately extracted the object signals from the visual and textual contrastive KV caches to extract the directions. Stage II: these directions are applied as multi-modal interventions to the initial KV cache of downstream input. The modified cache is then passed back to the decoder to generate responses.

Visual Direction Extraction. Given an image I i and its i target object segmentation label Mobj from MSCOCO [24], i i = I i ⊙ Mobj as a we define the object-only signal Ipos positive sample. Conversely, the background-only context i i Ineg = I i ⊙ (1 − Mobj ) is defined as the negative sample. Let Iimg denote the indices for all the visual tokens. Using a fixed textual prompt T to maintain the uniformity of Iimg , we perform two separate prefill-time forward passes through the LVLM. We obtain the positive cache i,l i,l i,l i,l (Kpos , Vpos ) and the negative cache (Kneg , Vneg ) at layer l respectively. Then, we compute the steering vectors from the visual tokens, denoted as: \begin {aligned} \Delta C^{i,l}_{\text {img}} = \text {AP}(C^{i,l}_{\text {pos}}- C^{i,l}_{\text {neg}})[\mathcal {I}_{\text {img}}], \quad C \in \{ K, V\} \end {aligned}

(3)

where AP(·) denotes the average pooling operation across i,l i,l and the visual token dimension [10, 52]. Sk,img = ∆Kimg i,l i,l Sv,img = ∆Vimg represent the visual steering vectors of key and value, respectively. Notably, we obtain the final visual l l , through averaging over N samdirections, Sk,img and Sv,img ples as follows:

\begin {aligned} {S}^{l}_{\text {k,img}} = \frac {1}{N} \sum _{i=1}^{N} {S}^{i,l}_{\text {k,img}}, \quad {S}^{l}_{\text {v,img}} = \frac {1}{N} \sum _{i=1}^{N} {S}^{i,l}_{\text {v,img}}. \end {aligned}

(4)

In addition, we apply PCA [1] via singular value decomposition to remove extra noise following [23, 29, 55]. These directions capture the target shift in representation associated with focusing on object signals rather than background context within visual tokens.

Textual Direction Extraction. Inspired by [12], we use a natural language processing tool [41] to construct contrastive textual inputs. Specifically, we focus on the visual concept (e.g., “entity”) associated with target object mentions within the caption. Consider the corresponding caption T i from MSCOCO [24] of the given image I i (e.g., “A i cat is sitting on top of a vehicle.”), we define Tpos as the anchor words that comprise the mentioned objects (e.g., “cat” i represents the control set and “vehicle”). Conversely, Tneg comprising non-anchor words from T i (i.e., the remaining context after masking the anchor words). Similar to the visual extraction process, we perform two separate prefill-time forward passes, while keeping the image constant. Subsequently, we obtain the positive cache i,l i,l i,l i,l (K̂pos , V̂pos ) and the negative cache (K̂neg , V̂neg ) at layer l. Following prior work [29], we focus on the last textual token in the input sequence, denoting its index as Nx − 1. We compute the steering vectors at this token index as, \begin {aligned} \Delta \hat C^{i,l}_{\text {txt}} = (\hat C^{i,l}_{\text {pos}}- \hat C^{i,l}_{\text {neg}})[N_x - 1], \quad \hat C \in \{\hat K, \hat V\} \end {aligned}

(5)

i,l i,l i,l i,l where Sk,txt = ∆K̂txt and Sv,txt = ∆V̂txt represent the textual steering vectors of key and value, respectively. We further calculate the averaging textual directions as:

\begin {aligned} {S}^l_{\text {k,txt}}=\frac {1}{N} \sum _{i=1}^{N} {S}^{i,l}_{\text {k,txt}}, \quad {S}^l_{\text {v,txt}}=\frac {1}{N} \sum _{i=1}^{N} {S}^{i,l}_{\text {v,txt}} \end {aligned}

(6)

and subsequently apply PCA similarly. These vectors capture the linguistic shift associated with explicitly grounding the text in the object concepts.

3.3. Downstream Intervention During inference on a downstream task, the standard prefill computation yields the initial KV cache for input samples. As shown in Figure 3 (Stage II), we inject the extracted visual and textual directions into the corresponding token position of the KV cache of all layers. Multi-modal Intervention. It is important to clarify that the visual and textual interventions are order-invariant and applied independently. For clarity of presentation, we first detail the intervention on the visual part. The KV cache (K̃ l , Ṽ l ) in layer l at the visual token positions Iimg is shifted as follows: \begin {aligned} {\tilde {K}^l}[\mathcal {I}_{\text {img}}] \mathrel {+}= \lambda _{\text {k,img}} {S}^{l}_{\text {k,img}}, \quad {\tilde {V}^l}[\mathcal {I}_{\text {img}}] \mathrel {+}=\lambda _{\text {v,img}} {S}^{l}_{\text {v,img}} \\ \end {aligned}

(7)

where λk,img and λv,img denote the scalar coefficients controlling the intervention strength for the keys and values, respectively. Following [23], we further apply normalization to maintain stability. This intervention adjusts the initial state of the visual tokens based on the contrast between the object and its background. Apart from visual cache intervention, we refine the grounding of its linguistic initial state. Specifically, we apply these textual directions only to the cache entry corresponding to the textual token position Itxt as follows: \begin {aligned} {{\tilde {K}}^l}[\mathcal {I}_{\text {txt}}] \mathrel {+}= \lambda _{ \text {k,txt}} {S}^{l}_{\text {k,txt}}, \quad {{\tilde {V}}^l}[\mathcal {I}_{\text {txt}}] \mathrel {+}= \lambda _{\text {v,txt}} {S}^{l}_{ \text {v,txt}} \end {aligned}

(8)

where λk,txt and λv,txt control the intervention strength, followed by a similar normalization step. This adjustment steers the textual representation toward the object-grounded concepts before generation starts. Model Response Generation. The enhanced initial KV cache serves as the starting state for the autoregressive decoding phase. The model then proceeds with the standard decoding process, as detailed in Section 3.1. Notably, PTI operates only on the initial cache and introduces negligible computational overhead and no modifications during the subsequent decoding steps.

4. Experiments 4.1. Experimental Setup Model Architectures. We evaluate PTI on three representative LVLMs with distinct architectures: LLaVA-1.5 [27], Qwen-VL-Chat [4], and DeepSeek-VL-Chat [31]. Decoding strategies. To demonstrate the sustained benefits of PTI, we verify it across three widely used decoding strategies: 1) Greedy, which deterministically selects the most probable token at each step. 2) Beam Search with a beam size of 5. 3) Nucleus Sampling with top-p=1.0. The temperature is fixed at 1.0 for all scenarios. Baselines. We compared PTI with several state-of-the-art (SOTA) training-free hallucination mitigation methods. For

baselines optimizing decoding strategy, we adopt VCD [17] and OPERA [13]. For the other decoding-time solutions: we adopt PAI [30], VTI [29], and VISTA [23]. We reproduce all baseline results using identical evaluation data and settings (e.g., prompt, temperature). To prevent implementation bias, we omitted methods lacking official support for code implementation and specific decoding strategies.

4.2. Implementation Details We extract multi-modal directions on a holdout training set containing 100 randomly selected VQA pairs from the MSCOCO [24]. To assess the effectiveness and robustness of PTI, we evaluated it on Object Hallucination Benchmarks containing CHAIR [33], POPE [21], and AMBER [46], as well as the comprehensive benchmarks including MMHAL [35] and MME [57] with significant distributional differences. In the experiment, we set λk,img = λk,txt and λv,img = λv,txt , and determine the optimal values through a grid search. Detailed hyperparameter configurations are provided in Appendix B.

4.3. Results on Object Hallucination Benchmarks CHAIR Evaluation. CHAIR [33] is a standard metric that quantifies object hallucinations by comparing objects mentioned in generated descriptions against the ground-truth object annotations. Following [23], We randomly select 500 images from the MSCOCO validation set with the prompt “Please help me describe this image in detail.”. CHAIR comprises two evaluation dimensions: object}| and sentenceinstance-level CHAIRI = |{hallucinated |{object}| hallucinated objects}| level CHAIRS = |{caption w/|{caption}| . As shown in Table 1, distinct from VCD [17] and OPERA [13], which support specific decoding strategies with minor improvements, PTI demonstrates strong generalization across various decoding strategies. Moreover, PTI outperforms both VTI [29] and VISTA [23] in most decoding strategies and LVLMs. We attribute this robust performance to the modality-specific and one-time intervention design, which brings fine-grained enhancement to the initial representation at the very onset of the decoding phase. Furthermore, we provide specific case studies in Appendix C. POPE Evaluation. POPE [21] quantitatively assesses object hallucination by posing targeted, closed-ended questions of the form “Is there a <object> in the image?”. The benchmark comprises three distinct splits of increasing difficulty designed to probe model robustness: Random, Popular, and Adversarial. Each split comprises 3000 VQA pairs drawing from the MSCOCO validation set and is framed as a binary classification task (“Yes”/“No”). We report the accuracy (ACC) and F1-score (F1) metrics. The results in Table 2 demonstrate PTI’s consistent superiority across LVLMs. Notably, while VTI [29] employs an additional visual intervention to the encoder, its improve-

Table 1. CHAIR hallucination evaluation results across different decoding strategies and LVLMs. PTI is compared against SOTA trainingfree methods that operate during decoding, where “Vanilla” stands for the original model. The maximum new token is set to 512. The best result is highlighted in bold, while the second-best is marked with an underline. We provide the latency comparison in Appendix B.

Decoding

LLAVA-1.5

Method

Greedy

Beam Search

Nucleus Sampling

Vanilla PAI [30] VTI [29] VISTA [23] PTI (Ours) Vanilla PAI [30] OPERA [13] VTI [29] VISTA [23] PTI (Ours) Vanilla PAI [30] VCD [17] VTI [29] VISTA [23] PTI (Ours)

CHAIRS ↓ 47.4 22.8 35.4 20.4 15.4 ( ↓ 32.0) 47.8 22.3 45.2 35.8 17.4 13.2 (↓ 34.6) 50.2 43.4 51.6 42.6 29.8 25.8 (↓ 24.4)

Qwen-VL-Chat

CHAIRI ↓ 13.7 7.0 11.8 6.9 5.4 (↓ 8.3) 14.2 6.8 12.4 11.1 6.3 5.5 (↓ 8.7) 16.2 14.7 15.3 14.5 10.6 9.2 (↓ 7.0)

CHAIRS ↓ 39.6 41.6 29.2 39.0 20.6 (↓ 19.0) 43.6 39.4 39.6 32.2 30.0 18.8 (↓ 24.8) 42.0 41.6 42.8 30.4 42.8 18.2 (↓ 23.8)

DeepSeek-VL-Chat

CHAIRI ↓ 12.0 12.0 8.8 16.8 7.0 (↓ 5.0) 11.8 10.6 11.7 8.2 12.0 6.3 (↓ 5.5) 14.9 12.4 13.3 10.0 17.6 9.1 (↓ 5.8)

CHAIRS ↓ 25.8 25.6 24.0 22.2 19.2 (↓ 6.6) 27.0 27.8 24.0 26.2 24.0 15.6 (↓ 11.4) 34.0 32.2 28.0 28.6 22.2 27.0 (↓ 7.0)

CHAIRI ↓ 8.2 6.5 8.4 7.0 6.7 (↓ 1.5) 7.5 6.5 7.8 7.2 6.3 4.8 (↓ 2.7) 11.4 8.2 8.2 10.0 10.2 8.0 (↓ 3.4)

Table 2. Experiment results on POPE Benchmark. We report the average accuracy and F1-score computed across three object splits, as well as the specific results on the most challenging adversarial split. We set the maximum new token to 32 and use the nucleus sampling strategy. The complete table can be found in Appendix B.

LLAVA-1.5 Method

Vanilla PAI [30] VCD [17] VTI [29] VISTA [23] PTI (Ours)

Adversarial Acc ↑ 75.40 76.93 76.63 76.33 75.53 77.40

Qwen-VL-Chat

Average

F1 ↑ 77.67 79.13 79.01 78.47 78.61 78.75

Acc ↑ 79.88 81.73 81.53 80.41 80.08 82.21

Adversarial

F1 ↑ 81.23 82.95 82.80 81.64 81.98 82.85

Acc ↑ 80.26 81.63 83.10 80.17 80.80 83.37

ments are marginal. For Qwen-VL-Chat, it even led to a decrease from 82.92% to 82.19% (−0.73%) in F1-score. This degradation is also observed with VISTA (−0.80%), which indicates the necessity of modality-specific differentiation within the dominant decoder. Table 3. Experiment results on AMBER Benchmark. The generative task’s parameters and metrics follow CHAIR, while the discriminative task’s align with POPE. Model LLAVA-1.5 Qwen-VL-Chat DeepSeek-VL-Chat

Method Vanilla PTI Vanilla PTI Vanilla PTI

Greedy CI ↓ 6.1 3.8 6.5 5.7 6.0 4.0

Acc ↑ 81.5 82.0 84.0 84.6 85.2 85.4

Sampling CI ↓ 9.9 7.3 8.0 7.4 8.1 7.0

Acc ↑ 73.5 74.6 79.9 80.9 83.7 84.1

F1 ↑ 79.76 81.12 82.04 79.29 79.46 82.41

DeepSeek-VL-Chat

Average Acc ↑ 83.69 84.91 85.06 83.27 83.54 85.69

F1 ↑ 82.92 84.15 83.72 82.19 82.12 84.62

Adversarial Acc ↑ 82.73 84.03 83.86 83.46 84.80 83.77

F1 ↑ 81.97 83.23 82.89 83.48 83.39 83.63

Average Acc ↑ 83.89 85.57 85.45 84.81 85.88 85.14

F1 ↑ 83.06 84.82 84.47 84.65 84.51 85.01

AMBER Evaluation. AMBER [46] is a human-annotated benchmark collected from the MSCOCO test set and “UnSplash” website. The benchmark comprises a generative task with 1, 004 questions and a discriminative task with 14, 216 questions, evaluated by CHAIRI (CI ) and accuracy (ACC), respectively. All these questions are designed to assess three categories of hallucination: existence, attributes, and relations. As shown in Table 3, PTI outperforms the vanilla decoding strategies across all LVLMs consistently. These results demonstrate the effectiveness of PTI in mitigating object hallucinations and enhancing the model’s discriminative understanding ability.

4.4. Results on Comprehensive Benchmarks MMHAL Evaluation. MMHal-Bench [35] is a specialized benchmark that assesses LVLM hallucinations using 96 meticulously crafted image-question pairs spanning 8

Figure 4. Performance comparison on MMHal-Bench, with results disaggregated by its eight question categories: attributes (ATTR), adversarial objects (ADV), comparisons (COMP), counting (COUNT), spatial relations (SPAT), environmental inference (ENV), holistic descriptions (HOL), and others (OTHER). All the model responses are evaluated using GPT-5 for alignment with ground-truth answers.

distinct categories. It emphasizes logical reasoning and complex visual understanding, and uses GPT-assistance to rate the model’s responses. Figure 4 presents the experimental results comparing PTI with PAI [30] and VTI [29] based on greedy decoding. Evidently, PTI delivers significant gains in average scores while concurrently achieving substantial reductions in hallucination rates. These improvements are particularly pronounced for the LLaVA-1.5 and Qwen-VL-Chat. Furthermore, we observe that PTI exceptionally excels in challenging tasks, such as counting (COUNT), spatial relations (SPAT), and attributes (ATTR). This is attributed to the fact that these categories fundamentally require a fine-grained capture of object details, which is essential for accurate responses. In summary, these results demonstrate the effectiveness of PTI in mitigating a broader range of hallucination challenges. Table 4. Evaluation of MME Benchmark. “LLAVA.”: LLAVA1.5, “Qwen.”: Qwen-VL-Chat, “DeepSeek.”: DeepSeek-VL-Chat. Method Vanilla PAI [30] VTI [29] VISTA [23] PTI (ours)

LLAVA. Acc ↑ 611.6 625.0 633.3 615.0 651.6

∆ − ↑ 13.4 ↑ 21.7 ↑ 3.4 ↑ 40.0

Qwen. Acc ↑ 598.3 605.0 626.6 611.6 638.3

∆ − ↑ 6.7 ↑ 28.3 ↑ 13.3 ↑ 40.0

DeepSeek. Acc ↑ 651.6 656.6 661.6 646.6 671.6

∆ − ↑ 5.0 ↑ 10.0 ↓ 5.0 ↑ 20.0

MME Evaluation. The MME benchmark [57] is designed to assess LVLM performance across multiple dimensions comprehensively. We mainly focus on its cognition-related subsets, which specifically address two types of hallucinations: object-level (existence and count) and attribute-level (position and color). As shown in Table 4, we report the performance across all tasks is measured using accuracy, based on greedy decoding. Obviously, PTI achieved the best total performance, while VTI [29] also demonstrated competitive results. This significant performance gap relative to other methods underscores the critical role of enhancing visual signals in boosting the model’s perceptual capabilities. The complete table is detailed in Appendix B.

5. Analysis and Discussions 5.1. Ablation Study and Error Analysis To validate our modality-specific intervention design, we conduct ablation studies both on the intervention modalities and positions. The results in Table 5 reveal a clear, modality-dependent dichotomy: Textual intervention is more effective when precise, steering the final generative state, while visual intervention yields optimal results when applied globally to enhance the entire visual initial state. Specifically, the Visual intervention is the most crucial component for hallucination reduction, driving significant performance improvements. However, it also induced a drop in the F1 score. This suggests that an overemphasis on the visual modality may bias the model towards fine-grained details, potentially at the expense of degraded generation quality. In summary, our final PTI method pursues this tradeoff. The additional precise textual intervention not only complements the visual concepts but also recovers the F1 score. This confirms that PTI successfully combines these two strategies, achieving the best overall performance. Table 5. Ablation study for intervention modality and position on LLAVA-1.5. “CS ”: CHAIRS , “CI ”: CHAIRI , “F1”: F1-score. Modality

Position

Textual Visual Last token All tokens × × × × ✓ × ✓ × ✓ × × ✓ × ✓ ✓ × × ✓ × ✓ PTI: Last textual token & All visual tokens

CS ↓

CI ↓

F1 ↑

47.4 40.8 45.2 41.2 16.8 15.4

13.7 12.0 14.3 12.4 6.2 5.4

75.3 76.5 75.6 76.4 70.3 72.7

5.2. Internal Interpretability Analysis Given the critical role of visual intervention shown in Section 5.1, we further conduct an internal interpretability analysis. Specifically, we decouple the interventions on the key and value caches to analyze their respective contributions. Value Cache Intervention. As shown in the left panel of Figure 5 (a), directly perturbing the original images via ran-

(a) Visual Value Cache Intervention

(b) Visual Key Cache Intervention

Figure 5. Internal interpretability analysis of visual cache intervention on LLAVA-1.5 across 300 randomly selected images from MSCOCO. (a): Ablation study on value intervention strategies, validating that the maximal contrast (object vs. background) yields the largest hallucination reduction. (b): Analysis of the key cache intervention, demonstrating its dual effect: mitigating global visual attention decay during generation (left) and enhancing local, object-centric attention (right). The change rate formula is detailed in Appendix A.

dom masking leads to a severe exacerbation of hallucinations. We attribute these results to the fact that random masking increases the risk of losing object signals. Therefore, a straightforward strategy is to leverage the contrastive directions derived from the original and perturbed images to enhance model robustness. However, the efficacy of this strategy appears saturated with no clear differences between masking ratios. These results motivated us to search for a more principled and potent contrast. Our intuition is that the contrast between the isolated object and its background yield the maximal object-centric signal. The right panel of the Figure 5 (a) validates this point of view, and achieves a maximal hallucination reduction (−9.7% of CHAIRI ), significantly outperforming all random-mask-based contrast and confirming our methodological choice. Key Cache Intervention. We further analyzed the impact of our visual key cache intervention in Figure 5 (b). The left part shows that LVLMs’ attention proportion allocated to the overall visual tokens progressively decays during generation. In contrast, our intervention consistently mitigates this decay. Critically, the relative change rate demonstrates that the positive impact of our intervention becomes increasingly pronounced as the inference phase progresses. Beyond this global improvement, we verified that this enhanced visual attention is meaningfully focused. The right part visualizes a targeted shift in attention toward objectlevel details across layers and heads (attention maps are visualized in Appendix C). Overall, these results validate that our intervention not only preserves global visual grounding but also effectively guides attention toward local details.

5.3. Generalizability Analysis Generalizability between Different LVLMs. We first evaluate the cross-model generalizability of our PTI to determine if the extracted steering vectors capture modelagnostic mechanisms for mitigating hallucinations. As shown in Table 6, applying PTI derived from one to another yields accuracy improvements compared to the vanilla.

Table 6. Cross-models and Comb-methods generalization study. Performance on POPE Adversarial subset. “L ⇐⇒ Q”: We restrict our cross-model generalization analysis to LLaVA-1.5 and Qwen-VL-Chat, as they share identical KV cache dimensions. Setting Cross LVLMs Integrated Methods

Method Vanilla PTI (L ⇐⇒ Q) PAI [30] PAI w/ PTI VISTA [23] VISTA w/ PTI

LLAVA. Acc. ↑ 75.40 75.53 76.93 78.76 75.53 76.33

∆ − ↑ 0.13 − ↑ 1.83 − ↑ 0.80

Qwen. Acc. ↑ 80.26 81.47 81.63 82.30 80.80 80.96

∆ − ↑ 1.21 − ↑ 0.67 − ↑ 0.16

DeepSeek. Acc. ↑ 82.73 − 84.03 84.60 84.80 85.56

∆ − − − ↑ 0.57 − ↑ 0.76

This positive transfer suggests that PTI captures representational properties related to objects that possess a degree of universality. This finding underscores the potential applicability of PTI beyond the architecture of the source model. Generalizability integrated with other methods. Furthermore, we investigate the complementarity of PTI by integrating it with other methods operating at different stages. Table 6 shows consistent and additive performance gains across all three diverse LVLMs. These synergistic improvements indicate that PTI operates through an orthogonal mechanism distinct from existing methods, highlighting PTI’s value not only as a standalone technique but also as a computationally efficient and effective module.

6. Conclusion In this paper, we propose a novel multi-modal steering paradigm, PTI, to mitigate hallucination in LVLMs. Unlike existing decoding-stage methods, PTI shifts the intervention to the prefill stage to overcome potential progressive error cascades. PTI precisely targets the initial KV cache with modality-aware and position-sensitive corrections, enabling decoupled Key and Value interventions to simultaneously enhance object-centric attention and robustness to background noise. Extensive experiments demonstrate that PTI not only achieves state-of-the-art performance across diverse benchmarks and LVLMs, but also is orthogonal to existing methods.

Acknowledgements This work was supported by the Natural Science Foundation of China under Grant 62571507.

References [1] Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433–459, 2010. 4 [2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 1 [3] Wenbin An, Feng Tian, Sicong Leng, Jiahao Nie, Haonan Lin, QianYing Wang, Ping Chen, Xiaoqin Zhang, and Shijian Lu. Mitigating object hallucinations in large visionlanguage models with assembly of global and local attention. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29915–29926, 2025. 1 [4] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023. 1, 2, 3, 5 [5] Max Belitsky, Dawid J Kopiczko, Michael Dorkenwald, M Jehanzeb Mirza, Cees GM Snoek, and Yuki M Asano. Kv cache steering for inducing reasoning in small language models. arXiv e-prints, pages arXiv–2507, 2025. 2, 3 [6] Jinhe Bi, Yujun Wang, Haokun Chen, Xun Xiao, Artur Hecker, Volker Tresp, and Yunpu Ma. Llava steering: Visual instruction tuning with 500x fewer parameters through modality linear representation-steering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15230– 15250, 2025. 2 [7] Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Linfeng Zhang, Lijie Wen, and Xuming Hu. Ict: Image-object cross-level trusted intervention for mitigating object hallucination in large vision-language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 4209–4221, 2025. 1, 2, 3 [8] Long Chen, Oleg Sinavski, Jan Hünermann, Alice Karnsund, Andrew James Willmott, Danny Birch, Daniel Maund, and Jamie Shotton. Driving with llms: Fusing object-level vector modality for explainable autonomous driving. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 14093–14100. IEEE, 2024. 1 [9] Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023. 1 [10] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 3, 4

[11] Yu Fu, Zefan Cai, Abedelkadir Asi, Wayne Xiong, Yue Dong, and Wen Xiao. Not all heads matter: A head-level KV cache compression method with integrated retrieval and reasoning. In The Thirteenth International Conference on Learning Representations, 2025. 2 [12] Woody Haosheng Gan, Deqing Fu, Julian Asilis, Ollie Liu, Dani Yogatama, Vatsal Sharan, Robin Jia, and Willie Neiswanger. Textual steering vectors can improve visual understanding in multimodal large language models. arXiv preprint arXiv:2505.14071, 2025. 4 [13] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multimodal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13418–13427, 2024. 5, 6 [14] Wei Huang, Xudong Ma, Haotong Qin, Xingyu Zheng, Chengtao Lv, Hong Chen, Jie Luo, Xiaojuan Qi, Xianglong Liu, and Michele Magno. How good are low-bit quantized llama3 models? an empirical study. CoRR, 2024. 1 [15] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 3 [16] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015–4026, 2023. 3 [17] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13872–13882, 2024. 3, 5, 6, 13, 14 [18] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Fanyi Pu, Joshua Adrian Cahyono, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. 1 [19] Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems, 36:41451–41530, 2023. 3 [20] Qiming Li, Zekai Ye, Xiaocheng Feng, Weihong Zhong, Libo Qin, Ruihan Chen, Baohang Li, Kui Jiang, Yaowei Wang, Ting Liu, et al. Cai: Caption-sensitive attention intervention for mitigating object hallucination in large visionlanguage models. arXiv preprint arXiv:2506.23590, 2025. 2, 12 [21] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 1, 3, 5 [22] Yichen Li, Zhiting Fan, Ruizhe Chen, Xiaotang Gai, Luqi Gong, Yan Zhang, and Zuozhu Liu. FairSteer: Inference

time debiasing for LLMs with dynamic activation steering. In Findings of the Association for Computational Linguistics: ACL 2025, pages 11293–11312, Vienna, Austria, 2025. Association for Computational Linguistics. 2 [23] Zhuowei Li, Haizhou Shi, Yunhe Gao, Di Liu, Zhenting Wang, Yuxiao Chen, Ting Liu, Long Zhao, Hao Wang, and Dimitris N Metaxas. The hidden life of tokens: Reducing hallucination of large vision-language models via visual information steering. arXiv preprint arXiv:2502.03628, 2025. 1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15 [24] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014. 3, 4, 5, 12 [25] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023. 1 [26] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 1, 2 [27] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024. 1, 2, 5 [28] Luyang Liu, Jonas Pfeiffer, Jiaxing Wu, Jun Xie, and Arthur Szlam. Deliberation in latent space via differentiable cache augmentation. arXiv preprint arXiv:2412.17747, 2024. 3 [29] Sheng Liu, Haotian Ye, Lei Xing, and James Zou. Reducing hallucinations in vision-language models via latent space steering. arXiv preprint arXiv:2410.15778, 2024. 1, 2, 3, 4, 5, 6, 7, 13, 14, 15 [30] Shi Liu, Kecheng Zheng, and Wei Chen. Paying more attention to image: A training-free method for alleviating hallucination in lvlms. In European Conference on Computer Vision, pages 125–140. Springer, 2024. 5, 6, 7, 8, 12, 13, 14 [31] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Hao Yang, et al. Deepseek-vl: towards real-world visionlanguage understanding. arXiv preprint arXiv:2403.05525, 2024. 1, 2, 3, 5 [32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PmLR, 2021. 3 [33] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156, 2018. 1, 2, 3, 5, 14 [34] Deepak Sridhar, Kartikeya Bhardwaj, Jeya Pradha Jeyaraj, Nuno Vasconcelos, Ankita Nayak, and Harris Teague. Video reasoning without training. arXiv preprint arXiv:2510.17045, 2025. 2 [35] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui,

Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023. 1, 5, 6 [36] Wei Suo, Lijun Zhang, Mengyang Sun, Lin Yuanbo Wu, Peng Wang, and Yanning Zhang. Octopus: Alleviating hallucination via dynamic contrastive decoding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 29904–29914, 2025. 1, 3 [37] Feilong Tang, Chengzhi Liu, Zhongxing Xu, Ming Hu, Zile Huang, Haochen Xue, Ziyang Chen, Zelin Peng, Zhiwei Yang, Sijin Zhou, et al. Seeing far and clearly: Mitigating hallucinations in mllms with attention causal decoding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 26147–26159, 2025. 2, 3, 12 [38] Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503.19786, 2025. 1 [39] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 1, 2 [40] Dezhan Tu, Danylo Vashchilenko, Yuzhe Lu, and Panpan Xu. Vl-cache: Sparsity and modality-aware kv cache compression for vision-language model inference acceleration. arXiv preprint arXiv:2410.23317, 2024. 2 [41] Yuli Vasiliev. Natural language processing with Python and spaCy: A practical introduction. No Starch Press, 2020. 4 [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 2 [43] Arthur Vogels, Benjamin Wong, Yann Choho, Annabelle Blangero, and Milan Bhan. In-distribution steering: Balancing control and coherence in language model generation. arXiv preprint arXiv:2510.13285, 2025. 2 [44] Zhongwei Wan, Ziang Wu, Che Liu, Jinfa Huang, Zhihong Zhu, Peng Jin, Longyue Wang, and Li Yuan. Look-m: Lookonce optimization in kv cache for efficient multimodal longcontext inference. arXiv preprint arXiv:2406.18139, 2024. 2 [45] Zhongwei Wan, Hui Shen, Xin Wang, Che Liu, Zheda Mai, and Mi Zhang. Meda: Dynamic kv cache allocation for efficient multimodal long-context inference. arXiv preprint arXiv:2502.17599, 2025. 2 [46] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Jiaqi Wang, Haiyang Xu, Ming Yan, Ji Zhang, and Jitao Sang. Amber: An llm-free multidimensional benchmark for mllms hallucination evaluation, 2024. 1, 3, 5, 6 [47] Mengyue Wang, Shuo Chen, Kristian Kersting, Volker Tresp, and Yunpu Ma. METok: Multi-stage event-based token compression for efficient long video understanding. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 18881–18895,

Suzhou, China, 2025. Association for Computational Linguistics. 2 [48] Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. Mitigating hallucinations in large vision-language models with instruction contrastive decoding. arXiv preprint arXiv:2403.18715, 2024. 3 [49] Zheng Wang, Boxiao Jin, Zhongzhi Yu, and Minjia Zhang. Model tells you where to merge: Adaptive kv cache merging for llms on long-context tasks. arXiv preprint arXiv:2407.08454, 2024. 2 [50] Chengyue Wu, Hao Zhang, Shuchen Xue, Zhijian Liu, Shizhe Diao, Ligeng Zhu, Ping Luo, Song Han, and Enze Xie. Fast-dllm: Training-free acceleration of diffusion llm by enabling kv cache and parallel decoding. arXiv preprint arXiv:2505.22618, 2025. 2 [51] Yuanchen Wu, Lu Zhang, Hang Yao, Junlong Du, Ke Yan, Shouhong Ding, Yunsheng Wu, and Xiaoqiang Li. Antidote: A unified framework for mitigating lvlm hallucinations in counterfactual presupposition and object perception. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14646–14656, 2025. 1 [52] Jianjun Xu, Yuxin Wang, Hongtao Xie, and Yongdong Zhang. Ote: Exploring accurate scene text recognition using one token. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28327– 28336, 2024. 4 [53] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. 1 [54] Dingkang Yang, Dongling Xiao, Jinjie Wei, Mingcheng Li, Zhaoyu Chen, Ke Li, and Lihua Zhang. Improving factuality in large language models via decoding-time hallucinatory and truthful comparators. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 25606–25614, 2025. 1 [55] Le Yang, Ziwei Zheng, Boxu Chen, Zhengyu Zhao, Chenhao Lin, and Chao Shen. Nullu: Mitigating object hallucinations in large vision-language models via halluspace projection. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14635–14645, 2025. 1, 2, 4 [56] Hao Yin, Guangzong Si, and Zilei Wang. Clearsight: Visual signal enhancement for object hallucination mitigation in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 14625–14634, 2025. 1, 3 [57] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12): nwae403, 2024. 1, 5, 7 [58] Qifan Yu, Juncheng Li, Longhui Wei, Liang Pang, Wentao Ye, Bosheng Qin, Siliang Tang, Qi Tian, and Yueting Zhuang. Hallucidoctor: Mitigating hallucinatory toxicity in visual instruction data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12944–12953, 2024. 1 [59] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training.

In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986, 2023. 3 [60] Muru Zhang, Ofir Press, William Merrill, Alisa Liu, and Noah A. Smith. How language model hallucinations can snowball. In Forty-first International Conference on Machine Learning, 2024. 2

Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models Supplementary Material A. Details of Internal Interpretability Analysis. In this section, we provide the mathematical formulation and calculation steps of the change rate of attention for the internal interpretability analysis presented in Section 5.2. Specifically, we randomly sample 300 images from the MSCOCO [24] dataset. For each image, we denote Atl,h ∈ RN as the last token attention distribution of the h-th head in the l-th layer at generation step t over the sequence of length N , Iimg represent the set of indices corresponding to visual tokens. Then, we quantitatively measure the rate of change in both 1) Global Visual Attention and 2) Local Object-Centric Attention before and after our intervention.

A.1. Global Visual Attention Dynamics To quantify how the model’s reliance on visual information evolves throughout the generation process, we employ a stage-wise attention analysis. This allows us to monitor the “attention decay” phenomenon [20, 30, 37] and verify whether our intervention effectively counteracts this trend. Visual Attention Proportion. First, for any given generation step t, we define the global visual attention proportion t Pimg as follows:

wise relative change rate ∆Rk is defined as follows: \Delta R^{k} = \frac {{\hat P}_{\text {img}}^{t_k} - P_{\text {img}}^{t_k}}{ \left | P_{\text {img}}^{t_k} \right | + \epsilon } \times 100\%.

Here, a positive ∆Rk indicates that the intervention effectively enhances the model’s visual grounding capabilities relative to the Vanilla. Crucially, by analyzing the trend of ∆R = {∆R0 , ∆R1 , . . . , ∆RK }, we can verify whether the intervention specifically counteracts the attention decay that typically occurs in the later stages of long-form generation.

A.2. Local Object-Centric Attention Shift Beyond preserving the global magnitude of visual signals, we further verify whether the enhanced visual attention is meaningfully focused on the relevant object regions rather than the background. Visual Attention Re-normalization. We first disentangle the spatial distribution from the total visual attention weight. For a specific layer l and head h at step t, we compute the conditional probability distribution Ãtl,h over the visual tokens: \tilde {\mathbf {A}}^{t}_{l,h}[v] = \frac {\mathbf {A}^{t}_{l,h}[v]}{\sum _{j \in \mathcal {I}_{\text {img}}} \mathbf {A}^{t}_{l,h}[j]}, \quad \forall v \in \mathcal {I}_{\text {img}}

P_{\text {img}}^{t} = \frac {1}{L \cdot H} \sum _{l=1}^{L} \sum _{h=1}^{H} \frac {\sum _{v \in \mathcal {I}_{\text {img}}} \mathbf {A}^{t}_{l,h}[v]}{\sum _{n\in N} \mathbf {A}^{t}_{l,h}[n] + \epsilon }

(9)

which represents the average probability mass allocated to visual tokens across all layers L and heads H. Here, ϵ is a small constant for numerical stability. Temporal Alignment via Staged Sampling. Since the total length of generated tokens N varies between the original (before) and intervened (after) models, we normalize the generation process into discrete progression stages to ensure a fair alignment. Specifically, we divide the generation into K discrete stages. For the k-th stage, we sample the proportion of visual attention at step tk as follows: t_k= \lfloor \frac {k}{K} \cdot N \rfloor , \quad k \in \{0, 1, \dots , K\}

(10)

which effectively captures the behavior of the model from the onset (0%) to the completion (100%) of the response. Global Relative Change Rate. To evaluate the impact of our intervention, we calculate the relative change in visual tk tk attention at each aligned stage. Let Pimg and P̂img denote the visual attention proportions for the vanilla (before) and intervened (after) models at stage k, respectively. The stage-

(11)

(12)

This re-normalization reflects the spatial focus of the head, given that it is attending to the image. Object-Centric Attention Score. Let M ∈ {0, 1}|Iimg | be the binary mask of the ground-truth object, where M [v] = 1 indicates token v falls within the object region. We calculate the aggregated attention mass allocated to the object as: S_{\text {obj}}^{t}(l, h) = \sum _{v \in \mathcal {I}_{\text {img}}} \tilde {\mathbf {A}}^{t}_{l,h}[v] \cdot M[v]

(13)

This metric (Sobj ) provides a granular view of the model’s spatial grounding capability at the head level. Local Targeted Shift Finally, to visualize the layer-wise and head-wise impact of our intervention, we compute the t absolute shift in object-centric attention: Let Sobj (l, h) and t Ŝobj (l, h) denote the proportions of object attention for the vanilla (before) and intervened (after) models at step t, ret spectively. The object-centric attention shift ∆Sobj (l, h) is defined as follows: \Delta S^t_{\text {obj}}(l, h) = \hat S_{\text {obj}}^{t}(l, h) - S_{\text {obj}}^{t}(l, h).

(14)

Table 7. More results on the random and popular splits of the POPE benchmark.

LLAVA-1.5 Method

Vanilla PAI [30] VCD [17] VTI [29] VISTA [23] PTI (ours)

Random Acc ↑ 83.74 85.73 85.36 83.40 84.74 85.43

Qwen-VL-Chat

Popular

F1 ↑ 84.54 86.32 86.00 84.14 85.78 85.72

Acc ↑ 80.50 82.53 82.60 81.50 79.97 83.80

Random

F1 ↑ 81.49 83.40 83.41 82.33 81.55 84.09

Acc ↑ 86.52 87.69 86.93 85.77 85.91 87.53

DeepSeek-VL-Chat

Popular

F1 ↑ 85.59 86.94 85.46 84.68 84.66 86.54

Acc ↑ 84.30 85.40 85.17 83.86 83.90 86.17

F1 ↑ 83.39 84.40 83.68 82.59 82.25 84.89

Random Acc ↑ 84.81 86.83 86.70 86.15 86.70 86.22

Popular

F1 ↑ 84.18 86.29 85.97 86.01 85.54 86.25

Acc ↑ 84.13 85.86 85.80 84.83 86.13 85.43

F1 ↑ 83.24 84.93 84.56 84.48 84.58 85.16

Table 8. Evaluation of MME Benchmark. “Exist.”: Existence. “Posit.”: Position. The maximum new token is set to 32. LLAVA-1.5

Method Vanila PAI [30] VTI [29] VISTA [23] PTI (ours)

Exist. 180.0 190.0 185.0 195.0 195.0

Count 143.3 148.3 153.3 138.3 163.3

Posit. 133.3 126.6 130.0 121.6 128.3

Color 155.0 160.0 165.0 160.0 165.0

Qwen-VL-Chat Total 611.6 625.0 633.3 615.0 651.6

Exist. 180.0 180.0 185.0 175.0 185.0

Count 115.0 120.0 145.0 125.0 140.0

A positive ∆Sobj (l, h) signifies that the h-th head in the l-th layer has successfully redistributed probability mass from the background to the object region. We visualize these shifts as heatmaps (see Figure 5 Right) to identify which specific components of the Transformer are responsible for the improved object grounding.

B. Additional Experiments of PTI. B.1. More Experimental Results of POPE. Table 7 presents the detailed performance on the Random and Popular splits of the POPE benchmark, supplementing the Adversarial results discussed in Section 4.3. Consistent with the performance in Table 2, PTI demonstrates robust generalization capabilities across varying difficulty levels. Specifically, in the Popular split, which challenges models with high-frequency objects prone to statistical language priors, PTI achieves superior performance, surpassing the Vanilla baseline by substantial margins (e.g., +2.60% F1 on LLaVA-1.5 and +1.50% F1 on Qwen-VL-Chat) and outperforming competitive baselines like PAI [30] and VCD [17]. In the Random split, where performance is generally saturated due to lower difficulty, PTI maintains competitive results, consistently ranking within the top-two across all evaluated LVLMs. Notably, PTI consistently outperforms both VTI [29] and VISTA [23], offering a significantly more effective steering paradigm.

B.2. Detailed Experimental Results of MME. Table 8 details the complete performance on the MME benchmark for evaluating object-level and attribute-level hallucination. Notably, PTI exhibits substantial gains in the

Posit. 128.3 130.0 116.6 141.6 148.3

Color 175.0 175.0 180.0 170.0 165.0

DeepSeek-VL-Chat Total 598.3 605.0 626.6 611.6 638.3

Exist. 190.0 190.0 195.0 185.0 195.0

Count 153.3 158.3 158.3 158.3 163.3

Posit. 133.3 133.3 133.3 128.3 138.3

Color 175.0 175.0 175.0 175.0 175.0

Total 651.6 656.6 661.6 646.6 671.6

most challenging fine-grained perception tasks, specifically “Count” and “Position” where standard models frequently struggle. While the competitive VTI method shows strength in specific attributes like color, PTI maintains a superior balance across all dimensions. This consistent superiority in spatially sensitive and quantitative metrics validates that our PTI effectively sharpens the model’s initial visual grounding, enabling it to better resolve distinct objects and their spatial relationships before the decoding phase begins.

B.3. Inference Efficiency Analysis. As shown in Table 9, we evaluate the inference efficiency of our method against competing approaches in terms of both latency and throughput. Existing methods often introduce significant computational overhead. Methods apply case-specific operations, such as VCD [17] and PAI [30], introduce significant overhead due to contrastive computations, nearly doubling the latency (e.g., ×1.99 for VCD on LLaVA-1.5). Consequently, their throughput suffers a substantial degradation, dropping to approximately 0.50× of the performance of vanilla. Additionally, DTI methods like VTI [29], which apply steering vectors across multiple steps, also incur notable costs (e.g., ×1.07, ×1.06, and ×1.08, respectively), resulting in a throughput decline lower than ×0.95. In contrast, PTI eliminates the need for sample-specific operations or multi-step interventions by modifying the initial KV cache only once. As a result, it incurs negligible latency overhead (lower than ×1.02 across all models), and sustains near-lossless throughput (over ×0.98 across all models). This consistent superiority highlights that PTI is a highly efficient, plug-and-play solution suitable for delay-sensitive real-world applications.

Table 9. Measure of Latency (ms/token) and Throughput (token/s) on CHAIR benchmark. All results use the Nucleus Sampling decoding strategy on a NVIDIA 4090 GPU.

LLAVA-1.5

Method Vanilla VCD [17] PAI [30] VISTA [23] VTI [29] PTI (ours)

ＫｅｙＣａｃｈＩｎｔｒｖｉｏＳｇ

Latency ↓ 19.52 (×1.00) 38.92 (×1.99) 37.62 (×1.93) 26.20 (×1.34) 20.84 (×1.07) 19.58 (×1.00)

Throughput ↑ 51.22 (×1.00) 25.69 (×0.50) 26.58 (×0.52) 38.17 (×0.74) 47.97 (×0.93) 51.06 (×0.99)

ＣＨＡＩＲｓＡｂｌａｔｉｏｎＭａｔｒｉｘ（↓）

３４．８

３．０（↓１２．６）

Qwen-VL-Chat Latency ↓ 20.55 (×1.00) 41.08 (×1.99) 42.07 (×2.04) 33.81 (×1.64) 21.86 (×1.06) 20.88 (×1.01)

ＣＨＡＩＲ，ＡｂｌａｔｉｏｎＭａｔｒｉｘ（↓）

２６．０

２０．２

１５．８

（↓ ２１．４）

（↓ ２７．２）

（↓ ３１．６）

４５

１１．３（↓ ２．４）

８．８

（↓ ４．９）

３５．０

２９．０

２１．０

１７．４

（↓ １８．４）

（↓ ２６．４）

（↓ ３０．０）

０．１

３９．２

３３．４

２２．０

１５．４

（↓８．２）

（↓１４．０）

（↓ ２５．４）

（↓ ３２．０）

０．０

４７．４

３８．２

２６．４

１７．８

３０２５

Latency ↓ 18.29 (×1.00) 36.38 (×1.98) 35.16 (×1.92) 26.79 (×1.46) 19.81 (×1.08) 18.56 (×1.02)

６．２

（↓７．１）

（↓７．５）

１３

７４．７

７５．８

７３．９

（↓ ０．６）

（个０．５）

（↓１．４）

８．７

（↓２．３）

１２．５

（↓５．０）

１０．４

６．８

５．９

（↓ ６．９）

（↓７．８）

７．０

５．４

（↓１．２）

（↓ ３．３）

（↓ ６．７）

（↓８．３）

１３．７

１１．６

８．２

６．１

（↓２．１）

（↓ ５．５）

（↓７．６）

０．２

０．４

０．６

１１１０

９８

７７

（↓４．９）

０．０

（↓９．２）０．２

（↓ ２１．０）（↓ ２９．６）

０．０

０．６

０．４

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

６

７６．４

７６．８

７５．５

（个１．１）

（个１．５）

（个０．２）

（↓１１．１）

８４．２

７４．５

８０．７

（↓１４．７）（↓ １８．２）

（↓ ２４．４）

９５

７１．５

７５

（↓ ３．８）

９１．２

８６．８

８２．８

７９．１

（↓７．７）

（↓ １２．１）

（↓ １６．１）

（↓１９．８）

９０

８５

７４

７２．７

７７．２

７７．３

７５．７

（个１．９）

（个２．０）

（个０．４）

（↓ ２．６）

７７．２

７５．５

（个１．９）

（１０．２）

７２．４

（↓ ２．９）

０．２

０．４

０．６

７３

９４．５

９０．７

８４．１

７９．５

（↓ ４．４）

（↓８．２）

（↓１４．８）

（↓１９．４）

９１．７

８５．０

８０．１

（↓７．２）

（↓１３．９）

（↓ １８．８）

０．２

０．４

０．６

７２

７５．３０．０

７１

８０

９８．９０．０

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

８７．８

７６

７

２０

Throughput ↑ 54.67 (×1.00) 27.49 (×0.50) 28.44 (×0.52) 37.33 (×0.68) 50.47 (×0.92) 53.89 (×0.99)

ＡｖｅｒａｇｅＧｅｎｅｒａｔｉｏｎＬｅｎｇｔｈ７０．４

１２

１１．４３５

Throughput ↑ 48.66 (×1.00) 24.34 (×0.50) 23.77 (×0.48) 29.57 (×0.60) 45.75 (×0.94) 47.90 (×0.98) Ｆ１ＡｂｌａｔｉｏｎＭａｔｒｉｘ（个）

６．６

４０

０．２（↓ １２．４）

DeepSeek-VL-Chat

７５

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

Figure 6. Ablation matrices for multi-modal KV cache intervention strength on LLAVA-1.5 with greedy decoding strategy. Brighter colors indicate better performance, while red boxes highlight the parameter combinations used in Table 1.

ＫｅｙＣａｃｈＩｎｔｒｖｉｏＳｇＫｅｙＣａｃｈＩｎｔｒｖｉｏＳｇ

ＣＨＡＩＲｓＡｂｌａｔｉｏｎＭａｔｒｉｘ（↓）

０．３

ＣＨＡＩＲ，ＡｂｌａｔｉｏｎＭａｔｒｉｘ（↓）

３８．８

３１．２

１５．８

７．０

（↓ ０．８）

（↓８．４）

（↓ ２３．８）

（↓ ３２．６）

４０

１１．９（↓ ０．１）

１０．０

（↓２．０）

４．７

（↓ ４．４）

（↓ ７．３）

１２

３５

０．２

４１．０

３４．４

２０．６

１０．８

（个１．４）

（↓ ５．２）

（↓１９．０）

（↓ ２８．８）

３０

３９．０

３５．２

（↓ ０．６）

（↓ ４．４）

（↓１６．４）

（↓ ２６．２）

３９．６

３６．６

２５．２

１５．４

２３．２

１３．４

１２．６（个０．６）

２０

１０．２

（↓１．８）

７．０

９．４

（↓５．０）

（↓ ２．６）

０．０

（↓ ３．０）

（↓１４．４）

（↓ ２４．２）

０．２

０．４

０．６

１０

１２．３

１０．９

８．１

７．２

（个０．３）

（↓１．１）

（↓３．９）

（↓ ４．８）

１２．０

１１．０

７．２

（↓１．０）

９．０（↓ ３．０）

（↓ ４．８）

０．２

０．４

０．６

０．０

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

７５．２

７１．０

（↓ ２．２）

（↓１．８）

（↓ ６．０）

６０．１

７６

（↓１６．９）

１０

８

一

７

８５．６

７９．８

（↓ ２０．８）（↓ ２６．６）

５９．７

１８．２

（↓ ４６．７）

（↓ ８８．２）

７５．２

７４．１

６４．７

（↓１．８）

（↓ ２．９）

（↓１２．３）

７０．１

７６．１

７５．８

７４．９

（↓ ０．９）

（↓１．２）

（↓２．１）

（↓６．９）

７７．０

７５．６

７５．０

（↓ １．４）

（↓ ２．０）

７０．２

（↓ ６．８）

０．２

０．４

０．６

７２

一

７０

９０

９２．１

８７．５

７５．７

（↓ １４．３）（↓１８．９）（↓ ３０．７）

６８

６６

９３．１

８８．７

８１．２

０．０

６２

６１．０

（↓１３．３）（↓ １７．７）（↓ ２５．２）（↓ ４５．４）

７６．５

６０．７

（↓１５．６）（↓ ２９．９）（↓ ４５．７）０．２

０．０

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

１０６．４

９０．８

８０

４９．８（↓ ５６．６）

６４

６

５

１００

７４

９

１５

０．０

７４．８

１１

２５

０．１

ＡｖｅｒａｇｅＧｅｎｅｒａｔｉｏｎＬｅｎｇｔｈ

Ｆ１ＡｂｌａｔｉｏｎＭａｔｒｉｘ（个）

７．６

一

７０

一

５０

６０

４０

３０

２０

０．６

０．４

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

Figure 7. Ablation matrices for multi-modal KV cache intervention strength on Qwen-VL-Chat with greedy decoding strategy. ＣＨＡＩＲｓＡｂｌａｔｉｏｎＭａｔｒｉｘ（↓）

０．３

ＣＨＡＩＲ，ＡｂｌａｔｉｏｎＭａｔｒｉｘ（↓）

２４．２

１８．０

１２．６

１２．０

（↓１．６）

（↓７．８）

（↓１３．２）

（↓ １３．８）

８．９２４

（个０．７）

１０．０

（↑１．８）

ＡｖｅｒａｇｅＧｅｎｅｒａｔｉｏｎＬｅｎｇｔｈ

Ｆ１ＡｂｌａｔｉｏｎＭａｔｒｉｘ（个）

７．６

４．７

（↓ ０．６）

（↓ ３．５）

１０

７１．６

７０．８

６６．８

（↓ ２．７）

（↓ ３．５）

（↓７．５）

５９．０

７４

（↓１５．３）

２３．２

０．１

０．０

１７．６

１６．６

１４．６

（↓８．２）

（↓９．２）

（↓１１．２）

２１．８

１９．２

１４．６

１６．８

（↓ ４．０）

（↓ ６．６）

（↓１１．２）

（↓９．０）

２５．８０．０

１９．８

１５．２

１５．８

（↓ ６．０）

（↓ １０．６）

（↓ １０．０）

０．２

０．４

０．６

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

２２２０

１８１６

１４

６．５

５．１

５．９

６．２

（↓１．７）

（↓ ３．１）

（↓２．３）

（↓２．０）

６．８

６．７

４．８

７．４

（↓１．４）

（↓１．５）

（↓ ３．４）

（↓ ０．８）

８

７

７２．９

７１．７

６５．４

（↓１．４）

（↓ ２．６）

（↓８．９）

５８．３

（↓１６．０）

８．２０．０

５．４

７．１

（↓１．１）

（↓ ２．８）

（↓１．１）

０．２

０．４

０．６

７．１

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

５

８６．２

９５．７

８１．７

７５．８

一

７０６８

１２０

１１０

１０９．８

７０．８

（↓１２．５）（↓ ２６．６）（↓ ４０．６）

（↓ ５１．５）

９８．６

６７．５

１００

６６

７５．１

７４．２

６６．５

（个０．８）

（↓ ０．１）

（↓７．８）

５７．４

（↓ １６．９）

－６４

１０９．５

８０．７

７４．３０．０

７４．８

６６．８

（个０．５）

（↓７．５）

（↓１５．９）

０．２

０．４

０．６

５８．４

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

６０５８

９０

（↓１２．８）（↓ ２３．７）（↓ ４１．６）（↓ ５４．８）

６２

６

１２

９６．６

（↓１２．０）（↓ ２５．７）（↓ ３６．１）（↓ ４６．５）

７２

９

０．２（↓ ２．６）

１１０．３

８０

１２２．３０．０

７９．６９９．０６９．３（↓ ２３．３）（↓ ４２．７）（↓５３．０）０．２

０．４

７０

０．６

ＶａｌｕｅＣａｃｈｅＩｎｔｅｒｖｅｎｔｉｏｎＳｔｒｅｎｇｔｈ

Figure 8. Ablation matrices for multi-modal KV cache intervention strength on DeepSeek-VL-Chat with greedy decoding strategy.

B.4. Ablation Experiments of Hyperparameters. To investigate the sensitivity of PTI to intervention intensity, we present the ablation matrices for key and value intervention strengths on CHAIR [33] benchmark across LLaVA-1.5 (Figure 6), Qwen-VL-Chat (Figure 7), and

DeepSeek-VL-Chat (Figure 8). As mentioned in Section 4.2, we unify the visual and textual coefficients by setting λk,img = λk,txt and λv,img = λv,txt to reduce the hyperparameter search space. A consistent trend is observable across these architecture-distinct models: the value cache intervention exerts a more dominant influence on hal-

Question：Please help me describe this image in detail.

Image Question：Please help me describe this image in detail.

Image

Answer (LLAVA-1.5): The image features a large brown horse standing in a grassy field, with a black and white cat nearby...

Attention (“horse”)

Attention (“cat”)

Answer (LLAVA-1.5): The image features a small bathroom with a white toilet situated next to a white sink....

Attention (“toilet”)

Attention (“sink”)

Answer (LLAVA-1.5 w/ PTI): The image features a large brown horse standing in a grassy field, ... , a black and white dog is walking towards the horse...

Attention (“horse”)

Attention (“dog”)

Answer (LLAVA-1.5 w/ PTI): The image features a white sink placed in a bathroom, with a clear glass shower head on the right side...

Attention (“sink”)

Attention (“shower”)

Figure 9. Visual analysis of cross-modal attention maps on LLAVA-1.5. For each sample, the hallucinated content is highlighted in red, while the correct content is highlighted in green. The attention map of the target token represents the magnitude of attention weights Answer (LLAVA-1.5 w/ PTI): The image features a Answer (LLAVA-1.5): The image features a black Question：Please helpaveraged assigned to image tokens, across all layers and heads. close-up view of a black suitcase, with the focus on me describe this image in detail.

suitcase with a pink tag attached to it. The suitcase is placed on a surface, possibly a bed or a table....

the tag attached to it....

this distribution. On one hand, PTI intensifies the attenlucination mitigation, as evidenced by the substantial retional weights on the correctly identified dominant object duction in CHAIRS /CHAIRI metrics corresponding to in(“horse”), successfully rectifying the model’s gaze. On the creases in value intervention strength. However, the maother hand, PTI ensures a more robust and stable visual reptrices also reveal a critical sensitivity to excessive steering; resentation, enabling the model to identify the “dog” corsimultaneously maximizing both key and value intervention strengths precipitates a noticeable degradation in F1 scores rectly. This dual effect—strengthening valid signals while correcting misaligned features—confirms that PTI operates and a sharp decline in average generation length, indicating Image Attention (“bed”) Attention (“table”) Attention (“suitcase”) Attention (“tag”) by enforcing precise object-centric attention, thereby elimia compromise in the model’s generation quality. Consequently, the configurations highlighted in red boxes reprenating object-level hallucinations at their source. sent an optimal equilibrium, effectively minimizing halluC.2. Qualitative Examples. cination rates via robust visual grounding while preserving the fidelity and completeness of the textual response. We further extend our qualitative evaluation across all three distinct architectures to demonstrate PTI’s effectiveness in reducing hallucination. Figures 10-12 present comparative C. Additional Case Studies examples between vanilla models, DTI methods (i.e., VTI [29] and VISTA [23]), and our PTI for LLAVA-1.5, QwenC.1. Attention Map Visualization. VL-Chat, and DeepSeek-VL-Chat, respectively. As evident To intuitively understand how PTI mitigates hallucinations, across these scenarios, while vanilla models and DTI methwe visualize the cross-modal attention maps of the generods frequently suffer from severe object hallucinations and ated tokens with respect to the visual features. As illustrated context misinterpretation, PTI effectively suppresses the in Figure 9, PTI substantially enhances the object-centric atgeneration of non-existent entities and erroneous attributes. tention and robust visual recognition of LLAVA-1.5. Taking These qualitative examples demonstrate the superior abilthe first example as prominent cases, the vanilla model sufity of PTI to mitigate hallucination in large vision-language fers from severe perceptual misalignment and erroneously models by purifying initial representations before decoding. hallucinates a “cat”. In contrast, PTI effectively corrects

LLAVA-1.5 The image features a skateboarder performing a trick on a ramp at a skate park. The skateboarder is in the middle of the trick, with the skateboard flipping upside down. Another person is sitting nearby, watching the skateboarder's impressive display.\n\nThere are a few other objects in the scene, including a bottle located near the left side of the image, and a backpack placed on the ground to the left of the skateboarder. Additionally, there is a chair situated in the background, slightly to the right of the skateboarder.

w/ VTI The image features a skateboarder performing a trick on a ramp at a skate park. The skateboarder is in the middle of the trick, with their skateboard flipping in the air. Another person is sitting nearby, watching the skateboarder's performance. \n\nThere are two bottles in the scene, one located near the skateboarder and the other further away. Additionally, there is a backpack placed on the ground, possibly belonging to one of the skateboarders.

w/ VISTA The image depicts a pool with two people in it..One person is on a skateboard, performing a trick in the pool, while another person is sitting on a bench. The skateboarder is in the middle of a trick, and the other person is in the background. There are two bottles in the scene, one on the left side and another on the right side. The scene also features a skateboard and a bench in the pool.. .

w/ PTI (ours) Question: Please help me describe this image in detail.

The image features a skateboarder performing a daring trick in mid-air, showcasing his skill and balance. The skateboarder is the main focus of the scene, with his skateboard clearly visible beneath him.\n\nIn the background, there is another person who appears to be watching the skateboarder's performance. The scene takes place in a skate park, where skateboarders often practice and perform their tricks.

LLAVA-1.5 The image features a large gray cat sitting on a desk, occupying a significant portion of the space. The cat is positioned near a computer keyboard, which is placed on the desk. There are also several books scattered around the desk, with some placed near the cat and others further away.\n\nIn addition to the cat and the books, there is a TV in the background, located on the right side of the desk. A refrigerator can be seen in the far right corner of the image, indicating that the desk is likely situated in a living or dining area.

w/ VTI The image features a gray cat sitting on a desk, likely on a computer desk. The cat is positioned near a computer keyboard and appears to be looking at the camera. The desk also has a book and a cup placed on it. The cat seems to be relaxed and comfortable in its environment.",

w/ VISTA

Question: Please help me describe this image in detail.

The image features a cat sitting on a desk. The cat is in a relaxed position, and it appears to be a gray cat. The desk is a workspace with a computer and a printer. The cat is in a comfortable position on the desk, and it is a gray cat.

w/ PTI (ours) The image features a large, fluffy gray cat sitting on a desk, occupying a significant portion of the scene. The cat appears to be looking directly at the camera, capturing the viewer's attention. The cat is positioned in the center of the image, with its body stretched out and filling the frame.

Figure 10. Qualitative examples of LLAVA-1.5. Hallucinated contents are marked in red.

Qwen-VL-Chat In the image, a man is standing in a room holding a purple piece of luggage over his head. He appears to be struggling with the luggage, as it seems too large for him to lift easily. The man is standing in what seems to be an office or a similar professional setting.\n\nThe room has several chairs placed around the area, along with a couple of fire extinguishers located in different spots for safety. There is also a clock on the wall, which might serve as a timekeeping tool for the office.

w/ VTI The image features a man standing in a room with a piece of purple luggage on a tripod. The man appears to be holding the luggage or its handle, which is elevated since it is on a tripod. \n\nThe room has a few other pieces of furniture, including a chair towards the right side of the scene, and a bench in the far right corner. There is also a handbag placed on the floor near the left side of the room.

w/ VISTA A man is holding a purple case above his head. The man is wearing a gray shirt and blue jeans. The case appears to be a carrying case for a professional hair styling product. The man is in a room with a few other people, one of whom is a woman. A fire extinguisher is present in the room, and there is a bench in the background. A few other items are present in the scene, including a couple of handbags and a book. A dining table is also visible in the room. The man and other people are standing around the room, which appears to be a combination of a break room, a storage area, and a room for displaying a few items.

w/ PTI (ours) Question: Please help me describe this image in detail.

In this image, a man is standing and holding a purple object above his head. The object appears to be a piece of luggage or a bag, and the man is holding it using a shoulder strap. \n\nThere are a few other people in the scene, but they are located further away and are not actively interacting with the man or the luggage. Overall, the focus of the image is on the man holding the bag and the strap above his head.

Qwen-VL-Chat The image features a woman sitting in a comfortable chair, holding a cup in her hands. She appears to be enjoying a relaxing moment, possibly sipping on a beverage while reading the newspaper. \n\nThe room has a vase placed on a side table near the chair, adding a touch of decoration to the space. The woman is wearing jeans and a vest, and she is seated on a wooden rocking chair. The scene is a cozy one, with the woman taking some time for herself to relax and enjoy her cup of tea or coffee while reading the newspaper.

w/ VTI The image features a woman sitting comfortably in a chair, holding a newspaper. She appears to be enjoying her time as she reads the newspaper while sitting in the chair. \n\nThe chair is positioned in a room, with a vase placed nearby on a small table. The table holds another vase, adding a decorative touch to the scene. The room also has a door, which is open, and a window nearby. The window is covered by blinds, providing the woman with some shade or privacy as she reads her newspaper.

w/ VISTA A woman is sitting in a chair with a newspaper. She has a cup in her hand and a flower in the room. A clock is visible in the background. A person is wearing a watch on their left hand. A small dog is present in the scene, and a book is on the table. A vase is placed on a side table near the chair. The room has a window with a blind and a curtain. The chair is on a carpet. The woman is wearing a vest and jeans. The image has a close-up view of a person in a chair.

Question: Please help me describe this image in detail.

w/ PTI (ours) In the image, a woman is sitting in a comfortable chair, holding a newspaper. She appears relaxed and is smiling as she reads the paper. The chair she is sitting in is brown and appears to be a rocking chair, adding to the sense of relaxation. \n\nNext to the woman, there is a small vase, possibly containing a flower, adding a touch of decoration to the scene. The overall atmosphere seems tranquil, with the woman enjoying her time reading the newspaper in a cozy setting.

Figure 11. Qualitative examples of Qwen-VL-Chat. Hallucinated contents are marked in red.

DeepSeek-VL-Chat In the image, there is a bird perched on a person's hand. The bird appears to be a cockatoo, characterized by its white feathers and distinctive red cheek patches. The person holding the bird is wearing a pink shirt. In the background, there is a blue chair and a whiteboard with some drawings on it. The setting seems to be an indoor environment, possibly an office or a classroom.

w/ VTI In this image, we see a white parrot perched on a person's hand. The parrot has a distinctive red patch on its head. The person is wearing a pink shirt. In the background, there's a blue chair and a whiteboard with some drawings on it. The setting appears to be an indoor room, possibly an office or a classroom, given the presence of the whiteboard.

w/ VISTA In the image, there is a white parrot with red patches around its neck and face, perched on someone's hand. The parrot has blue eyes and is looking straight ahead. The person holding the parrot is wearing a pink shirt and black gloves. In the background, there's a blue plastic chair and a whiteboard with some drawings on it. There are also some books on shelves in the background. The setting appears to be an office or a room with windows allowing natural light in.

w/ PTI (ours) Question: Please help me describe this image in detail.

In the image, a person is holding a white bird with a red patch on its neck. The bird is perched on the person's hand. The person is wearing a pink shirt. In the background, there is a blue object that appears to be a toy. The bird is the main focus of the image, and it is positioned in the center. The person's hand is visible at the bottom of the image, supporting the bird. The blue object in the background is located towards the right side of the image.

DeepSeek-VL-Chat The image showcases a plate of food with two prominent items: a salad and a skewered meat. The salad is a mix of various greens, including arugula, and is accompanied by shredded carrots. The skewered meat appears to be grilled and has distinct grill marks. The plate is placed on a table, and there's a fork visible on the right side of the plate. The overall setting suggests a meal ready to be enjoyed.

w/ VTI The image predominantly showcases a plate with food items. Initially, I identified seven instances of carrots. Interestingly, upon a closer look, there's an apparent misclassification - some of the identified carrots actually correspond to celery, while several vegetables have been mistakenly classified as carrots. Upon a thorough analysis, I discovered a total of seventeen pieces of celery and twenty-four pieces of broccoli. All the vegetables are located within the bounding box.

w/ VISTA

Question: Please help me describe this image in detail.

In the image, there are two meatballs placed on a plate. The meatballs are placed on a plate which also has a salad on it. The meatballs are brown in color and appear to be grilled. There are also some green leaves on the plate which could be part of the salad. There are also some carrots visible in the image. The plate is placed on a table which appears to be made of wood. There are also some other objects like a chair and a table visible in the image.

w/ PTI (ours) The image showcases a plate with two skewers of grilled meat, each skewer holding a piece of meat. The meat appears to be well-cooked with visible grill marks. Accompanying the meat on the plate is a fresh salad composed of arugula and shredded carrots. The salad provides a vibrant contrast to the grilled meat, with its green leaves and orange shreds. The plate is placed on a surface, and the overall presentation of the dish is appetizing.

Figure 12. Qualitative examples of DeepSeek-VL-Chat. Hallucinated contents are marked in red.

Record · ID 141541 · SHA-256 f762c66f4be2053f
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.