Conceptio › Archive › arXiv CS

arXiv CSopen access

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

knowledge-representation reasoning

artificial intelligence, reasoning, knowledge representation

arXiv:2605.14966v1 [cs.CV] 14 May 2026

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs Wei Ding∗

Yilin Li∗†

Yudong Zhang∗‡

Tsinghua University Beijing, China [email protected]

Tsinghua University, Tencent Beijing, China [email protected]

Ruobing Xie

Jiansheng Chen

Xingwu Sun

Tencent Beijing, China [email protected]

University of Science and Technology Beijing Beijing, China [email protected]

University of Macau Macau, China [email protected]

Yu Wang‡ Tsinghua University Beijing, China [email protected]

Abstract

1 Introduction

Large vision-language models (LVLMs) have achieved remarkable performance across diverse multimodal tasks, yet they continue to suffer from hallucinations, generating content that is inconsistent with the visual input. Prior work DHCP (Detecting Hallucinations by Cross-modal Attention Pattern) has explored hallucination detection from the perspective of cross-modal attention, but does not address hallucination mitigation. In this paper, we propose MHSA (Mitigating Hallucinations via Steered Attention), a lightweight framework that mitigates hallucinations by learning to correct cross-modal attention patterns in LVLMs. MHSA trains a simple three-layer MLP generator to produce corrected attention, guided by supervisory signals from the DHCP discriminator and the LVLM itself. During inference, MHSA mitigates both discriminative and generative hallucinations across various datasets and LVLMs by simply replacing the original cross-modal attention with the corrected one, without modifying any LVLM parameters. By extending cross-modal attention mechanisms from hallucination detection to hallucination mitigation, MHSA offers a novel perspective on hallucination research in LVLMs and helps enhance their reliability.

Large vision-language models (LVLMs) have demonstrated exceptional capabilities across a wide range of multimodal tasks, including visual question answering, image captioning, and visual reasoning [2, 3, 5, 19], building upon earlier large-scale visionlanguage pre-training efforts such as CLIP [23]. By integrating visual encoders with large language models (LLMs) through carefully designed projectors, LVLMs can effectively interpret and reason about visual content [1, 14, 33]. Despite these impressive achievements, a critical challenge persists: the issue of hallucinations, where models generate content that is inconsistent with or absent from the input visual information [15, 24], as further highlighted by recent hallucination and MLLM evaluation benchmarks such as M-HalDetect [10], HallusionBench [9], and MME [7]. Hallucinations in LVLMs manifest in diverse forms, including object hallucinations (incorrectly identifying objects), attribute hallucinations (misidentifying states, numbers, or actions of objects), and relational hallucinations (erroneously describing spatial or contextual relationships between objects) [15, 28]. These hallucinations severely undermine the reliability and trustworthiness of LVLMs in practical applications, making hallucination mitigation a pressing research priority. Existing approaches for mitigating hallucinations can be broadly categorized into two classes. Training-based methods involve fine-tuning the LVLM with reinforcement learning from human feedback (RLHF) or instruction tuning to reduce hallucinated outputs [17, 26, 27]. While effective, these methods require substantial computational resources and high-quality annotated data. Inference-time methods modify the decoding process to suppress hallucinations. However, these methods rely on handcrafted, fixed heuristic rules (e.g., manually designed penalty functions or static re-weighting schemes) that do not adapt to the specific hallucination signature of each input sample.

CCS Concepts • Security and privacy → Intrusion/anomaly detection and malware mitigation.

Keywords Large vision-language model, Hallucination Mitigation, Crossmodal Attention, Lightweight.

∗

Equal contribution. This work was completed during a visiting period at Tsinghua University. ‡ Corresponding authors. †

Wei Ding, Yilin Li, Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang

Is there a plane in the image?

No!

Is there a plane in the image?

(2) Fix Attention

(1) Detect Attention

Yes!

Hallucinated!

Figure 1: Schematic diagram of the MHSA mechanism. The LVLM first generates hallucinated responses, and the supervision signal from the DHCP discriminator guides the MHSA corrector to repair cross-modal attention, thereby achieving hallucination mitigation.

Recently, a line of research has explored the relationship between cross-modal attention patterns and hallucinations in LVLMs. Specifically, DHCP [32] reveals that the cross-modal attention weights assigned by generated text tokens to visual tokens exhibit significant and distinguishable differences between hallucinated and non-hallucinated samples. Based on this finding, DHCP trains a lightweight two-layer MLP detector that identifies hallucinations by monitoring cross-modal attention during inference, without any additional LVLM training or extra inference steps. However, DHCP has two key limitations: (1) it can only detect hallucinations but cannot mitigate them; (2) it operates solely at the sentence level and cannot detect hallucinations at the token level. Despite these limitations, the finding opens a promising avenue: if hallucinations can be detected through attention patterns, can they also be mitigated by learning to correct these patterns in a data-driven manner? In this paper, we propose MHSA (Mitigating Hallucinations via Steered Attention), a lightweight framework that bridges the gap between hallucination detection and mitigation by learning to correct cross-modal attention patterns, as shown in Fig. 1. Unlike existing attention manipulation methods (e.g., OPERA [11], PAI [20]) that apply fixed heuristic rules, MHSA is the first to learn sampleadaptive attention corrections through a data-driven training procedure. Specifically, MHSA trains a lightweight MLP-based generator 𝐺 that takes the original cross-modal attention A as input and produces a correction term ΔA, yielding the corrected attention A′ = A + ΔA. The pre-trained DHCP detector 𝐷 is repurposed as a token-level discriminator, providing attention-guidance supervision signals to steer the corrected attention toward nonhallucinatory patterns. The framework is further regularized to preserve the original attention structure and maintain output quality. Importantly, MHSA does not modify any parameters of the LVLM backbone; the lightweight generator and DHCP are trained, and it replaces the cross-modal attention during LVLM inference. We evaluate MHSA on several mainstream LVLMs, including Qwen2.5-VL, InternVL2-8B, and LLaVA-v1.5. Experimental results demonstrate that MHSA achieves stable and reliable hallucination

mitigation on discriminative tasks. By extending DHCP’s sentencelevel detector to the token level, we further evaluate MHSA on generative tasks, where the results show that MHSA also has the potential to mitigate hallucinations in open-ended generation. Our contributions can be summarized as follows: (1) We propose MHSA, the first framework that mitigates hallucinations in LVLMs by learning sample-adaptive corrections of cross-modal attention patterns. MHSA trains a lightweight three-layer MLP generator with a three-component training objective to produce datadriven corrections guided by a pre-trained hallucination detector, enabling effective and stable hallucination mitigation. (2) We extend MHSA to token-level attention correction for generative tasks by upgrading the attention-based detection to the token level, enabling MHSA to mitigate hallucinations not only in discriminative VQA tasks but also in open-ended image captioning. (3) Extensive experiments across multiple LVLMs and datasets demonstrate that MHSA consistently improves hallucination mitigation metrics on both discriminative and generative tasks. We further provide attention visualizations to validate the causal role of attention correction.

2 Related Work 2.1 Hallucination Evaluation in LVLMs Existing evaluation protocols can be broadly divided into discriminative and generative paradigms. Discriminative evaluation frames hallucination assessment as binary classification. POPE [15, 21] converts object presence into Yes/No questions (e.g., “Is there a [object] in this image?”) and defines three difficulty levels—Random, Popular, and Adversarial— that progressively test the model’s susceptibility to co-occurrence priors. AMBER [28] extends this paradigm beyond objects to encompass attribute and relational hallucinations. These benchmarks are efficient and easy to interpret, making them widely adopted for comparing mitigation methods. Generative evaluation assesses hallucinations in open-ended outputs. CHAIR [24] is a representative metric for image captioning that measures hallucinated objects at both the sentence level (CHAIR𝑠 ) and instance level (CHAIR𝑖 ). By examining whether generated object mentions match ground-truth annotations, CHAIR provides a fine-grained measure of hallucination severity in freeform text. These two paradigms pose fundamentally different challenges for mitigation methods: discriminative tasks require only a binary decision correction, whereas generative tasks demand tokenlevel intervention within an autoregressive sequence. MHSA is designed to address both paradigms through a unified token-level attention correction framework, enabling real-time mitigation during LVLM inference.

2.2

Hallucination Mitigation

Hallucination mitigation methods aim to reduce or eliminate hallucinated content in LVLM outputs. We identify two major research lines, each operating at a different level of the model pipeline. Training-based methods tackle the problem at the model weight level by fine-tuning LVLMs on curated data. RLHF-based

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

approaches [26, 27] leverage human feedback to align model outputs with factual content, while instruction tuning methods [17] construct high-quality training data designed to improve model faithfulness. These methods are effective because they directly adjust the model’s internal representations to favor factual outputs. However, they depend on substantial computational resources and large-scale, high-quality annotated data, and the learned improvements are tightly bound to the training distribution—hallucination patterns not represented in the training set may remain unaddressed. Moreover, each new model or domain may require a fresh round of expensive fine-tuning. Inference-time methods intervene at the output logit level or attention pattern level during inference, requiring no additional training. VCD [13] contrasts logits from the original and visually distorted inputs, amplifying tokens that genuinely depend on visual evidence. ICD [29] introduces instruction contrastive decoding, distorting the instruction rather than the image. HALC [4] further explores adaptive focal-contrast decoding to reduce object hallucination. These methods provide training-free mitigation; however, they approximately double inference cost due to multiple forward passes, and their effectiveness relies on the assumption that hallucinated and non-hallucinated tokens respond differently to the chosen distortion—an assumption that may not hold uniformly across all hallucination types. Additionally, OPERA [11] introduces over-trust penalty and retrospection-allocation mechanisms to penalize attention columns with abnormally high aggregation, which correlate with hallucination onset. PAI [20] re-weights attention to allocate more focus to image tokens. However, these methods employ heuristic rules (e.g., manually designed penalty functions) rather than learned corrections, and OPERA in particular requires complex backtracking during generation, introducing additional latency. Crucially, these methods modify attention patterns using fixed rules that do not adapt to the specific hallucination signature of each sample. Summary. Training-based methods are computationally expensive; contrastive decoding doubles inference cost; and attention manipulation methods rely on hand-crafted heuristics that do not adapt to per-sample hallucination signatures. MHSA bridges this gap by learning a lightweight generator that produces data-driven, sample-adaptive attention corrections, guided by a pre-trained hallucination detector, without modifying any LVLM backbone parameters.

2.3

Attention-Based Mechanisms in Trustworthy AI

A separate but closely related research line investigates how crossmodal attention patterns can be leveraged to improve trustworthy AI. In the field of adversarial attacks and defenses, prior work published at ACM MM (BNI track) has utilized cross-modal attention to achieve adversarial sample detection and defense. Specifically, PIP [30] extracts cross-modal attention from clean and adversarial samples on unrelated probe questions and trains a lightweight support vector machine to detect adversarial samples; F3 [31] achieves adversarial purification by injecting untrained purification noise into adversarial samples through a cross-modal attention guidance

mechanism. The success of PIP and F3 demonstrates the potential of cross-modal attention for improving trustworthy AI. In the field of hallucination detection, prior work also published at ACM MM utilizes cross-modal attention patterns. Specifically, DHCP [32] extracts the average attention weights from generated text tokens to each visual token across all LLM layers and attention heads from both hallucinated and non-hallucinated samples, and then trains a lightweight two-layer MLP detector on these attention patterns, achieving strong hallucination detection performance across discriminative tasks (e.g., POPE, AMBER) and generative tasks (e.g., COCO-Caption) without requiring additional LVLM training or extra inference steps. The success of DHCP raises a natural question: if hallucinations can be reliably detected through attention patterns, can they also be mitigated by directly correcting these patterns? MHSA is motivated precisely by this gap. We adopt DHCP as the detection backbone and repurpose it as a token-level discriminator that provides adversarial supervision for a learned generator, transforming a detection-only tool into a full mitigation framework, enabling hallucination mitigation in both discriminative and generative tasks— a capability that attention-based detection alone cannot provide.

3 Method In this section, we present MHSA, a lightweight framework for mitigating hallucinations in LVLMs via learned attention correction. We first review the preliminaries of cross-modal attention and hallucination detection (Sec. 3.1), then formulate the problem (Sec. 3.2), describe the MHSA framework in detail (Sec. 3.3), and finally discuss the extension to token-level correction for generative tasks (Sec. 3.6).

3.1

Preliminaries: Cross-Modal Attention and Hallucination Detection

We first review the cross-modal attention in LVLMs as defined by DHCP. Consider an LVLM consisting of a visual encoder 𝑓𝑉 , a projector 𝑓𝑃 , and an LLM 𝑓LLM . Given an image 𝑥𝑖 and a text prompt 𝑥𝑡 , the visual encoder generates features 𝑓𝑉 (𝑥𝑖 ), which are then (𝑙,ℎ) projected to obtain 𝑁 visual tokens fed into the LLM. Let A𝑞→𝑛 denote the attention weight assigned by output position 𝑞 to the 𝑛-th visual token at layer 𝑙 and head ℎ. For discriminative tasks such as POPE (Yes/No question answering), the model generates a single answer token, and DHCP defines the cross-modal attention as the attention weights from the first output token to all visual tokens: (𝑙,ℎ)

A𝑛

(𝑙,ℎ)

= A𝑞1 →𝑛 ,

𝑙 = 1, … , 𝐿, ℎ = 1, … , 𝐻 , 𝑛 = 1, … , 𝑁 ,

(1)

yielding a tensor of shape (𝐿, 𝐻 , 𝑁 ), where 𝐿 is the number of LLM layers and 𝐻 is the number of attention heads. For example, LLaVAv1.5-7B has (𝐿 = 32, 𝐻 = 32, 𝑁 = 576). For models with dynamic visual token counts such as Qwen2.5-VL-7B, DHCP standardizes the input resolution by resizing all images to 336×336 pixels before processing, which yields a fixed number of visual tokens (𝑁 = 144) across all samples. This ensures a consistent attention tensor shape for the MLP-based generator and detector.

Wei Ding, Yilin Li, Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang

For generative tasks, DHCP averages the cross-modal attention across all output tokens: (𝑙,ℎ)

A𝑛

𝑀

1 ∑ A(𝑙,ℎ) , 𝑀 𝑖=1 𝑞𝑖 →𝑛

𝑙 = 1, … , 𝐿, ℎ = 1, … , 𝐻 , 𝑛 = 1, … , 𝑁 ,

(2) This means that DHCP can only detect hallucinations at the sentence level and cannot identify which specific token is responsible for the hallucination. Furthermore, the cross-modal attention of hallucinating tokens may be diluted by averaging over the entire sentence, thereby reducing the effectiveness of hallucination detection. In our framework, we adopt DHCP as the hallucination detector 𝐷 , which operates on cross-modal attention patterns. The detector is a two-layer MLP with a hidden dimension of 128 and an output dimension of 2, representing hallucinatory and non-hallucinatory classes. Given the flattened cross-modal attention as input, 𝐷 outputs the probability of the sample being hallucinatory or nonhallucinatory:

𝐷 (A) = 𝐷𝑙2 (𝐷𝑙1 (flatten (A))) ∈ ℝ2 ,

(3)

where “flatten” refers to reshaping the cross-modal attention into a one-dimensional vector, while 𝐷𝑙1 and 𝐷𝑙2 represent the first and second layers of the detector 𝐷 , respectively. The first output corresponds to the non-hallucination score, and the second output corresponds to the hallucination score.

3.2

Problem Formulation

While attention-based hallucination detection methods can effectively detect hallucinations through cross-modal attention patterns, they do not provide the capability for hallucination mitigation. We aim to develop a method that can directly correct the attention patterns causing hallucinations. Formally, given a cross-modal attention A that is detected as hallucinatory by the detector (i.e., arg max 𝐷(A) = 1), our goal is to find an attention correction ΔA such that the corrected attention: A′ = A + ΔA

(4)

satisfies the following desiderata: (1) A′ is no longer classified as hallucinatory by 𝐷 , i.e., arg max 𝐷(A′ ) = 0 (2) ΔA is minimal to preserve the original semantics and avoid introducing new artifacts (3) The LVLM output generated with A′ maintains or improves quality compared to the output with A

3.3

MHSA: Attention Steering Framework

We propose MHSA, which employs a three-goal guided training strategy to learn the attention correction. The framework consists of a generator 𝐺 and a discriminator 𝐷 (the pre-trained hallucination detector). 3.3.1 Generator 𝐺 . The generator 𝐺 takes the original crossmodal attention A as input and produces the attention correction ΔA: ΔA = 𝐺(A), (5)

where 𝐺 is implemented as a three-layer MLP. The input attention tensor A ∈ ℝ𝐿×𝐻 ×𝑁 is first flattened into a vector of dimension 𝑑 = 𝐿 ⋅ 𝐻 ⋅ 𝑁 . All hidden layers in the MLP have a dimension of 512, with the ReLU activation function applied between each layer, and the final output dimension is the same as the input dimension. All linear layer weights are initialized with small values (𝒰(−10−5 , 10−5 )) and zero bias, ensuring that the initial correction ΔA is near-zero at the start of training. Since all input images are resized to a fixed resolution (see Sec. 3.1), 𝑁 is constant across all samples, ensuring that 𝐺 has a fixed architecture that can be trained across the entire dataset. The corrected attention is then added to the original attention, as shown in Eq. (4). We adopt a residual formulation A′ = A + ΔA rather than directly predicting A′ from scratch, so that the generator only needs to learn a small offset from the original cross-modal attention. This residual design preserves the pretrained LVLM’s attention structure as the starting point and confines the generator’s role to localized refinement. Together with the regularization loss in Eq. (7), it keeps A′ close to A, which we find empirically yields stable training and inference across all evaluated models and datasets. 3.3.2 Discriminator 𝐷 (Hallucination Detector). The discriminator 𝐷 is the pre-trained hallucination detector [32]. During MHSA training, 𝐷 evaluates whether the original attention A exhibits hallucinatory patterns. The parameters of 𝐷 are fine-tuned throughout the training process with a tiny learning rate, allowing it to serve as a nearly fixed yet slightly flexible supervisory signal that guides the generator to produce corrections moving attention patterns away from hallucination-associated regions in the detector’s learned feature space.

3.4

Training Objectives

The MHSA framework is optimized with three complementary loss functions, corresponding to the three desiderata in Sec. 3.2, as shown in Fig. 2. Detector-Guided Loss (ℒdg ). Inspired by the adversarial training paradigm [8], we design a detector-guided loss that drives the generator to produce corrections transforming hallucinated attention patterns into non-hallucinatory ones, as judged by the discriminator. For a hallucinatory sample with cross-modal attention A: ℒdg = − log 𝐷0 (A′ ) = − log 𝐷0 (A + 𝐺(A)),

(6)

where 𝐷0 (⋅) denotes the probability assigned to the nonhallucinatory class (class 0) by the discriminator. Minimizing ℒdg encourages the corrected attention to be classified as nonhallucinatory. Regularization Loss (ℒreg ). To prevent the generator from producing excessively large corrections that could disrupt the original attention semantics, we impose a regularization constraint on ΔA: (7) ℒreg = ‖ΔA‖22 = ‖𝐺(A)‖22 . This loss ensures that the correction remains small and localized, preserving the overall structure of the original cross-modal attention while making targeted adjustments to hallucination-inducing patterns. LVLM Output Quality Loss (ℒLVLM ). To ensure that the corrected attention leads to high-quality outputs from the LVLM, we

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

Training Phase Discriminator

Generator

🔥

ΔA

LVLM Forward Pass

❄

ℒLVLM

Push corrected attention toward non-hallucinatory.

⋅

❄

ℒD

ℒG = λdg ℒdg + λLVLM ℒLVLM + λreg ℒreg

LVLM Forward Pass

ℒdg = -logD0(A')

ℒdg

🔥

Training Samples

Loss Panel

Update D

ℒreg = ||ΔA||2

ℒreg

⋅

Update G

Inference Phase

⋅

Keep Corrections small and localized.

D checks if hallucinated.

Inputs

LVLM Forward Pass

Discriminator

ℒLVLM = CE(fLVLM(A'), ygt)

G corrects. Yes

Generator

ΔA

+ A'

LVLM Forward Pass

Outputs

Preserve output quality.

Figure 2: The MHSA method pipeline. Discriminator 𝐷 is responsible for detecting hallucinations, while Generator 𝐺 is responsible for correcting cross-modal attention, forming a two-stage “hallucination detection–mitigation” framework. include a loss term that measures the impact of the attention correction on the LVLM’s generation: ℒLVLM = CE(𝑓LVLM (A′ ), 𝑦gt ),

3.5

During inference, MHSA operates as illustrated in Fig. 2:

(8)

(1) Run the LVLM forward pass and extract the cross-modal attention A from the first output token over visual tokens (Eq. (1)). (2) Pass A through the pre-trained detector 𝐷 to determine whether the sample exhibits hallucinatory patterns. (3) If hallucination is detected, apply the generator: ΔA = 𝐺(A) and compute A′ = A + ΔA. (4) Replace the original cross-modal attention with A′ in the LVLM to produce the corrected output.

where CE(⋅, ⋅) denotes the cross-entropy loss, 𝑓LVLM (A′ ) represents the LVLM’s output distribution when using the corrected attention A′ , and 𝑦gt is the ground-truth label. This loss ensures that the attention correction not only reduces hallucination signatures but also maintains or improves the actual output quality. Overall Objective. The total training objective for the generator 𝐺 is: ℒtotal = 𝜆dg ⋅ ℒdg + 𝜆reg ⋅ ℒreg + 𝜆LVLM ⋅ ℒLVLM ,

(9)

where 𝜆reg and 𝜆LVLM are hyperparameters controlling the relative importance of the regularization and LVLM quality losses, respectively. At the same time, we train the discriminator 𝐷 using the original cross-modal attention A of the current sample and its hallucination label as supervision: ℒd = − [𝑦 log 𝐷1 (A) + (1 − 𝑦) log 𝐷0 (A)] ,

(10)

where 𝑦 ∈ {0, 1} is the hallucination label (𝑦 = 1 for hallucinatory, 𝑦 = 0 for non-hallucinatory), 𝐷1 (⋅) and 𝐷0 (⋅) denote the probabilities assigned to the hallucinatory and non-hallucinatory classes by the discriminator, respectively. Only the generator 𝐺 and the discriminator 𝐷 (with a tiny learning rate) are trained; the LVLM backbone remains frozen throughout.

Inference

Since both 𝐺 and 𝐷 are lightweight MLPs (a two-layer MLP with hidden dimension 128 for 𝐷 ; a three-layer MLP with hidden dimension 512 for 𝐺 ), the additional computation introduced by MHSA is marginal compared to the LVLM’s own inference cost involving billions of parameters. Importantly, MHSA does not modify any LVLM backbone parameters—only the external generator is trained offline.

3.6

Extension to Token-Level Correction for Generative Tasks

For generative tasks such as image captioning, hallucinations may occur at specific tokens within the generated sequence rather than affecting the entire response. To handle this, we extend MHSA to operate at the token level.

Wei Ding, Yilin Li, Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang

Instead of using a single attention tensor derived from one output position (as in Eq. (1) for discriminative tasks), we consider the cross-modal attention for each individual generated token 𝑚: (𝑙,ℎ)

A𝑚 = A𝑞𝑚 →𝑛 ,

𝑙 = 1, … , 𝐿, ℎ = 1, … , 𝐻 , 𝑛 = 1, … , 𝑁 ,

(11)

where 𝑞𝑚 denotes the output position corresponding to the 𝑚-th generated token. Each A𝑚 is a tensor of shape (𝐿, 𝐻 , 𝑁 ), capturing the cross-modal attention pattern specific to the generation of that token. To enable token-level hallucination detection, we upgrade the original sentence-level DHCP detector to a token-level detector. The attention training samples for the token-level detector are obtained as follows: we let the LVLM perform image captioning on training images and simultaneously extract the per-token crossmodal attention. For each generated caption, we detect all nouns in the generated text. For each noun, we check whether it appears in the CHAIR object whitelist [24]; if it does, we further verify whether it constitutes a hallucination (i.e., whether the object is absent from the ground-truth annotations). Nouns that are present in the whitelist and correctly describe objects in the image are labeled as non-hallucinated attention samples, while those that are hallucinated are labeled as hallucinated attention samples. The corresponding per-token cross-modal attention patterns are then used to train the token-level detector and the token-level generator 𝐺 :

ΔA𝑚 = 𝐺(A𝑚 ).

(12)

During generative inference, MHSA operates at the decoding steps covered by the CHAIR protocol, before the corresponding token is sampled. At such a step 𝑚, we first run a standard LVLM forward pass to obtain the cross-modal attention A𝑚 for the candidate token at position 𝑚. The token-level discriminator 𝐷 then evaluates whether A𝑚 exhibits a hallucinatory pattern. If so, the generator produces a correction ΔA𝑚 = 𝐺(A𝑚 ), and the corrected attention A′𝑚 = A𝑚 + ΔA𝑚 replaces A𝑚 in the LVLM forward pass to recompute the output logits, from which the token at position 𝑚 is then sampled.

Experiment

In this section, we evaluate the effectiveness of MHSA on both discriminative and generative hallucination mitigation tasks. We first describe the experimental setup (Sec. 4.1), then present the main results on discriminative POPE tasks (Sec. 4.2), followed by dataset generalization (Sec. 4.3), model generalization (Sec. 4.4), cross-dataset out-of-distribution generalization (Sec. 4.5), generative captioning results (Sec. 4.6), and ablation studies (Sec. 4.7).

4.1

Experimental Setup

4.1.1 Models. We conduct experiments on three representative LVLMs to demonstrate the generality of MHSA: Qwen2.5-VL7B [2], InternVL2-8B [3], and LLaVA-v1.5-7B [18, 19]. For models with variable visual token numbers, Qwen2.5-VL and InternVL have visual token sizes set to 144 and 256, respectively. For each model, the hallucination detector 𝐷 is pre-trained on the corresponding training split’s cross-modal attention data following the protocol described in [32]. The generator 𝐺 is a three-layer MLP with hidden dimension 512.

Table 1: POPE results on MSCOCO (Qwen2.5-VL-7B, 𝑁 =3000). Best results in bold.

Method

Accuracy

Precision

Recall

Yes%

Baseline MHSA

86.83 92.77

95.27 92.54

77.62 93.40

85.55 92.97

40.9 50.5

+5.94

−2.73

+15.78

+7.42

+9.6

4.1.2 Datasets. For POPE tasks, we evaluate on four datasets: MSCOCO [16], Objects365 [25], OpenImagesV7 [12], and ImageNet [6]. POPE questions are generated following the POPE framework [15] with three evaluation clusters: Random, Popular, and Adversarial. For captioning tasks, we evaluate on two datasets: MSCOCO [16] and Flickr30k [22], assessed using the CHAIR metric [24]. 4.1.3 Evaluation Metrics. For discriminative tasks, we report Accuracy, Precision, Recall, F1-score, and Yes Ratio. The F1-score is computed for the “Yes” label following the POPE convention. In some cases, overtrained models may produce invalid outputs which cannot be assigned to any of the standard classification categories, i.e., TP, TN, FP, or FN. To address this, we report accuracy; these unresolved samples are included in the denominator. For generative tasks, we report CHAIR𝑠 (lower is better), CHAIR𝑖 (lower is better), and Recall (higher is better). 4.1.4 Implementation Details. The MHSA generator 𝐺 is implemented as a three-layer MLP with hidden dimension 512. The flattened attention dimension 𝑑 = 𝐿 ⋅ 𝐻 ⋅ 𝑁 differs across models: 𝑑 = 28×28×144 = 112,896 for Qwen2.5-VL-7B, 𝑑 = 32×32×256 = 262,144 for InternVL2-8B, and 𝑑 = 32 × 32 × 576 = 589,824 for LLaVA-v1.5-7B. The linear layer weight of 𝐺 is initialized with small weights (𝒰(−10−5 , 10−5 )), and the bias is initialized to zero. The hallucination detector 𝐷 is pre-trained following the protocol in [32]. The loss weights are set to 𝜆LVLM = 1 and 𝜆dg = 0.01 by default. 𝜆reg differs between models. The attention correction is applied across all LLM layers by default. For the captioning task, we adopt an offline training mode: pertoken cross-modal attention tensors are pre-extracted and stored, and the generator 𝐺 is trained without loading the LVLM, using only the adversarial loss and regularization loss (ℒLVLM is not applicable in offline mode).

4.2

Main Results on POPE-COCO

We begin by presenting detailed POPE results with Qwen2.5-VL7B on MSCOCO, as shown in Tab. 1. This serves as the primary evidence for MHSA’s effectiveness. MHSA yields a substantial improvement of +7.42 F1 points. Notably, the Yes Ratio shifts from 40.9% to 50.5%, close to the balanced 50%, indicating that the attention correction alleviates the model’s tendency to over-predict “No” answers. The Recall improvement of +15.78 points is particularly significant, demonstrating that the corrected attention enables the model to correctly recognize present objects that were previously missed. While Precision decreases

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

Table 2: Dataset generalization on POPE (Qwen2.5-VL-7B). All experiments use the same hyperparameter configuration as the COCO experiments without dataset-specific tuning. Best results in bold.

Table 4: Cross-dataset OOD generalization (F1 scores). Rows = training dataset, columns = test dataset. Diagonal entries (bold) = in-domain. (a) InternVL2-8B

Dataset

Method

Acc

Prec

Recall

Yes%

Objects365

Baseline MHSA

83.93 91.23

92.84 91.87

73.52 90.46

82.06 91.16

39.6 49.2

ImageNet

Baseline MHSA

82.80 86.67

96.33 84.25

68.20 90.20

79.86 87.12

35.4 53.5

(a) InternVL2-8B

Method

Acc

Prec

Recall

Yes%

COCO

Baseline MHSA

87.07 93.87

90.86 90.16

82.54 98.54

86.50 94.16

45.6 54.9

Obj365

Baseline MHSA

83.97 90.53

89.60 89.19

76.90 92.28

82.77 90.71

43.0 51.8

OpenImg

Baseline MHSA

79.53 83.73

74.77 75.74

89.77 99.74

81.58 86.10

60.6 66.5

COCO

Obj365

COCO Obj365 Baseline

94.16 95.77 86.50

90.57 90.71 82.77

(b) LLaVA-v1.5-7B

Table 3: Model generalization on POPE across three datasets. Best results in bold.

Dataset

Datasets

COCO

Obj365

COCO Obj365 Baseline

92.27 93.97 84.53

89.50 91.47 80.69

Datasets

COCO

Obj365

COCO Obj365 Baseline

92.97 93.48 85.55

91.02 91.16 82.06

(b) LLaVA-v1.5-7B

Dataset

Method

Acc

Prec

Recall

Yes%

COCO

Baseline MHSA

85.57 92.10

91.49 90.70

78.55 93.89

84.53 92.27

43.1 52.0

Obj365

Baseline MHSA

82.57 90.37

90.56 85.66

72.77 98.12

80.69 91.47

40.2 57.0

OpenImg

Baseline MHSA

78.50 81.07

72.69 74.20

90.09 96.05

80.46 83.72

60.9 63.3

slightly (−2.73), the overall F1 improvement confirms a net gain in discriminative quality.

4.3

Dataset Generalization

To evaluate whether MHSA generalizes across different visual domains, we fix the model as Qwen2.5-VL-7B and test it on additional POPE benchmarks. For each dataset, the detector and generator are trained and tested on the same dataset. Tab. 2 presents the results. MHSA achieves consistent improvements on Objects365 (ΔF1=+9.10) and ImageNet (ΔF1=+7.26), both without any datasetspecific hyperparameter tuning. Detailed per-category results are provided in the Appendix.

4.4

Model Generalization

To demonstrate that MHSA generalizes across different model architectures, we evaluate InternVL2-8B and LLaVA-v1.5-7B across MSCOCO, Objects365, and OpenImagesV7. Tab. 3 presents the results.

MHSA improves F1 across all six model-dataset combinations, with gains ranging from +3.26 (LLaVA on OpenImages) to +10.78 (LLaVA on Objects365). InternVL2-8B on COCO achieves the highest F1 of 94.16 (ΔF1=+7.66). These results confirm that the learned attention correction generalizes well across diverse LVLM architectures.

4.5

Cross-Dataset Out-of-Distribution Generalization

A critical question is whether the attention correction patterns learned by MHSA can transfer across data distributions. We train MHSA on one dataset and test it on another, constructing a crossdataset generalization matrix. Tab. 4 presents the results in terms of F1 scores, with in-domain results on the diagonal and out-ofdistribution (OOD) results off the diagonal. All OOD entries surpass the corresponding baselines, demonstrating that the attention correction learned by MHSA transfers well across data distributions. Remarkably, InternVL2-8B trained on Objects365 achieves 95.77 F1 on COCO, exceeding its indomain performance of 94.16, suggesting that the learned correction captures fundamental hallucination patterns that generalize beyond the training data. Similarly, for Qwen2.5-VL-7B, the OOD setting (Obj365→COCO: 93.48) outperforms the in-domain setting (COCO→COCO: 92.97). These findings strongly support that MHSA learns generalizable attention correction rather than dataset-specific artifacts.

Wei Ding, Yilin Li, Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang

Table 5: Caption Generation Results (CHAIR, Qwen2.5-VL7B) Method

CHAIRi ↓

CHAIRs ↓

Recall ↑

Flickr30k

Baseline MHSA Δ

16.43 9.20 -44.0%

37.50 21.00 -44.0%

86.63 83.28 -3.9%

Baseline MHSA Δ

5.68 5.19 -8.6%

21.00 18.00 -14.3%

56.42 55.14 -2.3%

COCO

A: Layer-wise Correction Intensity

3 2 1 0

Layer Index

ed at

😀

rre

cte

Figure 4: Attention visualization before and after MHSA correction of the POPE discriminative task.

B: Spatial Attention Entropy

6.25 6.00 5.75 5.50 5.25 5.00 4.75 4.50

cin

llu

Pre-correction 0.2 Post-correction Entropy (Right Axis)

0.0

Entropy (Post - Pre)

Total Absolute Delta

Entropy (Lower = More Focused)

Dataset

😞

0.2 0.4 0

Layer Index

😞

Hallucinated: ... looking out through an open window or door ...

Corrected: ... looking out through an open window or hatch ...

0.8 0.7 0.6 0

Layer Index

0.6

e at

0.5 0.4 0.3

cte

0.1 0 2 4 6 8 10 12 14 16 18 20 22 24 26

😀

Co rre

0.2 0.0

Head Index

Figure 3: Statistical analysis of attention modifications between pre-correction (hallucinated) and post-correction (factual) states. (A) Layer-wise correction intensity, measured by the total absolute delta of attention weights. Red asterisks denote the top three most modified layers. (B) Spatial attention entropy before (gray) and after (green) correction. Purple bars (right axis) represent the change in entropy (Δ Entropy). Lower entropy values indicate a higher degree of spatial focus. (C) Layer-wise cosine similarity between preand post-correction attention maps. (D) Heatmap detailing the absolute modification magnitude (delta) for each specific attention head. Error bars and shaded regions in panels A, B, and C represent the standard error of the mean (SEM).

4.6

cin

llu

Delta Magnitude

0.9

Layer Index

Cosine Similarity

1.0

0.5

D: Head-wise Modification Heatmap 262422201816141210 8 6 4 2 0

C: Pre-correction/Post-correction Cosine Similarity

Generative Hallucination Mitigation (Captioning)

For generative tasks, we apply the token-level MHSA extension (Sec. 3.6) to image captioning. The generator 𝐺token is trained offline on pre-extracted per-token attention tensors. Tab. 5 presents the results with Qwen2.5-VL-7B on Flickr30k and MSCOCO. On Flickr30k, MHSA reduces both CHAIR𝑖 and CHAIR𝑠 by 44% with only a 3.9% drop in Recall, demonstrating a highly favorable trade-off between hallucination suppression and content preservation. On MSCOCO, where the baseline hallucination rate is already low, MHSA still achieves meaningful reductions (CHAIR𝑖 : −8.6%, CHAIR𝑠 : −14.3%) with minimal Recall loss (−2.3%). These results demonstrate that the token-level attention correction effectively extends MHSA from discriminative to generative tasks.

Figure 5: Attention visualization before and after MHSA correction of the COCO generative task.

4.7

Ablation Studies

Table 6: Ablation study on loss function combinations. Setting Baseline w/o ℒdg w/o ℒLVLM w/o ℒreg ℒreg only MHSA (full)

ℒLVLM

ℒdg

ℒreg

w/ w/o w/ w/o w/

w/o w/ w/ w/o w/

w/ w/ w/o w/ w/

F1 85.55 88.51 84.18 89.60 83.11 92.97

We conduct ablation studies on Qwen2.5-VL-7B using POPECOCO to analyze the contribution of each component in MHSA. Tab. 6 presents the results for different combinations of loss functions. Removing any single loss term leads to a clear performance drop, and using ℒreg alone provides little benefit. The full combination achieves the best F1, confirming that all three losses are effective and complementary.

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

4.8

Mechanistic Analysis of Attention Correction

Figure 3 provides a statistical characterization of the modifications induced by MHSA in cross-modal attention. The correction is not distributed uniformly across the network; instead, it is concentrated in a limited set of intermediate layers, where the total absolute change in attention is the largest and the cosine similarity between pre-correction and post-correction attention shows the greatest reduction. At the same time, post-correction attention entropy is generally lower than pre-correction attention entropy, indicating that MHSA makes the spatial distribution of attention more concentrated. The head-wise heatmap further shows that these modifications are sparse, with substantial updates confined to only a small subset of layer-head pairs. Taken together, these observations suggest that MHSA functions as a targeted correction mechanism that intervenes primarily at the layers and heads where hallucination-related cross-modal misalignment is most pronounced, rather than broadly perturbing the full attention structure. Figures 4 and 5 provide qualitative examples that are consistent with the statistical evidence in Fig. 3. In the POPE discriminative example, the hallucinated prediction is associated with attention that is diffuse and partially misaligned, whereas after correction, the attention shifts toward the queried object itself, enabling a more visually grounded judgment. A similar pattern appears in the captioning example: before correction, the generated token is supported by attention assigned to an irrelevant region, whereas after correction, the attention is redirected to the visually relevant region, resulting in a more faithful description. These examples complement the statistical results and further support the conclusion that MHSA mitigates hallucination by selectively improving the spatial grounding of cross-modal attention.

Conclusion

In this paper, we propose MHSA (Mitigating Hallucinations via Steered Attention), a lightweight framework that learns sampleadaptive corrections of cross-modal attention patterns to mitigate hallucinations in large vision-language models. Unlike existing heuristic-based attention manipulation methods, MHSA trains a lightweight MLP generator guided by a pre-trained hallucination detector through adversarial supervision, without modifying any LVLM backbone parameters. Extensive experiments on three representative LVLMs (Qwen2.5-VL-7B, InternVL2-8B, and LLaVA-v1.5-7B) across multiple POPE benchmarks demonstrate consistent improvements in F1-score, with strong dataset, model, and cross-dataset out-of-distribution generalization. On generative captioning tasks, MHSA substantially reduces CHAIR scores. Future work will explore more expressive generator architectures, broader model coverage, and synergistic combinations with complementary mitigation strategies.

Wei Ding, Yilin Li, Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang

References [1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. 2022. Flamingo: a Visual Language Model for Few-Shot Learning. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022, Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (Eds.). http://papers.nips.cc/paper_files/paper/ 2022/hash/960a172bc7fbf0177ccccbb411a7d800-Abstract-Conference.html [2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, MingHsuan Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Report. CoRR abs/2502.13923 (2025). arXiv:2502.13923 doi:10.48550/ARXIV.2502.13923 [3] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. 2023. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks. CoRR abs/2312.14238 (2023). arXiv:2312.14238 doi:10.48550/ARXIV.2312.14238 [4] Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. 2024. HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 (Proceedings of Machine Learning Research), Ruslan Salakhutdinov, Zico Kolter, Katherine A. Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (Eds.). PMLR / OpenReview.net, 7824–7846. https://proceedings.mlr.press/v235/chen24bi.html [5] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. 2023. InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (Eds.). http://papers.nips.cc/paper_files/paper/2023/hash/ 9a6a435e75419a836fe47ab6793623e6-Abstract-Conference.html [6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA. IEEE Computer Society, 248–255. doi:10.1109/CVPR. 2009.5206848 [7] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. CoRR abs/2306.13394 (2023). arXiv:2306.13394 doi:10.48550/ ARXIV.2306.13394 [8] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, Zoubin Ghahramani, Max Welling, Corinna Cortes, Neil D. Lawrence, and Kilian Q. Weinhttps://proceedings.neurips.cc/paper/2014/hash/ berger (Eds.). 2672–2680. 5ca3e9b122f61f8f06494c97b1afccf3-Abstract.html [9] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. 2024. Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. IEEE, 14375–14385. doi:10.1109/ CVPR52733.2024.01363 [10] Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and Preventing Hallucinations in Large Vision Language Models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan (Eds.). AAAI Press, 18135–18143. doi:10.1609/AAAI.V38I16.29771 [11] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and

Retrospection-Allocation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. IEEE, 13418– 13427. doi:10.1109/CVPR52733.2024.01274 [12] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. 2018. The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale. CoRR abs/1811.00982 (2018). arXiv:1811.00982 http://arxiv.org/abs/1811.00982 [13] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating Object Hallucinations in Large VisionLanguage Models through Visual Contrastive Decoding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. IEEE, 13872–13882. doi:10.1109/CVPR52733.2024.01316 [14] Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. 2023. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA (Proceedings of Machine Learning Research), Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (Eds.). PMLR, 19730–19742. https: //proceedings.mlr.press/v202/li23q.html [15] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating Object Hallucination in Large Vision-Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Computational Linguistics, 292–305. doi:10.18653/V1/2023.EMNLP-MAIN.20 [16] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V (Lecture Notes in Computer Science), David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer, 740–755. doi:10.1007/978-3-319-10602-1_48 [17] Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2024. Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https: //openreview.net/forum?id=J44HfH4JCg [18] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved Baselines with Visual Instruction Tuning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024. IEEE, 26286–26296. doi:10.1109/CVPR52733.2024.02484 [19] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023, Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and http://papers.nips.cc/paper_files/paper/2023/hash/ Sergey Levine (Eds.). 6dcf277ea32ce3288914faf369fe6de0-Abstract-Conference.html [20] Shi Liu, Kecheng Zheng, and Wei Chen. 2024. Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29October 4, 2024, Proceedings, Part LXXXIII (Lecture Notes in Computer Science), Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol (Eds.). Springer, 125–140. doi:10.1007/978-3-031-73010-8_8 [21] Holy Lovenia, Wenliang Dai, Samuel Cahyawijaya, Ziwei Ji, and Pascale Fung. 2023. Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models. CoRR abs/2310.05338 (2023). arXiv:2310.05338 doi:10.48550/ARXIV.2310.05338 [22] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2017. Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models. Int. J. Comput. Vis. 123, 1 (2017), 74–93. doi:10.1007/S11263-016-0965-7 [23] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event (Proceedings of Machine Learning Research), Marina Meila and Tong Zhang (Eds.). PMLR, 8748–8763. http://proceedings.mlr.press/v139/radford21a.html [24] Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object Hallucination in Image Captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, 4035–4045. doi:10.18653/V1/D18-1437 [25] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019. Objects365: A Large-Scale, High-Quality Dataset for Object Detection. In 2019 IEEE/CVF International Conference on Computer Vision,

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs

ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 8429–8438. doi:10.1109/ICCV.2019.00852 [26] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.). https://proceedings.neurips.cc/paper/2020/hash/ 1f89885d556929e98d3ef9b86448f951-Abstract.html [27] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liangyan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. 2024. Aligning Large Multimodal Models with Factually Augmented RLHF. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL), Lun-Wei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 13088–13110. doi:10.18653/V1/2024.FINDINGSACL.775 [28] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023. An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation. CoRR abs/2311.07397 (2023). arXiv:2311.07397 doi:10.48550/ARXIV.2311.07397 [29] Xintong Wang, Jingheng Pan, Liang Ding, and Chris Biemann. 2024. Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024 (Findings of ACL), LunWei Ku, Andre Martins, and Vivek Srikumar (Eds.). Association for Computational Linguistics, 15840–15853. doi:10.18653/V1/2024.FINDINGS-ACL.937 [30] Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang. 2024. PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questions. In Proceedings of the 32nd ACM International Conference on Multimedia, MM 2024, Melbourne, VIC, Australia, 28 October 2024 - 1 November 2024, Jianfei Cai, Mohan S. Kankanhalli, Balakrishnan Prabhakaran, Susanne Boll, Ramanathan Subramanian, Liang Zheng, Vivek K. Singh, Pablo César, Lexing Xie, and Dong Xu (Eds.). ACM, 11175–11183. doi:10.1145/3664647.3685510 [31] Yudong Zhang, Ruobing Xie, Yiqing Huang, Jiansheng Chen, Xingwu Sun, Zhanhui Kang, Di Wang, and Yu Wang. 2025. Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMs. In Proceedings of the 33rd ACM International Conference on Multimedia, MM 2025, Dublin, Ireland, October 27-31, 2025, Cathal Gurrin, Klaus Schoeffmann, Min Zhang, Luca Rossetto, Stevan Rudinac, Duc-Tien Dang-Nguyen, Wen-Huang Cheng, Phoebe Chen, and Jenny Benois-Pineau (Eds.). ACM, 12266– 12275. doi:10.1145/3746027.3758152 [32] Yudong Zhang, Ruobing Xie, Xingwu Sun, Yiqing Huang, Jiansheng Chen, Zhanhui Kang, Di Wang, and Yu Wang. 2025. DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models. In Proceedings of the 33rd ACM International Conference on Multimedia, MM 2025, Dublin, Ireland, October 27-31, 2025, Cathal Gurrin, Klaus Schoeffmann, Min Zhang, Luca Rossetto, Stevan Rudinac, Duc-Tien Dang-Nguyen, Wen-Huang Cheng, Phoebe Chen, and Jenny Benois-Pineau (Eds.). ACM, 3555–3564. doi:10.1145/3746027. 3755118 [33] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2024. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net. https: //openreview.net/forum?id=1tZbq88f27

Wei Ding, Yilin Li, Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang

Inference Efficiency

We benchmark the MHSA pipeline on POPE-COCO (1,000 samples, Qwen2.5-VL-7B, NVIDIA H20). Since MHSA only corrects the 12.3% of samples detected as hallucinated, the vast majority (87.7%) incur no correction overhead, resulting in an amortized mean latency increase of only 0.43×.

Table 9: Attention configurations for each LVLM. 𝐿: number of transformer layers; 𝐻 : number of attention heads per layer; 𝑁 : number of visual tokens; 𝑑 = 𝐿 ⋅ 𝐻 ⋅ 𝑁 : total number of cross-modal attention entries used as input to the generator and detector.

Table 7: Overall inference efficiency (POPE-COCO, Qwen2.5VL-7B).

Avg latency (ms) Throughput (samples/s)

Baseline

MHSA

113.1 8.84

161.2 6.20

+0.43× −0.30×

Ratio

Avg (ms)

Median (ms)

Non-Halluc. (no correction) Hallucinated (corrected)

87.7% 12.3%

115.1 486.4

114.9 205.5

All

100%

161.2

115.1

This +0.4× overhead benefits from the two-stage detect-thencorrect design: the lightweight detector first screens all samples, and the more expensive MHSA generator is invoked only for the small fraction flagged as hallucinated. By contrast, contrastive decoding methods such as VCD [13] and ICD [29] require at least two full forward passes for every sample, incurring a fixed ∼+1× additional latency cost regardless of whether a given sample is hallucinated. Attention-based methods such as OPERA [11] further introduce backtracking overhead during generation. Furthermore, the current +0.4× figure is measured on POPE [15, 21] that employs adversarial negative sampling to deliberately maximize hallucination-prone scenarios, resulting in an elevated hallucination rate (e.g., 12.3% on COCO as detected by DHCP). In real-world deployments, where queries are not adversarially constructed, the proportion of samples triggering MHSA correction would be substantially lower, driving the amortized overhead closer to +0×— while contrastive decoding methods would still incur a constant +1× additional cost on every sample.

𝐿

𝐻

𝑁

𝑑 =𝐿⋅𝐻 ⋅𝑁

Qwen2.5-VL-7B InternVL2-8B LLaVA-v1.5-7B

28 32 32

144 256 576

112,896 262,144 589,824

Table 10: Generator 𝐺 and Detector 𝐷 architecture. 𝑑 is model-specific (Tab. 9). Input

Hidden

Output

Structure

𝑑 𝑑

512 128

𝑑 2

Lin-ReLU-Lin-ReLU-Lin LN-Lin-ReLU-Lin

𝐺 𝐷

Table 8: MHSA latency breakdown. 87.7% of samples skip correction and run close to baseline speed. Sample Type

Model

Table 11: Training hyperparameters for MHSA. lr𝐺 : generator learning rate; lr𝐷 : discriminator learning rate; 𝜆LVLM : LVLM quality loss weight; 𝜆dg : detector-guided loss weight; 𝜆reg : regularization loss weight. Task

Model

lr𝐺

lr𝐷

𝜆LVLM

𝜆dg

𝜆reg

Epochs

Batch

POPE POPE POPE Caption

Qwen2.5-VL LLaVA-v1.5 InternVL2 Qwen2.5-VL

1e-4 1e-4 1e-3 1e-3

1e-5 1e-5 1e-4 1e-7

1.0 1.0 1.0 0.0

0.01 0.01 0.01 0.5

1e-4 5e-4 1e-4 0.01

1 1 1 1

16 8 8 32

each). For each model×dataset, we extract per-layer cross-modal attention and split 80/20 by question ID (seed=42). DHCP classifies entries into Class 0/1 (non-hallucination) and Class 2/3 (hallucination). MHSA training oversamples: all Class 2/3 retained, Class 0/1 subsampled to half of Class 2+3 total. Table 12: MHSA training set after oversampling. Cls 0/1 = non-halluc., Cls 2/3 = halluc. Model

Data

Cls 0

Cls 1

Cls 2

Cls 3

Total

Qwen2.5-VL

COCO Obj365 OImages

20.8k 29.3k 35.5k

36.1k 47.4k 22.1k

5.5k 11.3k 48.9k

83.3k 117.4k 142.0k

Tab. 9 lists the attention configurations for each LVLM. Tab. 10 details the lightweight generator 𝐺 and detector 𝐷 .

LLaVA-v1.5

COCO Obj365 OImages

22.3k 31.3k 40.5k

34.6k 47.3k 20.9k

10.0k 15.4k 60.1k

89.0k 125.2k 161.9k

InternVL2

COCO Obj365 OImages

21.4k 30.3k 38.8k

29.7k 42.6k 18.2k

13.2k 18.0k 59.3k

85.7k 121.2k 155.1k

Model Architecture Details

Training Hyperparameters

Tab. 11 lists all training hyperparameters. All configurations use weight decay = 1e-4 and train for 1 epoch.

Dataset and Training Statistics

Each POPE dataset contains 3,000 Yes/No questions (1,000 per category: Popular, Random, Adversarial; 500 positive + 500 negative

Hyperparameter Sensitivity Analysis

We conduct a comprehensive hyperparameter sensitivity study on POPE-COCO with Qwen2.5-VL-7B. For each hyperparameter, we

MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs POPE-COCO Overall Metrics vs lr_d Accuracy ***

1.00

Sampling strategy. Fig. 6 compares class-balanced vs. oversampling. Oversampling yields significantly higher Recall and F1, confirming its effectiveness.

Score

0.90 0.85

1e-6

1e-5

Recall

Score

0.85

1e-6

1e-4

F1 Score

1.00

***

1e-5

1e-3

***

0.88 0.80

0.90 0.85

*** 0.72

0.80

1e-6

1e-5

0.92

lr_d

1e-4

1e-3

1e-6

1e-5

lr_d

1e-4

1e-3

0.88

0.80

bal

bal class

amp

class

overs

Recall

Figure 8: POPE-COCO metrics vs. lr𝐷 .

amp

overs

F1 Score

1.00

***

0.96

POPE-COCO Overall Metrics vs lr_g / lr_d

0.95

0.80

0.90

1.00

0.85

0.72

sampling

Score

bal class

amp

overs

amp

sampling

overs

0.96

0.90 0.85

0.92 0.88

0.80

Figure 6: POPE-COCO metrics vs. sampling strategy.

Precision

1.00

0.95

0.80

bal

class

Accuracy

Score

0.88

Score

1e-3

Score

Precision

0.96

0.90

1e-4

0.95

1.00

0.95

0.92 0.88

0.80

POPE-COCO Overall Metrics vs sampling Accuracy

0.96

1.00

Precision

1.00

0.95

Score

vary it while keeping all others at their default values, and report Accuracy, Precision, Recall, and F1 across multiple random seeds. Box plots show the distribution; diamonds denote means; asterisks indicate statistical significance (∗ 𝑝<0.05, ∗∗ 𝑝<0.01, ∗∗∗ 𝑝<0.001).

0.1

Recall

1.00

0.96

0.1

F1 Score

0.88

Score

0.95

Generator learning rate (lr𝐺 ). Fig. 7 shows that lr𝐺 =1e-4 achieves the best F1 and Accuracy. Too small (1e-5) under-corrects; too large (1e-3) increases variance.

0.80

0.90 0.85

0.72

0.80

lr_g / lr_d

0.1

lr_g / lr_d

0.1

POPE-COCO Overall Metrics vs lr_g Accuracy

1.00

***

0.96

0.85

POPE-COCO Overall Metrics vs (DG)

0.92 0.88

1e-5

1e-4

1e-3

1e-5

Recall

1e-4

***

Score

0.80

0.96

0.90 0.85

0.90

0.01

0.85

1e-4

lr_g

1e-3

1e-5

1e-4

lr_g

0.1

0.95

1e-3

Figure 7: POPE-COCO metrics vs. lr𝐺 .

0.01

F1 Score

1.00

0.96

0.80

1e-5

0.1

Recall

0.88

Score

0.72

0.92 0.88

0.80

Score

0.95 0.88

Precision

1.00

***

0.95

1e-3

F1 Score

1.00

***

0.96

1.00

Score

0.80

Accuracy

Score

0.90

Score

0.95

Figure 9: POPE-COCO metrics vs. lr𝐺 /lr𝐷 ratio.

Precision

1.00

***

0.80

0.90 0.85

0.72

0.80

0.01

(DG)

0.1

0.01

(DG)

0.1

Discriminator learning rate (lr𝐷 ). Fig. 8 shows that smaller lr𝐷 (1e-5 to 1e-6) yields better Recall and F1, consistent with keeping the discriminator nearly frozen as a stable supervisory signal.

Figure 10: POPE-COCO metrics vs. 𝜆dg .

Learning rate ratio (lr𝐺 /lr𝐷 ). Fig. 9 shows that the ratio has limited impact on F1, suggesting MHSA is robust to the relative learning rate scale.

Detector-guided loss weight (𝜆dg ). Fig. 10 shows that 𝜆dg =0.01 provides the best Accuracy. Removing it (𝜆dg =0) degrades performance significantly.

Wei Ding, Yilin Li, Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, and Yu Wang

Regularization weight (𝜆reg ). Fig. 11 shows that moderate regularization (1e-4 to 1e-3) achieves the best F1. Without regularization (𝜆reg =0), variance increases; too strong (1e-2) suppresses useful corrections. POPE-COCO Overall Metrics vs (REG) Accuracy

1.00 0.95

***

0.85

0.92

1e-4

1e-3

1e-4

1e-3

1e-2

F1 Score

1.00

***

0.96

1e-2

Recall

Score

0.88 0.80

Precision

Recall

Popular

Baseline MHSA

85.90 93.70

95.54 95.16

76.94 93.52

85.24 94.33

Random

Baseline MHSA

88.20 93.70

98.40 94.09

76.72 92.92

86.21 93.50

Adversarial

Baseline MHSA

86.40 90.90

92.25 88.57

79.23 93.75

85.25 91.09

Table 14: Per-category POPE: Qwen2.5-VL-7B on Objects365. Category

Method

Accuracy

Precision

Recall

Popular

Baseline MHSA

84.10 92.70

92.19 93.25

73.29 91.51

81.66 92.37

Random

Baseline MHSA

84.10 91.90

97.37 94.69

71.29 89.40

82.31 91.97

Adversarial

Baseline MHSA

83.60 89.10

89.36 87.89

76.06 90.54

82.17 89.20

0.90 0.85

0.72

0.80

1e-4

(REG)

1e-3

1e-2

1e-4

(REG)

1e-3

1e-2

Figure 11: POPE-COCO metrics vs. 𝜆reg . LVLM quality loss weight (𝜆LVLM ). Fig. 12 shows that including ℒLVLM (𝜆LVLM ≥ 0.1) substantially improves Recall and F1 over the no-LVLM-loss setting (𝜆LVLM =0), validating its role in preserving output quality. POPE-COCO Overall Metrics vs (LVLM) Accuracy

1.00

***

Precision

1.00

***

0.95

***

Table 15: Per-category POPE: Qwen2.5-VL-7B on OpenImages. Category

Method

Accuracy

Precision

Recall

Popular

Baseline MHSA

83.20 83.70

78.97 78.02

88.80 98.92

83.59 87.24

Random

Baseline MHSA

93.20 94.50

98.20 95.49

87.93 97.49

92.78 96.48

Adversarial

Baseline MHSA

68.90 65.30

63.53 61.13

88.15 98.96

73.84 75.57

0.96

0.90

Score

Accuracy

***

0.95

0.85

0.92 0.88

0.80

***

0.96

0.1

Recall

***

0.1

F1 Score

1.00

***

Score

0.88 0.80