UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations

UniProbe: A Learnable Token-Level Hallucination Detector for Large VLMs using Multi-Structural Internal Representations Dvir Samuel1 Guy Bar-Shalom2 Fabrizio Frasca2 Ethan Fetaya1,3 Yftah Ziser1,4 Gal Chechik1,3 Haggai Maron1,2 1 NVIDIA Research, Tel Aviv, Israel 2 Technion, Haifa, Israel 3 Bar-Ilan University, Ramat Gan, Israel 4 University of Groningen, Groningen, The Netherlands Abstract

What is the child wearing? GLM-4V-Thinking:

arXiv:2608.10835v1 [cs.CV] 11 Aug 2026

Large Vision-Language Models (LVLMs) achieve impressive visual reasoning and dialogue capabilities, yet frequently hallucinate content unsupported by the visual input. Effective mitigation requires token-level localization, enabling targeted intervention without discarding the entire response. Existing detectors require expensive full-model fine-tuning, rely on external verifiers that ignore the model’s generation process, or reduce internal signals to isolated features and hand-crafted statistics, discarding spatial, sequential, and relational structure. We introduce UniProbe, a lightweight, unified, learnable detector that models a frozen LVLM’s heterogeneous computational trace from a single forward pass. UniProbe constructs a directed graph over image patches, query tokens, and generated tokens, with attention weights encoding their relations. It processes this trace with alternating structureaware modules: a GNN for relational evidence, a ViT for 2-D visual geometry, and a GRU for response order. Interleaving them allows spatial, relational, and sequential evidence to interact throughout the detector. We further develop a streaming variant for hallucination-aware decoding, which detects and resamples hallucinated tokens during generation, and a selfadaptation strategy aligning the detector with the LVLM’s own generations. Across diverse LVLM backbones, UniProbe achieves state-ofthe-art token-level and object-hallucination detection. During decoding, it reduces object hallucinations by up to 55% at 1.06× the latency of standard generation. Project Page

1

The child is wearing white pants and a blue shirt. Detection (denoted by a yellow mark): HaloDet: The child is wearing white pants and a blue shirt. HalLocalizer: The child is wearing white pants and a blue shirt. Ours: The child is wearing white pants and a blue shirt. Mitigation (Ours) The child is wearing blue pants and a white shirt.

Figure 1: Token-level hallucination detection and mitigation. Given an input image, a LVLM generates hallucinated response texts (red). Existing SOTA detectors either misidentify non-hallucinated tokens (HaloDet) or fail to capture the full hallucination (HalLocalizer). Yellow highlights denote the tokens identified as hallucinated by each detector. Our method accurately detects the hallucinated tokens and actively mitigates them during generation to produce a grounded response (green).

ing, and interactive dialogue. Despite their impressive capabilities, LVLMs remain notoriously prone to hallucinations: generating text responses that are ungrounded in, or directly contradict, the visual input. These unfaithful generations severely impair trustworthiness and hinder real-world applications. Crucially, effective mitigation requires not only determining whether a response contains a hallucination, but also precisely identifying the affected tokens, enabling targeted intervention without discarding or regenerating the entire response. Accurate token-level hallucination detection is therefore a central challenge in trustworthy multimodal AI.

Introduction

Large Vision-Language Models (LVLMs) (Liu et al., 2024; Dai et al., 2023; Chen et al., 2024; Hong et al., 2025) have achieved remarkable progress in bridging vision and language, enabling advanced visual reasoning, detailed image caption-

Prior approaches to token-level hallucination detection generally fall into three categories: finetuning the LVLM to tag hallucinated spans (Cai et al., 2025; Whitehead et al., 2024); training an external verifier that compares the response with the 1

image (Park et al., 2025); or probing the model’s internal signals for evidence of grounding (Kogilathota et al., 2026; Nguyen et al., 2026; Fieback et al., 2024; Zhang et al., 2024). Full fine-tuning updates billions of parameters and may degrade the model it is meant to safeguard, while external verification typically requires an additional visionlanguage model and ignores how the response was produced. Internal probes avoid both costs by reusing signals already computed during generation, but existing methods reduce these signals to isolated features or hand-crafted statistics, discarding much of their underlying structure. Recent text-only (LLM) detectors (Bar-Shalom et al., 2026, 2025; Frasca et al., 2026) show that treating a model’s computational trace as structured data, such as a sequence, tensor, or attention graph, is substantially more effective than approaches that use internal features but discard this structure. Extending this idea to LVLMs is challenging because their traces are inherently heterogeneous, combining a 2-D grid of image patches, 1-D query and response sequences, and cross-modal relations induced by attention. We therefore introduce UniProbe, a lightweight detector that reads this heterogeneous trace from a single forward pass of a frozen LVLM. It constructs a directed graph over image, query, and response tokens and processes it with alternating structureaware modules: a GNN (Gilmer et al., 2017) for relational evidence, a ViT (Dosovitskiy et al., 2021) for visual geometry, and a GRU (Cho et al., 2014) for response order. Interleaving these modules allows spatial, relational, and sequential evidence to interact throughout the network while keeping the backbone LVLM frozen. Beyond identifying hallucinations in completed responses, we use the same computational-trace formulation to intervene during generation. We train an online variant of UniProbe that enables hallucination-aware decoding: During decoding, each candidate token is assessed against the current computational trace. If the detector flags a token as hallucinated, we reject that token and resample an alternative, so the hallucination is corrected during generation and never reaches the final response. Additionally, we identify and address an important practical challenge: hallucination detectors are trained on benchmarks that aggregate hallucinated responses from a variety of LVLM backbones, yet in real applications a detector must monitor and score its own generations. Because different back-

bones hallucinate in systematically different ways, a detector trained on other models’ outputs faces a severe train–test distribution shift at deployment. We mitigate this mismatch by adapting UniProbe using responses sampled directly from the target model, aligning the detector with the host LVLM’s specific generation dynamics. We evaluate UniProbe across a variety of opensource LVLM backbones on three tasks: tokenlevel hallucination detection, object-hallucination detection, and real-time detection and mitigation of hallucinations in generated responses. It consistently outperforms state-of-the-art detectors on all three setups: token-level F1 improves by 4–6 points over the strongest trained baseline while keeping the backbone frozen, and object-hallucination detection on POPE rises from 41.0 to 63.1 F1. Finally, once adapted to the model’s own outputs, our hallucination-aware decoding cuts object hallucinations by up to 55% at just 1.06× vanilla latency, without degrading response quality or substantially altering the model’s output distribution.

2

Related Work

Hallucination Detection from LLM Internals Recent work on LLMs has explored detecting hallucinations directly from their internal representations. Azaria and Mitchell (2023) probe hidden states to predict statement truthfulness, and Li et al. (2024) steer activations toward truthful behavior at inference time. Beyond single-vector probes, recent methods treat the computational traces as structured data: attention maps reveal contextual hallucination (Chuang et al., 2024; Sriramanan et al., 2024), ACT-ViT reads activation tensors with a vision transformer (Bar-Shalom et al., 2025), CHARM runs message passing on attention graphs (Frasca et al., 2026), and LOS-Net learns detectors over output-distribution sequences (BarShalom et al., 2026). The usefulness of internal representations extends beyond hallucination detection. Features extracted from frozen diffusion models support personalized segmentation and retrieval (Samuel et al., 2024), intermediate ViT layers enable state-of-the-art place recognition (Tzachor et al., 2025), and intermediate MLLM layers encode substantial task-relevant information for video–text retrieval (Tzachor et al., 2026). Together, these results suggest that intermediate computations provide rich signals that can be exploited without modifying the underlying model. We build 2

on this perspective for multimodal hallucination detection, where computational trace has a distinctive heterogeneous structure spanning image patches, query tokens, and generated response tokens.

tor, and TruthPrInt (Duan et al., 2025), which reads an LVLM’s latent states but steers generation along a learned truthful direction. In contrast, our guardrail extends detector-guided decoding to the multimodal setting using a learned streaming detector over the full internal trace—not attention alone—and rejects and resamples the tokens it flags. This targeted intervention reduces hallucinations while preserving the model’s output distribution.

Hallucination Detection in LVLMs LVLMs often generate fluent content that is unsupported by their visual input. Existing approaches rely on external verification, full-model fine-tuning, or internal probing. HalLocalizer (Park et al., 2025) trains an external model to compare the image with the generated response, whereas HALP (Kogilathota et al., 2026) probes the backbone’s internal representations but predicts only a global hallucination label. HaloDet (Cai et al., 2025) and Whitehead et al. (2024) obtain token-level predictions by fine-tuning the LVLM, the latter optionally using pretraining on synthetically corrupted grounding data. In contrast, a growing family of methods flags object hallucinations from hand-crafted internal signals: the balance of attention between image and text tokens and its alignment with token representations (Nguyen et al., 2026; Jiang et al., 2024; Hoang-Xuan et al., 2026), per-token confidence together with cross-modal attention patterns (Fieback et al., 2024; Zhang et al., 2024), or projections of visual features into the vocabulary space (Jiang et al., 2025). A recent method, ZINA (Wada et al., 2026), combines a hallucination detector with an external LVLM reviewer. Although effective, it requires a human-written reference caption at inference time, which is rarely available in real-world settings. In contrast, our method is fully reference-free. Unlike all existing approaches, our lightweight learnable detector keeps the backbone frozen and treats its computational trace as a structured representation for general token-level hallucination localization.

3

Notation and Problem Setup

We consider a frozen pre-trained large visionlanguage model M (the backbone) that receives an image I and a text query x, and autoregressively generates a response r = (r1 , . . . , rT ) consisting of T tokens. Internally, M processes both inputs and outputs as a single sequence of n tokens, segmented into 3 disjoint groups: image patches, query tokens, and response tokens we wish to verify. For a given generation, one can extract two primary signals from an intermediate layer ℓ of M : (1) Hidden States: Let hi ∈ Rd be the activation in the residual stream at position i, summarizing the model’s computation for that token. (2) Attention Maps: Let A ∈ [0, 1]n×n be the attention matrix (averaged across all heads), where Aij represents the attention weight from position i to position j. Problem Statement. A token-level hallucination detection is defined as learning a detector gθ that outputs a sequence of probabilities p ∈ [0, 1]T . Each pi represents the likelihood that token ri is a hallucination (i.e., not supported by I or x). The detector is trained on responses annotated with hallucinated spans.

4

Method

Overview. A single generation of M yields three structured sources of information: the input image, represented as a 2-D grid of patches; the generated response, represented as a 1-D token sequence; and the attention, which captures interactions among image, query, and response tokens. Because these sources have different structures, we process each with a dedicated structure-aware module and couple them in a unified learnable detector built from alternating blocks. Each block applies the three modules in turn to a shared per-token representation, allowing image, attention, and response evidence to interact throughout the network. The detector is trained jointly, end-to-end, while M remains frozen. See Figure 2 for an illustration.

Hallucination Mitigation in LLMs A complementary line of work intervenes during or after decoding to mitigate hallucinations. OPERA (Huang et al., 2024) penalizes over-trust attention patterns and backtracks when they occur; VCD (Leng et al., 2024) contrasts predictions from the original and distorted images to suppress language priors; and Woodpecker (Yin et al., 2024) corrects completed responses using external verifiers. These methods rely on fixed heuristics or additional models. Learned alternatives include Lookback Lens (Chuang et al., 2024), which guides text-only decoding with an attention-based detec3

“Describe the image”

only the highest-Aij connections per token and drop the weak ones, which bounds the edge count. By causality, response-to-response edges originate only from preceding tokens, so the neighborhood of i represents the visual and textual context used to generate ri . We denote its type-t neighbors by Nt (i), where t ∈ {img, query, response}.

Large Vision-Language Model 𝑴

“A dog catches a green frisbee…”

𝑙 hidden states ℎ! + attention 𝐴!" Computational-trace graph query response image ... es edg n ntio atte

UniProbe

GNN

ViT

GRU

mixes evidence across attention edges (Eq.2)

updates image nodes (Eq.4)

updates response nodes (Eq.5)

Per-token hallucination probability (Eq.6)

catches green frisbee ... 0.08 0.94 0.12 ...

The UniProbe Architecture Given the computational-trace graph above, UniProbe learns to combine its cross-modal, spatial, and sequential structure to identify hallucinated response tokens. We first project all node features into a shared representation space using type-specific linear maps:

×𝑳

Figure 2: UniProbe architecture. From a single forward pass of the frozen LVLM M , UniProbe reads layer-ℓ hidden states and attention to construct a computational-trace graph over image, query, and response tokens. L alternating blocks of GNN, ViT, and GRU, integrate cross-modal, spatial, and sequential evidence, followed by a linear head that predicts a hallucination probability for each response token.

(0)

xi

= Wt(i) hi + 1[t(i)=img] γi ,

(1)

where t(i) is the type of node i. The network then stacks L identical blocks; within block b we ab(b−1) breviate xi ≡ xi . Each block runs the three modules in turn. First, a GNN (Gilmer et al., 2017) mixes evidence across modalities, updating every response node from its typed attention neighbors, X X mi = aij Wt xj + Wm ci , t j∈Nt (i) (2)
x̂i = LN xi + ReLU(Ws xi + mi ) ,

Computational-trace Graph From one forward pass we build a directed, attributed graph G = (V, E) that records, from M ’s own internals, the visual and contextual evidence behind each emitted response token. The graph carries two complementary kinds of information. The first is internal signals directly from M : the layer-ℓ hidden states and the attention weights. The second is lightweight structural metadata, namely each node’s type and, for image nodes, its patch coordinates; this metadata acts as a positional encoding providing each structure-aware module with the node’s position within the image or sequence. Nodes are tokens of three types: (1) all T response tokens and a subset of (2) image patches and (3) query tokens. Specifically, following the observation (Frasca et al., 2026) that attention magnitude indicates which context a token relies on, we keep the graph lightweight by scoring each image position by theP total attention it receives from the response, sj = i∈resp Aij , and retaining the Nimg highest-scoring positions. The same selection procedure is applied to query tokens. Node v carries its layer-ℓ hidden state hv as a feature; image nodes additionally carry a 2-D encoding γv of their patch coordinates. Edges are derived directly from attention. For each response token i, we retain the strongest incoming connections from image patches, query tokens, and earlier response tokens j < i, with edge weight aij = Aij . Specifically, we keep

where every type t has its own projection Wt , Ws is a self-transform, and ci ∈ R3 stacks the incoming attention mass per modality, P  ci = a , (3) ij j∈Nt (i) t∈{img,qry,resp}

a coarse summary of how much the token leaned on vision vs. text. Separate Wt , for each modality, let the detector read “attends to an image patch” and “attends to the query” as distinct evidence. Then, two within-modality modules capture spatial and sequential structure not explicitly modeled by the graph step: a ViT (Dosovitskiy et al., 2021) over the image grid and a BiGRU (Cho et al., 2014) along the response in generation order, both as residual updates,
xI ← LN x̂I + ViT(x̂I ) , (4)
xR ← LN x̂R + GRU(x̂R ) , (5) where xR ∈ RT ×d and xI ∈ RNimg ×d stack the response- and image-node embeddings (where d is the shared space dimension) and become x(b) . 4

Alternating the three modules L times couples them into a single detector trained jointly, end-toend. A linear head on the final response embeddings gives the per-token scores (L)

pi = σ w⊤ xi

,

i ∈ {1, . . . , T }.

further fine-tune UniProbe on these automatically labeled generations. This aligns our detector with M ’s own generations, thereby reducing the distribution shift.

5

(6)

Experiments

We evaluate our approach on three setups: (1) token-level hallucination detection, on both teacher-forced and self-generated responses; (2) object hallucination detection; and (3) hallucination-aware decoding during streaming generation. Datasets and metrics. (1) Token-level: MHALO (Cai et al., 2025), a fine-grained detection benchmark whose responses are annotated with hallucinated spans (five eval sets, ∼500 samples each, spanning general, knowledge, and reasoning prompts); and HalLoc (Park et al., 2025), a largescale localization benchmark with per-token labels in several categories (object, attribute, relationship, scene) across VQA, instruction, and captioning. We report F1M , a word-overlap F1 between predicted and GT hallucination spans; F1IoU , the mean over gold spans of the best word-level IoU with any predicted span; and IF, the fraction of well-formed predictions. On HalLoc we report per-category Precision/Recall/F1 following (Park et al., 2025). (2) Object hallucination detection: We evaluate object presence on POPE (Li et al., 2023), following the same training data and evaluation protocol as Nguyen et al. (2026); Jiang et al. (2025). Specifically, we ask the detector to identify incorrect answers rather than directly scoring the LVLM’s responses, and report F1 and AUC. (3) Self-generated captions: We evaluate detection on the model’s own COCO (Lin et al., 2014) captions using CHAIR (Rohrbach et al., 2018). CHAIRi measures the fraction of mentioned objects absent from the image, and CHAIRs the fraction of captions containing at least one such object. Labels are generated automatically by matching caption words to COCO categories and synonyms and comparing them with GT annotations, without a judge model. We also report detection F1 against these labels. Backbones. We test our detector across various LVLMs: GLM-4.1V-9B (Hong et al., 2025), Qwen-3-VL (Bai et al., 2025), LLaVA-1.5-7B (Liu et al., 2024), InternVL2-8B (Chen et al., 2024), and InstructBLIP-7B (Dai et al., 2023). Baselines. We compare against baselines for

With the graph size bounded by fixed node and edge budgets, UniProbe remains lightweight in practice and adds only modest overhead beyond the backbone computation it reads from. Online Detection and Hallucination-aware Decoding We turn UniProbe into a streaming (online) detector by simply replacing its bidirectional response GRU with a unidirectional one. Because the graph already follows generation order and the ViT processes only the static image, this change ensures that the score of ri depends only on the prefix r≤i , without otherwise modifying the architecture. The streaming detector further turns detection into prevention. As M decodes, we score each newly generated token using the causal computational trace. If its hallucination probability exceeds a threshold τ , we reject it, banning its first token and re-decoding, following the detector-guided decoding of Chuang et al. (2024). Otherwise we accept and continue. Mitigating Distribution Shift in Self-generation Current token-level datasets (Cai et al., 2025; Park et al., 2025) share a fixed protocol: the detector is supervised on hallucination annotations of responses produced by a set of other LVLMs, and is then run over M while M is teacher-forced on those same pre-collected responses. In real deployment, however, the detector must judge M ’s own, freely sampled generations, which are more fluent and confident than the third-party responses seen in training. This train–test distribution shift lowers detection precision dramatically. We address this gap through self-adaptation, without requiring manual hallucination annotations. We sample a small subset of Objects365 (Shao et al., 2019) images with ground-truth object annotations, generate free-form captions using the target model M , and automatically label each generated object mention using CHAIR (Rohrbach et al., 2018). Specifically, an object mention is marked as hallucinated if it does not correspond to any ground-truth object in the image. We then 5

MHALO

F1M F1IoU

IF

Zero-shot prompted LVLMs MiniCPM-V 2.6 InternVL2-76B Qwen-VL-Max Llama-3.2-90B-V Abab7-chat Claude-3.5-Sonnet GLM-4V-Plus Gemini-1.5-Pro GPT-4o Claude-4.8-Opus

18.4 28.5 30.4 34.9 36.9 49.8 35.9 51.9 56.8 58.6

13.1 21.2 22.9 23.6 28.0 30.1 30.3 36.7 40.6 43.9

82.1 88.1 99.7 94.1 95.2 98.9 94.2 99.0 99.2 99.3

Trained detectors (Qwen-3-VL backbone) Whitehead et al. (Whitehead et al., 2024) 49.7 HalLocalizer (Park et al., 2025) 49.9 HaloDet (Cai et al., 2025) 55.4 UniProbe (Ours) 61.7

45.1 45.3 46.7 51.2

99 100 91 100

Trained detectors (GLM-4V backbone) Whitehead et al. (Whitehead et al., 2024) HalLocalizer (Park et al., 2025) HaloDet (Cai et al., 2025) UniProbe (Ours)

47.3 47.9 49.8 52.9

99 100 92 100

56.0 56.0 59.1 63.2

HalLoc

P

R

F1

10.1

100

16.6

any

Always-1

InternVL2

HalLocalizer 69.5 73.1 70.6 UniProbe (Ours) 73.8 75.2 73.9

LLaVA-1.5

HalLocalizer 70.8 71.1 69.8 UniProbe (Ours) 74.6 75.0 74.3

InstructBLIP HalLocalizer 64.1 68.9 65.3 UniProbe (Ours) 65.1 71.1 67.4

Table 2: Token-level localization on HalLoc. Ours beats the trained HalLocalizer on all three backbones (overall P/R/F1 over the 12 subset×category cells). COCO

Table 1: Token-level detection on MHALO. Our approach beats zero-shot frontier LVLMs and SoTA trained detectors on all metrics and across different backbones.

POPE

Method

F1

F1

AUC

MetaToken-GB (Fieback et al., 2024) HalLoc (Park et al., 2025) SVAR (Jiang et al., 2024) DHCP (Zhang et al., 2024) ProjectAway (Jiang et al., 2025) HALP (flat probe) (Kogilathota et al., 2026) Token Grounding (Nguyen et al., 2026) UniProbe (Ours)

68.0 71.0 76.0 73.0 79.0 – 82.0 92.3

22.0 29.0 25.0 27.0 38.0 39.5 41.0 63.1

73.0 71.0 – 69.0 70.0 68.9 75.0 90.0

Table 3: Self-generation and object-hallucination detection. UniProbe outperforms prior detectors on selfgenerated COCO captions and POPE across all metrics.

token-level detection, object-level detection, and hallucination-aware decoding. For token-level detection, we include zero-shot frontier LVLMs using Analyze-then-Judge (Cai et al., 2025), as well as trained methods. HaloDet (Cai et al., 2025) finetunes the backbone to regenerate responses with inline hallucination tags, while Whitehead et al. (2024) replaces the LM head with a token classifier. HalLocalizer (Park et al., 2025) instead trains an external verifier over the image and response. For object-hallucination detection, we also compare with methods that read signals from a frozen backbone. HALP (Kogilathota et al., 2026) probes pre-generation representations at the sample level; token-wise probing of response hidden states is covered by our sequence-only ablation (Sec. 6). MetaToken (Fieback et al., 2024) and DHCP (Zhang et al., 2024) train lightweight classifiers over confidence and cross-modal attention features. SVAR (Jiang et al., 2024), Token Grounding (Nguyen et al., 2026), and PAS (HoangXuan et al., 2026) use hand-crafted attention or representation-alignment scores, while ProjectAway (Jiang et al., 2025) projects visual features into the language space. For hallucination-aware decoding, we apply the same guardrail with HaloDet (Cai et al., 2025) and PAS(Hoang-Xuan et al., 2026).

Results Token-level detection. Table 1 shows that zeroshot prompting of frontier LVLMs is weak at finegrained detection (F1IoU 40.6 for GPT-4o, 43.9 for Claude-4.8-Opus). Trained detectors close part of the gap, but our frozen read-out beats them all on both backbones: F1M /F1IoU 63.2/52.9 vs. 59.1/49.8 for HaloDet on GLM-4V, and 61.7/51.2 vs. 55.4/46.7 on Qwen-3-VL. This holds even though HaloDet fine-tunes every backbone weight and ours updates none. The advantage carries over to HalLoc (Table 2), where UniProbe beats the trained HalLocalizer on all three backbones. The accuracy gains come with modest computational overhead. Relative to vanilla generation (1.00×), UniProbe scores the full response at 1.15× latency, compared with 1.30× for HaloDet, 1.21× for HalLocalizer, and 1.23× for Whitehead et al. Thus, UniProbe provides the strongest accuracy–efficiency trade-off, achieving higher detection performance while adding less latency than all trained baselines. Figure 3 qualitatively illustrates the advantage of UniProbe on generations produced by different LVLM backbones. The examples expose several limitations of existing token-level detectors: they 6

Describe the image.

Which animals are shown in the image?

GLM-4V-Thinking:

Qwen-3-VL:

The image features a large, golden passenger jet parked on the runway at an airport. There are 2 people, one closer to the left side of the image and the other to the right.

There is a cat and a dog in the foreground of the image, and three birds on the tree behind them.

HaloDet: The image features a large, golden passenger jet parked on the runway at an airport. There are 2 people, one closer to Miss all the left side of the image and the other to the right. HalLocalizer: The image features a large, golden passenger jet parked on the runway at an airport. There are 2 people, one closer to the left side of the image and the other to the right. Ours (Detection): The image features a large, golden passenger jet parked on the runway at an airport. There are 2 people, one closer to the left side of the image and the other to the right.

HaloDet: There is a cat and a dog in the foreground of the image, and three birds on the tree behind them. HalLocalizer: There is a cat and a dog in the foreground of the image, and three birds on the tree behind them. Ours (Detection): There is a cat and a dog in the foreground of the image, and three birds on the tree behind them. Ours (Mitigation): There are two dogs in the foreground of the image. <eos>

Ours (Mitigation): The image features a large, golden passenger jet parked on the runway at an airport. <eos>

Figure 3: Qualitative token-level detection and mitigation across different LVLM backbones. Existing SOTA methods exhibit different failure modes: they incorrectly flag visually grounded text, detect only part of the hallucinated span, or miss it entirely. In contrast, UniProbe accurately localizes the hallucinated tokens in both examples and uses its predictions during hallucination-aware decoding to produce the grounded outputs shown in green. Each column presents an example from a different backbone, demonstrating that the observed behavior is not model-specific. Backbone-generated hallucination is shown in red, while yellow highlights mark the tokens predicted as hallucinated by each detector. Detection Method

F 1obj

Vanilla VCD (Leng et al., 2024) PAS (Hoang-Xuan et al., 2026) HaloDet (Cai et al., 2025) + FT on self-generation Ours (streaming) + FT on self-generation

— – 24.9 11.2 34.1 32.6 63.8

(a) Each module alone

Mitigation CHAIRi CHAIRs JS Div SPICE 18.0 16.8 17.1 17.9 13.1 15.3 8.2

37.2 36.1 35.8 37.0 27.4 31.4 16.6

– 0.031 0.029 0.0004 0.011 0.012 0.013

0.214 0.203 0.205 0.214 0.213 0.218 0.222

Lat. 1.00× 2.00× 1.14× 1.30× 1.30× 1.06× 1.06×

F1M F1IoU

F1M F1IoU

MLP (flat probe) 31.9 25.7 Transformer (flat probe) 33.2 27.8

Ours 63.2 52.9 − ViT 61.5 52.5 − query 60.2 51.7 − BiGRU 58.6 51.6 − graph 53.1 46.7

ViT only GNN only GRU only Ours

Table 4: Detection and mitigation on streaming selfgenerated captions on COCO. The same streaming guardrail driven by different detectors: our approach reduces object hallucination the most, at 1.06× vanilla latency, while preserving caption quality (SPICE) and the output distribution.

(b) Remove one module

51.8 45.1 56.9 49.8 53.1 46.7 63.2 52.9

Table 5: Architecture and component ablations on MHALO. (a) MLP and Transformer on flat probes and individual modules underperform the full model. (b) Removing any component hurts, with the graph and response modules contributing most.

can over-detect by marking visually supported content, under-detect by identifying only a short portion of a hallucinated statement, or fail to detect the hallucination altogether. In contrast, UniProbe consistently recovers the complete unsupported span across both backbones. Importantly, this improved localization translates directly into more effective intervention: when used for hallucination-aware decoding, the same detector prevents the unsupported continuation and yields a response that remains grounded in the visual input. Object hallucination detection. Table 3 evaluates the detection of hallucinated object mentions with LLaVA-1.5-7B, both in self-generated COCO captions and among the model’s POPE answers. UniProbe reaches 92.3 F1 on COCO captions and

63.1 F1 / 90.0 AUC on POPE, compared to 82.0 and 41.0 / 75.0 for Token Grounding, the strongest prior detector. All prior detectors plateau at 69–75 AUC on POPE, whether they are external verifiers or flat, hand-crafted read-outs of the same frozen internals; structuring that same evidence as a computational trace lifts detection to 90.0 AUC on this heavily imbalanced task, where the wrong answers to be caught are rare. Streaming detection and hallucination-aware decoding. We evaluate deployment on GLM4.1V’s freely generated captions, with hallucinated object mentions labeled by CHAIR. The streaming detector achieves 32.6 F 1obj before adaptation and 63.8 after self-adaptation, while retaining its 7

best (layer 20)

65

16 i

CHAIR

F1M

60 55 50 45 4

8

14 12 10

12 16 20 24 28 32 36 40

GLM-4.1V read-out layer

0.4

0.5

0.6

guardrail threshold τ

0.7

best (layer 14)

Figure 5: The threshold controls how much hallucination the guardrail removes. CHAIRi on GLM-4.1V falls as τ tightens; we operate at τ =0.70 (marked), the most aggressive setting that still leaves the correctly mentioned objects intact (lower τ removes more but starts deleting correct content)

F1

70

60

50

2

6

10

14

18

22

26

30

LLaVA-1.5 read-out layer

Figure 4: The grounding signal lives in the middle layers, across backbones. Detection F1 vs. the readout layer: GLM-4.1V (top, MHALO) peaks at layer 20 and LLaVA-1.5 (bottom, HalLoc) at layer 14, both degrading toward the output.

structure-aware modules. Flat MLP and Transformer probes perform substantially worse, trailing the full model by at least 30.0 F1M , highlighting the importance of preserving the trace’s heterogeneous structure. Among individual modules, the GNN performs best but reaches only 56.9 F1M , compared with 63.2 for the full model. Conversely, removing any component hurts, with the largest drops caused by removing the relational graph (−10.1) and response recurrence (−4.6). These results show that hallucination evidence is distributed across the relational, spatial, and sequential structure of the trace and is most effective when jointly modeled.

MHALO performance at 63.5/53.0 F 1M /F 1IoU with IF = 100. Table 4 applies the same reject-and-resample guardrail to every detector, isolating the effect of detection quality. PAS and HaloDet, with 24.9 and 11.2 F 1obj , only modestly reduce CHAIRi /CHAIRs to 17.1/35.8 and 17.9/37.0, while adding more latency. Our unadapted streaming detector lowers them from 18.0/37.2 to 15.3/31.4, and self-adaptation further reduces them to 8.2/16.6, a 55% reduction, at only 1.06× vanilla latency. Adapting HaloDet raises its F 1obj to 34.1 and lowers CHAIR to 13.1/27.4, but it remains well behind ours, showing that adaptation alone does not explain the gain. The improvement also does not come from suppressing content: guarded captions retain 95.4% of the vanilla word count (143.7 vs. 150.7), preserve correctobject coverage (2.14 vs. 2.23), and eliminate the six empty captions produced by vanilla decoding. Perplexity remains below vanilla’s (2.27 vs. 2.30), with only a small distribution shift (JS = 0.013). A complementary user study appears in the appendix.

6

Read-out layer. UniProbe reads from a single mid-network layer. Fig. 4 sweeps this choice on GLM-4.1V: detection climbs through the early layers, peaks at layer 20, and drops toward the output, where representations specialize for next-token prediction rather than grounding. The other backbones behave the same way, peaking mid-network (layer 14 of 32 for LLaVA-1.5 and InternVL2); the peak layer is chosen per backbone on validation data. Reading and training from several layers at once, consistently underperforms the single peak layer. Because grounding is concentrated in a narrow mid-network band, depth pooling dilutes it with early generic features and late next-tokenprediction features. Varying activation scales further bias naive averaging toward high-norm, lessinformative layers.

Ablation Study

We analyze the key design choices underlying UniProbe through ablations of its architecture and hallucination-aware decoding. Unless stated otherwise, we evaluate the detector on MHALO with the GLM-4V backbone; additional ablations are provided in the appendix. Architecture and components. Table 5 compares UniProbe with flat probes and ablates its

Decoding threshold. The threshold τ trades hallucination reduction against content preservation (Fig. 5). We use τ = 0.70, the most aggressive setting that reduces hallucinations without removing correctly generated content. Lowering it to τ = 0.40 reduces CHAIRi by 36%, but also removes 12% of correct mentions. 8

7

Conclusion

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, and 1 others. 2024. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198.

We introduced UniProbe, a lightweight learnable detector that models the internal attention, visual, and response representations of a frozen LVLM. Across multiple backbones and benchmarks, it improves token-level and object-level hallucination detection while keeping the underlying model frozen. We also propose a streaming variant that further reduces object hallucinations during decoding while preserving response quality and adding little latency. These results show that structured internal model traces provide a practical signal for both hallucination detection and mitigation.

Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). Yung-Sung Chuang, Linlu Qiu, Cheng-Yu Hsieh, Ranjay Krishna, Yoon Kim, and James Glass. 2024. Lookback lens: Detecting and mitigating contextual hallucinations in large language models using only attention maps. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Limitations Despite its effectiveness, our approach has several limitations. It requires access to hidden states and attention maps, and applicable only to open-source models or directly by closed-source providers. We also train a separate readout for each backbone, as internal representations differ across models, and leave cross-backbone transfer to future work.

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. InstructBLIP: Towards general-purpose visionlanguage models with instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS).

References Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR).

Amos Azaria and Tom Mitchell. 2023. The internal state of an LLM knows when it‘s lying. In Findings of the Association for Computational Linguistics: EMNLP 2023. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631.

Jinhao Duan, Fei Kong, Hao Cheng, James Diffenderfer, Bhavya Kailkhura, Lichao Sun, Xiaofeng Zhu, Xiaoshuang Shi, and Kaidi Xu. 2025. Truthprint: Mitigating large vision-language models object hallucination via latent truthful-guided pre-intervention. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).

Guy Bar-Shalom, Fabrizio Frasca, Yaniv Galron, Yftah Ziser, and Haggai Maron. 2025. Beyond token probes: Hallucination detection via activation tensors with ACT-ViT. In Advances in Neural Information Processing Systems (NeurIPS).

Laura Fieback, Jakob Spiegelberg, and Hanno Gottschalk. 2024. MetaToken: Detecting hallucination in image descriptions by meta classification. arXiv preprint arXiv:2405.19186.

Guy Bar-Shalom, Fabrizio Frasca, Derek Lim, Yoav Gelberg, Yftah Ziser, Ran El-Yaniv, Gal Chechik, and Haggai Maron. 2026. Beyond next token probabilities: Learnable, fast detection of hallucinations and data contamination on LLM output distributions. In Proceedings of the AAAI Conference on Artificial Intelligence.

Fabrizio Frasca, Guy Bar-Shalom, Yftah Ziser, and Haggai Maron. 2026. Neural message-passing on attention graphs for hallucination detection. In International Conference on Learning Representations (ICLR).

Yishuo Cai, Renjie Gu, Jiaxu Li, Xuancheng Huang, Junzhe Chen, Xiaotao Gu, and Minlie Huang. 2025. MHALO: Evaluating MLLMs as fine-grained hallucination detectors. In Findings of the Association for Computational Linguistics: ACL 2025.

Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. 2017. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning (ICML).

9

Nhat Hoang-Xuan, Minh Vu, My T. Thai, and Manish Bhattarai. 2026. Pas: Prelim attention score for detecting object hallucinations in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Common objects in context. In European Conference on Computer Vision (ECCV). Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng, Ji Qi, Junhui Ji, Lihang Pan, Shuaiqi Duan, Weihan Wang, Yan Wang, Yean Cheng, Zehai He, Zhe Su, Zhen Yang, Ziyang Pan, Aohan Zeng, and 73 others. 2025. GLM-4.1V-Thinking: Towards versatile multimodal reasoning with scalable reinforcement learning.

Tuan Dung Nguyen, Minh Khoi Ho, Qi Chen, Yutong Xie, Nguyen Cam-Tu, Minh Khoi Nguyen, Dang Huy Pham Nguyen, Anton van den Hengel, Johan W. Verjans, Phi Le Nguyen, and Vu Minh Hieu Phan. 2026. Beyond the global scores: Fine-grained token grounding as a robust detector of LVLM hallucinations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.

Eunkyu Park, Minyeong Kim, and Gunhee Kim. 2025. Halloc: Token-level hallucination localization for vision-language models. In CVPR. Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallucination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Nicholas Jiang, Anish Kachinthaya, Suzanne Petryk, and Yossi Gandelsman. 2025. Interpreting and editing vision-language representations to mitigate hallucinations. In The Thirteenth International Conference on Learning Representations (ICLR).

Dvir Samuel, Rami Ben-Ari, Matan Levy, Nir Darshan, and Gal Chechik. 2024. Where’s waldo: Diffusion features for personalized segmentation and retrieval. In Advances in Neural Information Processing Systems (NeurIPS).

Zhangqi Jiang, Junkai Chen, Beier Zhu, Tingjin Luo, Yankun Shen, and Xu Yang. 2024. Devils in middle layers of large vision-language models: Interpreting, detecting and mitigating object hallucinations via attention lens. arXiv preprint arXiv:2411.16724.

Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 2019. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).

Sai Akhil Kogilathota, Sripadha Vallabha E G, Luzhe Sun, and Jiawei Zhou. 2026. HALP: Detecting hallucinations in vision-language models without generating a single token. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

Gaurang Sriramanan, Siddhant Bharti, Vinu Sankar Sadasivan, Shoumik Saha, Priyatham Kattakinda, and Soheil Feizi. 2024. LLM-check: Investigating detection of hallucinations in large language models. In Advances in Neural Information Processing Systems (NeurIPS).

Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large visionlanguage models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Issar Tzachor, Boaz Lerner, Matan Levy, Michael Green, Tal Berkovitz Shalev, Gavriel Habib, Dvir Samuel, Noam Korngut Zailer, Or Shimshi, Nir Darshan, and Rami Ben-Ari. 2025. Effovpr: Effective foundation model utilization for visual place recognition. In International Conference on Learning Representations (ICLR).

Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. 2024. Inferencetime intervention: Eliciting truthful answers from a language model. Advances in Neural Information Processing Systems. Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP).

Issar Tzachor, Dvir Samuel, and Rami Ben-Ari. 2026. Vidvec: Unlocking video mllm embeddings for video-text retrieval. arXiv preprint arXiv:2602.08099. Yuiga Wada, Kazuki Matsuda, Komei Sugiura, and Graham Neubig. 2026. ZINA: Multimodal Fine-grained Hallucination Detection and Editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. 2014. Microsoft COCO:

10

Spencer Whitehead, Jacob Phillips, and Sean Hendryx. 2024. Pre-training multimodal hallucination detectors with corrupted grounding data. arXiv preprint arXiv:2409.00238. Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. 2024. Woodpecker: Hallucination correction for multimodal large language models. Science China Information Sciences. Yudong Zhang, Ruobing Xie, Xingwu Sun, Yiqing Huang, Jiansheng Chen, Zhanhui Kang, Di Wang, and Yu Wang. 2024. Dhcp: Detecting hallucinations by cross-modal attention pattern in large visionlanguage models. Proceedings of the 33rd ACM International Conference on Multimedia.

11

Appendix

Method

A

UniProbe (Ours) PAS (Hoang-Xuan et al., 2026) HaloDet (Cai et al., 2025)

Implementation Details

We extract internal signals from a single midnetwork layer of each frozen backbone, chosen per backbone by validation F1 over a coarse layer sweep: layer 20 for GLM-4.1V and InstructBLIP7B, and layer 14 for LLaVA-1.5 and InternVL2. UniProbe uses a hidden width of h = 256 and L = 2 alternating blocks across all backbones (∼16M trainable parameters). Each block applies one message-passing GNN layer, a single-layer bidirectional GRU over the response tokens, and one ViT encoder layer over the image nodes (4 attention heads, feed-forward width 2h); dropout is 0.2 (0.1 within the ViT layers). Pruning is done following (Frasca et al., 2026). We train for 8 epochs with per-graph updates (one graph per step) using Adam with a learning rate of 10−3 , weight decay of 10−4 , and class-balanced binary crossentropy. For hallucination-aware decoding, we use the streaming detector with a threshold τ = 0.7. The threshold is applied to the detector probability pi and is selected on a held-out development set. Unless otherwise noted, all architectural, graphconstruction, training, and decoding hyperparameters reported below were selected through ablation studies on held-out validation data. For selfadaptation, we sample the target model’s own captions on 500 Objects365 images. All experiments, for both UniProbe and the trainable detectors are conducted on a single H100 GPU.

B

Preferred (%) 55 30 15

Table 6: Human preference study on mitigated captions (100 raters × 20 comparisons = 2,000 judgments). Fraction of judgments in which each method’s caption was chosen as the best description of the image.

rate above 90%. Each participant received 10 cents for each rating. Instructions. Participants received the following instruction: “For each image, you will see three captions presented in random order. Select the caption that best describes the visible content of the image. Consider factual accuracy and completeness, and do not prefer a caption solely because it is longer or more fluent.” Each comparison required exactly one selection. Consent and ethics. Before participating, raters were informed that their anonymous judgments would be used for research and provided consent. No personally identifying information was collected or retained. Results. Table 6 reports the outcome. Raters prefer UniProbe’s mitigated caption in 55% of judgments, against 30% for PAS and 15% for HaloDet. Human preference thus mirrors the automatic CHAIR ranking of Table 4: the detector that localizes hallucinations most precisely also produces the captions people find most faithful, by a wide margin.

User Study

Automatic object-hallucination rates (Table 4) measure whether the guardrail removes ungrounded objects, but not whether the resulting caption reads, to a person, as a better description of the image. We therefore complement Table 4 with a human preference study over the mitigated captions of the three guardrail-driven methods: UniProbe, PAS, and HaloDet.

C

Additional Qualitative Results

UniProbe can be applied across backbones and across hallucination types. Figure 6 collects additional detection-and-mitigation examples on three backbones (GLM-4.1V, Qwen-VL, and LLaVA1.5) that span the error modes we target: object hallucination (a phantom “2 cats” invented in an otherwise empty train station), attribute hallucination (swapped colors and lighting, such as “white pants and a yellow shirt” or “at night”), and objectpresence errors on POPE-style questions, where the backbone denies an object (“skis”, “oven”) that is in fact present. In each case UniProbe localizes the hallucinated span while the trained detectors HaloDet and HalLocalizer miss it; conditioned on that signal, our mitigation rewrites the response

Protocol. We assemble 100 comparisons; each shows an image alongside the mitigated caption produced by each of the three methods, presented in random order and without method labels. 100 raters recruited through Mechanical Turk each judge 20 comparisons drawn at random from this set (2,000 judgments in total). Participants were required to be fluent in english and have an approval 12

Attention edge features

F1M

F1IoU

Mean over heads (UniProbe) All heads, single layer All heads, all layers

63.1 61.9 62.5

52.4 51.8 50.8

Describe the image. GLM-4V-Thinking: The image features a red and white train parked, at night, inside a train station. There are 2 cats in the scene, both located near the entrance of the train. HaloDet: The image features a red and white train parked, at night, inside a train station. There are 2 cats in the scene, both located near the entrance of the train. HalLocalizer: The image features a red and white train parked, at night, inside a train station. There are 2 cats in the scene, both located near the entrance of the train. Ours (Detection): The image features a red and white train parked, at night, inside a train station. There are 2 cats in the scene, both located near the entrance of the train.

Table 7: Head averaging on MHALO. Collapsing attention to the mean over heads (one scalar per edge) beats retaining all heads as per-edge features with a learned gate, which adds parameters and L×H features per edge without improving detection.

Ours (Mitigation): The image features a red and white train parked, at daytime, inside a train station. There are windows visible in the background, providing natural light to the area.

into a grounded one, removing the cats and correcting “night” to “daytime”, swapping the colors back, or answering “Yes”. The presence cases are the most striking: the backbone verbally denies an object that its own internal state encodes as present, and UniProbe recovers it directly from that trace.

D

Describe the image. Qwen-3-VL: The skateboarder is wearing white pants and a yellow shirt.

HaloDet: The skateboarder is wearing white pants and a yellow shirt.

Miss all

HalLocalizer: The skateboarder is wearing white pants and a yellow shirt. Ours (Detection): The skateboarder is wearing white pants and a yellow shirt.

Additional Ablation Results

Ours (Mitigation): The skateboarder is wearing yellow pants and a white shirt.

We present additional ablation studies that complement the results reported in the main paper.

Is there a skis in the image? LLaVA-1.5: No

Budget HaloDet:

UniProbe keeps the trace graph compact by bounding two quantities: the number of nodes it retains (at most 200 image and 64 query patches, ranked by the attention they receive from the response), and the number of edges each response token draws from every modality. (1) Visual budget: Sweeping the image-node cap ({100, 200, 400, 600}) and the per-token image edges (kimg ∈ {8, 16, 32, 64}) leaves F1 essentially flat (F1M 62.1–63.0, no trend): a small set of the most-attended patches already carries the visual evidence, and top-16 image edges per token suffice. (2) Query/response edge budget: Varying the per-token edges to query tokens (kqry ∈ {4, 8, 12, 16}) and to preceding response tokens (kresp ∈ {4, 8, 12}) leaves F1 within seed noise (F1M 62.4–63.2): a handful of edges to the most-attended context is enough; the defaults are top-12 query and top-8 response edges.

HalLocalizer:

No No Miss

Ours (Detection): Ours (Mitigation):

No Yes

Is there an oven in the image? LLaVA-1.5: No

HaloDet: HalLocalizer: Ours (Detection): Ours (Mitigation):

No No

Miss

No Yes

Figure 6: Additional detection and mitigation examples across backbones and hallucination types. For each example we show the backbone response with the tokens flagged as hallucinated by HaloDet, HalLocalizer, and UniProbe (Detection), followed by UniProbe’s corrected generation (Mitigation). The cases cover object hallucination (an invented “2 cats”), attribute hallucination (swapped colors and “night” → “daytime”), and object-presence errors on POPE questions (“skis” and “oven” denied though present). UniProbe localizes and repairs hallucinations that the baseline detectors miss, on GLM-4.1V, Qwen-3-VL, and LLaVA-1.5.

Head Averaging Ablation UniProbe summarizes each attention edge by the mean attention over heads. Table 7 compares this against keeping all heads as separate per-edge features (with a learned gate over them), evaluated under an identical protocol. Averaging over heads is both the most accurate and the lightest: retaining every head adds an L×H-dimensional feature per edge plus a gating layer, yet slightly lowers 13

detection, whether the heads are read at the single mid-network layer or across all layers. Streaming Read-out Making the response GRU unidirectional, as tokenby-token deployment requires, costs ∼2 F1M on teacher-forced MHALO (63.2 → 61.2, F1IoU tied), and nothing where it is actually used: on the model’s own generations the streaming and bidirectional detectors tie (CHAIR detection F1 32.6 vs. 32.2).

14

Related documents

• From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs#632874
• Scalable Token-Level Hallucination Detection in Large Language Models#178887
• GenCOPE: Syn2Real Generalized Category-Level Object Pose Estimation for Robotic Picking#1157029
• HalluTracer: Hallucination Detection via Depth-Averaging Truth Signals#620753
• HalluTracer: Hallucination Detection via Depth-Averaging Truth Signals#619074