arXiv CSopen access

ClinHallu: A Benchmark for Diagnosing Stage-Wise Hallucinations in Medical MLLM Reasoning

ClinHallu: A Benchmark for Diagnosing Stage-wise Hallucinations in Medical MLLM Reasoning Sicheng Yang∗1,2 , Hangjie Yuan∗‡2,3,4 , Wenjun Zhang2 , Jinwang Wang2,3 , Yichen Qian2,3 , Weihua Chen†2,3 , Fan Wang2 , Lei Zhu†1 1

The Hong Kong University of Science and Technology (Guangzhou) DAMO Academy, Alibaba Group 3 Hupan Lab 4 Zhejiang University 2

arXiv:2606.14697v1 [cs.CV] 12 Jun 2026

* Equal contribution; ‡ Project Lead; † Corresponding authors.

Building trustworthy medical multimodal large language models (MLLMs) is critical for reliable clinical decision support. Existing medical hallucination benchmarks mainly focus on data collection, but often ignore where hallucinations originate within the reasoning process. We find that hallucination sources vary across samples: errors may arise from visual misrecognition, incorrect medical knowledge recall, or flawed reasoning integration. To enable source-level hallucination diagnosis, we introduce ClinHallu, a benchmark for stage-wise hallucination diagnosis in medical MLLM reasoning. ClinHallu contains 7,031 validated instances, where each instance is augmented with a structured reasoning trace decomposed into Visual Recognition, Knowledge Recall, and Reasoning Integration. We also use stagereplacement interventions to measure how correcting specific stages affects the final answer. Beyond evaluation, we show that trace-supervised fine-tuning reduces stage-wise hallucinations. ClinHallu provides a fine-grained hallucination testbed for diagnosing and mitigating reasoning failures in medical MLLMs. The benchmark is publicly available at https://github.com/alibaba-damo-academy/ClinHallu.

Date: June 15, 2026

1

Introduction

Recent medical hallucination benchmarks have made important progress in evaluating unreliable model outputs. For example, Med-HALT (Pal et al., 2023) examines hallucination in medical LLMs, while multimodal benchmarks such as CARES (Xia et al., 2024) and Med-HallMark (Chen et al., 2024a) extend hallucination evaluation to medical vision-language models. Despite these advances, most existing evaluations remain centered on the final output: they judge whether the model’s answer or response is correct, and then use this judgment to determine whether hallucination occurs. Such evaluations can identify that a model produces an unreliable answer, but provide limited evidence about how the error arises during multimodal reasoning. As illustrated in Fig. 1, the same wrong answer may be caused by different trace-level failures: the model may misrecognize the visual evidence, recall incorrect medical knowledge, or fail to properly integrate relevant evidence and knowledge. When these distinct failure sources are collapsed into a single final-answer judgment, current benchmarks have limited ability to diagnose where

MLLMs are increasingly used in medical scenarios (Li et al., 2023a; Chen et al., 2024b; Jiang et al., 2025), including medical visual question answering (VQA) (Liu et al., 2021; Zhang et al., 2023; Zuo et al., 2025; Yao et al., 2026), report generation (Zambrano Chaves et al., 2025), and clinical decision support (Singhal et al., 2025; Tanno et al., 2025; Yang et al., 2026). These applications place high demands on reliability. However, in real-world medical use, a model may describe a non-existent lesion in an image, associate it with an incorrect clinical implication, and still present the response in a confident manner (Xia et al., 2024; Asgari et al., 2025). Such seemingly plausible but unsupported outputs are referred to as “hallucinations” (Li et al., 2023b; Liu et al., 2024; Huang et al., 2025; Ji et al., 2023). They remain a central obstacle to the reliable use of MLLMs in highstakes medical settings, as they can mislead clinical interpretation and compromise downstream medical decision-making (Pal et al., 2023; Kim et al., 2025). 1

hallucinations originate and how they propagate.

Question: What hypoattenuated tissue is between the abdominal wall and skin?

To address this limitation, we introduce ClinHallu, a benchmark for stage-wise hallucination diagnosis in medical MLLM reasoning. We construct ClinHallu from four medical VQA datasets, yielding 7,031 validated instances. ClinHallu augments each medical VQA instance with a validated reference trace decomposed into Visual Recognition, Knowledge Recall, and Reasoning Integration, and uses stage-replacement interventions to test how correcting specific stages affects the final answer. Experiments on representative MLLMs show that ClinHallu reveals stagedependent failure patterns, and quantifies how visual and knowledge errors propagate into downstream reasoning. These results demonstrate the value of ClinHallu as a fine-grained diagnostic testbed for medical MLLMs. In summary, our contributions are:

Golden Answer

MLLM

Reasoning

The CT shows a continuous hypoattenuated layer between the skin and abdominal wall. Since fat is normally subcutaneous and appears hypoattenuated on CT, the answer is fat.

Visual Hallucination

The CT shows a localized fluid-like pocket between the skin and abdominal wall. A focal subcutaneous collection can appear hypoattenuated on CT, the answer is abscess.

Knowledge Hallucination

The CT shows a continuous hypoattenuated layer between the skin and abdominal wall. Since subcutaneous hypoattenuation usually indicates abscess, the answer is abscess.

Reasoning Hallucination

The CT shows a continuous hypoattenuated layer between the skin and abdominal wall. Although this matches normal subcutaneous fat, the term “hypoattenuated” is mistaken as a pathological sign, so the answer is abscess.

Figure 1 Different reasoning failures can produce the

same wrong answer in medical VQA. In this example, the correct answer is “fat”, but visual misrecognition, incorrect knowledge recall, and flawed reasoning integration can each lead the model to answer “abscess”. This motivates ClinHallu, which diagnoses hallucinations by localizing them to specific reasoning stages rather than only judging final-answer correctness.

• We present a data curation pipeline for constructing ClinHallu, a benchmark for sourcelevel hallucination diagnosis in medical MLLMs. ClinHallu contains 7,031 validated medical VQA instances, each augmented with structured reference traces covering visual recognition, knowledge recall, and reasoning integration.

swer in incorrect visual evidence (Lyu et al., 2023), relying on inaccurate medical knowledge, or drawing an unsupported conclusion (Chang et al., 2025). Therefore, hallucination evaluation is essential for building trustworthy medical MLLMs.

• We design a stage-wise evaluation pipeline and replacement-based interventions, enabling hallucination diagnosis by identifying which reasoning stage limits final-answer correctness. We further evaluate 11 representative closed-source and open-source MLLMs on ClinHallu.

Medical hallucination benchmarks. Medical hallucina-

• We provide a fine-grained analysis of hallucination bottlenecks across datasets and models. Beyond diagnosis, we show that annotated structured traces can serve as supervision for reducing stage-wise hallucinations.

2

Knowledge

Gold (GT) Answer: fat Wrong Answer (all three): abscess

Visual

tion benchmarking has seen rapid progress. Textonly benchmarks, such as Med-HALT (Pal et al., 2023), MedHalu (Agarwal et al., 2024), and MedHallu (Pandit et al., 2025), mainly focus on hallucination detection in medical question answering, healthcare queries, and clinical knowledge assessment. Multimodal benchmarks, including CARES (Xia et al., 2024), Med-HallMark (Chen et al., 2024a), MedVH (Gu et al., 2026), MedHallBench (Zuo and Jiang, 2024), MedHallTune (Yan et al., 2025), and MedHEval (Chang et al., 2025), further extend hallucination evaluation to medical VLMs through visual question answering or trustworthiness assessment.

Related Work

Reasoning in medical MLLMs. Medical MLLMs have

recently shown strong potential in visual question answering, report understanding, and clinical decision support (Li et al., 2023a; Saab et al., 2024). Built upon general-purpose MLLMs such as GPT-4V (OpenAI, 2023), Gemini (Team et al., 2023), LLaVA (Liu et al., 2023), and Qwen-VL (Bai et al., 2023), medical variants (e.g., Med-Gemma (Sellergren et al., 2025)) adapt multimodal reasoning capabilities to specialized medical scenarios. Recent efforts also enhance medical MLLM reasoning, for example through CoT (Wei et al., 2022) and in-context learning (Brown et al., 2020; Dong et al., 2024). Nevertheless, models may produce an explanation while grounding its an-

However, existing medical hallucination benchmarks remain largely answer-centric: they can identify hallucinated outputs, but offer limited insight into their underlying sources. To address this limitation, we introduce ClinHallu, a benchmark and evaluation framework that uses structured reasoning traces to diagnose not only whether hallucination occurs, but also where it originates. A detailed comparison with existing benchmarks is provided in Table 1. 2

Table 1 Comparison with representative medical hallucination benchmarks.

We compare ClinHallu with existing benchmarks in terms of data scale, reasoning-process supervision, and hallucination evaluation. ClinHallu uniquely supports structured chain-of-thought (CoT) annotations, stage-wise traces, source localization, and hallucination rate evaluation, enabling fine-grained diagnosis of medical MLLM hallucinations.

3

Benchmark

Data Size

Multimodal

Structured CoT

Stage-wise Trace

Med-HALT MedHalu MedHallu CARES Med-HallMark MedVH MedHallTune MedHEval

18,866 2,077 10,000 41K 7,341 N/A 100K 15,976

✗ ✗ ✗ ✓ ✓ ✓ ✓ ✓

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✗

✗ ✗ ✗ ✗ ✗ ✗ ✗ ✓

✓ ✓ ✓ ✗ ✓ ✓ ✗ ✓

ClinHallu

7,031

✓

✓

✓

✓

✓

ClinHallu Benchmark

trace generator Gref produces τi = Gref (xi , qi ),

We introduce ClinHallu, a stage-wise hallucination diagnosis benchmark for MLLMs. Let xi denote the medical image or image set, qi the question, ai the ground-truth answer, and G the MLLM under evaluation. Conventional VQA evaluation compares the model prediction âi = G(xi , qi ) with ai , which only measures final-answer correctness and leaves the reasoning process unexamined.

(1)

(3)

Trace validation and filtering. Since reference traces

where τi records the reference reasoning process leading to the answer. Accordingly, each model is asked to generate both a trace and an answer, (τ̂i , âi ) = G(xi , qi ),

τi = (vi , ki , ri ).

Here, vi , ki , and ri denote Visual Recognition, Knowledge Recall, and Reasoning Integration, respectively. For example, in Fig. 2, the trace first observes that the brain image contains bright fluid, then recalls that fluid is bright in “T2-weighted MRI” but dark in “T1-weighted MRI”, and finally connects the observation with this rule to answer “T2-weighted MRI”. This decomposition separates visual evidence, medical knowledge, and their integration, enabling stagewise hallucination analysis.

As illustrated in Fig. 2, ClinHallu augments each VQA sample di with a validated structured trace: di ClinHallu = (xi , qi , τi , ai ),

Hallucination Hallucination Source Loc. Rate Eval.

are generated at scale, we further filter them to ensure their reliability. For each generated trace τi = (vi , ki , ri ), we apply an LLM-as-judge model J(·) to evaluate two criteria: format validity and answer consistency:

(2)

so that hallucinations can be localized by comparing the generated τ̂i with the reference τi , with detailed definitions provided below.

ans (cfmt i , ci ) = J(τi , xi , ai ),

(4)

tive medical VQA datasets: VQA-RAD (Lau et al., 2018), PathVQA (He et al., 2020), MedFrameQA (Yu et al., 2025), and MedXpertQA (Zuo et al., 2025). They provide complementary coverage across medical domains, imaging modalities, and task formulations.

where cfmt checks whether the trace follows the rei quired three-stage format and whether all stages are non-empty, and cans checks whether the trace supi ports the ground-truth answer ai without introducing conflicting conclusions. We retain a trace only when both criteria are satisfied:   (5) ϕ(τi ) = 1 cfmt ∧ cans . i i

Structured reasoning trace construction. We augment

The final benchmark is then defined as

Source data. ClinHallu integrates four representa-

each standardized VQA sample with a structured reference reasoning trace. Given xi and qi , a reference

DClinHallu = {(xi , qi , τi , ai ) | ϕ(τi ) = 1}. 3

(6)

Source datasets

Trace generation

VQA-RAD

Trace Filtering

PathVQA (52.81%) MedFrameQA (24.3%)

Path-VQA

MedXpertQA (18.1%)

Format Filtering

Trace MedFrameQA

MedXpertQA

VQA-RAD (4.79%)

R

K

V

Visual Recognition

ClinHallu

Q: What imaging modality is this?

Knowledge Recall

V: Brain image; fluid appears bright K: T1: fluid dark; T2: fluid bright R: Fluid matches T2 characteristics

Answer Acc

Answer

Reasoning Integration

V

K

R

A: T2-weighted MRI

Figure 2 Overview of the ClinHallu construction pipeline. ClinHallu integrates four medical VQA datasets and augments

each sample with a structured reasoning trace covering Visual Recognition (V), Knowledge Recall (K), and Reasoning Integration (R). Generated traces are filtered by format validity and answer consistency, yielding validated stage-wise annotations for diagnosing hallucination sources in medical MLLM reasoning.

4.2

This filtering step ensures that retained traces are complete and answer-consistent, providing reliable references for downstream evaluation.

We first evaluate whether the final answer produced by the candidate MLLM is correct. For each instance, an answer judge J(·) compares the predicted answer âi with the ground-truth answer ai and assigns a binary correctness label:

After filtering, DClinHallu contains 7,031 validated VQA instances from four source datasets. Each instance includes the original multimodal sample and a three-stage reference trace, supporting stage-wise hallucination analysis. As shown in Table 1, prior benchmarks typically focus on text-only hallucination or lack structured reasoning traces for multimodal settings. ClinHallu instead combines multimodal inputs, structured CoT annotations, source localization, and hallucinationrate evaluation, enabling fine-grained diagnosis of medical MLLM failures. Released benchmark instances.

4

Evaluation

4.1

Evaluation Overview

Answer-Level Evaluation

ci = J(xi , âi , ai ),

ci ∈ {0, 1},

(7)

where ci = 1 indicates a correct answer and ci = 0 otherwise. The answer-level accuracy is: |D|

Acc =

1 X ci . |D| i=1

(8)

However, final-answer accuracy cannot identify the source of an error. We therefore introduce stagewise hallucination evaluation to localize failures in different sources, i.e. visual recognition, knowledge recall, and reasoning integration.

4.3

ClinHallu evaluates both final-answer correctness and the source of hallucinations in the reasoning process. As illustrated in Fig. 3, for each instance xi , the evaluated model first generates a structured trace τ̂i = (v̂i , k̂i , r̂i ) and then produces answer âi . Given the reference trace τi = (vi , ki , ri ) and answer ai , ClinHallu conducts three evaluations: (1) answerlevel evaluation, which measures whether âi matches ai ; (2) stage replacement intervention, which replaces selected generated stages with reference stages (e.g., v̂i → vi ) to obtain decoupled stage-wise evaluations; and (3) stage-wise diagnosis, which reports hallucination rates at each stage and measures replacementinduced answer-accuracy changes.

Stage-Wise Evaluation

Reasoning hallucinations may arise from upstream errors in visual recognition or knowledge recall, leading to a cascading effect. To disentangle these and identify which stage contributes most to hallucination, we apply stage replacement interventions to decouple the structured CoT, and then analyze hallucination rates and answer accuracy before and after replacement. As illustrated in Fig. 3 (b), for each intervention, one or more generated stages are replaced with their reference counterparts, and the evaluated MLLM G is asked to continue the remaining reasoning process and produce a new answer. Specifically, for each instance xi , Stage replacement intervention.

4

(a) Model Prediction Q: imaging modality?

MLLM

(b) Stage-wise Replacement Rep-V

V

Rep-K

V

Rep-VK

V

Vision Knowledge Reasoning

K ...R ...A ... A R K ... ... R K A ... ...

MLLM

Rep-V

V Brain image; V: fluid appears dark

V Brain image; V: fluid appears bright

K: K T1: fluid dark; T2: fluid bright

K T1: fluid dark;

V

K

R

A

MLLM

V

K

R

A

A

T1 MRI

80.4

65.9 Qwen2.5

82.5

25.7

R Fluid matches T2 characteristics

T1 characteristics

A

GT

T2: fluid bright

R: Fluid matches R

Answer

(c) Stage-wise Judging

Gemini3

T2 MRI

H𝑉

H𝐾

H𝑅

Acc

Figure 3 Evaluation protocol of ClinHallu. Given a medical VQA sample, the evaluated MLLM generates a structured

trace and final answer. We then replace selected generated stages with validated reference stages and ask the model to complete the remaining reasoning process. The resulting traces and answers are judged against the references to compute stage-wise hallucination rates (HV , HK , HR ) and answer accuracy (Acc), enabling diagnosis of the main bottleneck in medical MLLM reasoning.

let (vi , ki , ri , ai ) denote the reference output, and let (v̂i , k̂i , r̂i , âi ) denote the output generated by G.

the reasoning stage generated after replacing both visual and knowledge stages, as defined in Eq. 11. A value of 1 indicates that the corresponding stage contains hallucinated content. We then compute the hallucination rate for each stage:

For visual-stage replacement, the reference visual stage vi is provided, and G generates the remaining knowledge, reasoning, and answer: Rep-V :

(k̂i , r̂i , âi ) = G(xi , vi ).

(9)

|D|

1 X s h , H = |D| i=1 i s

For knowledge-stage replacement, the generated visual stage v̂i is retained while the reference knowledge stage ki is provided: Rep-K :

(r̂i , âi ) = G(xi , v̂i , ki ).

(r̂i , âi ) = G(xi , vi , ki ).

(10)

In addition to hallucination rates, we use answer accuracy changes to diagnose which upstream stage most affects final-answer correctness. For each replacement setting s ∈ {Rep-V, Rep-K, Rep-VK}, we compute the average accuracy gain over all evaluated models:

Accuracy diagnosis.

(11)

Hallucination rate evaluation. We assess each stage

under the intervention context where its upstream stages are fixed to reference counterparts. Specifically, the hallucination labels for each stage are defined as:

∆sAcc =

1 X Accsm − AccORG , m |M|

(14)

m∈M

hVi = J(xi , v̂i , vi ), hK i = J(xi , k̂i,REP-V , ki ),

(13)

By controlling upstream stages through replacement, these hallucination rates directly reflect the model’s hallucination tendency at each stage.

For joint visual-and-knowledge replacement, both reference stages vi and ki are provided, and G generates only the remaining r̂i and âi : Rep-VK :

s ∈ {V, K, R}.

where AccORG denotes the original answer accuracy m of model m defined by the answer judge in Eq. 8, and Accsm denotes its answer accuracy under replacement setting s. A larger ∆sAcc indicates that correcting the corresponding stage leads to a greater improvement in final-answer correctness, suggesting that this stage is a more important bottleneck in the reasoning process.

(12)

hR i = J(xi , r̂i,REP-VK , ri ), R where hVi , hK i , hi ∈ {0, 1}. Here, k̂i,REP-V denotes the knowledge stage generated after replacing the visual stage, as defined in Eq. 9, and r̂i,REP-VK denotes

5

Table 2 Accuracy and stage-wise hallucination rates on ClinHallu. We report accuracy (Acc; Eq. 8) and hallucination rates (Eq. 13) for Visual Recognition (HV ), Knowledge Recall (HK ), and Reasoning Integration (HR ). Within each model group, the best value for each metric is highlighted in bold. VQA-RAD

Model

Acc↑ HV ↓ HK ↓

PathVQA HR ↓

Acc↑ HV ↓ HK ↓

MedFrameQA HR ↓

Acc↑ HV ↓ HK ↓

MedXpertQA

AVG

HR ↓

Acc↑ HV ↓ HK ↓

HR ↓

Acc↑ HV ↓ HK ↓

HR ↓

Closed-source MLLMs Qwen3-VL-Flash 70.9 Qwen3-VL-Plus 74.2 Gemini-3-Flash 82.5

50.7 44.5

9.2 6.8

6.8 3.9

72.4 72.9

45.7 41.9

13.5 6.7

7.0 5.2

64.0 65.9

51.0 46.7

17.8 10.9

9.1 7.6

47.1 55.9

61.1 55.8

39.4 25.6

7.4 2.1

63.6 67.2

52.1 47.2

20.0 12.5

7.6 4.7

22.9

3.6

3.0

81.6

21.5

2.8

1.9

71.3

31.0

5.5

3.0

85.0

27.6

4.2

1.3

80.1

25.8

4.0

2.3

54.9 65.0 65.3 71.8 69.7 78.6 77.7

59.4 57.0 45.1 38.0 35.0 47.8 38.9

34.1 17.2 12.2 24.3 13.7 9.5 14.0

12.5 4.2 8.9 14.2

46.0 55.0 64.8 60.2 58.7 63.6 69.7

61.2 55.4 48.8 50.1 44.3 47.7 50.3

38.0 23.4 19.8 29.5 21.6

8.4 5.2 5.1 13.6 3.2

40.8

43.9 32.6 25.6 30.1 21.1

2.7

65.9 60.5 52.2 51.1 45.6 50.8 52.0

45.5 33.2 27.3 33.4 26.6 18.8 30.5

32.3

6.2

72.7

34.7

2.2

70.7

29.7 18.6

42.7 51.5 52.7 53.2 53.9 63.8 64.3

80.4

49.7 53.0 41.5

65.8 59.7 51.5 49.8 49.9 37.8 54.2

18.1 7.8 13.6 30.5 6.6

24.2 14.2

24.7 32.0 27.7 26.4 26.5 47.2 43.8

22.3 7.0 16.9 61.7 10.2

2.2

29.1 14.7 23.5 32.4 10.2 9.4 8.6

78.2 71.4 65.6 64.1 62.5

11.4

45.3 53.9 53.0 54.6 60.5 65.7 66.0

8.1

52.6

69.1

41.9

18.7

Open-source MLLMs Qwen2.5-VL-7B Qwen3-VL-8B Lingshu-7B MedGemma-4B InternVL3.5-8B Qwen3-VL-32B Qwen3.5-4B Qwen3.5-9B

4.4

2.7

3.0 3.3 3.3

Evaluation after Training with Traces

Experiments

5.1

Experimental Setup

16.3

58.1

66.0 58.9

35.8

3.1

5.9 5.6

4.4

5.1 4.8

Implementation details. All local open-source models

are served using the vLLM framework (Kwon et al., 2023). For CoT generation, we set the temperature to 0.7 to encourage diverse reasoning traces. For finalanswer judging and stage-wise hallucination judging, we use a lower temperature of 0.01 to ensure deterministic and reproducible evaluation. The prompts used for each stage are provided in Appendix C.

To examine whether the structured traces in ClinHallu can also serve as effective supervision, we conduct trace-supervised fine-tuning on Qwen3.5-9B. Since MedFrameQA and MedXpertQA do not include training sets, we restrict fine-tuning to VQARAD and PathVQA. We construct golden traces for their training splits using the same trace generation pipeline and evaluate the fine-tuned models on the corresponding test sets in ClinHallu. Detailed finetuning configurations are provided in Appendix A.

5

64.8 58.2 49.4 52.3

5.2

Results Analysis

Finding 1.Visual hallucination is generally severe; VQA-RAD is visual-bottlenecked, MedXpertQA is knowledge-bottlenecked, while PathVQA and MedFrameQA are relatively balanced. Table 2 first reveals distinct dataset-level hallucination patterns. Across all subsets, visual hallucination is consistently high, with average rates exceeding 40%. On VQA-RAD, visual hallucination is the dominant error source, with an average rate of 42.9% across models, far higher than knowledge hallucination at 13.7%. MedXpertQA exhibits a different pattern: knowledge hallucination becomes much more severe, reaching 43.1%. By contrast, PathVQA and MedFrameQA show more balanced visual–knowledge error profiles. Tables 3 and 4 lead to the same conclusion based on answer-accuracy changes. Correcting the visual stage on VQA-RAD improves Acc by 15.5%, much larger than the 4.6% gain from correcting knowledge. In contrast, MedXpertQA benefits more from knowledge replacement, with a 33.4% gain compared with 13.8% from Rep-V. PathVQA and MedFrameQA again show more comparable gains between V/K replacement. These results indicate hallucination bottlenecks are dataset-dependent.

Evaluation models. We evaluate a set of both closed-

and open-source MLLMs. The closed-source models comprise Qwen3-VL-Flash (Bai et al., 2025a), Qwen3-VL-Plus (Bai et al., 2025a), and Gemini-3Flash. The open-source models include Qwen2.5-VL7B (Bai et al., 2025b), Qwen3-VL-8B (Bai et al., 2025a), Lingshu-7B (Xu et al., 2025), MedGemma4B (Sellergren et al., 2025), InternVL3.5-8B (Wang et al., 2025), Qwen3-VL-32B (Bai et al., 2025a), and two more recent Qwen variants: Qwen3.5-4B (Qwen Team, 2026), and Qwen3.5-9B (Qwen Team, 2026). For benchmark construction and evaluation, Qwen3.5Plus (Qwen Team, 2026) serves as the generator Gref for golden CoT trace construction, as defined in Eq. 3. We use Qwen3.5-27B (Qwen Team, 2026) as the judge model J for both trace validation and evaluation, including answer correctness and hallucination analysis, as defined in Eqs. 4, 7, and 12. 6

Table 3 Accuracy diagnosis under stage-replacement interventions on ClinHallu. For each subset, we report the original VK answer accuracy (Acc), computed using Eq. 8, and the corresponding accuracy gains, ∆VAcc , ∆K Acc , and ∆Acc , following Eq. 14. Darker blue cells indicate larger gains. Higher ↑ is better.

VQA-RAD

Model

PathVQA

Acc↑

VK ∆VAcc ↑ ∆K Acc ↑ ∆Acc ↑ Acc↑

VK ∆VAcc ↑ ∆K Acc ↑ ∆Acc ↑ Acc↑

70.9 74.2

+19.9 +17.5 +12.5

+11.0 +10.9 +4.8

MedFrameQA

MedXpertQA

VK ∆VAcc ↑ ∆K Acc ↑ ∆Acc ↑ Acc↑

VK ∆VAcc ↑ ∆K Acc ↑ ∆Acc ↑

+15.6 +16.3 +27.6 +16.0 +13.5 +25.8 +14.3 +15.7 +22.1

+16.3 +32.3 +44.2 +18.5 +25.4 +40.8 +8.3 +7.5 +13.5

Closed-source MLLMs Qwen3-VL-Flash Qwen3-VL-Plus Gemini-3-Flash

82.5

+4.8 +2.7 +1.5

+22.3 +19.0 +12.2

72.4 72.9 81.6

+9.9 +7.9 +5.4

+18.7 +18.2 +8.7

64.0 65.9 71.3

47.1 55.9 85.0

Open-source MLLMs Qwen2.5-VL-7B Qwen3-VL-8B Lingshu-7B MedGemma-4B InternVL3.5-8B Qwen3-VL-32B Qwen3.5-4B Qwen3.5-9B

54.9 65.0 65.3 71.8 69.7 78.6 77.7 80.4

+22.2 +21.9 +18.1 +6.8 +16.9 +11.0 +11.9 +11.6

+6.8 +3.9 +5.9 +9.5 +5.1 +2.1 +4.2 +3.9

+32.6 +28.2 +23.4 +11.3 +25.5 +15.7 +15.1 +14.2

46.0 55.0 64.8 60.2 58.7 63.6 69.7 72.7

+19.8 +19.7 +10.5 +8.8 +17.3 +19.1 +8.7 +9.2

+16.7 +13.3 +15.3 +16.0 +17.9 +12.3 +10.0 +10.3

Table 4 Average gains under stage-replacement interventions.

     

VQA-RAD PathVQA MedFrameQA MedXpertQA

+15.5 +12.7 +13.2 +13.8

+4.6 +12.3 +15.7 +33.4

%UHDN5DWH

VK ∆VAcc ↑ ∆K Acc ↑ ∆Acc ↑

)L[5DWH

Each value reports the model-averaged accuracy gain corresponding to Table 3. Darker cells indicate larger gains. The varying patterns suggest that hallucination bottlenecks differ across datasets. Subset

+35.5 +32.7 +21.3 +17.4 +30.1 +26.6 +17.8 +16.1

+20.0 +22.1 +24.7 +43.4

45.3 53.9 53.0 54.6 60.5 65.7 66.0 70.7

+9.7 +18.3 +12.6 +6.0 +15.1 +15.5 +13.0 +9.4

94$5$'    9 . 9.

+15.6 +18.1 +18.5 +12.3 +19.2 +12.8 +16.7 +14.0

+26.2 +34.5 +26.5 +10.2 +27.8 +25.9 +25.0 +20.0

3DWK94$ 

24.7 32.0 27.7 26.4 26.5 47.2 43.8

+8.8 +18.4 +11.4 +7.9 +15.3 +18.0 +16.4 +12.7

52.6

0HG)UDPH4$ 

 

 

9 . 9.

9 . 9.





+50.6 +60.6 +56.0 +8.6 +62.9 +47.6 +50.3 +42.0

0HG;SHUW4$   

9 . 9.



 

+38.4 +42.1 +47.8 +24.7 +46.7 +33.9 +37.9 +30.9

  

  

9 . 9.

9 . 9.



 

 

9 . 9.

9 . 9.

Figure 4 Fix and break rates under stage-replacement

interventions. We report gains after replacing V (Visual), K (Knowledge), and VK (both) across four subsets.

Finding 2.Replacing visual and/or knowledge stages improves performance, with datasetdependent gains. Average gains cannot distinguish whether replacement corrects wrong answers or breaks originally correct ones. We therefore report “Fix” and “Break” rates to measure samplelevel changes under each replacement setting s ∈ {V, K, VK}. Let ci ∈ {0, 1} denote whether the origi(s) nal answer is correct, and ci ∈ {0, 1} denote whether the answer is correct after replacement. We define i P|D| h (s) 1 c = 0 ∧ c = 1 i i i=1 , Fix(s) = P|D| i=1 1[ci = 0] i (15) P|D| h (s) 1 ci = 1 ∧ ci = 0 i=1 Break(s) = . P|D| i=1 1[ci = 1]

pertQA is knowledge-driven: Rep-K achieves a higher Fix Rate than Rep-V (64.0% vs. 42%) and a lower Break Rate (10% vs. 22%). PathVQA and MedFrameQA show more balanced Fix/Break patterns, suggesting mixed visual and knowledge failure sources.

Finding 3.Reasoning ability is not the primary bottleneck for reliable prediction. Table 2 shows that reasoning-stage hallucination is generally lower than visual- and knowledge-stage hallucinations across most models and datasets. Consistently, Table 3 shows that Rep-VK usually brings the largest accuracy gains. These results suggest that reliability failures mainly arise from upstream visual grounding and knowledge recall, rather than the final reasoning step. MedGemma-4B is an exception, where high reasoning hallucination is partly due to frequent failed or incomplete CoT generation after intervention.

As shown in Fig. 4, replacement effects vary across datasets. On VQA-RAD, Rep-V fixes more wrong answers than Rep-K (73% vs. 29%), consistent with its visual-dominant bottleneck. In contrast, MedX7

Table 5 Ablation study of trace-supervised fine-tuning on Qwen3.5-9B. We compare no fine-tuning (w/o FT),

Table 6 Human validation of automatic evaluation. H-H

denotes agreement between two human annotators. HJ denotes the average agreement between each human annotator and the automatic judge J. We report both agreement accuracy and Cohen’s κ.

answer-only fine-tuning (Ans-only), and trace-supervised variants using visual-recognition (V), knowledge-recall (K), visual-and-knowledge (V+K), and full trace (V+K+R) supervision. All variants are trained on constructed training sets from VQA-RAD and PathVQA and evaluated on their corresponding subsets in ClinHallu. We report answer accuracy (Eq. 8) and stage-wise hallucination rates (Eq. 13). Dataset

Variant

Acc ↑

HV ↓

HK ↓

HR ↓

VQA-RAD

w/o FT Ans-only V K V+K V+K+R

80.4 82.8 82.2 81.9 82.5

32.3 28.2 22.8 31.5 27.0

6.2 5.9 5.6 11.0 5.9

3.3 3.9 3.0 3.6 3.0

83.7

22.6

4.2

2.7

72.7 73.6 77.1 75.6 77.9

34.7 32.8 30.2 44.8 28.0

14.2 11.6 13.9 15.3 9.8

2.2 2.3 2.1 2.5 2.3

78.8

27.6

7.8

2.0

PathVQA

w/o FT Ans-only V K V+K V+K+R

Acc. HV HK HR

H-J

Acc.↑

κ↑

Acc.↑

κ↑

0.962 0.932 0.915 0.910

0.919 0.861 0.822 0.820

0.940 0.917 0.897 0.903

0.872 0.831 0.785 0.805

label, we report agreement accuracy and Cohen’s κ (Cohen, 1960) for both human-human (H-H) and average human-judge (H-J) agreement. As shown in Table 6, J closely matches human judgments, with 94.0% agreement and a Cohen’s κ of 0.872 for Acc, close to human-human agreement (96.2%, κ of 0.919). Stage-wise labels also reach 89.7–91.7% agreement with κ values of 0.785–0.831, supporting J for large-scale evaluation.

Finding 4.Trace-supervised fine-tuning helps mitigate stage-wise hallucinations. Beyond diagnosis, we examine whether the structured traces in ClinHallu can be used to fine-tune models for improved reliability. Specifically, we construct finetuning samples by using different parts of the annotated reasoning trace as supervision signals. The Ans-only variant uses only the final answer as the target output, without any intermediate trace. For trace-supervised variants, we supervise the model with Visual Recognition (V), Knowledge Recall (K), their combination (V+K), or the full trace including Reasoning Integration (V+K+R). As shown in Table 5, Ans-only fine-tuning improves Acc but provides limited stage-wise gains and slightly increases HR . In contrast, V-only supervision reduces HV , while K-only supervision is less stable without visual information. Combining V+K is more effective than either alone, and full trace supervision achieves the highest Acc and lowest hallucination rates. These results show that complete trace supervision is more effective for hallucination mitigation.

5.3

H-H

Label

6

Conclusion

We introduce ClinHallu, a benchmark for diagnosing stage-wise hallucinations in medical multimodal reasoning. Unlike answer-centric evaluations, ClinHallu augments medical VQA instances with structured reference traces covering Visual Recognition, Knowledge Recall, and Reasoning Integration, enabling hallucination sources to be localized within the reasoning process. Experiments on ClinHallu show that hallucination bottlenecks vary across datasets. We also demonstrate that these traces provide effective supervision: full trace-supervised fine-tuning improves answer accuracy and reduces stage-wise hallucinations. Overall, ClinHallu offers a fine-grained testbed for understanding and mitigating hallucinations in medical MLLMs, supporting the development of more reliable medical multimodal systems.

7

Limitations

ClinHallu has its limitations. The current benchmark focuses on medical VQA-style tasks and does not cover long-form report generation or real-world clinical decision-support scenarios. We choose VQA as the initial setting because it offers a controlled testbed. Future work will extend ClinHallu to broader medical reasoning scenarios.

Human Evaluation

We conduct human evaluation to assess whether the automatic judge J aligns with human annotations. We sample a 10% stratified subset of ClinHallu and ask two evaluators with medical backgrounds to annotate answer correctness (Acc) and three stagewise hallucination labels (HV , HK , and HR ). For each 8

References

and 1 others. 2024b. Towards injecting medical visual knowledge into multimodal llms at scale. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 7346–7370.

Vibhor Agarwal, Yiqiao Jin, Mohit Chandra, Munmun De Choudhury, Srijan Kumar, and Nishanth Sastry. 2024. Medhalu: Hallucinations in responses to healthcare queries by large language models. arXiv preprint arXiv:2409.19492.

Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46.

Elham Asgari, Nina Montaña-Brown, Magda Dubois, Saleh Khalil, Jasmine Balloch, Joshua Au Yeung, and Dominic Pimenta. 2025. A framework to assess clinical safety and hallucination rates of llms for medical text summarisation. NPJ digital medicine, 8(1):274.

Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Baobao Chang, and 1 others. 2024. A survey on in-context learning. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 1107–1128.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.

Zishan Gu, Jiayuan Chen, Fenglin Liu, Changchang Yin, and Ping Zhang. 2026. Medvh: Toward systematic evaluation of hallucination for large vision language models in the medical context. Advanced Intelligent Systems, 8(1):2500255.

Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025a. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631.

Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. 2020. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In Proceedings of the International Conference on Learning Representations (ICLR).

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, and 8 others. 2025b. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.

Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1–55.

Aofei Chang, Le Huang, Parminder Bhatia, Taha Kass-Hout, Fenglong Ma, and Cao Xiao. 2025. Medheval: Benchmarking hallucinations and mitigation strategies in medical large vision-language models. arXiv preprint arXiv:2503.02157.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM computing surveys, 55(12):1–38.

Jiawei Chen, Dingkang Yang, Tong Wu, Yue Jiang, Xiaolu Hou, Mingcheng Li, Shunli Wang, Dongling Xiao, Ke Li, and Lihua Zhang. 2024a. Detecting and evaluating medical hallucinations in large vision language models. arXiv preprint arXiv:2406.10185.

Songtao Jiang, Yuan Wang, Sibo Song, Tianxiang Hu, Chenyi Zhou, Bin Pu, Yan Zhang, Zhibo Yang, Yang Feng, Joey Tianyi Zhou, and 1 others. 2025. Hulu-med: A transparent generalist model towards holistic medical vision-language understanding. arXiv preprint arXiv:2510.08668.

Junying Chen, Chi Gui, Ruyi Ouyang, Anningzhe Gao, Shunian Chen, Guiming Hardy Chen, Xidong Wang, Zhenyang Cai, Ke Ji, Xiang Wan,

Yubin Kim, Hyewon Jeong, Shan Chen, Shuyue Stella Li, Chanwoo Park, Mingyu Lu, Kumail Alhamoud, Jimin Mun, Cristina Grau, Minseok Jung, and 1 9

others. 2025. Medical hallucinations in foundation models and their impact on healthcare. arXiv preprint arXiv:2503.05777.

OpenAI. 2023. Gpt-4v(ision) system card. https: //openai.com/index/gpt-4v-system-card/. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models. In Proceedings of the 27th Conference on Computational Natural Language Learning (CoNLL), pages 314–334.

Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.

Shrey Pandit, Jiawei Xu, Junyuan Hong, Zhangyang Wang, Tianlong Chen, Kaidi Xu, and Ying Ding. 2025. Medhallu: A comprehensive benchmark for detecting medical hallucinations in large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 2858–2873.

Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. 2018. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):180251. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023a. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36:28541– 28564.

Qwen Team. 2026. Qwen3.5: Towards native multimodal agents. Khaled Saab, Tao Tu, Wei-Hung Weng, Ryutaro Tanno, David Stutz, Ellery Wulczyn, Fan Zhang, Tim Strother, Chunjong Park, Elahe Vedadi, and 1 others. 2024. Capabilities of gemini models in medicine. arXiv preprint arXiv:2404.18416.

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. 2023b. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 292– 305.

Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, and 1 others. 2025. Medgemma technical report. arXiv preprint arXiv:2507.05201.

Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. 2021. Slake: A semanticallylabeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th international symposium on biomedical imaging (ISBI), pages 1650–1654. IEEE.

Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, and 1 others. 2025. Toward expert-level medical question answering with large language models. Nature medicine, 31(3):943–950.

Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253.

Ryutaro Tanno, David GT Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Charles Lau, Tao Tu, Shekoofeh Azizi, and 1 others. 2025. Collaboration between clinicians and vision–language models in radiology report generation. Nature Medicine, 31(2):599–608.

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916.

Gemini Team, Rohan Anil, Sebastian Borgeaud, JeanBaptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.

Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Callison-Burch. 2023. Faithful chain-ofthought reasoning. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 305–329.

Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, and 1 others. 2025. Internvl3. 5: Advancing open-source 10

multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265.

evaluation metric for chest x-ray findings. Nature Communications, 16(1):3108.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.

Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Pmc-vqa: Visual instruction tuning for medical visual question answering. arXiv preprint arXiv:2305.10415. Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo. 2024. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 3: system demonstrations), pages 400–410.

Peng Xia, Ze Chen, Juanxi Tian, Yangrui Gong, Ruibo Hou, Yue Xu, Zhenbang Wu, Zhiyuan Fan, Yiyang Zhou, Kangyu Zhu, and 1 others. 2024. Cares: A comprehensive benchmark of trustworthiness in medical vision language models. Advances in Neural Information Processing Systems, 37:140334–140365.

Kaiwen Zuo and Yirui Jiang. 2024. Medhallbench: A new benchmark for assessing hallucination in medical large language models. arXiv preprint arXiv:2412.18947.

Weiwen Xu, Hou Pong Chan, Long Li, Mahani Aljunied, Ruifeng Yuan, Jianyu Wang, Chenghao Xiao, Guizhen Chen, Chaoqun Liu, Zhaodonghui Li, and 1 others. 2025. Lingshu: A generalist foundation model for unified multimodal medical understanding and reasoning. arXiv preprint arXiv:2506.07044.

Yuxin Zuo, Shang Qu, Yifei Li, Zhang-Ren Chen, Xuekai Zhu, Ermo Hua, Kaiyan Zhang, Ning Ding, and Bowen Zhou. 2025. Medxpertqa: Benchmarking expert-level medical reasoning and understanding. In International Conference on Machine Learning, pages 80961–80990. PMLR.

Qiao Yan, Yuchen Yuan, Xiaowei Hu, Yihan Wang, Jiaqi Xu, Jinpeng Li, Chi-Wing Fu, and PhengAnn Heng. 2025. Medhalltune: An instructiontuning benchmark for mitigating medical hallucination in vision-language models. arXiv preprint arXiv:2502.20780. Sicheng Yang, Haipeng Zhou, Yijun Yang, Weiming Wang, Shifu Chen, Guang Yang, Huazhu Fu, and Lei Zhu. 2026. Lcm-net: Llm-driven crossmodality moe feature fusion network for cancer survival analysis. IEEE Transactions on Medical Imaging. Zonghai Yao, Benlu Wang, Yifan Zhang, Junda Wang, Iris Xia, Zhipeng Tang, Shuo Han, Feiyun Ouyang, Zhichao Yang, Arman Cohan, and 1 others. 2026. Medical thinking with multiple images. In The Fourteenth International Conference on Learning Representations. Suhao Yu, Haojin Wang, Juncheng Wu, Luyang Luo, Jingshen Wang, Cihang Xie, Pranav Rajpurkar, Carl Yang, Yang Yang, Kang Wang, and 1 others. 2025. Medframeqa: A multi-image medical vqa benchmark for clinical reasoning. arXiv preprint arXiv:2505.16964. Juan Manuel Zambrano Chaves, Shih-Cheng Huang, Yanbo Xu, Hanwen Xu, Naoto Usuyama, Sheng Zhang, Fei Wang, Yujia Xie, Mahmoud Khademi, Ziyi Yang, and 1 others. 2025. A clinically accessible small multimodal radiology model and 11

Appendix of ClinHallu A

Fine-Tuning Configuration

Training Data Construction. Of the four source datasets, only VQA-RAD and PathVQA provide official training

splits, so we use them for trace-supervised fine-tuning. For each dataset, we apply the same data curation pipeline used to construct ClinHallu to obtain faithful structured traces. After curation, we obtain 1,221 training samples for VQA-RAD and 10,187 for PathVQA. These curated samples are then used to train the model.

Fine-Tuning Hyperparameters. We fine-tune Qwen3.5-9B (Qwen Team, 2026) using LoRA (Hu et al., 2022) with rank r = 8 and scaling factor α = 16. Training is conducted with LLaMA-Factory (Zheng et al., 2024) using a cosine learning-rate schedule, an initial learning rate of 1 × 10−4 , and a warmup ratio of 0.1. All other hyperparameters follow the default settings of LLaMA-Factory.

B

Case Study

We present a representative case study to illustrate the stage-replacement behavior in Fig. 5 and Fig. 6. The model misidentifies an “AIIS avulsion fracture” as a “femoral neck fracture” and recalls incorrect anatomical knowledge, leading to the wrong answer “gluteus medius”. Replacing knowledge recall alone is insufficient, while replacing both stages enables the model to correctly infer “rectus femoris”, revealing a coupled visual-knowledge failure hidden by answer-level evaluation alone.

C

Prompt Templates

We summarize the prompt templates used throughout the ClinHallu pipeline. The prompts are organized according to the main stages of our framework, including benchmark construction, stage replacement, and automatic judging. The complete prompt templates are provided in Figs. 7–13.

D

Use of AI Assistants

We used AI to assist with English writing polish. All scientific content, experimental design, and conclusions are solely our own. No AI-generated text was used without human review and revision.

E

Datasets and Licenses

Data source. We use four publicly available medical VQA datasets: VQA-RAD Lau et al. (2018), PathVQA He

et al. (2020), MedXpertQA Zuo et al. (2025), and MedFrameQA Yu et al. (2025). VQA-RAD is released under the CC0 1.0 Universal License; PathVQA and MedXpertQA are released under the MIT License; and MedFrameQA is released under the CC BY 4.0 License. All datasets are used for research purposes, and we follow their license terms and attribution requirements. Privacy and content screening. All source datasets used in ClinHallu are benchmark datasets released for

research purposes. We do not collect new patient data or personally identifying information. During data curation, we only use the released images, questions, answers, and metadata provided by the original datasets. The curated benchmark does not contain information that directly identifies individual patients. Ethics review. The study uses publicly available benchmark datasets and does not collect new patient data.

Human evaluation only involved annotating model outputs and benchmark instances for research validation, without collecting sensitive personal information from annotators. 12

F

Human Evaluation Details

Two annotators with medical backgrounds were recruited from our group and compensated as part of their regular research appointments. Each annotator performed the evaluation independently. Annotators were informed that their annotations would be reported in this paper.

G

Intended Use and Risk Analysis

Intended use. ClinHallu is designed for research on medical MLLM evaluation, hallucination diagnosis,

and model analysis. It should be used to measure and compare model behavior under controlled benchmark settings. Out-of-scope use. ClinHallu is not intended for direct patient-care decision making.

Potential risks. A possible risk is that benchmark improvements may be over-interpreted as clinical reliability. To mitigate this, we report stage-wise hallucination rates in addition to answer accuracy and explicitly analyze failure sources.

13

Input Question: A radiograph (Figure A) shows a hip injury sustained by an adolescent male during a track competition. Based on the radiographic findings, which muscle is most likely responsible for this injury? Options: A. Tensor fascia lata / B. Gluteus medius / C. Biceps femoris / D. Gluteus minimus / E. Rectus femoris (Figure A) MedXpert Case ID: MM-5

Ground-truth Answer: E. Rectus femoris Qwen2.5-VL-7B Prediction: B. Gluteus medius Answer Correct: No

Golden

Visual Recognition The image is an anteroposterior (AP) radiograph of the pelvis. The bony structures of the pelvis, including the iliac crests, sacrum, coccyx, pubic symphysis, and proximal femurs, are visible. A radiopaque gonadal shield is present centrally at the bottom of the image. On the patient's right side (indicated by the 'R' marker on the left of the image), there is a small, detached bony fragment located just inferior to the anterior inferior iliac spine (AIIS), adjacent to the superior aspect of the acetabulum. The rest of the hip joints appear congruent, and no other obvious fractures are seen in the proximal femurs or pelvic ring. Knowledge Recall Avulsion fractures are common injuries in adolescents, particularly athletes, because the apophyses (sites of tendon attachment) are often weaker than the tendons themselves during periods of rapid growth. Different muscles attach to specific bony landmarks on the pelvis: The Tensor fascia lata and Sartorius attach to the Anterior Superior Iliac Spine (ASIS). The Rectus femoris (straight head) attaches to the Anterior Inferior Iliac Spine (AIIS). The Hamstrings (Biceps femoris, Semitendinosus, Semimembranosus) attach to the Ischial Tuberosity. The Gluteus medius and Gluteus minimus attach to the Greater Trochanter. The Iliopsoas attaches to the Lesser Trochanter. The Adductors attach to the Pubis. Injuries typically occur during forceful muscle contraction, such as sprinting or kicking. Reasoning The visual evidence shows an avulsion fracture at the Anterior Inferior Iliac Spine (AIIS) on the right side. Based on anatomical knowledge, the straight head of the Rectus femoris muscle originates from the AIIS. In an adolescent male participating in a track competition (which involves sprinting and forceful leg extension), a sudden, powerful contraction of the Rectus femoris can avulse the apophysis at its origin. Therefore, the Rectus femoris is the muscle responsible for this specific injury pattern. The other muscles listed attach to different locations (ASIS, Ischial Tuberosity, Greater Trochanter) which do not match the radiographic finding. Answer E. Rectus femoris

Figure 5 Case study.

14

Qwen2.5-VL Trace

Visual Recognition The image shows a radiograph of a pelvis and hips. There appears to be a fracture involving the proximal aspect of the femur, specifically the femoral neck region. The bones and joint spaces are visible, but there is no obvious dislocation. The surrounding soft tissues are not clearly defined due to the nature of the radiograph. Knowledge Recall Hip fractures can occur due to direct trauma, such as a fall, or indirect trauma like a sports injury. The femoral neck is a common site for fractures in older adults, but in younger individuals, it may be associated with muscle strain or overuse injuries. The gluteus medius is a major abductor of the hip and is particularly susceptible to injury when the hip is adducted and the foot is dorsiflexed. Reasoning Given the radiographic finding of a femoral neck fracture, the most likely muscle responsible for this injury would be one that is involved in the mechanics of hip abduction. The gluteus medius is the primary abductor of the hip and is known to be injured in certain hip abduction patterns. The tensor fascia lata and rectus femoris are primarily responsible for knee flexion and extension, and the biceps femoris is more associated with the lower leg's functions rather than hip abduction. The gluteus minimus is a smaller muscle and less commonly associated with femoral neck fractures compared to the gluteus medius. Answer B. Gluteus medius

Rep-K

Visual Recognition- <<model own>> Knowledge Recall - <<replaced with golden>> Reasoning Given the radiograph shows a fracture at the femoral neck region, we need to consider which muscle is most likely to cause such an injury through avulsion at the greater trochanter. The tensor fascia lata, rectus femoris (straight head), and hamstrings (biceps femoris, semitendinosus, semimembranosus) attach more distally and would not typically cause a fracture at the femoral neck through avulsion. The iliopsoas and adductors attach to the pubis and lesser trochanter, respectively, and are less likely to cause this type of injury. Therefore, the muscle most likely responsible for the femoral neck injury seen in the radiograph is the Gluteus medius. Answer B. Gluteus medius

Rep-VK

Visual Recognition -<<replaced with golden>> Knowledge Recall - <<replaced with golden>> Reasoning The radiographic image shows an avulsion fracture at the anterior inferior iliac spine (AIIS). This specific location is where the rectus femoris muscle attaches, particularly its straight head. Avulsion fractures at the AIIS are commonly associated with the rectus femoris muscle, as it contracts forcefully during activities like sprinting and quick directional changes, leading to this type of injury. Given the radiographic finding and the known anatomical attachments of the muscles involved, the muscle most likely responsible for this injury is the rectus femoris. Answer E. Rectus femoris Figure 6 Case study.

15

[REFERENCE TRACE CONSTRUCTION PROMPT] ------------------------------------You are an expert medical professional. Your task is to analyze a medical image (or medical question) and provide a structured Chainof-Thought (CoT) reasoning. You MUST follow this exact 3-step reasoning framework:

=== THE 3 STEPS ARE INDEPENDENT === - Step 1 (Visual Recognition): Describe what you see in the image. Do NOT reference the question or attempt any diagnosis. - Step 2 (Knowledge Recall): Recall medical knowledge related to the question. Do NOT reference the image or your visual observations. - Step 3 (Reasoning): Integrate Step 1 and Step 2 to derive the final answer. === OUTPUT FORMAT (follow EXACTLY) ===

[Visual Recognition] Describe the medical objects, structures, and visual features in the image. Focus on objective visual findings. Do NOT make diagnostic judgments. [Knowledge Recall] Based on the question, recall relevant medical concepts, definitions, or criteria. Do NOT reference the image. [Reasoning] Integrate your visual findings with your medical knowledge to derive the answer. [Answer] your_final_answer_here. === RULES === 1. Each step MUST start with the exact header: [Visual Recognition], [Knowledge Recall], [Reasoning] 2. [Reasoning] MUST end with [Answer] followed by the final answer 3. [Visual Recognition] must NOT contain any diagnosis or reference to the question 4. [Knowledge Recall] must NOT reference the image or visual observations 5. Do NOT fabricate observations - only describe what you genuinely observe 6. Use proper medical terminology 7. Be honest about uncertainty 8. If and only if explicit answer options are provided in the prompt, treat the question as multiple-choice. 9. For multiple-choice questions, [Answer] MUST be the option letter followed by the option content (e.g., [Answer] C. Lung cancer) 10. If no explicit answer options are provided, do NOT invent option letters, labels, or choices. 11. For yes/no questions without explicit options, [Answer] MUST be exactly 'yes' or 'no'.

Figure 7 Prompt templates used in the ClinHallu pipeline.

16

[TRACE VALIDATION PROMPT] ------------------------------------You are a strict evaluator for medical reasoning traces. Given a question, a ground-truth answer, and a structured reasoning trace, judge whether the trace is valid. A valid trace must satisfy both criteria: 1. Format Validity: the trace must contain all required stages, and each stage must be non-empty and clearly separated. 2. Answer Consistency: the trace must support the ground-truth answer without introducing conflicting conclusions. Question: {question} Ground-truth Answer: {ground_truth} Generated Trace: {trace} Return the result in the following format: Format Validity: Yes/No Answer Consistency: Yes/No Reason: ...

[Stage-Wise Evaluation PROMPT] ------------------------------------You are an expert medical professional. Your task is to analyze a medical image (or medical question) and provide a structured Chainof-Thought (CoT) reasoning. You MUST follow this exact 3-step reasoning framework: [Visual Recognition] Describe the medical objects, structures, and visual features in the image. Focus on objective visual findings. Do NOT make diagnostic judgments. [Knowledge Recall] Based on the question, recall relevant medical concepts, definitions, or criteria. Do NOT reference the image. [Reasoning] Integrate your visual findings with your medical knowledge to derive the answer. [Answer] your_final_answer_here Question: {question} {options_text}If no options are shown above, answer directly and do not invent option letters. Generate the 3-step CoT reasoning ([Visual Recognition] -> [Knowledge Recall] -> [Reasoning] + [Answer]).

Figure 8 Prompt templates used in the ClinHallu pipeline.

17

[REP-V PROMPT] ------------------------------------You are continuing a structured Chain-of-Thought reasoning for a medical question. Question: {question} The [Visual Recognition] step has already been completed: [Visual Recognition] {visual_recognition} Now continue with the remaining steps: 1. [Knowledge Recall] - Recall relevant medical knowledge (do NOT reference the image) 2. [Reasoning] - Integrate visual findings with knowledge to derive the answer. End the [Reasoning] step with [Answer] followed by the final answer.

Output ONLY [Knowledge Recall] and [Reasoning]. End [Reasoning] with [Answer] followed by the final answer.

[REP-K PROMPT] ------------------------------------You are continuing a structured Chain-of-Thought reasoning for a medical question. Question: {question} The first two steps have already been completed: [Visual Recognition] {visual_recognition} [Knowledge Recall] {knowledge_recall} The visual-recognition step is generated by the evaluated model, while the knowledge-recall step is replaced with the reference one. Now continue with the final step: [Reasoning] - Integrate the visual findings with the medical knowledge to derive the answer. End with [Answer] followed by your final answer. Output ONLY [Reasoning]. End [Reasoning] with [Answer] followed by the final answer.

Figure 9 Prompt templates used in the ClinHallu pipeline.

18

[REP-VK PROMPT] ------------------------------------You are continuing a structured Chain-of-Thought reasoning for a medical question. Question: {question} The first two steps have already been completed: [Visual Recognition] {reference_visual_recognition} [Knowledge Recall] {reference_knowledge_recall} Now continue with the final step: [Reasoning] - Integrate the visual findings with the medical knowledge to derive the answer. End with [Answer] followed by your final answer. Output ONLY [Reasoning]. End [Reasoning] with [Answer] followed by the final answer.

[ANSWER JUDGE SYSTEM PROMPT] ------------------------------------You are an expert medical AI evaluator. Your task is to determine whether a model's predicted answer is correct by comparing it against the ground truth answer for a medical visual question answering task. You MUST respond with a valid JSON object (no markdown, no extra text).

Figure 10 Prompt templates used in the ClinHallu pipeline.

19

[ANSWER JUDGE USER PROMPT] ------------------------------------Question: {question} Ground Truth Answer: {ground_truth} Model's Predicted Answer: {predicted_answer} Determine whether the predicted answer is semantically correct. Be lenient on wording but strict on medical facts. Respond with this exact JSON format: { "answer_correct": 0 or 1, "answer_reasoning": "brief explanation of why the answer is correct or wrong" }

[STAGE-WISE HALLUCINATION JUDGE SYSTEM PROMPT] ------------------------------------You are an expert medical AI evaluator. Your task is to determine whether a single reasoning step in a medical visual question answering chain-of-thought contains hallucinated, unsupported, or medically incorrect content. Judge the candidate step against the provided question, answer, reference step, and any support context. When images are provided, use them as evidence. You MUST make a binary decision and respond with a valid JSON object (no markdown, no extra text).

Figure 11 Prompt templates used in the ClinHallu pipeline.

20

[VISUAL HALLUCINATION JUDGE USER PROMPT] ------------------------------------Step Type: Visual Recognition Question: {question} Ground Truth Answer: {ground_truth} Reference Visual Recognition: {reference_text}

Candidate Visual Recognition: {candidate_text} Determine whether the candidate visual-recognition step contains hallucinated visual facts, unsupported findings, or materially incorrect observations. Use the reference step as guidance, but prioritize the actual image evidence when available. Respond with this exact JSON format: { "hallucinated": 0 or 1, "reason": "brief explanation focused on factual support and hallucination risk" }

[KNOWLEDGE HALLUCINATION JUDGE USER PROMPT] ------------------------------------Step Type: Knowledge Recall Question: {question} Ground Truth Answer: {ground_truth} Reference Knowledge Recall: {reference_text} Candidate Knowledge Recall: {candidate_text} Available support context: {support_context} Determine whether the candidate knowledge-recall step introduces hallucinated, unsupported, irrelevant, or medically incorrect knowledge. Be strict on factual correctness and relevance to the case. Set hallucinated to 1 if there is any material hallucination or unsupported medical claim; otherwise 0. Respond with this exact JSON format: { "hallucinated": 0 or 1, "reason": "brief explanation focused on factual support and hallucination risk" }

Figure 12 Prompt templates used in the ClinHallu pipeline.

21

[REASONING HALLUCINATION JUDGE USER PROMPT] ------------------------------------Step Type: Reasoning Question: {question} Ground Truth Answer: {ground_truth} Reference Reasoning: {reference_text} Candidate Reasoning: {candidate_text} Available support context: {support_context} Determine whether the candidate reasoning step contains hallucinated claims, unsupported logical jumps, or medically incorrect inference. Focus on whether the conclusion is justified by the available evidence and context. Set hallucinated to 1 if there is any material hallucination or unsupported inference; otherwise 0. Respond with this exact JSON format: { "hallucinated": 0 or 1, "reason": "brief explanation focused on factual support and hallucination risk" }

Figure 13 Prompt templates used in the ClinHallu pipeline.

22