Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMs Zhixiao Zheng
Zheren Fu∗
Zhiyuan Yao
[email protected] University of Science and Technology of China Hefei, China
[email protected] University of Science and Technology of China Hefei, China
[email protected] University of Science and Technology of China Hefei, China
Chunxiao Liu
Dongming Zhang
Zhendong Mao
[email protected] Xiaomi Corporation Beijing, China
[email protected] State Key Laboratory of Communication Content Cognition, People’s Daily Online Beijing, China
[email protected] University of Science and Technology of China Hefei, China
arXiv:2607.13712v1 [cs.CV] 15 Jul 2026
Abstract Despite the rapid progress of Multimodal Large Language Models (MLLMs), they still suffer from untruthfulness issues, such as visual hallucinations, content fabrication, and unfaithful reasoning, which substantially undermine their faithfulness and practical utility. Alignment methods based on human preference, such as Direct Preference Optimization (DPO), have been widely adopted to address these issues. However, multimodal reasoning errors often propagate across stages, and final-answer errors can often be traced to mistakes in early grounding stages, yet standard DPO typically applies preference optimization at the final-answer level. This credit-assignment challenge means that supervision for early grounding stages is indirect rather than stage-specific, making it difficult to suppress error propagation arising from grounding drift and context inconsistency. To address this, we propose Grounded Context Preference Optimization (Groc-PO), a grounded preference optimization framework for MLLMs. We further construct the Grounded Context Preference Dataset (GCPD), organizing multistage preference samples around three stages of Object Grounding, Contextual Grounding, and Grounded Reasoning, to capture the formation, integration, and utilization of grounded context. By introducing more explicit preference supervision over multiple grounded stages, Groc-PO strengthens context-dependent reasoning and mitigates cross-stage error propagation. Extensive experiments show that, compared with standard DPO and other strong baselines, Groc-PO achieves improved performance in hallucination mitigation, faithful reasoning, and overall reliability, supporting the value of more explicit grounded supervision for trustworthy multimodal reasoning.
He is holding a black
remote control and using it to
Wrong EarlyGrounding stage (s) Misread: camera → remote control (in the right man’s hands)
control the electric skateboard, helping What is the man on the right holding, and what does this suggest about his intention ?
the boy complete the jump.
Wrong Reasoning
Laterstage
(a) (↑)
(b)
Figure 1: Motivating example of error propagation across stages in MLLMs. (a) A case where an early grounding error propagates to the later reasoning stage and leads to an incorrect answer. (b) Statistical experiments with LLaVA-v1.57B [20] on GCPD dataset (constructed from RLHF-V [35]), showing that introducing errors into 0, 1, or 2 grounding stages is associated with progressively lower final reasoning accuracy, consistent with error propagation in MLLMs.
CCS Concepts • Computing methodologies → Machine learning.
∗ Corresponding author.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]. Conference’17, Washington, DC, USA © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM ISBN 978-x-xxxx-xxxx-x/YYYY/MM https://doi.org/10.1145/nnnnnnn.nnnnnnn
Keywords Multimodal LLMs, Preference Optimization, Truthful Models
1
Introduction
Multimodal Large Language Models (MLLMs), by integrating powerful vision and language capabilities, are transforming humancomputer interaction [2, 3, 19]. MLLMs also demonstrate astonishing potential in tasks like Visual Question Answering (VQA) and
Conference’17, July 2017, Washington, DC, USA
Zhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, and Zhendong Mao
medical analysis [24, 32, 36]. However, despite these advancements, MLLMs suffer from the unfaithfulness problems, such as generating content that contradicts visual facts (e.g., fabricated objects, attributes, or relationships) [5, 10]. This significantly hinders their reliability and utility in real-world applications [18, 22]. To mitigate the above issues, preference learning based on human feedback has become a mainstream alignment paradigm. Among these methods, Direct Preference Optimization (DPO) [25] has been widely adopted because it directly optimizes the policy from preference data without requiring an explicit reward model. Existing DPO methods apply preference supervision at the final-answer-level to guide the model in generating more accurate and reliable outputs. However, multimodal reasoning often involves several stages, including early grounding and later reasoning, and the quality of the final answer depends on the robustness of the full process. As illustrated in Fig. 1a, an error arising in an early grounding stage can propagate to later reasoning stages and lead to reasoning failure. Therefore, it remains unclear whether standard DPO, which mainly relies on holistic final-answer-level preferences, can adequately address reasoning errors rooted in early grounding stages. To this end, we conduct further controlled explorations. As shown in Fig. 1b, introducing errors into more grounding stages is associated with lower final reasoning accuracy, indicating that errors can propagate and accumulate across stages along the reasoning process. This observation reveals a key structural limitation of final-answer-level preference optimization, namely the propagation of MLLM errors: later reasoning failures often do not arise solely at the answer-generation stage, but instead originate from early grounding stages and then accumulate along the reasoning process, affecting the final reasoning outcome. Yet standard DPO typically supervises only at the final-answer-level, offering holistic guidance while providing little stage-specific supervision for these early grounding stages. Consequently, an important question for MLLMs preference alignment is how to provide more targeted preference supervision for early grounding stages, so as to reduce error propagation and improve faithful multimodal reasoning. To address these problems, we propose Grounded Context Preference Optimization (GrocPO), a grounded preference optimization framework for MLLMs. Built upon the newly constructed Grounded Context Preference Dataset (GCPD), Groc-PO leverages preference signals from three contextual stages: Object Grounding, Contextual Grounding, and Grounded Reasoning. This design provides more explicit supervision for upstream grounded stages, thereby enhancing the model’s ability to robustly construct and faithfully utilize the grounded context. The framework incorporates the full context and adopts an adaptive stage-aware optimization strategy, improving complex reasoning while mitigating the propagation of upstream errors. Extensive experiments demonstrate that Groc-PO yields notable gains in hallucination mitigation, contextual understanding, and faithful reasoning. The main contributions of this paper are as follows:
• We propose the Groc-PO framework. By introducing explicit supervision over grounded pre-final stages, it improves faithful multimodal reasoning under multi-round contexts and effectively mitigates error propagation in MLLMs.
• We construct the Grounded Context Preference Dataset (GCPD). GCPD organizes multi-stage preference data around Visual Grounding, Context Grounding, and Faithful Complex Reasoning, providing support for explicit stage-wise supervision of grounded context. • Built on GCPD, Groc-PO employs an adaptive grounded preference optimization mechanism that dynamically allocates learning emphasis across different stages and sample complexities, enabling more targeted alignment. • We conduct systematic experiments across multiple datasets and benchmarks. The results show that Groc-PO consistently outperforms standard DPO and several strong baselines on hallucination and complex capability evaluations, validating the effectiveness of explicit supervision for pre-final grounded stages.
2 Related Works 2.1 Unfaithfulness in MLLMs Unfaithfulness (or hallucination) in MLLMs refers to the generation of content inconsistent with the visual input, typically manifested as fabricated objects, incorrect attributes, or misinterpreted relationships [5]. Hallucinations stem from training data flaws [16]; module biases [9]; suboptimal training paradigms [6]; and inference-stage defects [12]. To address hallucinations, approaches fall into two categories: training-free methods (e.g., Opera [12], VCD [14]) and training-based techniques (e.g., RLHF, PPO) [26, 33].
2.2
Preference Learning for faithful MLLMs
Preference learning initially applied to LLM alignment via RLHF, but DPO has recently gained widespread adoption as a simpler and stable alternative. V-DPO [34] extends DPO by incorporating visual context learning. POVID [38] creates a fine-grained dataset by injecting noise to texts and images. RLHF-V [35] collects segmentlevel human preference data and performs dense DPO. In addition, SPO-Task Planning [17] constructs preference pairs using curriculum learning to improve long-horizon planning. MM-RLHF [37] constructed a preference dataset and proposed a novel reward model to achieve MLLM alignment. SPO [27] treats questioning and answering jointly as a policy trajectory, co-optimizing them via a structured reward function to enhance the model’s consideration of visual dependency in dialogue. The mrDPO [29] utilizes multiround DPO and Rebirth Tuning to optimize audio-visual LLMs.
3
Methodology
The overview of our Groc-PO framework is illustrated in Fig. 2. We first introduce the DPO, followed by the novel Grounded Context Preference Dataset (GCPD) and its generation pipeline, and the adaptive Groc-PO Loss.
3.1
Preliminaries: Direct Preference Optimization
DPO directly optimizes the model through a contrastive learning objective, making it more inclined to generate human-preferred responses while reducing the probability of generating dispreferred responses. DPO learns from preference data (𝑥, 𝑦 +, 𝑦 − ) ∼ D, where
Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMs
Pipeline
Conference’17, July 2017, Washington, DC, USA
Groc-PO Training
Grounded Context Preference Dataset (GCPD) Stage 1: Object Grounding Prompt:
+ Query: List each entity and its key attributes.
Chosen: a boy, a skateboard , a man, a dog (yellow)... Rejected: a boy, a scooter, a man, a cat (yellow) ...
Stage-1 Chosen
Stage-1 Rejected
Stage 2: Contextual Grounding Teacher’s Drafting + Model-Centric Sampling
+ Query
Teacher MLLM-Self
Chosen Sample (Draft)
Rejected Sample
Prompt:
Chosen Sample (Revised)
(1) Random 10% of All
Human
(2) Flagged by Self-Correction
Stage-2 Rejected
Stage-3 Chosen
Stage-3 Rejected
Rejected: He is performing roller blasting. Stage 3: Grounded Reasoning Prompt:
+ Stage 1 (Query + Chosen) + Stage 2 (Query + Chosen) +
Human-in-the-Loop Verification
Stage-2 Chosen
Implict Reward: �(�, �) = �� (�|�) �log ���� (�|�)
Chosen: He is playing skateboarding.
Iterative Correction Chosen Sample Model-Centric (Draft) Verify
+ Stage 1 (Query +Chosen) +
Query: What sport is the boy doing?
Hardnessaware Loss
Query: If he loses balance, who is most likely to help first and why? Chosen: The man is closest and watching him, so he can react fastest. Rejected: The dog will jump to help him because it is trained to assist.
Stageaware Loss
Focus: +
Focus
1
2
3 Stage
Focus: + +
Figure 2: Overview of our Grounded Context Preference Optimization (Groc-PO) framework, including GCPD dataset construction. The left panel shows dataset construction, where multi-stage preference pairs are generated through teacher-assisted drafting, model-centric sampling, iterative correction, and human verification. The middle panel presents three stages of grounded preference supervision: Stage 1 for object grounding, Stage 2 for contextual grounding, and Stage 3 for grounded reasoning. The right panel shows Groc-PO training, which jointly uses preference pairs from all three stages with a stage-aware, hardness-aware loss to improve context-dependent reasoning and mitigate cross-stage error propagation. 𝑥 is the input prompt, 𝑦 + is the human-preferred /chosen response, 𝑦 − is the dispreferred /rejected response, and D is the dataset. The DPO objective function assumes that the human preference probability 𝑝 ∗ (𝑦 + ≻ 𝑦 − | 𝑥) can be modeled via a latent reward function 𝑟 ∗ (𝑥, 𝑦): 𝑝 ∗ (𝑦 + ≻ 𝑦 − | 𝑥) = 𝜎 (𝑟 ∗ (𝑥, 𝑦 + ) − 𝑟 ∗ (𝑥, 𝑦 − )). DPO further relates the reward function to the model’s policy 𝜋𝜃 and a reference policy 𝜋ref : 𝑟 ∗ (𝑥, 𝑦) = 𝛽 (log(𝜋𝜃 (𝑦 | 𝑥)) − log(𝜋ref (𝑦 | 𝑥))). where 𝛽 is a hyperparameter controlling the ratio between reward function and policy deviation. DPO’s loss can directly optimize MLLM to maximize the probability of generating 𝑦 + and minimize generating 𝑦 − . Let us define the log-likelihood ratio for the preferred response as 𝑟 + = log (𝜋𝜃 (𝑦 + | 𝑥)/𝜋 ref (𝑦 + | 𝑥)) and for the dispreferred response as 𝑟 − = log (𝜋𝜃 (𝑦 − | 𝑥)/𝜋 ref (𝑦 − | 𝑥)). Then the DPO loss function is defined as: LDPO = − log 𝜎 𝛽 (𝑟 + − 𝑟 − ) . (1) By minimizing this loss function, the model 𝜋𝜃 is trained to increase the difference between the log-probabilities of 𝑦 + and 𝑦 − , It makes DPO simpler and demonstrates comparable or superior performance to RLHF.
3.2
Grounded Context Preference Dataset (GCPD)
Failures in multimodal reasoning often do not emerge only at the final response, but can originate from imperfect grounding and accumulated inconsistencies in pre-final stages. To explicitly supervise these upstream stages, we construct the Grounded Context Preference Dataset (GCPD), a structured preference dataset organized around progressively accumulated grounded context.
Specifically, GCPD is built as a 3-stage context-dependent preference dataset. Its three stages move from basic visual grounding, to context-grounded understanding, and finally to faithful complex reasoning. This formulation places preference supervision not only on the final answer, but also on the grounded context that supports it, providing a more direct signal for reducing error propagation and improving multimodal faithfulness. Cumulative Multi-stage Context: For any stage 𝑠, the prompt includes all historical context from stage 1 to 𝑠 − 1, ensuring continuous information flow. Let I denote the image, Q denote the question, A denote the answer, A𝑠+ means the chosen response (human-preferred). Then, we will have the following prompt structure for every stage: • Stage 1 (S1): Prompt = {𝐼, 𝑄 1 } • Stage 2 (S2): Prompt = {𝐼, 𝑄 1, 𝐴𝑠+1 , 𝑄 2 } • Stage 3 (S3): Prompt = {𝐼, 𝑄 1, 𝐴𝑠+1 , 𝑄 2, 𝐴𝑠+2 , 𝑄 3 } Then, we detail the progressive stages design: • S1: Object Grounding. S1 marks the starting point of CoT process—identifying basic facts. We present the model with a standardized question (e.g., List each entity and its key attributes) to extract core visual elements. • S2: Contextual Grounding. S2 simulates the intermediate steps of CoT. Building upon S1, S2 focuses on tasks such as relationship description, comprehensive captioning, or visual question answering. It requires the model not only to identify individual entities but also to understand how they form a meaningful whole.
Conference’17, July 2017, Washington, DC, USA
Zhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, and Zhendong Mao
• S3: Grounded Reasoning. S3 is the culmination of the CoT simulation. We pose complex questions that require integrating the image with context from S1 and S2, and performing logical inference, intent prediction, or reasoning tasks. This compels the model to perform high-level cognition based on established, reliable context. In this way, our GCPD dataset is no longer fragmented questionanswer pairs but offers a progressive learning process aligned with human cognitive laws.
3.3
Pipeline of GCPD’s Generation and Features
3.3.1 Basic Workflow. Our pipeline begins with a widely recognized RLHF-V dataset [35], which contains 5,733 images along with human-annotated, high-quality preference pair for tasks like captioning, relational description. Our goal is to generate a structured 3-stage context for each image. For any stage 𝑠, the core tasks is to generate a specific question 𝑄𝑠 and a high-quality preference pair (𝑦𝑠+, 𝑦𝑠− ), consisting of a chosen and a rejected response. This three-stage structure is designed with progressively increasing complexity, following a perception-understanding-reasoning path. For the Stage 1, we define a universal base question Q1 : “List each entity and its key attributes,” to establish a factual foundation. The chosen response, 𝑦𝑠+1 , is initially generated by an advanced teacher model; the rejected response, 𝑦𝑠−1 , is derived from 𝑦𝑠+1 via introduced rule-based deficiencies, retaining structural soundness while including factual inaccuracies. Subsequently, 𝑦𝑠+1 undergoes rigorous verification and refinement to ensure its accuracy. For the Stage 2, aiming for both efficiency and quality, we adopt the existing data from the RLHF-V dataset corresponding to each image, as its design philosophy aligns perfectly with our goals for this stage. This means that the question 𝑄 2 , the chosen response 𝑦𝑠+2 , and the rejected response 𝑦𝑠−2 are all sourced from this high-quality, human-validated dataset. It guarantees the superiority of the data for relational description and understanding. In the Stage 3, our objective is to enhance model’s ability of complex thinking. We leverage an advanced teacher model to generate a new, more profound, and complex question, 𝑄 3 , based on the context of the original image and the preceding dialogue (𝑄 1, 𝑦𝑠+1 , 𝑄 2, 𝑦𝑠+2 ). Following the pattern of the 1st stage, we then generate a high-quality chosen response, 𝑦𝑠+3 , via the teacher model combined with rigorous verification and refinement, and a rejected response, 𝑦𝑠−3 , from our LLaVA base model to complete the final preference pair. 3.3.2 Iterative Self-Correction for Chosen Samples (𝑦𝑠+ ). To maximize the chosen response quality and factual accuracy, we introduce an iterative self-correction mechanism. • In Stage 1, the initial list of entities in the chosen response, generated by the advanced teacher model, is fed back into the teacher model with a detailed verification prompt. The teacher model is instructed to comprehensively check and provide a score. If the response contains hallucinations or the average score is too low, it triggers a rewrite by the advanced teacher model. • Similarly, in Stage 3, the complex reasoning answer in the chosen response is sent back for a second review. In this step, the teacher model acts as a "Critic", inspecting the
reasoning chain for logical fallacies and ensuring it is fully grounded in the provided visual and textual context. This "generate-and-refine" closed-loop process significantly enhances the quality of our chosen responses, providing the model with a clear and reliable learning target. 3.3.3 Model-Centric Sampling. To further improve data quality and make training more efficient, we draw inspiration from the on-policy concept in RLHF [7] and adopt a Model-Centric Sampling strategy. The core idea is to ensure that the distribution of training data, aligns as closely as possible with the generation distribution of our targeted fine-tuned model. This approach enables the model to directly confront and rectify its own predominant error patterns, making the fine-tuning process highly targeted. This strategy is reflected in two key aspects: • The rejected responses in Stage 3 (𝑦𝑠−3 ) are generated by our target LLaVA model. Consequently, these samples are representative of the model’s intrinsic failure modes, particularly in areas like long-range dependency, contextual understanding, and complex reasoning, which manifest as logical fallacies or cumulative hallucinations. In contrast to negative samples from a more capable, external model teacher model, which often suffer from a distribution mismatch in 𝑦𝑟− . These ’model-centric’ samples provide a highly targeted and valuable learning signal for DPO. • In Stage 2, a key characteristic of the RLHF-V dataset lies in its preference pair construction: the rejected responses (𝑦𝑠−2 ) are generated by the MLLM family, while the chosen responses (𝑦𝑠+2 ) are human-revised versions of these same rejected samples. This approach ensures high distributional and stylistic alignment with our target model, effectively forming a tailored "problem-solution" paradigm for its specific weaknesses. This mechanism provides the highly valuable and targeted learning signal that is the core rationale for our adoption of this dataset. It is worth noting that the Stage 1 responses are highly uniform, making generation variance across models minimal, and thus Model-Centric Sampling has little impact. 3.3.4 Human-in-the-Loop Verification. To ensure the rigor and quality of our GCPD dataset, we introduced Human-in-the-Loop verification. The audit team consisted of three MLLM-familiar PhD students who adhered to a guideline for all checks and corrections. The samples reviewed included: first, a 10% random sample of the entire dataset; second, a targeted review of 𝑦𝑠+ samples flagged by the teacher model Critic as having major issues during the "Iterative Self-Correction" process. Overall, approximately 12% (∼2k) of the 𝑦𝑠+ samples were manually audited, leading to the revision or rewriting of nearly 2% of severely problematic samples (primarily in R3). The total time cost was approximately 57 hours per reviewer (30h for auditing, 27h for revision). This mechanism ensures the reliability and faithfulness of the dataset.
3.4
Customized Groc-PO Loss
The standard DPO loss (Equation 1) treats all samples in the dataset equally. This uniform approach overlooks the inherent gradient of
Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMs
Conference’17, July 2017, Washington, DC, USA
Table 1: Performance comparison with leading methods on various hallucination and general benchmarks. On LLaVA-v1.5-7B, LLaVA-v1.5-13B [20], Groc-PO achieves significant leads on key faithfulness metrics (e.g., AMBER, MM-Hal) and simultaneously enhances general abilities (e.g., LLAVA-Bench, SEED). Here, AMBER-Gene. refers to AMBER-Generation, and AMBER-Discri. denotes the AMBER-Discrimination. MM-Hal [28]
Methods
AMBER-Gene. [31]
AMBER-Discri. [31]
LLaVA [21]
SEED [15]
74.3 82.1 76.1 74.7 81.6 78.5 85.0
65.6 69.1 68.9 68.2 68.0 72.9
66.1 66.4 65.9 66.1 66.1 67.1
73.1 82.1 82.3 88.2
73.1 72.8 76.7 76.5
68.2 68.6 68.2 68.8
Score↑
Hal-Rate↓
CHAIR↓
Hal-Rate↓
Cog↓
Acc↑
F1↑
LLaVA-1.5-7B [20] + DPO [25] + CSR [39] + POVID [38] + V-DPO [34] + RLHF-V [35] + mDPO [30] + Groc-PO (Ours)
2.01 2.14 2.05 2.26 2.16 2.02 2.39 2.76
61.4 58.3 60.4 55.2 56.0 60.4 54.0 47.0
7.8 5.7 5.4 5.7 5.6 5.5 4.4 4.2
36.4 27.3 25.5 26.9 27.3 26.3 24.5 25.2
4.2 2.6 2.6 3.0 2.7 2.5 2.4 1.5
71.7 71.3 73.2 71.9 74.8 78.0
LLaVA-1.5-13B [20] + DPO [25] + RLHF-V [35] + Groc-PO (Ours)
2.38 2.47 2.50 2.85
53.1 51.0 52.1 45.0
7.0 6.1 6.3 3.8
33.1 26.3 25.1 24.4
3.3 2.7 2.1 1.3
71.4 71.9 79.2 83.5
cognitive depth and sample difficulty within our GCPD dataset. To better leverage this rich and structured information, we propose a Groc-PO Loss, with a sample-level adaptive weight, 𝑤𝑖 , enabling the model to dynamically focus on samples that are more informative and have higher learning value. The Groc-PO loss is defined as: LGroc-PO = −E (𝑥𝑖 ,𝑦𝑤,𝑖 ,𝑦𝑙,𝑖 )∼D [𝑤𝑖 · log 𝜎 (𝑟𝑖 (𝜃 ))]
(2)
𝜋 (𝑦 + |𝑥𝑖 ) 𝜋 (𝑦 − |𝑥𝑖 ) where 𝑟𝑖 (𝜃 ) = 𝛽 log 𝜋 𝜃 (𝑦𝑖+ |𝑥 ) − 𝛽 log 𝜋 𝜃 (𝑦𝑖− |𝑥𝑖 ) is the implicit reref 𝑖 ref 𝑖 𝑖
ward difference for preference pair 𝑖. The core innovation lies in the design of adaptive weight 𝑤𝑖 , which is composed of two multiplicative components: 𝑤𝑖 = 𝜆𝑟 (𝑖 ) · 𝛾𝑖
(3)
3.4.1 Stage-aware Importance Weight (𝜆𝑟 ). This weight is designed to reflect the learning value of different rounds. Later rounds represent more complex tasks that demand stronger long-range dependency and comprehensive abilities. Thus, for any stage 𝑠, we design 𝜆𝑠 as a monotonically increasing function of 𝑠 to encourage model to focus on these advanced knowledge: 𝜆𝑟 = 1 + 𝛼 (𝑟 − 1)
(4)
where 𝑟 ∈ {1, 2, 3} is the dataset stage, and 𝛼 ≥ 0 is a hyperparameter that controls the growth rate of stage importance. When 𝛼 > 0, samples from the latter 2 stages are assigned a higher loss weight. 3.4.2 Hardness-aware Focusing Weight (𝛾𝑖 ). This weight aims to make model focus more on "hard samples" that are difficult to distinguish. When the model can easily distinguish between 𝑦 + and 𝑦 − , the sample is "easy" and has low learning value. Conversely, when model perceives two responses as having similar quality, the sample is "hard" and should be prioritized. We define 𝛾𝑖 as: 𝛾𝑖 = (1 − 𝜎 (𝑟𝑖 (𝜃 )))𝜂
(5)
where 𝜂 ≥ 0 is a focusing parameter. For 𝜂 > 0, this term significantly decreases the loss for well-distinguished samples (where
𝜎 (𝑟𝑖 ) → 1), thereby directing the optimization process toward the most challenging pairs. Through this dual-weighting mechanism, our Groc-PO Loss adaptively evaluates the importance of each training sample, considering both its role in the progressive rounds (via 𝜆𝑡 ) and sample’s learning value (via 𝛾𝑖 ).
4 Experiments 4.1 Datasets, Metrics and Implementation Details Training Data: Based on our GCPD dataset generation pipeline described in Section 3.2, we constructed the 3-stages preference dataset comprising 5,733 diverse images and 17,199 high-quality preference pairs. Evaluation Benchmarks: To comprehensively evaluate models’ performance, we employ several widely used benchmarks: For faithfulness evaluation, AMBER [31] is a LLM-free benchmark for evaluating hallucinations, which has two components: (a) Discrimination: deciding whether a statement is correct; (b) Generation: describing for an image. MM-Hal [28] evaluates response-level hallucination rate and informativeness. For general capability evaluation, LLaVA-Bench [21] is a benchmark spanning diverse scenarios. SEED-Bench [15] is a large-scale benchmark to assess model abilities likes visual understanding and reasoning. For complex tasks, some sub-tasks of benchmarks are adopted, such as LLaVABench-complex reasoning [21], LLaVA-Bench-conversation [21], SEEDvisual reasoning [15], MME-commonsense-reasoning [8], and multiturn dialogue benchmark MM-MT [1]. Implementation Details: Our experiments leverage the widely adopted LLaVA-v1.5-7B and 13B [20] and Qwen2.5-VL-7B [4] models to evaluate scalability and effectiveness of our method. We employed LoRA [11] and AdamW optimizer [23]. Training was
Conference’17, July 2017, Washington, DC, USA
Zhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, and Zhendong Mao
Table 2: Comparison with different preference optimization methods on Qwen2.5-VL-7B [4]. Method Qwen2.5-VL-7B [4] + DPO [25] + POVID [38] + CSR [39] + Groc-PO (Ours)
MM-Hal Score (↑)
MM-Hal Rate (↓)
3.55 3.59 3.73 3.71 3.83
0.40 0.38 0.37 0.41 0.32
Table 4: Ablation study of the loss components and hyperparameters (𝛼, 𝜂) on MM-Hal [28], as described in Section 3.4.
Table 3: Comparison of Groc-PO and DPO on complex tasks. Tasks cover reasoning (LLaVA-Bench-complex reasoning, SEED-visual reasoning, MME-commonsense-reasoning), single-turn conversation (LLaVA-Bench-conversation), and multi-turn dialogue (MM-MT ). Groc-PO shows superiority.
Models DPO Groc-PO
𝜂
MM-Hal Score (↑)
MM-Hal Rate (↓)
(A) Component Ablation DPO Loss 0 + Stage-aware 0.25 + Difficulty-aware 0 + Groc-PO Loss 0.25
0 0 2 2
2.24 2.46 2.42 2.76
60.0 55.0 57.0 47.0
(B) Sensitivity to 𝛼 w/ 𝛼 = 0 w/ 𝛼 = 0.25 w/ 𝛼 = 0.5
0 0.25 0.5
2 2 2
2.42 2.76 2.40
57.0 47.0 55.5
(C) Sensitivity to 𝜂 w/ 𝜂 = 0 w/ 𝜂 = 1 w/ 𝜂 = 2
0.25 0.25 0.25
0 1 2
2.46 2.57 2.76
55.0 51.0 47.0
LLaVAMM LLaVASEED- MMEcomplex conversation reason reason(%) -MT [1] [15] [8] reason [21] [21] 56.9 82.5 (+45%)
62.0 67.2 (+8%)
74.3 76.5 (+3%)
38.3 51.8 (+35%)
1.88 2.55 (+36%)
performed over 2 epochs with an effective batch size of 32. The teacher model we used is GPT-4o [13].
4.2
𝛼
Loss Setting
Main Results
On multiple mainstream faithfulness and general benchmarks, we conduct a comprehensive comparison of Groc-PO with a series of representative baselines, such as llava-v1.5-7B [20], DPO [25], CSR [39], POVID [38], RLHF-V [35], V-DPO [34], and mDPO [30]. On Faithfulness and General Abilities Evaluation. Table 1 shows that Groc-PO achieves leading performance across almost all key evaluation metrics. These evaluations cover different types of faithfulnesss and general ability tests. These demonstrate that through our progressive preference data and adaptive training framework, the model can not only significantly suppress hallucinations but also improve comprehensive abilities Furthermore, Groc-PO shows its scalability by delivering consistent and substantial gains across models from 7B to 13B, validating its efficiency and broad applicability. In addition, to validate the scalability and generalizability of our framework, we applied Groc-PO to Qwen2.5VL-7B [4]. As detailed in Table 2, the model achieved improvements on the faithfulness test. On Complex Understanding and Reasoning. Table 3 demonstrates the particularly prominent superiority of Groc-PO in complex reasoning tasks. Across demanding benchmarks, including LLaVA-bench-complex, LLaVA-conversation, SEED-reasoning, and MME-Commonsense-reasoning, our model consistently outperforms the baseline, achieving up to a 45% relative improvement. Furthermore, Groc-PO demonstrates superior conversational capabilities on MM-MT benchmark (multi-round dialogue), validating the effectiveness of its structured context.
4.3
Ablation Study
4.3.1 Contribution of Groc-PO Loss Components. To verify the effectiveness of our Groc-PO Loss, we compared it against 3 variants: (1) DPO Loss: uses GCPD data with standard DPO loss; (2) Stage-aware Only; and (3) Difficulty-aware Only. Table 4 shows that the full Groc-PO Loss achieves the best performance. The individual components each provide significant gains. In addition, We also performed ablation studies on the key hyperparameters of the loss, 𝛼 (Stage-aware) and 𝜂 (Hardness-aware). 4.3.2 Effect of Model-Centric Sampling. We conduct an ablation of the data construction strategy, comparing a model trained solely on Teacher-generated preference data with one trained on our final dataset using "model-centric sampling". Figure 3a shows that the latter performs better, suggesting that learning from its own imperfect responses provides a closer data distribution and more targeted alignment signals, thereby improving self-alignment. 4.3.3 Contribution of Each Stage. To quantify the contribution of each context stage, we established three independent training settings: S1, S2, and S3. These models were exclusively trained on data from their respective stages, lacking historical context in S2 and S3. Figure 3b revealed that S2 outperformed S1 and S3. Because S3 relies on preceding information, the absence of context leads to misalignment and performance decline. This affirms the necessity of collaboration among three context rounds, asserting that optimal performance requires structured integration. 4.3.4 Impact of Multi-stage Context Depth. To investigate the influence of contextual learning depth, we used three training settings: (1) using the 1st-round data (S1-Only); (2) using the data of 1st and 2nd rounds (S1+S2); and (3) using full data of 3 stages. Figure 3c shows the model performance monotonically improves with increasing context depth and complexity. This demonstrates that our designed "perception→understanding→reasoning" progressive learning path is indispensable for building model’s compressive capabilities and faithfulness.
Base
40
2.5
50
2.0 1.5
Not Sampling -Sampling
60
(a) Model-Centric Sampling or Not
S1
S2
S3
40
(b) Contribution of each Round
3.5
70
MM-Hal Score ( ) Hal-Rate ( )
3.0
60
2.5
50
2.0 1.5
3.5
S1
S1+ S1+ S2 S2+S3
MM-Hal Score
2.0
3.0
70
Hal-Rate(%)
50
MM-Hal Score ( ) Hal-Rate ( )
MM-Hal Score
60
2.5
1.5
3.5
Hal-Rate(%)
3.0
70
MM-Hal Score
MM-Hal Score ( ) Hal-Rate ( )
Hal-Rate(%)
MM-Hal Score
3.5
Conference’17, July 2017, Washington, DC, USA
40
MM-Hal Score ( ) Hal-Rate ( )
3.0
60
2.5
50
2.0 1.5
(c) Impact of Context Depth
70
Hal-Rate(%)
Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMs
S3
S1+ S1+ S3 S2+S3
40
(d) Sensitivity to Context History
Figure 3: Ablations and Analysis. (a) Effectiveness of Model-Centric Sampling or not. (b) Contribution of individual stages: S2 peaks while S3 degrades from misalignment without history. (c) Impact of context depth: monotonic improvement with progressive history. (d) Sensitivity to history length: performance on a fixed stage-3 query improves as context is added.
Complex Question: What is the setting or environment in which the image takes place? Diff. of log-prob
Table 6: Training Overhead Comparison. The average persample processing time and peak memory usage in DPO and Groc-PO, indicating that Groc-PO adds only minor overhead. Model
Avg. time (s)
Peak Memory Usage (MB)
DPO Groc-PO (Ours)
1.99 2.05 (+3%)
40420 40432 (+0.03%)
DPO
log-prob of DPO & Groc-PO
Player 10's kicking motion suggest that he intends to pass the ball to a teammate, and his action may be influenced by player 7's passive stance, which could potentially block the passing lane. Player 10's kicking motion suggest that he
Complex Question: What does
player 10's raised arm and kicking motion reveal about his intended play, and how might player 7's passive stance influence the success of that action?
Token
Figure 4: Token-level log-probability (log-prob) comparison on chosen responses 𝑦𝑟+ for complex tasks: Groc-PO vs. DPO. The curves (green:Groc-PO, brown:DPO) show log-prob assigned to each token. The top yellow bar chart illustrates the difference (Groc-PO - DPO), which is almost entirely positive. This suggests Groc-PO exhibits higher internal confidence on complex tasks. Table 5: Comparison of training with and without grounded context. The "Flattened DPO" (the same 3-stage 17k dataset but without grounded context) shows lower faithfulness than Groc-PO (with grounded context). Setting
4.4
MM-Hal Score(↑)
Hal-Rate(↓)
Flattened DPO
2.17
61.0
Full Structured Context
2.76
47.0
Analysis and Discussion
4.4.1 Dependence on Structured Context. To verify the importance of structured context history, we designed a baseline named
Groc- intends to attack aggressively, while player 7 remains without active running or PO
The cows' calm and relaxed state, along with the lush green field, suggest that the farm's management practices are likely responsible , which can positively impact the herd's future well-being on the farm. The cows' calm and relaxed state, along
cows' calm, relaxed state and the lush field suggest about the farm's management practices, and how might this impact the farm’s future?
Evidence-based, well-founded,
and coherent offering a passing option, potentially affecting the success of player 10's action.
DPO
Complex Question: What does the
With a certain degree of errors
Groc- with the lush field, suggest that the farm's management practices prioritize the wellPO
being of the herd, which can bring the farm healthier and higher-yielding products.
With a certain degree of errors
Evidence-based, well-founded, and coherent
Figure 5: Comparison on complex tasks: Groc-PO vs. DPO. While DPO shows certain degree of errors, Groc-PO demonstrates robust, evidence-based, and coherent reasoning. "Flattened DPO", which uses the same 17k preference pairs but removes all context history, degrading all training data to singlestage question-answer pairs. Table 5 shows the "Flattened DPO" is far below structured context Groc-PO. A likely reason is that many Stage 3 require information from the first two stages, and removing history has left the model misaligned. This result underscores the necessity of incorporating structured context to cultivate contextual coherence and enhance faithfulness. 4.4.2 Sensitivity to History Length. We evaluated the model’s ability to leverage contextual history of varying lengths by testing
Conference’17, July 2017, Washington, DC, USA
Zhixiao Zheng, Zheren Fu, Zhiyuan Yao, Chunxiao Liu, Dongming Zhang, and Zhendong Mao
the same third-stage question (S3) under three conditions: Zeroshot (image + S3), 1-stage context (image + S1 + S3), and 2-stage context (full history: image + S1 + S2 + S3). Figure 3d shows that performance exhibited a clear improvement: 2-stage > 1-stage > Zero-shot. This empirically validates the critical role of multi-stage context in multimodal dialogue, demonstrating that more complete history enables the model to better localize the question, perform logical reasoning, and generate accurate responses. 4.4.3 Training Overhead. We compared the training overhead of Groc-PO with DPO. Table 6 reports the average processing time per sample and peak memory usage. Results show that Groc-PO introduces only marginal overhead. 4.4.4 Discussion: Effectiveness of Grounded Context Supervision. Our analysis highlights the effectiveness of grounded context supervision. As shown in Figure 3c, model performance consistently improves as more complete grounded context is incorporated during training. This suggests that more targeted and stage-specific supervision on early grounding stages may improve the reliability of later-stage reasoning. Notably, the improvement is most evident on complex reasoning tasks. As reported in Table 3, the full Groc-PO achieves clear gains on these tasks. At the same time, it maintains strong performance on general and faithfulness benchmarks. These results indicate that grounded context supervision improves reasoning quality, is beneficial for challenging multimodal tasks, and maintains competitive performance on general and faithfulness benchmarks.
4.5
Case Study and Visualization
4.5.1 Token-level Log-Probabilities on Chosen Responses of Complex Tasks: Groc-PO vs. DPO. To further investigate Groc-PO’s generation confidence for complex tasks, we conducted a case study for token-level log-probability (log-prob) comparison. In Figure 4, we observe that Groc-PO (green curve) assigns a higher log-prob to the majority of tokens in the chosen responses compared to DPO (brown curve), where the difference (yellow bars) is almost entirely positive. This indicates that Groc-PO exhibits higher internal confidence when handling complex tasks. 4.5.2 Visualization of Qualitative Comparison. Figure 5 visually confirms that while the DPO relatively fails on complex queries, Groc-PO generates logically accurate responses that are well-supported by explicit visual evidence.
5
Conclusion
In this paper, we proposed Grounded Context Preference Optimization (Groc-PO), a framework that improves MLLM faithfulness through explicit preference supervision over grounded pre-final stages. To support this objective, we introduced the Grounded Context Preference Dataset (GCPD) and novel adaptive loss function. Extensive evaluations demonstrate that Groc-PO comprehensively enhances multiple capabilities, contributing to developing more faithful MLLMs.
6
Acknowledgements
This work was supported by the Artificial Intelligence-National Science and Technology Major Project (2023ZD0121200) and the
Fundamental and Interdisciplinary Disciplines Breakthrough Plan of the Ministry of Education of China (No. JYB2025XDXM103).
References [1] Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, et al. 2024. Pixtral 12B. arXiv preprint arXiv:2410.07073 (2024). [2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736. [3] Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, et al. 2024. Understanding alignment in multimodal llms: A comprehensive study. arXiv preprint arXiv:2407.02477 (2024). [4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923 (2025). [5] Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Zongbo Han, Zheng Zhang, and Mike Zheng Shou. 2024. Hallucination of multimodal large language models: A survey. arXiv preprint arXiv:2404.18930 (2024). [6] Assaf Ben-Kish, Moran Yanuka, Morris Alper, Raja Giryes, and Hadar AverbuchElor. 2023. Mocha: Multi-objective reinforcement mitigating caption hallucinations. arXiv preprint arXiv:2312.03631 2 (2023). [7] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems 30 (2017). [8] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. 2023. MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models. arXiv preprint arXiv:2306.13394 (2023). [9] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. 2024. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 14375–14385. [10] Anisha Gunjal, Jihan Yin, and Erhan Bas. 2024. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 18135–18143. [11] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR 1, 2 (2022), 3. [12] Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, and Nenghai Yu. 2024. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and retrospection-allocation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13418–13427. [13] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024). [14] Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. 2024. Mitigating object hallucinations in large vision-language models through visual contrastive decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13872–13882. [15] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023. Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125 (2023). [16] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. 2023. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023). [17] Xiwen Liang, Min Lin, Weiqi Ruan, Rongtao Xu, Yuecheng Liu, Jiaqi Chen, Bingqian Lin, Yuzheng Zhuang, and Xiaodan Liang. 2025. Structured preference optimization for vision-language long-horizon task planning. arXiv preprint arXiv:2502.20742 (2025). [18] Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. 2024. A Survey of Multimodel Large Language Models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering. 405–409. [19] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 26296–26306.
Groc-PO: GROunded Context Preference Optimization for Truthful Multimodal LLMs
[20] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual Instruction Tuning. [21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems 36 (2023), 34892–34916. [22] Hanchao Liu, Wenyuan Xue, Yifei Chen, Dapeng Chen, Xiutian Zhao, Ke Wang, Liping Hou, Rongjun Li, and Wei Peng. 2024. A survey on hallucination in large vision-language models. arXiv preprint arXiv:2402.00253 (2024). [23] Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017). [24] Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. 2024. Strengthening multimodal large language model with bootstrapped preference optimization. In European Conference on Computer Vision. Springer, 382–398. [25] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36 (2023), 53728–53741. [26] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017). [27] Guohao Sun, Can Qin, Yihao Feng, Zeyuan Chen, Ran Xu, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. 2025. Structured Policy Optimization: Enhance Large Vision-Language Model via Self-referenced Dialogue. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 741–751. [28] Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. 2023. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525 (2023). [29] Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zujun Ma, and Chao Zhang. 2024. Enhancing multimodal LLM for detailed and accurate video captioning using multi-round preference optimization. arXiv preprint arXiv:2410.06682 (2024). [30] Fei Wang, Wenxuan Zhou, James Y Huang, Nan Xu, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. mdpo: Conditional preference optimization for
Conference’17, July 2017, Washington, DC, USA
multimodal large language models. arXiv preprint arXiv:2406.11839 (2024). [31] Junyang Wang, Yuhang Wang, Guohai Xu, Jing Zhang, Yukai Gu, Haitao Jia, Ming Yan, Ji Zhang, and Jitao Sang. 2023. An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation. arXiv preprint arXiv:2311.07397 (2023). [32] Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang, and Linchao Zhu. 2024. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback. arXiv preprint arXiv:2404.14233 (2024). [33] Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Fangxun Shu, Hao Jiang, and Linchao Zhu. 2025. Detecting and mitigating hallucination in large vision language models via fine-grained ai feedback. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 25543–25551. [34] Yuxi Xie, Guanzhen Li, Xiao Xu, and Min-Yen Kan. 2024. V-dpo: Mitigating hallucination in large vision language models via vision-guided direct preference optimization. arXiv preprint arXiv:2411.02712 (2024). [35] Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. 2024. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13807–13816. [36] Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Haocheng Feng, Jingdong Wang, et al. 2024. Automated multi-level preference for mllms. Advances in Neural Information Processing Systems 37 (2024), 26171–26194. [37] Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, et al. 2025. Mm-rlhf: The next step forward in multimodal llm alignment. arXiv preprint arXiv:2502.10391 (2025). [38] Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. 2024. Aligning modalities in vision large language models via preference fine-tuning. arXiv preprint arXiv:2402.11411 (2024). [39] Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, and Huaxiu Yao. 2024. Calibrated self-rewarding vision language models. arXiv preprint arXiv:2405.14622 (2024).