arXiv CSopen access

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

arXiv:2607.12752v1 [cs.CV] 14 Jul 2026

Hongbo Wang1,2 · Huaibo Huang1,2 · Jie Cao1,2 · Jin Liu1,3 · Haoyang Tong1,2 · Ran He1,2

Abstract While recent progress in 3D generation has enabled impressive visual synthesis, most existing methods still primarily rely on 2D diffusion-based supervision without mechanisms for enforcing geometric consistency, often resulting in spatial hallucinations such as duplicated structures or misaligned geometry. These challenges intensify in 4D generation, where maintaining consistency across viewpoints and temporal progression is substantially more difficult and often leads to temporal artifacts such as jitter, identity flicker, and structural drift. To address these limitations, we present Hallo4D, a unified and model-agnostic framework for mitigating spatiotemporal hallucinations in both 3D and 4D content generation. Hallo4D introduces a generation-detection-correction paradigm that leverages the reasoning capabilities of large multimodal language models (LMMs) to locate and summarize spatial and temporal inconsistencies from multi-view, multi-frame renderings. To prevent compounding errors from single-pass edits and ensure robust geometric fidelity, these insights guide a consensus-driven image-space consistency optimization, where an LMM selector evaluates multiple candidate corrections via multi-model voting. This process is achieved without requiring retraining or architectural modifications. To further enhance temporal consistency, Hallo4D incorporates a motion-saliency-driven keyframe sampling strategy based on optical flow, enabling more targeted and efficient refinement. The framework also includes an LMMguided initialization scheme and an attention-based appearance alignment module to improve early optimization and 1 MAIS&NLPR, Institute of Automation, Chinese Academy of Science, Beijing, 100190, China 2 University of Chinese Academy of Sciences, Beijing, 101408, China 3 Shanghaitech University, Shanghai, 201210, China E-mail: [email protected], {huaibo.huang, jie.cao}@cripac.ia.ac.cn, [email protected], [email protected], [email protected]

cross-view fidelity. Additionally, we address exposure instability with two losses: Contrastive Semantic Exposure Alignment (CSEA), a foreground-masked contrastive objective that favors well-exposed semantics while penalizing over- and under-exposure, and a log-dynamic-range (LDR) loss that regularizes luminance contrast. Together with union-of-frusta visibility pruning to remove out-ofview clutter and reduce pseudo under-exposure, these additions mitigate exposure-driven collapse under non-frontal views. Extensive experiments demonstrate that Hallo4D consistently outperforms strong baselines across diverse generation settings, offering a scalable and generalizable solution for consistency-aware 3D and 4D content generation. Additional visualizations are available on our project page: https://wafer-bob.github.io/Hallo3D-4D/. Keywords 3D Generation · 4D Generation · Spatiotemporal Consistency · Multi-Modal Reasoning.

1 Introduction Three-dimensional content generation has become a foundational capability for AI-driven visual understanding and simulation. With the advent of diffusion-based supervision, recent methods can synthesize high-fidelity 3D assets from minimal input, using text prompts (Poole et al., 2022; Jun and Nichol, 2023), single-view images (Liu et al., 2023b), or sparse multiview data (Jiang et al., 2023a; Shi et al., 2024). A particularly successful direction leverages 2D diffusion models as supervisory priors by training 3D representations to match the distribution of rendered views with that of a pre-trained image diffusion model. This is operationalized through Score Distillation Sampling (SDS) (Poole et al., 2022), which iteratively updates 3D parameters to minimize view-level discrepancies against the learned 2D prior. While remarkably effective, these approaches rely exclusively on view-dependent 2D supervision, which often

H. Wang et al.

Hallo4D

Baselines

Hallo4D

Baselines

2

Hallo4D

Baselines

View 1

Hallo4D

Baselines

View 2

Fig. 1: Comparison of 3D and 4D generation results between Hallo4D and the baseline. Our method achieves significant improvements in spatiotemporal consistency. leads to hallucinated or duplicated structures in unobserved views—a failure mode commonly referred to as the Janus problem(Armandpour et al., 2023). This issue reflects a core weakness of current pipelines: the absence of mechanisms to explicitly model and enforce geometric consistency across viewpoints. Several follow-up works attempt to address this by incorporating 3D-aware priors(Zhao et al., 2023; Liu et al., 2024c) or explicitly enforcing multiview coherence (Yang et al., 2023), but such solutions remain constrained by limited supervision and still face challenges in achieving robust generation. Importantly, while existing efforts primarily focus on improving spatial consistency within static 3D content, realworld scenarios demand coherence that extends across both space and time. Applications such as animation (Xu et al., 2024b) and tracking (Wu et al., 2024c) require modeling dy-

namic geometry that evolves continuously over time, naturally calling for 4D content generation to capture the spatiotemporal continuity of the physical world. Transitioning from static to dynamic geometry, however, introduces a new set of challenges that current 4D generation approaches have yet to resolve in a unified way. Some methods build upon dynamic NeRF frameworks (Pumarola et al., 2020; Li et al., 2022) by decoupling the canonical representation from the deformation field to model motion over time, achieving high-quality 4D content. Yet, their reliance on latent-space encoding often compromises geometric consistency across frames, leading to temporal artifacts such as jittering and ghosting (Zheng et al., 2024; Jiang et al., 2023b). Meanwhile, other approaches adopt explicit neural structures (Cao and Johnson, 2023) to model deformation, improving inference efficiency and robustness to local non-rigid motion.

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

Nevertheless, maintaining high-fidelity textures and ensuring stable spatiotemporal consistency across complex dynamic scenes remains highly challenging (Ren et al., 2023; Bahmani et al., 2024b; Singer et al., 2023). To alleviate the above limitations, we propose Hallo4D, a spatiotemporal consistency optimization framework for detecting and mitigating hallucinations in both 3D and 4D content generation. Central to our approach is the observation that LMMs exhibit strong spatial reasoning capabilities, allowing them to identify and summarize inconsistencies that emerge across both views and timesteps in dynamic scenes. Leveraging this capability, Hallo4D implements a novel generation-detection-correction paradigm that operates directly on multi-view, multi-frame renderings—without requiring any additional training data or task-specific supervision. By combining this paradigm with a consensus-driven diffusion-based 2D editing technique, which evaluates multiple candidate corrections to prevent error accumulation, our framework can be seamlessly applied to a wide range of existing 3D and 4D generation models without modifying their architectures or retraining procedures. Beyond consistency correction, Hallo4D further improves appearance quality in 3D and temporal coherence in 4D. We introduce an attention-based alignment module to enhance cross-view texture consistency, and a motionsaliency-driven keyframe sampling strategy to improve temporal stability and efficiency. In addition, we address exposure instability with two exposure-aware losses, Contrastive Semantic Exposure Alignment (CSEA) and a log-dynamicrange (LDR) loss, and apply union-of-frusta visibility pruning to suppress out-of-view clutter and reduce pseudo underexposure; these additions mitigate exposure-driven collapse under non-frontal views. These components together support reliable and high-quality generation across diverse scenarios. Specifically, for 3D generation, we first render images from multiple viewpoints and construct a 3D-inconsistencyaware inquiry to query the LMM. Through standardized responses, the LMM identifies cross-view inconsistencies and generates enhanced negative prompts, which are then used to guide targeted image-space optimization. This process leverages a Multi-view Appearance Alignment strategy, where a focal view supplies key and value features to a cross-attention mechanism, ensuring texture consistency across different viewpoints. For 4D generation, we introduce three additional mechanisms to address the more severe spatiotemporal challenges. First, an LMM-guided initialization stage. This stage jointly analyzes static 3D assets and animated sequences to detect cross-view and cross-timestep inconsistencies. The identified hallucinations are transformed into targeted initialization losses, ensuring that early optimization is guided by stable supervision and reducing the risk of compounding errors in subsequent stages. Second,

3

OF-Range for keyframe sampling. This optical-flow-based strategy prioritizes frames with salient motion while maintaining temporal dispersion. By focusing optimization on dynamically informative frames and avoiding redundant updates on static regions, OF-Range improves temporal coherence and enhances computational efficiency during 4D training. Third, an exposure-aware semantic alignment module. This module prevents view-dependent collapse by introducing two complementary losses: Contrastive Semantic Exposure Alignment (CSEA), a foreground-masked contrastive objective that aligns with well-exposed semantics while penalizing over- and under-exposure, and a log-dynamic-range (LDR) term that regularizes contrast in the log-luminance domain. Together with union-of-frusta visibility pruning to remove out-of-view clutter and alleviate pseudo underexposure, this module stabilizes optimization under nonfrontal views and improves overall robustness. Across both 3D and 4D stages, identified inconsistencies are corrected through a Prompt-Enhanced Re-consistency module, which uses diffusion-based image-space editing conditioned on enhanced negative prompts. Across both 3D and 4D stages, identified inconsistencies are corrected through a Consensus-Driven Re-consistency via Prompt Enhancement module. Instead of relying on a single deterministic edit that may introduce compounding errors, this module generates multiple candidate corrections guided by enhanced negative prompts, and employs an LMM Consensus Selector to determine the optimal geometrically faithful result via multi-model voting. To ensure theoretical validity, we establish an image-space reformulation of SDS that supports these corrections without disrupting the underlying training dynamics. Extensive experiments show that Hallo4D consistently outperforms strong baselines in spatial, temporal, and perceptual quality metrics, demonstrating its robustness and broad applicability across diverse 3D and 4D generative tasks. Our Contributions. We summarize our contributions as follows: – We propose Hallo4D, a unified spatiotemporal consistency framework for detecting and mitigating hallucinations in 3D and 4D generation. Hallo4D operates without additional data or retraining, ensuring broad applicability across diverse models. – We develop a novel generation-detection-correction paradigm that leverages LMMs for inconsistency detection and a consensus-driven image-space optimization for correction. We demonstrate that LMMs can not only infer spatial structures and diagnose multi-view and temporal inconsistencies but also act as an effective ensemble selector to ensure robust geometric fidelity via multimodel voting. – We design a 4D initialization scheme and an opticalflow-based motion saliency sampling strategy to im-

4

H. Wang et al.

prove early-stage optimization and enhance temporal stability in 4D generation. Additionally, we introduce two exposure-aware losses, together with unionof-frusta visibility pruning to reduce exposure instability under non-frontal views – Extensive experiments show that Hallo4D consistently improves visual quality and spatiotemporal consistency over strong baselines across various 3D and 4D tasks.

Differences from Our Prior Work Hallo3D (Wang et al., 2024a) published in NeurIPS 2024. This paper presents Hallo4D, a substantial extension of (Wang et al., 2024a), with major advances in scope, modeling capabilities, and theoretical grounding:

– Methodological Enhancement: Hallo4D extends spatial-only correction to unified spatiotemporal optimization. Crucially, we upgrade Hallo3D’s deterministic single-pass editing to a Consensus-Driven PromptEnhanced Re-consistency mechanism, mitigating compounding errors via multi-model voting. Coupled with LMM-guided initialization and motion-saliencybased keyframe sampling, this ensures high-quality, temporally coherent 4D generation. Furthermore, two novel exposure-aware losses (CSEA and LDR) are introduced to stabilize illumination and reduce pseudo under-exposure in non-frontal views. To our knowledge, this is the first plug-and-play, model-agnostic consistency optimization strategy for 4D generation. – Functional Generalization: The role of LMMs is further generalized in both scope and depth of application. Originally used to detect spatial inconsistencies in 3D outputs, LMMs are now shown to effectively identify temporal inconsistencies in 4D sequences, demonstrating their capacity to reason over both multi-view geometry and cross-frame dynamics. In addition, we extend their usage beyond post-hoc analysis by incorporating LMM-derived signals into the early optimization phase, enabling broader participation in consistencyaware generation. – Theoretical Grounding and Expanded Evaluation: We provide new theoretical analysis that explains why image-space consistency methods can effectively transfer to 3D and 4D geometry refinement, offering a stronger conceptual basis for our correction strategy. In addition, we broaden the empirical evaluation to include high-quality baselines not previously covered (Wang et al., 2023; Long et al., 2023), demonstrating the generality and robustness of the proposed framework across diverse generative settings.

2 Related Work 2.1 3D Content Generation The advent of diffusion models has revolutionized text-to3D generation, enabling the synthesis of high-fidelity 3D assets from textual descriptions. DreamFusion (Poole et al., 2022) pioneered this field by leveraging Score Distillation Sampling (SDS) to optimize 3D structures, integrating MipNeRF 360 (Barron et al., 2022) for neural rendering and Imagen (Saharia et al., 2022) for high-quality text-toimage generation. While NeRF-based approaches (Mildenhall et al., 2022; Lorraine et al., 2023; Zhou et al., 2023; Wang et al., 2023; Li et al., 2023; Gao et al., 2023; Lin et al., 2023) improve photorealism and lighting effects, their high computational cost limits scalability. To address this, 3D Gaussian Splatting (3DGS) (Kerbl et al., 2023) proposed an efficient alternative by representing 3D scenes with optimizable Gaussians, enabling real-time rendering. Subsequent refinements (Chen et al., 2023; Yi et al., 2023; Di et al., 2025) have improved compositionality and efficiency. On the optimization side, CoGrad3D (Tong et al., 2026) couples spatially-aware timestep scheduling with orthogonal gradient fusion across viewpoints, further improving view consistency and texture fidelity. Beyond conventional methods, GAN-based models (Attaiki et al., 2024; Li et al., 2024b) integrate GANs with diffusion models for fast, eliminating iterative optimization. Meanwhile, scene-level methods (Zhou et al., 2024b; Li et al., 2024c) improve structural and semantic coherence, expanding text-to-3D synthesis capabilities. Images from specific viewpoints provide stronger visual consistency for 3D generation. Image-based methods (Tang et al., 2023a; Fu et al., 2023; Liu et al., 2023c; Tang et al., 2023b; AlBahar et al., 2023; Qian et al., 2023; Long et al., 2023) often outperform text-based approaches by leveraging accurate multi-view supervision. 3D-aware image generation techniques (Xiang et al., 2023; Deng et al., 2022) enhance rendering consistency across viewpoints but still face challenges due to training data scarcity. Recent diffusionbased models (Jun and Nichol, 2023; Nichol et al., 2022) incorporating 3D priors have improved geometric realism, reducing perceptual errors and advancing the fidelity of image-to-3D synthesis. More recently, feed-forward generative models built on structured 3D latents (Xiang et al., 2025) have substantially improved the scalability and fidelity of direct 3D asset generation.

2.2 4D Content Generation Generating 4D content (Miao et al., 2025), which extends 3D synthesis by introducing temporal dynamics, requires balancing motion realism, efficiency, and control to create

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

5

coherent, high-fidelity dynamic scenes. Early efforts built on NeRF-based models (Du et al., 2021; Pumarola et al., 2020) or theoretical methods based on geometry (Laga et al., 2021), but these struggled with high computational costs and temporal inconsistencies. More recent diffusion-based techniques (Bahmani et al., 2024b,a; Ling et al., 2024) introduced score distillation and trajectory conditioning, significantly improving motion coherence and structure. Parallel to this, video-driven methods (Chu et al., 2024; Jiang et al., 2023b; Yang et al., 2025; Pan et al., 2025) refined multi-object interactions, viewpoint consistency, and frame-wise feature disentanglement, addressing challenges in scene complexity. Meanwhile, physics-aware approaches (Zhang et al., 2024; Huang et al., 2024) incorporated realworld constraints, enhancing realism in object dynamics. As generative models advanced, the demand for efficient scene representations grew. Traditional volumetric techniques were computationally expensive, leading to the rise of Gaussian splatting (Wu et al., 2024a; Lin et al., 2024; Ren et al., 2023; Zeng et al., 2024; Ling et al., 2024), enabling real-time rendering while preserving fidelity. Diffusion-driven motion synthesis (Blattmann et al., 2023; Sun et al., 2024) further improved scalability, while LLM-guided compositional approaches (Xu et al., 2024a) added structural priors for greater scene control. Beyond efficiency, structured motion control remains crucial. Hybrid representations (Li et al., 2024e; Xie et al., 2024) enhanced multi-view consistency and sparse motion control, while high-resolution 4D synthesis (Li et al., 2024d) improved detail retention in large-scale environments, enhancing adaptability and realism in complex dynamic settings. More recently, multi-view video diffusion models such as CAT4D (Wu et al., 2025) and feed-forward 4D generative frameworks (Chen et al., 2025) enable direct 4D creation from monocular inputs, further broadening the applicability of dynamic content generation.

preference optimization (Sun et al., 2026b) refine alignment, while geometric corrections (Liu et al., 2023a) and videoinspired frameworks (Voleti et al., 2024) offer alternative solutions. Additionally, methods like Liu et al. (2024b); Liang et al. (2023) enhance consistency and fidelity across dynamic and complex 3D content, and recent think-driven approaches (Jiao et al., 2026; Guo et al., 2026; Liu et al., 2026; Fan et al., 2026) leverage the reasoning capabilities of large multimodal models to enforce structural consistency in visual and 3D generation. However, these approaches still face challenges with non-orthogonal viewpoints and adaptability across different architectures. Building on multi-view alignment in 3D, recent efforts in 4D generation increasingly focus on ensuring temporal consistency across frames. Techniques such as spatiotemporal anchoring and feature synchronization (Zeng et al., 2024; Grammatikopoulou et al., 2023) improve frame coherence and enable more consistent motion generation. Meanwhile, temporal-consistent diffusion models and fast spatiotemporal optimization strategies (Zhou et al., 2024a; Liang et al., 2024b) reduce flickering artifacts. Further advances target long-range temporal correspondence and deformation minimization (Yang et al., 2024a; Ouyang et al., 2024), while others (Xu et al., 2024c; Wang et al., 2024b) focus on producing smoother and more stable motion trajectories. Progress in motion tracking and flow refinement (Chen et al., 2024; Liang et al., 2024a) has also contributed to enhancing the robustness of dynamic content generation. While most existing methods focus on addressing consistency issues within specific models, efforts toward improving generality (Qiu et al., 2023b; Wang et al., 2025; Liu et al., 2025) remain confined to either 3D or 4D generation within limited domains. In contrast, our proposed Hallo4D leverages LMMs and image-space optimization to achieve scalable and efficient consistency enhancement across both 3D and 4D generation tasks.

2.3 Spatio-temporal Consistency Enhancement

3 Preliminaries

Ensuring consistency is fundamental for generating coherent 3D and 4D content. In 3D generation, the primary focus lies in maintaining multi-view consistency, however, a key challenge arises from the reliance on 2D diffusion models, which often introduces viewpoint-specific hallucinations. To mitigate this, fine-tuning approaches (Shi et al., 2024; Zhao et al., 2023; Yang et al., 2023; Liu et al., 2024c; Jiang et al., 2023a) incorporate 3D constraints into diffusion models, improving consistency for 3D subjects (Ruiz et al., 2023; Raj et al., 2023), transparent objects (Zhang and Agrawala, 2024), and multiple viewpoints (Seo et al., 2023; Ji et al., 2026). Beyond fine-tuning, prompt optimization (Hong et al., 2023; Armandpour et al., 2023), objectlevel self-correction (Sun et al., 2026a), and instance-aware

3.1 Diffusion Models Diffusion models (Ho et al., 2020) has a forward diffusion process with diffusion steps from 0 to T , which degrades the original sample x0 into pure noise xT over T steps via: √ √ xt = αt x0 + 1 − αt ϵ, ϵ ∼ N (0, I), (1) where α := (α1 , . . . , αT ) ∈ RT≥0 controls the noise schedule. The reverse process then reconstructs x0 from xT . This paradigm now underpins a broad spectrum of visual synthesis and restoration tasks, including image restoration and enhancement (Li et al., 2025) and faithful image superresolution (Wang et al., 2026). In text-guided diffusion (Takagi and Nishimoto, 2022), conditioning on a text prompt P

6

H. Wang et al. Appearance Attention

Score Distillation Sampling

Focal View

K V

2D Diffusion

3D Content

Multi-view Renderings

Q

×4

×4

AAttn

" A cartoon fox"

Inquiry LMM

Consensus-Driven Re-consistency via Prompt Enhancement

Video

“Which two adjacent frames differ the most?" Input Inquires

“ , the latter is more severe, suffering distorted facial features ……" Output Analysis

LMM

2D Diffusion

Image

“Are there any Inconsistency?" Input Inquires

Input Image

Pos: "A cartoon fox" Neg: " Abnormal 3 ears , …"

Output Images

Consensus Voting

“ Abnormal three ears , ……"

Selected Images LMM

Output Analysis

Fig. 2: Pipeline overview of the 3D consistency optimization stage. We jointly optimize our model using LSDS and LCG . For LSDS , a focal view is selected based on camera pose and used as keys and values to align all views via attention. For LCG , hallucinations are identified using LMMs and translated into enhanced negative prompts. These prompts guide the generation of multiple candidate corrections, from which an MLLM consensus selector determines the optimal result via multi-model voting to enforce re-consistency. is achieved via a text encoder (e.g., CLIP (Radford et al., 2021)). The denoiser ϵϕ is trained to predict the noise ϵ given the noisy input xt and text condition: L(ϕ) = Et∼U (1,T ),ϵ∼N (0,I) ∥ϵ − ϵϕ (xt ; t, P )∥22 ,

(2)

To enhance text-image alignment, Classifier-Free Guidance (CFG) (Ho and Salimans, 2021) interpolates between conditional and unconditional predictions: ϵ̃ϕ (xt ; t,P, s) = ϵϕ (xt ; t, ∅)+s (ϵϕ (xt ; t,P )−ϵϕ (xt ; t, ∅)) ,

ϵ defines the gradient for updating θ, thereby improving the 3D model. The SDS gradient is computed as:   ∂x ∇θ LSDS (ϕ, x = g(θ)) ≜ Et,ϵ w(t) (ϵ̃ϕ (xt ; t, P )−ϵ) , ∂θ (4) where w(t) is a time-dependent weighting function.

4 Methodology

(3) where s is the guidance scale and ∅ denotes the null prompt. In practice, negative prompts P − (Du et al., 2020) are used instead of ∅ to avoid undesired content in the generated samples.

3.2 Score Distillation Sampling SDS, first proposed in DreamFusion (Poole et al., 2022), employs vision priors from pre-trained 2D diffusion models to supervise 3D model optimization, establishing itself as a foundational learning paradigm for 3D generation. Given a 3D representation model parameterized by θ and a differentiable renderer g(θ, c) that generates a rendered image x from camera pose c, SDS aligns the probability distribution of x with the diffusion model prior p(ϕ). Specifically, SDS introduces a score function ϵϕ (xt ; t, P ), which predicts noise ϵ̂ for a noisy image xt conditioned on text prompt P . The discrepancy between ϵ̂ and the injected Gaussian noise

This section introduces the overall framework and underlying principles of Hallo4D. At the core of Hallo4D lies a consistency optimization strategy for 3D content generation, which serves as the foundation for addressing a more challenging aspect—achieving spatiotemporal consistency in the 4D domain. Building on this foundation, we first describe our method for enhancing multi-view consistency in 3D generation. We then extend this approach to incorporate temporal coherence, enabling joint optimization across both spatial and temporal dimensions.

4.1 Multi-view Appearance Alignment Conventional SDS optimizes isolated single-view images, which contradicts the principle of multi-view simultaneity required for consistent 3D representation. This singleview formulation limits cross-view interaction during training, leading to the loss of surface details during denoising, as shown in Fig.10. To address this, we propose a

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

7

Initialization for 4D Optimization “A cartoon fox is dancing”

Multi-modal Language Model

Text Prompt

view

3D Content

2D Image (optional, depending on baselines)

Prompt Enhanced Re-Consistency

: Abnormal three ears

360º Video

Inquiry LMM

Multi-modal Language Model

: Excessive white of eye

time

Consensus & Re-Consistency

Prompt Enhanced Re-Consistency

: Distorted facial features =

Animated Video

Inquiry LMM

Consensus & Re-Consistency

4D Optimization

Sampling Results:

OF-Range

4D Content

Multi-modal Language Model

4D Frame Sampling

Inquiry LMM

… : Rough Texture

…

Prompt Enhanced Re-Consistency

Consensus & Re-Consistency

Fig. 3: Pipeline overview of the 4D spatiotemporal consistency stage. We first identify the views and frames with the most severe inconsistencies through initialization to guide early-stage training. During training, we employ OF-Range, an optical flow-based inter-frame sampling strategy that improves both the efficiency and effectiveness of the 4D generation process. Multi-view Appearance Alignment strategy by introducing an appearance-attention mechanism AAttn(·) into the denoising function ϵ̃ϕ (·), replacing the original self-attention in the U-Net architecture(Ronneberger et al., 2015). Specifically, inspired by recent advancements in diffusion models (Liu et al., 2024a; Cao et al., 2023), which suggest that query features within the self-attention spaces primarily shape image structure and layout, while key and value features influence texture, our method leverages this insight. As illustrated in the bottom left corner of Fig. 2, we select a focal view i based on the camera pose, using the image from this viewpoint to provide the key and value features in the attention module. These are used to compute query features across all views, ensuring alignment of appearances. The cross attention is defined by the following formula:  AAttn(Q, Ki , Vi ) = Softmax

QKiT √ d

 · Vi ,

(5)

where AAttn(·) is the appearance attention, with Ki and Vi as the key and value features corresponding to the image rendered from the focal view i, and Q as the query feature from all views. The key and value are derived from the focal view, while each of the four views computes a distinct query. By jointly rendering and processing images from multiple viewpoints, our method enables view-aware denoising and enhances appearance consistency across views, outperforming conventional single-view SDS approaches (Poole et al., 2022; Wang et al., 2022)

4.2 Multi-modal Hallucination Detection Existing 3D generation methods often suffer from spatial inconsistencies, primarily due to the inherent limitations of 2D pre-trained models in understanding 3D geometry. The problem become even more pronounced in 4D generation, where temporal dynamics further amplify spatial artifacts. Moreover, while 3D generation benefits from the relatively standardized SDS framework (Poole et al., 2022), 4D generation lacks a unified paradigm. Its diverse conditioning inputs—ranging from text, images, and videos to 3D assets—combined with the higher-dimensional nature of the task, have led to fragmented and less generalizable solutions. This lack of coherence across methods impedes direct applicability to existing frameworks, highlighting the need for a pluggable and broadly compatible consistency optimization strategy. To bridge this gap, we propose an inconsistency detection module, Multi-modal Hallucination Detection, which leverages LMMs to analyze multi-view and multi-frame renderings—serving as the common basis for 4D generation. Benefiting from the strong cross-modal reasoning and temporal understanding capabilities of recent LMMs (Li et al., 2024a), and in line with recent progress on multimodal hallucination evaluation and detection (Chen et al., 2026), our approach builds on the hypothesis that LMMs can effectively detect and help mitigate spatiotemporal inconsistencies. To validate this hypothesis, we design a two-phase inquiry-based evaluation assessing three critical aspects: spatial structure reasoning, multi-view inconsistency diagnosis, and temporal inconsistency localization. As shown

8

H. Wang et al.

in Fig. 4, LMMs, even without explicit geometric supervision, can accurately identify inconsistencies across 3D renderings and video frames. Importantly, our use of LMMs is deliberately narrow: they act as lightweight view/framelevel consistency checkers in a constrained setting (single foreground subject, fixed-orbit cameras, standardized prompts), requiring pattern-level comparison rather than full 3D reconstruction. This aligns with their pretraining, which provides cross-view priors from multi-view/3D data. Concretely, LLaVA-OneVision-72B (Li et al., 2024a) is trained on multi-view/3D-related corpora, including (Dai et al., 2017; Caesar et al., 2020; Shridhar et al., 2020; Suhr et al., 2019; Jhamtani and Berg-Kirkpatrick, 2018; Hosseinzadeh and Wang, 2021; Azuma et al., 2022), providing cross-view and 3D localization priors leveraged by our pipeline. To stabilize and standardize outputs, we adopt one-shot Chainof-Thought prompting (Brown et al., 2020) with a reference template, from which we extract the indicated target views/frames and the corresponding negative prompts. In our framework, we consider three types of inputs: a single rendered image xr , a multi-view video xv generated from 3D object renderings, and a temporally ordered frontview video xf , each paired with a 3D-aware inquiry prompt PI = {PIr , PIv , PIf }. The LMM processes these inputs to generate enhanced negative prompts PE− , which are then used to guide subsequent image-space corrections. The formulation is given by: PE− = Dψ (xr , PIr ),

(6)

where Dψ denotes the LMM parameterized by ψ. For 4D generation, the LMM additionally performs inconsistency reasoning over both views and frames: v ∗ , PE− = Dψ (xv , PIv ), τm , τa1 , τa2 , τ

∗

, PE− = Dψ (xf , PIf ),

(7) (8)

where v ∗ is the inconsistent view from multi-view renderings, τ ∗ is the detected inconsistent frame in the animated video, τa1 and τa2 are adjacent frames with significant motion differences, and τm ∈ {τa1 , τa2 } is the frame where inconsistency arises. For clarity, we omit explicit distinctions between different PE− variants. 4.3 Consensus-Driven Re-consistency via Prompt Enhancement With the enhanced negative prompt PE− introduced in Sec. 4.2, one natural approach to improve consistency is to apply 2D image editing techniques directly on the rendered views. However, a conceptual gap emerges: while image editing operates in the pixel space, most existing 3D generation pipelines grounded in SDS supervise 3D models through the noise space. This apparent mismatch

raises concerns about the theoretical validity of applying image-space corrections under SDS-based training. Recent works(Kingma et al., 2021; Deng et al., 2024) suggest that diffusion models can be reinterpreted to predict images directly rather than denoising noise. Motivated by this perspective, we revisit the underlying formulation of SDS to explore whether it can be consistently extended to the image space. Starting from the DDPM, by substituting the expression of Eq. (1) into Eq. (4) to replace ϵ, we obtain the following equation:     √ xt − αt x0 ∂x ∇θ LSDS (ϕ, x) ≜Et,ϵ w(t) ϵ̂ϕ (xt ; t, P ) − √ , ∂θ 1−αt    ∂x  ≜Et,ϵ w′ (t) x0 −xϕ , 0 ∂θ (9)

where √ xϕ0 = xt −

1−αt ϵ̂ϕ (xt ; t, P ) , w′ (t) = √ αt

r

αt w(t), 1−αt (10)

Building on this foundation, we derive an image-space formulation of SDS, thereby establishing the methodological validity of directly operating on rendered images. However, existing image editing approaches predominantly focus on manipulating the null prompt (Mokady et al., 2023) or modifying the positive prompt (Hertz et al., 2022), which are generally ineffective at altering the geometric structure of 2D images rendered from 3D models. To overcome this limitation, we propose a novel module, termed Prompt-Enhanced Re-consistency, which introduces PE− to enhance the geometric fidelity of the rendered outputs. We regenerate the 2D rendered image x0 under the guidance of PE− . Specifically, to preserve the original semantic information of x0 , we employ DDIM (Song et al., 2022) to invert the image x0 to its noisy representation xT as follows. r αt−1 x̂t−1 = x̂t + δt ϵ̃ϕ (x̂t ; t, P, PE− ), (11) αt s p αt−1 (1 − αt ) δt = ( 1 − αt−1 − ), (12) αt Subsequently, we apply DDIM sampling to regenerate the consistent versions of the image from the DDIM inverted xT , denoted as x̂0 . This approach also ensures that the regenerated image retains its core semantic integrity while improving its multi-view consistency, and it can be readily derived by mathematical induction that: r T X α0 x̂0 = xT + κi ϵ̃ϕ (x̂t ; t, P, PE− ), (13) αT i=1 s s α0 (1 − αi−1 ) α0 (1 − αi ) κi = ( − ), (14) αi−1 αi

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

9

Spatial Structure Reasoning Capabilities This image was rendered from the 3D generated by AI, and you can look at the spatial structure of the object in the picture to see if there are any inconsistencies with body shape, perspective, surface texture, color, and so on. The whole photo, especially the dancer's face, is blurry. Subjects in the picture are confusing and there may be structural inconsistencies. You are a master of graphic restoration. Based on your analysis above, summarize the 3D inconsistency problems as negative prompts for restoring the image I provided. Reference is as bellow: Negative Prompt: “multi-head, unnatural lighting, smooth appearance, distorted color, long neck”. Negative Prompt: "blurry background, duplicated feature, unsharp edges, lack of detail clarity, artificial object".

Multi-view Inconsistency Diagnosis Capability This video is composed of AI-generated renderings of a 3D object from different views. Please verify consistency across views …… (Similarly above) The fourth view exhibits a clear Janus problem, with the eagle showing two heads, indicating a severe semantic inconsistency absent in other views. You are a master of graphic restoration. Based on your analysis above, point out the inconsistent view and summarize the 3D inconsistency problems as negative prompts for restoring the view. Reference is as bellow: View: “3”; Negative Prompt: “multi-head, unnatural lighting, smooth appearance, distorted color, long neck”. View: "4"; Negative Prompt: "duplicated eagle head, semantic inconsistency, structural artifact, incorrect anatomy".

Temporal Inconsistency Localization Capability This video is generated by AI and depicts a moving 3D object. Please verify consistency across frame …… (Similarly above) Additionally, check for any two consecutive frames with excessively large motion differences that may lead to instability or collapse in one of the frames. Frame 3 exhibits a clear collapse, with severe facial distortion and motion blur artifacts. Additionally, frame 2 shows hand blur due to a large motion transition from frame 1. You are a master of video understanding. Based on your analysis above, point the inconsistent frame and identify consecutive frames with large motion differences and indicate which frame exhibits a severe issue …… (Similarly above) Inconsistent View: “4”; Negative Prompt: “multi-head, unnatural lighting, smooth appearance, distorted color, long neck". High-motion adjacent frames: “3; 3, 4”; Negative Prompt: “multi-head, unnatural lighting, smooth appearance". Inconsistent View: "3"; Negative Prompt: "collapsed face, distorted head, multi-layered facial features". High-motion adjacent frames: "2; 1, 2"; Negative Prompt: "motion blur, smeared hand, low-definition hand region".

Fig. 4: A multi-modal reasoning case study designed to evaluate the capabilities of LMMs in spatial structure reasoning, multi-view inconsistency diagnosis, and temporal inconsistency localization. The first round of dialogue demonstrates that LMMs possess the necessary reasoning abilities, while the second round illustrates that LMMs can produce responses in specific formats, enabling the subsequent extraction of both negative prompts and the associated target views or specific frames to be processed using regular expressions. Meanwhile, a practical limitation of single-shot image editing is that it may introduce stochastic editing noise or local over-corrections, and repeated reliance on one edited output can accumulate error, especially when the editing model is imperfect. To reduce over-dependence on any single edit, we augment Prompt-Enhanced Re-consistency with a consensus-based selection module inspired by the LMM Consensus Selector. Instead of producing only one x̂0 , we generate a candidate set of edited images by varying the editing randomness which includes DDIM sampling noise and timesteps:

the same constrained setting used by our detector. Each evaluator selects the index of its preferred candidate:   (γ) (γ) (γ) eval γk = arg max Sk , Sk = Dψk x̂0 , PS (16) γ

(γ)

where k ∈ [m], Sk denotes the score assigned by the kth LMM to the γ-th image, and PS represents the scoring prompt provided to the LMMs. Then we aggregate votes across evaluators: (γ)

Vote(x̂0 ) =

m X

I[γk = γ],

(17)

k=1 (1)

(2)

(n)

Xcand = {x̂0 , x̂0 , . . . , x̂0 },

(15)

We then introduce an evaluator ensemble of LMMs eval eval eval Dψ = {Dψ1 , . . . , Dψm }, each independently scoring the candidates with respect to (i) structural integrity, (ii) prompt alignment, and (iii) cross-view/temporal plausibility under

The final corrected image is chosen by majority voting, (γ)

x̂∗0 = arg max Vote(x̂0 ), γ

(18)

This consensus mechanism explicitly targets the failure mode where a single edit introduces subtle artifacts that are

10

H. Wang et al.

hard to detect numerically but are consistently penalized by multiple independent LMM judges, thereby improving robustness and reducing the risk of error accumulation across optimization iterations. Finally, we train the 3D model θ using the MSE loss LCG between x0 and x̂0 in the image space: h i 2 LCG ≜ E |x̂∗0 − x0 | ,

Baselines

Under-Exposure

Hallo4D

Fig. 5: Attention Visualization in Prompt-Enhanced Reconsistency. Left: image shows an inconsistent rendered view. Middle and Right: images visualize U-Net attention from the DDIM-Inversion-based image editing model, with the middle and rightmost images corresponding to early ( t → T ) and late ( t → 0 ) diffusion steps, respectively. It is evident that, as diffusion progresses, the attention gradually decreases its focus on the inconsistent regions.

Hallo4D

Baselines

Over-Exposure

Fig. 6: Illustration of over-exposure and under-exposure situation. Baselines exhibit view-dependent collapse in non-frontal views, producing white-out/black-out frames, whereas Hallo4D, with losses of CSEA and LDR, maintains stable exposure and preserves structure across viewpoints.

(19)

By combining Eq. (9) and Eq. (19), the effectiveness of our method can be theoretically justified, provided that x̂0 exhibits stronger view consistency and better alignment with realistic semantics than xϕ0 . To support this, we visualize the attention maps from the intermediate steps of Eq. (13), as shown in Fig. 5. It can be seen that during the early stages of editing, the attention covers the entire fox in the image, while in the later stages, attention to inconsistent regions—such as the third ear on top of the fox’s head—is significantly reduced. This indicates that the negative promotbased image editing technique effectively corrects inconsistencies in the rendered image, producing a refined x̂0 that better aligns with realistic semantics. The overall process is illustrated in Fig. 2. Moreover, it is worth noting that we apply PromptEnhanced Reconsistency only when the rendered image exhibit complete semantic structure. Our detector, Dψ , assesses the semantic completeness of the image. If the semantic structure is deemed incomplete or unclear, Dψ returns None, precluding further processing. This ensures that enhancements are only applied to images that are adequately prepared. This dependency of our enhancement process on the state of semantic completeness directly influences the formulation of the final training loss for the 3D model, as detailed below: ( LSDS + w1 LCG , if Dψ (x, PI ) ̸= None, L(θ) = (20) LSDS , otherwise,

where w1 is set to balance the magnitude of LSDS and LCG . By incorporating LCG , which is only applied when Dψ confirms the semantic readiness of the image, we ensure that our model focuses on enhancing well-formed images. This selective application of LCG prevents further exacerbating the quality of images already of poor quality. Simultaneously, it avoids misallocating resources to images that do not benefit from the intended enhancements, thereby improving the efficiency and effectiveness of our training process.

4.4 Initialization for 4D Optimization As discussed in Sec. 4.2, 4D generation poses more severe spatiotemporal consistency challenges compared to 3D counterpart. Through our experiments, we further observe that the diverse input modalities in 4D generation—whether from the 3D object or the animated video—can independently introduce inconsistencies that propagate and degrade the final output. This highlights the necessity of early intervention during model training. To address this, we propose an initialization strategy for 4D optimization that preprocesses inconsistencies in both the 3D content and the animated video, thereby mitigating error accumulation during the subsequent 4D training process. Specifically, for the 3D object in the 4D generation input, we sample eight surrounding views at 45° intervals and concatenate them in order to form a video. This video is then

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

fed into the LMM, which is queried to identify the view exhibiting inconsistency and to summarize the issue as a negative prompt, yielding v ∗ , PE− . For the animated video, it is directly input to the LMM, which is asked to identify the inconsistent frame τ ∗ , the pair of adjacent frames with significant motion τa1 , τa2 , and the specific frame τm within the pair where the inconsistency occurs due to large motion. These inconsistency cues are then summarized into the corresponding PE− . The process is formalized in Eq. (7) and illustrated in Fig. 3. In addition, we apply the PromptEnhanced Re-Consistency module to the identified inconsistent views or frames, and following Eq. (19) compute the corresponding loss term LCG−init , which is weighted and applied during the first 10% of training iterations on top of the baseline. The formulation is as follows: ( L(θ) + w2 LCG−init , if e < 10%E, L4D (θ) = (21) L(θ), otherwise, where w2 is a weighting factor that balances the scale of different loss terms, e denotes the current training epoch, and E represents the total number of training epochs.

4.5 OF-Range based Sampling Strategies Although multi-modal hallucination detection via LMMs enables accurate identification of spatiotemporal inconsistencies, querying LMMs incurs non-negligible computational overhead, making full-frame or full-view detection impractical at every training epoch. In 3D generation, we alleviate this by randomly selecting one view per epoch. However, this strategy is unreliable for 4D keyframe selection: as noted in Sec. 4.4, the known frames τm and τ ∗ are easily oversampled, reducing optimization efficiency. Meanwhile, extensive research(Lai et al., 2018; Huang et al., 2023; Yang et al., 2024b) underscores the importance of effective frame selection for maintaining temporal consistency. A natural solution is to detect keyframes based on geometric changes across time. However, synthesized 3D objects often exhibit free-form motion without rigid-body assumptions, and localized deformations cannot be reliably captured by explicit geometric features. Therefore, we propose OF-Range, an optical flow-based inter-frame sampling strategy that improves both the efficiency and effectiveness of the 4D generation process. Specifically, we begin by rendering a sequence of frontview frames from the 4D asset, resulting in a frame set. For each frame τ , we extract Shi-Tomasi corner (Jianbo Shi and Tomasi, 1994) features Cτ = {cj } from geometrically stable interior regions, leveraging cross-frame coherence while avoiding transient surfaces that may introduce reconstruction ambiguity. Optical flow vectors {vj } between consecutive frames are then computed using the Lucas-Kanade

11

method(Lucas and Kanade, 1981). These are used to define a deformation-sensitive motion saliency score: P j exp(−λd(cj , Γτ ))|vj | Sτ = P , (22) j exp(−λd(cj , Γτ )) where exp(−λd(cj , Γτ )) is a spatial confidence weight that decays exponentially with the Euclidean distance d(cj , Γτ ) from each feature point cj to the nearest unstable boundary region Γτ . This weighting mechanism suppresses noisy motion near volatile areas while amplifying deformation cues from stable regions. Given that most 4D generation models operate on short sequences (typically n < 15) due to GPU memory constraints, it is crucial to prevent selection bias and avoid repeatedly sampling frames near the previously identified keyframes τm and τ ∗ . To this end, we formulate keyframe selection as a regularized stochastic optimization problem:
τ̂ = arg max{E[Sτ ] − γ I(τm ̸= τ ∗ ) · Qτ +Gτ }, (23) τ

Qτ = min(|τ − τm |, |τ − τ ∗ |)/n,

(24)

where Qτ penalizes the selection of frames temporally close to existing keyframes, and Gτ ∼ Gumbel(0, 1) introduces stochasticity to encourage diverse sampling across epochs. This strategy jointly accounts for motion saliency, temporal diversity, and stochastic exploration, leading to more robust and efficient keyframe selection for 4D training. 4.6 Adaptive Exposure-Aware Semantic Alignment Hard cases remain a persistent obstacle in 4D generation, with one of the most severe failure modes being viewdependent collapse under non-frontal viewpoints. In our empirical studies, we repeatedly observe that exposure instability—either as a primary error or via propagation through the generation pipeline—can precipitate catastrophic overexposure or underexposure, yielding outputs that are nearly uniform extreme white or black. In such cases, the underlying semantics are largely lost, where recovery is challenging for inversion-based post-hoc procedures. These exposuredriven failure modes are illustrated in Fig. 6. To address these hard cases, we augment our existing strategy with a contrast-aware reward specifically aimed at stabilizing exposure in challenging views. Because background handling varies across baselines, we first factor out background effects by deriving a soft foreground prior from the alpha channel, M ∈ [0, 1]H×W , and compositing a foreground-only image xfg onto a neutral canvas C: xfg = M ⊙ x + (1 − M ) ⊙ C,

(25)

This concentrates optimization on the object—routing gradients primarily through foreground pixels—while the neutral canvas suppresses spurious background influence. To

12

H. Wang et al.

GaussianDreamer

SJC

Magic3D

ProlificDreamer

Baselines

DreamFusion-IF

270º

90º

270º

90º

270º

90º

270º

90º

270º

Hallo4D

90º

Baselines

prompt: An elegant flamingo standing tall with long legs and pinkish-white feathers.

225º

45º

225º

45º

225º

45º

225º

45º

225º

285º

105º

Hallo4D

45º

Baselines

prompt: A sculpture of a dog in medieval style, sitting upright under a spotlight.

105º

285º

105º

285º

105º

285º

105º

Hallo4D

285º

prompt: An electric sports car with aerodynamic curves and glossy blue appearance.

Fig. 7: Qualitative comparison in text-driven 3D generation of Hallo4D and baseline models. For intuitive comparison, we use the same and complementary viewpoints to present the results, where our method shows significant improvements. combine signals from multiple exposure-re lated loss components in a stable and differentiable way, we define the logsum-exp operator LSE with parameter µ as: X LSE(Z; µ) = µ log exp(z/µ), (26) z∈Z

This operator behaves as a smooth maximum: for small µ it approximates the worst offender in Z, while larger µ yields a softer aggregation. It thus provides a tunable mechanism to focus the penalty on the most problematic exposure tendency without introducing non-differentiabilities. Concretely, a CLIP-based reward Rc is computed on the foreground-only image xfg using exposure-aware textual prompts. fg RP c = CLIP(P, x |P ∈ PΛ ),

(27)

combining evidence across exposure modes. The resulting Contrastive Semantic Exposure Alignment loss encourages strong alignment with well-exposed semantics while simultaneously suppressing alignment with over- and underexposed semantics, yielding a single differentiable objective that focuses learning on the most problematic exposure tendency. LCSEA = −Rc+ + LSE(Rcover , Rcunder ; µ),

(28)

We optimize a contrastive, exposure-aware objective that pulls the image toward the well-exposed description while pushing it away from both over- and under-exposed cues. During optimization, gradients are masked to the foreground so that background brightness does not become a lever for objective reduction:

where PΛ = {PΛ+ , PΛover , PΛunder } denote the well-exposed,

∇x LCSEA = M ⊙ (∂LCSEA /∂xfg ),

over-exposed, and under-exposed prompts respectively, and fg define RP c = CLIP(P, x ) as the corresponding CLIPScore. These rewards serve as the inputs to the log-sumexp operator, which provides a smooth, stable means of

This gating eliminates background-driven degeneracies, stabilizes updates on the object surface, and ensures that improvements in the loss correspond to genuine object-level corrections rather than exposure tricks.

(29)

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

13

DreamGaussian

Hallo4D

Zero-1-to-3

Hallo4D

Wonder3D

Hallo4D

DreamGaussian

Hallo4D

Zero-1-to-3

Hallo4D

Wonder3D

Hallo4D

Fig. 8: Qualitative comparison in image-driven 3D generation of Hallo4D and baseline models. For intuitive comparison, we enlarge the regions with noticeable differences, and the results demonstrate that our method achieves significant improvements. Moreover, we regularize the rendered appearance toward a target contrast bandwidth with a log-dynamic-range loss:
LLDR = | std log(ϑ + Y ) −σ0 |,

(30)

Table 1: Quantitative comparisons in text-driven 3D generation. CLIP Scores ↑

Methods

B/32

where std(·) is the per-image standard deviation evaluated over foreground pixels, Y denotes per-pixel luminance in the linear RGB domain, ϑ > 0 is a small stabilizer that prevents singular gradients in dark regions, and σ0 specifies the desired log-luminance standard deviation. Minimizing LLDR steers the contrast distribution toward the desired bandwidth, suppressing both insufficient contrast and excessive contrast that would otherwise push highlights or shadows into saturation. Across extensive experiments, we found that there exist cases where apparent under-exposure is in fact pseudo under-exposure: large swaths of pure-black floating geometry arise within the frustum in non-frontal views and occlude the subject. In parallel, we frequently observe floating clutter outside the camera frusta. For the out-of-frustum case, existing 3D/4D pipelines leave these regions unconstrained—such primitives are never rendered, receive no SDS supervision, and remain invisible until model export. To reduce the risk of unpredictable generation collapses and make effective use of limited memory, at each validation pass we apply deterministic visibility pruning that retains only primitives whose centers lie within the union of camera frusta (UoF) defined by the evaluation trajectory and discards the rest. Although introduced to suppress out-of-frustum clutter, this simple UoF pruning also alleviates the aforementioned pseudo under-exposure by removing the pure-black floating geometry that occludes nonfrontal views, thereby improving the robustness of the 4D generation pipeline.

B/16

L/14

User Study ↑ Cons.

Qual.

Align.

GSD

21.31

22.67

23.70

6.00

5.53

5.57

+Hallo4D

24.68

27.11

30.22

9.01

8.82

8.95

SJC

20.13

21.36

23.95

4.53

4.77

5.63 7.53

+Hallo4D

24.39

26.41

28.17

7.79

7.92

DF-IF

14.09

15.98

18.19

4.63

4.17

4.70

+Hallo4D

22.19

23.82

26.80

6.41

7.50

7.12

Magic3D

14.93

16.41

17.99

5.13

4.60

5.17 7.40

+Hallo4D

22.14

24.35

27.83

7.61

8.06

P.Dreamer

22.45

24.18

26.85

4.17

8.50

7.26

+Hallo4D

25.71

28.42

31.49

7.64

9.37

9.23

5 Experiments In this section, we conduct a comprehensive evaluation of Hallo4D on both 3D and 4D generation tasks, presenting comparative results against baseline models to showcase its effectiveness. To further demonstrate Hallo4D’s ability to enhance multi-view consistency in 3D and spatiotemporal consistency in 4D generation, we also perform an extensive user study. Finally, we carry out ablation experiments to validate the necessity of each component in the proposed framework.

5.1 Experiment Setup Implementation. We set w1 in Eq. (20) to 0.1, and decay w2 in Eq. (21) from 0.1 to 0.01 over training. Additionally,

14

H. Wang et al.

Hallo4D

Baselines

Consistent4D

Hallo4D

Baselines

DreamGaussian4D

Hallo4D

Baselines

4D-FY

Fig. 9: Qualitative comparison in 4D generation of Hallo4D and baseline models. Among them, Consistent4D and DreamGaussian4D are image-to-4D methods, while 4D-FY is a text-to-4D generation approach. due to the presence of LCG-init , we apply a decaying weight to Rτ in Eq. (23), reducing it from 1 to 0.0001. To select the focal view in Fig. 5, we use the camera’s vertical field of view (Fovy), choosing the first view where Fovy exceeds 120% of the baseline default. For LMM reasoning, we employ a streamlined setup using a single interaction for efficiency—merging the two-stage dialogue in Fig. 4 into one. To align with the general negative prompts used in existing baselines (Yi et al., 2023; Wang et al., 2022; Poole et al., 2022; Tang et al., 2023a; Lin et al., 2023; Liu et al., 2023b), we adopt: “unnatural colors, poor lighting, low quality, artifacts, smooth texture.” For the MLLM Consensus Selector, we generate four candidate images per pass and employ an ensemble of four LMMs to score them. For exposure adjustment, we set the LSE temperature to µ = 0.06. The prompts in PΛ are tailored by LMM prior to optimization. We compute luminance in the linear RGB domain as Y = 0.2126R + 0.7152G + 0.0722B and use ϑ = 0.01 and σ0 = 0.9, chosen empirically. We use LLaVA-OneVision72B (Li et al., 2024a) as the LMM backbone for this task. All experiments are conducted on NVIDIA A100 GPU.

Baselines. We evaluated our method against several established baselines, demonstrating strong performance across diverse frameworks. These include text-to-3D models like GaussianDreamer (Yi et al., 2023), Score Jacobian Chain (SJC) (Wang et al., 2022), DreamFusion-IF (Poole et al., 2022), Magic3D (Lin et al., 2023), and ProlificDreamer (Wang et al., 2023), as well as image-to-3D models such as DreamGaussian (Tang et al., 2023a), Zero1-to-3 (Liu et al., 2023b), and Wonder3D (Long et al., 2023). We also included methods based on NeRF (Mildenhall et al., 2022) and 3DGS (Kerbl et al., 2023) for a comprehensive comparison. For 4D generation, we adopt the most representative and efficient models in the field—DreamGaussian4D (Ren et al., 2023), 4DFY (Bahmani et al., 2024b), and Consistent4D (Jiang et al., 2023b)—as our baselines, including both text-to-4D and image-to-4D. Identical parameter configurations and seed values were maintained for fair comparison, using default hyperparameters from the baselines’ open-source implementations. We employed the Threestudio library (Guo

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

15

Table 2: Quantitative comparisons in image-driven 3D generation. CLIP Scores ↑

Methods

User Study ↑

Objective Metrics

B/32

B/16

L/14

CD ↓

Vol. IoU ↑

PSNR ↑

SSIM ↑

LPIPS ↓

Cons.

Qual.

Align.

DreamGaussian

31.77

32.13

32.94

0.0185

0.5861

16.502

0.8543

0.2025

8.40

9.23

8.55

+Hallo4D

32.14

35.03

35.68

0.0185

0.6116

16.530

0.8801

0.1734

9.29

9.66

9.13

Zero-1-to-3

24.45

25.17

26.01

0.0370

0.4824

13.433

0.7210

0.3926

7.25

6.10

8.30

+Hallo4D

25.13

25.71

26.89

0.0287

0.5614

14.939

0.7531

0.3335

7.86

7.25

9.01

Wonder3D

27.31

28.08

28.92

0.2147

0.5218

12.258

0.7831

0.3823

8.69

8.21

8.76

+Hallo4D

28.04

28.58

29.74

0.1917

0.5946

14.312

0.8010

0.3019

9.09

9.40

9.26

Table 3: Quantitative comparisons in 4D generation.

Methods

CLIP Scores ↑

User Study ↑

B/32

B/16

L/14

Cons.

Qual.

Align.

Cons.4D

22.34

23.05

23.72

5.17

6.91

8.60

+Hallo4D

25.31

26.12

26.84

8.02

8.65

9.32

DG4D

28.62

29.03

29.87

6.16

6.25

8.02

+Hallo4D

32.14

33.02

33.71

8.82

8.94

9.49

4DFY

19.83

20.46

21.07

3.42

4.93

7.11

+Hallo4D

28.52

29.21

29.65

8.17

8.51

9.08

et al., 2023) for SJC and Magic3D, and the official codebases for the other methods. Metrics. In both 3D and 4D generation, the absence of ground-truth references has long hindered the development of widely accepted quantitative metrics for evaluating view consistency and spatiotemporal coherence. After reviewing a broad range of existing literature, we adopt CLIPScore (Radford et al., 2021) as our primary evaluation metric. Additionally, for image-to-3D tasks, we follow the experimental setup established in prior works (Long et al., 2023; Wu et al., 2024b), enabled by the availability of GSO (Downs et al., 2022) and Objaverse (Deitke et al., 2022) datasets. This includes evaluating geometric accuracy using Chamfer Distance (CD) and Volume IoU (Vol. IoU) between ground-truth and reconstructed shapes, and assessing visual quality using PSNR, SSIM (Zhou Wang et al., 2004), and LPIPS (Zhang et al., 2018).

5.2 Qualitative Comparison with Baselines We perform a comprehensive qualitative comparison between Hallo4D and a range of baseline methods across three major tasks: text-to-3D (Fig. 7), image-to-3D (Fig. 8), and 4D generation (Fig. 9). For each task, we directly compare the original outputs of baseline models with the cor-

responding outputs after applying our Hallo4D optimization. To ensure a fair evaluation, we do not modify the original configurations or generation procedures of the baselines. Across all scenarios, Hallo4D demonstrates substantial improvements in both geometric fidelity and consistency. In 3D tasks, it produces sharper textures, more stable structures, and notably mitigates typical Janus artifacts such as multi-head or extra-limb distortions. In 4D generation, Hallo4D achieves higher spatial-temporal coherence with fewer frame-to-frame inconsistencies or identity flickers. Notably, for text-driven 4D generation, Hallo4D shows clear advantages over 4D-FY, significantly reducing geometry collapse and temporal drift. These results highlight the robustness and generalizability of Hallo4D across diverse generative conditions and modalities.

5.3 Quantitative Comparison with Baselines Computational results. Following (Yi et al., 2023; Qiu et al., 2023a; Tang et al., 2023a), we generated 80 unique 3D prompts using ChatGPT (OpenAI, 2022) and arranged 16 cameras uniformly in a 360-degree configuration around the z-axis to evaluate the CLIP-Score. For 3D generation, the average CLIP-Score across all views was used to evaluate multi-view consistency, while for 4D generation, we computed the average CLIP-Score across all frames to assess spatiotemporal consistency. For image-conditioned models, we selected 60 objects from the GSO (Downs et al., 2022) and Objaverse (Deitke et al., 2022), replacing overly simplistic instances to ensure a more challenging and comprehensive evaluation. Each object was rendered from a frontal view at a resolution of 256×256 and used as input for computing CD, Vol. IoU, PSNR, SSIM (Zhou Wang et al., 2004), and LPIPS (Zhang et al., 2018) to assess both geometric accuracy and visual quality. It should be noted that the existence of a ground truth corresponding to the front view in image-driven generation generally leads to higher generation quality. The quantitative results for text-to-3D, imageto-3D, and 4D generation are presented in Tab. 1, Tab. 2,

16

H. Wang et al.

Hallo4D Hallo4D

w/o AAttn dark

w/o Init-4D

bright

w/o LMM-D

w/o Re-Cons.

w/o OF-Range

w/o &

Fig. 10: Ablation study of Hallo4D. We conduct a quantitative ablation study by successively removing each module from the full model, confirming the necessity and effectiveness of the modular design in achieving high-quality spatiotemporal generation. Table 4: Quantitative ablation study. Modules

CLIP Scores ↑

User Study ↑

B/32

B/16

L/14

Cons.

Qual.

Align.

3D Baseline

21.27

22.67

23.71

6.00

5.53

5.57

w/o AAttn

23.98

25.88

29.36

7.64

7.79

7.93

w/o LMM-D

23.65

25.10

28.71

7.25

7.32

7.53

w/o Re-Cons.

22.46

23.59

26.92

6.83

7.01

6.80

− w/o & PE

22.23

23.23

25.58

6.54

6.69

6.43

Hallo4D

24.45

26.83

30.00

8.87

8.67

8.87

4D Baseline

28.62

29.03

29.87

6.16

6.25

8.02

w/o Init-4D

30.41

30.92

31.40

7.04

7.33

8.52

w/o OF-Range

29.28

29.82

30.36

6.91

7.21

8.40

w/o CESA & LDR

31.88

32.65

33.34

8.50

8.65

9.21

w/o Consensus

31.95

32.71

33.39

8.58

8.70

9.29

Hallo4D

32.14

33.02

33.71

8.82

8.94

9.49

and Tab. 3, respectively. The results show that Hallo4D consistently outperforms all baselines across all evaluation metrics, demonstrating its effectiveness in enhancing spatiotemporal consistency for both 3D and 4D generation. User study. We conducted a user study with 58 volunteers specializing in artificial intelligence to evaluate the quality of generated 3D and 4D content. To facilitate comprehensive comparison, participants were asked to rate each model on three criteria—“Multi-view Consistency,” “Overall Quality,” and “Alignment with Prompt”—using a 10point scale. For 3D generation, each asset was rendered at 15° intervals and compiled into a video to visualize view consistency. For 4D generation, we incorporated continuous camera rotation during playback, allowing simultaneous observation of spatial and temporal changes. This design of-

fers an intuitive and immersive way to assess spatiotemporal consistency. Final scores were obtained by averaging user ratings, as shown in the last columns of Tab. 1, Tab. 2, and Tab. 3. The results demonstrate that Hallo4D achieves significant improvements over baseline models, further validating the effectiveness of our approach.

5.4 Ablation Study We conducted ablation experiments on the individual modules of Hallo4D, as shown in Fig. 10. Starting from the full model, we removed each module independently to evaluate its impact. For clarity, we denote AAttn as Multi-view Appearance Alignment, LMM-D as Multi-modal Hallucination Detection, Re-Cons. as Prompt-Enhanced Re-Consistency and Init-4D as Initialization for 4D Optimization. Notably, for 3D generation under the “w/o Re-Cons.” setting, the LMM output PE− is still applied in LSDS to demonstrate the necessity of LCG . We further include a setting “w/o & PE− ”, where PE− is not used at all, to more clearly isolate and verify the effectiveness of the Re-Cons. module. In Fig. 10, we evaluate the contribution of each module. Removing AAttn leads to noticeable brightness imbalance and unnatural color shifts, which affect texture fidelity. Modules LMM-D and Re-Cons. are critical for ensuring cross-view consistency: without LMM-D, the lion’s head is visibly deformed or missing across views; without Re-Cons., duplicated facial features and extra limbs appear. When both Re-Cons. and PE− are removed (“w/o & PE− ”), these artifacts become more pronounced, reinforcing the necessity of LMM-guided correction. The “w/o Init-4D” setting results in significant degradation of both spatial and temporal consistency in 4D generation, highlighting the crucial role of initialization in improving early-stage optimiza-

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

tion. Additionally, removing module OF-Range weakens the impact of motion cues from the animated video, leading to reduced temporal alignment. Furthermore, we conduct a comprehensive ablation study to validate the effectiveness of the proposed modules. Specifically, CESA & LDR denotes the Adaptive ExposureAware Semantic Alignment, and Consensus stands for the LMM Consensus Selector. Note that we adopt an incremental ablation setup; thus, the baseline configuration excluding CESA & LDR inherently omits the Consensus module as well. The ablation results demonstrate that the sequential integration of each component steadily improves the model’s performance in 4D generation. These modules are critical for enhancing the overall generation quality, as they offer targeted strategies to resolve distinct technical challenges. To complement these qualitative findings, Tab. 4 presents quantitative results that further confirm the necessity and effectiveness of each module in the Hallo4D framework.

6 Conclusion In this work, we propose Hallo4D, a unified and modelagnostic framework for mitigating spatiotemporal hallucinations in both 3D and 4D content generation. By leveraging the spatial and temporal reasoning capabilities of LMMs, our generation-detection-correction paradigm identifies inconsistencies across views and frames and guides imagespace optimization accordingly. Crucially, to ensure robust geometric fidelity and prevent compounding errors inherent in single-pass edits, these insights guide a consensusdriven image-space optimization mechanism. This approach utilizes an MLLM Consensus Selector to evaluate multiple candidate corrections via multi-model voting. To further enhance 4D consistency, Hallo4D incorporates an optical flowbased keyframe sampling strategy, an LMM-guided initialization scheme, and an attention-based appearance alignment module. To address exposure instability, we introduce two complementary losses—Contrastive Semantic Exposure Alignment (CSEA) and a log-dynamic-range (LDR) term—and apply union-of-frusta visibility pruning to suppress out-of-view clutter and reduce pseudo under-exposure. Without requiring retraining or architectural changes, our method improves generation quality across spatial and temporal dimensions. Extensive experiments confirm that Hallo4D consistently outperforms strong baselines, offering a scalable and effective solution for consistency-aware 3D and 4D generation.

References AlBahar B, Saito S, Tseng HY, Kim C, Kopf J, Huang JB (2023) Single-Image 3D Human Digitization with Shape-

17

Guided Diffusion. In: SIGGRAPH Armandpour M, Sadeghian A, Zheng H, Sadeghian A, Zhou M (2023) Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond. arXiv:230404968 2304.04968 Attaiki S, Guerrero P, Ceylan D, Mitra NJ, Ovsjanikov M (2024) GANFusion: Feed-forward text-to-3D with diffusion in GAN space. In: WACV, 2412.16717 Azuma D, Miyanishi T, Kurita S, Kawanabe M (2022) ScanQA: 3D Question Answering for Spatial Scene Understanding. In: CVPR Bahmani S, Liu X, Yifan W, Skorokhodov I, Rong V, Liu Z, Liu X, Park JJ, Tulyakov S, Wetzstein G, Tagliasacchi A, Lindell DB (2024a) TC4D: Trajectory-conditioned textto-4D generation. In: ECCV, 2403.17920 Bahmani S, Skorokhodov I, Rong V, Wetzstein G, Guibas L, Wonka P, Tulyakov S, Park JJ, Tagliasacchi A, Lindell DB (2024b) 4D-fy: Text-to-4D Generation Using Hybrid Score Distillation Sampling. In: CVPR Barron JT, Mildenhall B, Verbin D, Srinivasan PP, Hedman P (2022) Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In: CVPR Blattmann A, Dockhorn T, Kulal S, Mendelevitch D, Kilian M, Lorenz D, Levi Y, English Z, Voleti V, Letts A, Jampani V, Rombach R (2023) Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv:231115127 2311.15127 Brown TB, Mann B, Ryder N, Subbiah M, Kaplan J, Dhariwal P, Neelakantan A, Shyam P, Sastry G, Askell A, Agarwal S, Herbert-Voss A, Krueger G, Henighan T, Child R, Ramesh A, Ziegler DM, Wu J, Winter C, Hesse C, Chen M, Sigler E, Litwin M, Gray S, Chess B, Clark J, Berner C, McCandlish S, Radford A, Sutskever I, Amodei D (2020) Language Models are Few-Shot Learners. In: NeurIPS Caesar H, Bankiti V, Lang AH, Vora S, Liong VE, Xu Q, Krishnan A, Pan Y, Baldan G, Beijbom O (2020) nuScenes: A multimodal dataset for autonomous driving. In: CVPR Cao A, Johnson J (2023) HexPlane: A Fast Representation for Dynamic Scenes. In: CVPR Cao M, Wang X, Qi Z, Shan Y, Qie X, Zheng Y (2023) MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing. In: ICCV, 2304.08465 Chen J, Wu Y, Tan J, Ma H, Furukawa Y (2024) MapTracker: Tracking with strided memory fusion for consistent vector HD mapping. In: ECCV, 2403.15951 Chen Z, Wang F, Liu H (2023) Text-to-3D using Gaussian Splatting. In: CVPR Chen Z, Liu T, Zhuo L, Ren J, Tao Z, Zhu H, Hong F, Pan L, Liu Z (2025) 4DNeX: Feed-Forward 4D Generative Modeling Made Easy. arXiv:250813154 2508.13154

18

Chen Z, Min Y, Zhang J, Yan B, Wang J, Wang X, Shan S (2026) A Survey of Multimodal Hallucination Evaluation and Detection. IJCV 2507.19024 Chu WH, Ke L, Fragkiadaki K (2024) DreamScene4D: Dynamic multi-object scene generation from monocular videos. In: NeurIPS, 2405.02280 Dai A, Chang AX, Savva M, Halber M, Funkhouser T, Nießner M (2017) ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In: CVPR Deitke M, Schwenk D, Salvador J, Weihs L, Michel O, VanderBilt E, Schmidt L, Ehsani K, Kembhavi A, Farhadi A (2022) Objaverse: A Universe of Annotated 3D Objects. In: CVPR, 2212.08051 Deng W, Luo W, Tan Y, Biloš M, Chen Y, Nevmyvaka Y, Chen RTQ (2024) Variational Schrödinger Diffusion Models. In: ICML, 2405.04795 Deng Y, Yang J, Xiang J, Tong X (2022) GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation. In: CVPR Di D, Yang J, Luo C, Xue Z, Chen W, Yang X, Gao Y (2025) Hyper-3DG: Text-to-3D Gaussian Generation via Hypergraph. IJCV 2403.09236 Downs L, Francis A, Koenig N, Kinman B, Hickman R, Reymann K, McHugh TB, Vanhoucke V (2022) Google Scanned Objects: A High-Quality Dataset of 3D Scanned Household Items. In: ICRA, 2204.11918 Du Y, Li S, Mordatch I (2020) Compositional visual generation with energy based models. In: NeurIPS, vol 33 Du Y, Zhang Y, Yu HX, Tenenbaum JB, Wu J (2021) Neural Radiance Flow for 4D View Synthesis and Video Processing. In: ICCV Fan C, Cheng J, Yang W, Li Z, Zhang W, Hu W, Zhang Y, Zeng P (2026) MOC-3D: Manifold-Order Consistency for Text-to-3D Generation. arXiv:260501743 2605. 01743 Fu Y, Liu S, Kulkarni A, Kautz J, Efros AA, Wang X (2023) COLMAP-Free 3D Gaussian Splatting. In: CVPR Gao G, Liu W, Chen A, Geiger A, Schölkopf B (2023) GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs. In: CVPR Grammatikopoulou M, Sanchez-Matilla R, Bragman F, Owen D, Culshaw L, Kerr K, Stoyanov D, Luengo I (2023) A spatio-temporal network for video semantic segmentation in surgical videos. IJCARS 2306.11052 Guo YC, Liu YT, Shao R, Laforte C, Voleti V, Luo G, Chen CH, Zou ZX, Wang C, Cao YP, Zhang SH (2023) Threestudio. https://github.com/ threestudio-project/threestudio, opensource project Guo Z, Zhang R, Li H, Zhang M, Chen X, Wang S, Feng Y, Pei P, Heng PA (2026) Thinking-while-Generating: Interleaving Textual Reasoning throughout Visual Generation. In: CVPR, 2511.16671

H. Wang et al.

Hertz A, Mokady R, Tenenbaum J, Aberman K, Pritch Y, Cohen-Or D (2022) Prompt-to-Prompt Image Editing with Cross Attention Control. In: ICLR Ho J, Salimans T (2021) Classifier-Free Diffusion Guidance. In: NeurIPS Ho J, Jain A, Abbeel P (2020) Denoising Diffusion Probabilistic Models. In: NeurIPS Hong S, Ahn D, Kim S (2023) Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation. In: NeurIPS Hosseinzadeh M, Wang Y (2021) Image Change Captioning by Learning from an Auxiliary Task. In: CVPR Huang T, Zhang H, Zeng Y, Zhang Z, Li H, Zuo W, Lau RWH (2024) DreamPhysics: Learning Physics-Based 3D Dynamics with Video Diffusion Priors. In: AAAI Huang Z, He Y, Yu J, Zhang F, Si C, Jiang Y, Zhang Y, Wu T, Jin Q, Chanpaisit N, Wang Y, Chen X, Wang L, Lin D, Qiao Y, Liu Z (2023) VBench: Comprehensive Benchmark Suite for Video Generative Models. In: CVPR, 2311.17982 Jhamtani H, Berg-Kirkpatrick T (2018) Learning to Describe Differences Between Pairs of Similar Images. In: EMNLP Ji C, Ye W, Chen Z, Gao J, Huang X, Wang X, Zhang G, Zhang S, He T, Ouyang W, Zhao C (2026) DiffPano++: Scalable and Consistent Multi-View Panorama Generation with Spherical Epipolar-Aware Diffusion. IJCV Jianbo Shi, Tomasi (1994) Good features to track. In: CVPR Jiang Y, Tang H, Chang JHR, Song L, Wang Z, Cao L (2023a) Efficient-3DiM: Learning a Generalizable Single-image Novel-view Synthesizer in One Day. In: ICLR Jiang Y, Zhang L, Gao J, Hu W, Yao Y (2023b) Consistent4D: Consistent 360◦ dynamic object generation from monocular video. In: ICLR, 2311.02848 Jiao S, Lin Y, Zhong Y, She Q, Zhou W, Lan X, Huang Z, Yu F, Yu Y, Zhao Y, Zhao Y, Wei Y (2026) ThinkGen: Generalized Thinking for Visual Generation. In: CVPR, 2512.23568 Jun H, Nichol A (2023) Shap-E: Generating Conditional 3D Implicit Functions. arXiv:230502463 2305.02463 Kerbl B, Kopanas G, Leimkuehler T, Drettakis G (2023) 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans Graph 42(4) Kingma DP, Salimans T, Poole B, Ho J (2021) Variational Diffusion Models. In: ICML Laga H, Padilla M, Jermyn IH, Kurtek S, Bennamoun M, Srivastava A (2021) 4D Atlas: Statistical Analysis of the Spatiotemporal Variability in Longitudinal 3D Shape Data. IEEE Trans Pattern Anal Mach Intell 2101. 09403 Lai WS, Huang JB, Wang O, Shechtman E, Yumer E, Yang MH (2018) Learning Blind Video Temporal Consistency.

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

In: ECCV, 1808.00449 Li B, Zhang Y, Guo D, Zhang R, Li F, Zhang H, Zhang K, Zhang P, Li Y, Liu Z, Li C (2024a) LLaVA-OneVision: Easy Visual Task Transfer. arXiv:240803326 2408. 03326 Li H, Ma L, Shi H, Hao Y, Liao Y, Cheng L, Zhou P (2024b) 3D-GOI: 3D GAN omni-inversion for multifaceted and multi-object editing. In: ECCV, 2311.12050 Li H, Shi H, Zhang W, Wu W, Liao Y, Wang L, Lee Lh, Zhou P (2024c) DreamScene: 3D gaussian-based text-to3D scene generation via formation pattern sampling. In: ECCV, 2404.03575 Li M, Zhou P, Liu JW, Keppo J, Lin M, Yan S, Xu X (2023) Instant3D: Instant Text-to-3D Generation. IJCV Li R, Pan P, Yang B, Xu D, Zhou S, Zhang X, Li Z, Kadambi A, Wang Z, Tu Z, Fan Z (2024d) 4K4DGen: Panoramic 4D generation at 4K resolution. arXiv:240613527 2406. 13527 Li T, Slavcheva M, Zollhoefer M, Green S, Lassner C, Kim C, Schmidt T, Lovegrove S, Goesele M, Newcombe R, Lv Z (2022) Neural 3D Video Synthesis from Multi-view Video. In: CVPR, 2103.02597 Li X, Ren Y, Jin X, Lan C, Wang X, Zeng W, Wang X, Chen Z (2025) Diffusion Models for Image Restoration and Enhancement: A Comprehensive Survey. IJCV 2308.09388 Li Z, Chen Y, Liu P (2024e) DreamMesh4D: Video-to-4D generation with sparse-controlled gaussian-mesh hybrid representation. In: NeurIPS Liang F, Wu B, Wang J, Yu L, Li K, Zhao Y, Misra I, Huang JB, Zhang P, Vajda P, Marculescu D (2024a) FlowVid: Taming Imperfect Optical Flows for Consistent Video-toVideo Synthesis. In: CVPR Liang H, Yin Y, Xu D, Liang H, Wang Z, Plataniotis KN, Zhao Y, Wei Y (2024b) Diffusion4D: Fast spatialtemporal consistent 4D generation via video diffusion models. In: NeurIPS, 2405.16645 Liang Y, Yang X, Lin J, Li H, Xu X, Chen Y (2023) LucidDreamer: Towards high-fidelity text-to-3D generation via interval score matching. In: CVPR Lin CH, Gao J, Tang L, Takikawa T, Zeng X, Huang X, Kreis K, Fidler S, Liu MY, Lin TY (2023) Magic3D: High-Resolution Text-to-3D Content Creation. In: CVPR Lin Y, Dai Z, Zhu S, Yao Y (2024) Gaussian-Flow: 4D Reconstruction with Dynamic 3D Gaussian Particle. CVPR Ling H, Kim SW, Torralba A, Fidler S, Kreis K (2024) Align your gaussians: Text-to-4D with dynamic 3D gaussians and composed diffusion models. In: CVPR, 2312. 13763 Liu F, Wu D, Wei Y, Rao Y, Duan Y (2023a) Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior. In: CVPR

19

Liu J, Huang H, Cao J, He R (2024a) ZePo: Zero-Shot Portrait Stylization with Faster Sampling. In: ACM MM, 2408.05492 Liu R, Wu R, Hoorick BV, Tokmakov P, Zakharov S, Vondrick C (2023b) Zero-1-to-3: Zero-shot One Image to 3D Object. In: ICCV Liu T, Huang Z, Chen Z, Wang G, Hu S, Shen L, Sun H, Cao Z, Li W, Liu Z (2025) Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency. In: ICCV, 2503.20785 Liu X, Zhou C, Huang S (2024b) 3DGS-enhancer: Enhancing unbounded 3D gaussian splatting with viewconsistent 2D diffusion priors. In: NeurIPS Liu X, Liu J, Wang H, He R, Huang H (2026) ThinkThen-Generate: Structural Chain-of-Thought Reasoning for Consistent 3D Generation. In: CVPR Liu Y, Lin C, Zeng Z, Long X, Liu L, Komura T, Wang W (2024c) SyncDreamer: Generating Multiview-consistent Images from a Single-view Image. In: ICLR Liu Z, Feng Y, Xiu Y, Liu W, Paull L, Black MJ, Schölkopf B (2023c) Ghost on the Shell: An Expressive Representation of General 3D Shapes. In: ICLR Long X, Guo YC, Lin C, Liu Y, Dou Z, Liu L, Ma Y, Zhang SH, Habermann M, Theobalt C, Wang W (2023) Wonder3D: Single image to 3D using cross-domain diffusion. In: CVPR, 2310.15008 Lorraine J, Xie K, Zeng X, Lin CH, Takikawa T, Sharp N, Lin TY, Liu MY, Fidler S, Lucas J (2023) ATT3D: Amortized Text-to-3D Object Synthesis. In: ICCV Lucas BD, Kanade T (1981) An Iterative Image Registration Technique with an Application to Stereo Vision. In: IJCAI, vol 2 Miao Q, Li K, Quan J, Min Z, Ma S, Xu Y, Yang Y, Liu P, Luo Y (2025) Advances in 4D Generation: A Survey. arXiv:250314501 2503.14501 Mildenhall B, Srinivasan PP, Tancik M, Barron JT, Ramamoorthi R, Ng R (2022) NeRF: Representing scenes as neural radiance fields for view synthesis. Commun ACM Mokady R, Hertz A, Aberman K, Pritch Y, Cohen-Or D (2023) Null-text Inversion for Editing Real Images using Guided Diffusion Models. In: CVPR Nichol A, Jun H, Dhariwal P, Mishkin P, Chen M (2022) Point-E: A System for Generating 3D Point Clouds from Complex Prompts. arXiv:221208751 2212.08751 OpenAI (2022) ChatGPT Ouyang H, Wang Q, Xiao Y, Bai Q, Zhang J, Zheng K, Zhou X, Chen Q, Chen Q (2024) CoDeF: Content Deformation Fields for Temporally Consistent Video Processing. In: CVPR Pan Z, Yang Z, Zhu X, Zhang L (2025) Efficient4D: Fast Dynamic 3D Object Generation from a Single-view Video. IJCV 2401.08742

20

Poole B, Jain A, Barron JT, Mildenhall B (2022) DreamFusion: Text-to-3D using 2D Diffusion. In: ICLR Pumarola A, Corona E, Pons-Moll G, Moreno-Noguer F (2020) D-NeRF: Neural Radiance Fields for Dynamic Scenes. In: CVPR, 2011.13961 Qian G, Mai J, Hamdi A, Ren J, Siarohin A, Li B, Lee HY, Skorokhodov I, Wonka P, Tulyakov S, Ghanem B (2023) Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors. In: ICLR Qiu L, Chen G, Gu X, Zuo Q, Xu M, Wu Y, Yuan W, Dong Z, Bo L, Han X (2023a) RichDreamer: A Generalizable Normal-Depth Diffusion Model for Detail Richness in Text-to-3D. In: CVPR, 2311.16918 Qiu S, Anwar S, Barnes N (2023b) PnP-3D: A Plug-andPlay for 3D Point Clouds. IEEE Trans Pattern Anal Mach Intell 45(1) Radford A, Kim JW, Hallacy C, Ramesh A, Goh G, Agarwal S, Sastry G, Askell A, Mishkin P, Clark J, Krueger G, Sutskever I (2021) Learning Transferable Visual Models From Natural Language Supervision. In: ICML Raj A, Kaza S, Poole B, Niemeyer M, Ruiz N, Mildenhall B, Zada S, Aberman K, Rubinstein M, Barron J, Li Y, Jampani V (2023) DreamBooth3D: Subject-Driven Textto-3D Generation. In: ICCV Ren J, Pan L, Tang J, Zhang C, Cao A, Zeng G, Liu Z (2023) DreamGaussian4D: Generative 4D Gaussian Splatting. arXiv:231217142 abs/2312.17142, 2312.17142 Ronneberger O, Fischer P, Brox T (2015) U-Net: Convolutional Networks for Biomedical Image Segmentation. In: MICCAI, 1505.04597 Ruiz N, Li Y, Jampani V, Pritch Y, Rubinstein M, Aberman K (2023) DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In: CVPR Saharia C, Chan W, Saxena S, Li L, Whang J, Denton E, Ghasemipour SKS, Ayan BK, Mahdavi SS, GontijoLopes R, Salimans T, Ho J, Fleet DJ, Norouzi M (2022) Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding. In: NeurIPS Seo J, Jang W, Kwak MS, Ko J, Kim H, Kim J, Kim JH, Lee J, Kim S (2023) Let 2D Diffusion Model Know 3DConsistency for Robust Text-to-3D Generation. In: ICLR Shi Y, Wang P, Ye J, Long M, Li K, Yang X (2024) MVDream: Multi-view Diffusion for 3D Generation. In: ICLR Shridhar M, Thomason J, Gordon D, Bisk Y, Han W, Mottaghi R, Zettlemoyer L, Fox D (2020) ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks. In: CVPR Singer U, Sheynin S, Polyak A, Ashual O, Makarov I, Kokkinos F, Goyal N, Vedaldi A, Parikh D, Johnson J, Taigman Y (2023) Text-to-4D dynamic scene generation. In: ICML, 2301.11280

H. Wang et al.

Song J, Meng C, Ermon S (2022) Denoising Diffusion Implicit Models. In: ICLR Suhr A, Zhou S, Zhang A, Zhang I, Bai H, Artzi Y (2019) A Corpus for Reasoning About Natural Language Grounded in Photographs. In: ACL Sun J, Wang H, Cao J, Huang H, He R (2026a) Marmot: Object-Level Self-Correction via Multi-Agent Reasoning. Machine Intelligence Research 2504.20054 Sun J, Wang P, Wang H, Liu X, Huang H, He R (2026b) Towards Fine-Grained Attribution: Instance-Aware Preference Optimization for Aligning Diffusion Models. In: CVPR Sun W, Chen S, Liu F, Chen Z, Duan Y, Zhang J, Wang Y (2024) DimensionX: Create any 3D and 4D scenes from a single image with controllable video diffusion. arXiv:241104928 2411.04928 Takagi Y, Nishimoto S (2022) High-resolution image reconstruction with latent diffusion models from human brain activity. In: CVPR Tang J, Ren J, Zhou H, Liu Z, Zeng G (2023a) DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation. In: ICLR Tang J, Wang T, Zhang B, Zhang T, Yi R, Ma L, Chen D (2023b) Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior. In: ICCV Tong H, Wang H, Liu J, Wang Q, Cao J, He R (2026) CoGrad3D: Spatially-Coupled Timestep Optimization with Orthogonal Gradient Fusion for 3D Generation. In: AAAI Voleti V, Yao CH, Boss M, Letts A, Pankratz D, Tochilkin D, Laforte C, Rombach R, Jampani V (2024) SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion. In: ECCV Wang H, Du X, Li J, Yeh RA, Shakhnarovich G (2022) Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation. In: CVPR Wang H, Cao J, Liu J, Zhou X, Huang H, He R (2024a) Hallo3D: Multi-modal hallucination detection and mitigation for consistent 3D content generation. In: NeurIPS Wang H, Huang H, Wang P, Hao J, Zhou C, He R (2026) Coloring the Noise: Adversarial Sobolev Alignment for Faithful Image Super Resolution. In: ICML, 2605. 23264 Wang J, Ma Y, Guo J, Xiao Y, Huang G, Li X (2024b) COVE: Unleashing the Diffusion Feature Correspondence for Consistent Video Editing. In: NeurIPS, 2406. 08850 Wang Y, Liu G, Wang X, Chen Z, Li J, Liang X, Sun F, Zhu J (2025) Video4DGen: Enhancing Video and 4D Generation through Mutual Optimization. IEEE Trans Pattern Anal Mach Intell Wang Z, Lu C, Wang Y, Bao F, Li C, Su H, Zhu J (2023) ProlificDreamer: High-Fidelity and Diverse Text-to-3D Gen-

Hallo4D: Multi-Modal Hallucination Mitigation for Consistent Spatio-Temporal Generation

eration with Variational Score Distillation. In: NeurIPS Wu G, Yi T, Fang J, Xie L, Zhang X, Wei W, Liu W, Tian Q, Wang X (2024a) 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. In: CVPR, 2310.08528 Wu K, Liu F, Cai Z, Yan R, Wang H, Hu Y, Duan Y, Ma K (2024b) Unique3D: High-quality and efficient 3D mesh generation from a single image. In: NeurIPS, 2405. 20343 Wu R, Gao R, Poole B, Trevithick A, Zheng C, Barron JT, Holynski A (2025) CAT4D: Create Anything in 4D with Multi-View Video Diffusion Models. In: CVPR, 2411. 18613 Wu Z, Zheng J, Ren X, Vasluianu FA, Ma C, Paudel DP, Van Gool L, Timofte R (2024c) Single-Model and AnyModality for Video Object Tracking. In: CVPR Xiang J, Yang J, Huang B, Tong X (2023) 3D-aware Image Generation using 2D Diffusion Models. In: ICCV Xiang J, Lv Z, Xu S, Deng Y, Wang R, Zhang B, Chen D, Tong X, Yang J (2025) Structured 3D Latents for Scalable and Versatile 3D Generation. In: CVPR, 2412.01506 Xie Y, Yao CH, Voleti V, Jiang H, Jampani V (2024) SV4D: Dynamic 3D content generation with multi-frame and multi-view consistency. arXiv:240717470 2407.17470 Xu D, Liang H, Bhatt NP, Hu H, Liang H, Plataniotis KN, Wang Z (2024a) Comp4D: LLM-guided compositional 4D scene generation. arXiv:240316993 2403.16993 Xu Z, Zhang J, Liew JH, Yan H, Liu JW, Zhang C, Feng J, Shou MZ (2024b) MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model. In: CVPR Xu Z, Zhang J, Liew JH, Yan H, Liu JW, Zhang C, Feng J, Shou MZ (2024c) MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model. In: CVPR Yang J, Cheng Z, Duan Y, Ji P, Li H (2023) ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion. In: CVPR Yang L, Liu C, Zhu Z, Liu A, Ma H, Nong J, Liang Y (2025) Not all frame features are equal: Video-to-4D generation via decoupling dynamic-static features. arXiv:250208377 2502.08377 Yang S, Zhou Y, Liu Z, Loy CC (2024a) Fresco: SpatialTemporal Correspondence for Zero-Shot Video Translation. In: CVPR Yang S, Zhou Y, Liu Z, Loy CC (2024b) Fresco: SpatialTemporal Correspondence for Zero-Shot Video Translation. In: CVPR Yi T, Fang J, Wang J, Wu G, Xie L, Zhang X, Liu W, Tian Q, Wang X (2023) GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models. In: CVPR Zeng Y, Jiang Y, Zhu S, Lu Y, Lin Y, Zhu H, Hu W, Cao X, Yao Y (2024) STAG4D: Spatial-temporal anchored gen-

21

erative 4D gaussians. In: ECCV, 2403.14939 Zhang L, Agrawala M (2024) Transparent Image Layer Diffusion using Latent Transparency. ACM Trans Graph Zhang R, Isola P, Efros AA, Shechtman E, Wang O (2018) The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. In: CVPR Zhang T, Yu HX, Wu R, Feng BY, Zheng C, Snavely N, Wu J, Freeman WT (2024) PhysDreamer: Physics-based interaction with 3D objects via video generation. In: ECCV, 2404.13026 Zhao M, Zhao C, Liang X, Li L, Zhao Z, Hu Z, Fan C, Yu X (2023) EfficientDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior. In: CVPR Zheng Y, Li X, Nagano K, Liu S, Hilliges O, De Mello S (2024) A Unified Approach for Text-and Image-Guided 4D Scene Generation. In: CVPR Zhou L, Shih A, Meng C, Ermon S (2023) DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling. In: CVPR Zhou S, Yang P, Wang J, Luo Y, Loy CC (2024a) UpscaleA-Video: Temporal-Consistent Diffusion Model for RealWorld Video Super-Resolution. In: CVPR Zhou X, Ran X, Xiong Y, He J, Lin Z, Wang Y, Sun D, Yang MH (2024b) GALA3D: Towards text-to-3D complex scene generation via layout-guided generative gaussian splatting. In: ICML, 2402.07207 Zhou Wang, Bovik A, Sheikh H, Simoncelli E (2004) Image quality assessment: From error visibility to structural similarity. Trans Image Processing 13(4)