PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions

PhyGenHOI: Physically-Aware 4D Generation of Dynamic Human-Object Interactions

Omer Benishu

Gal Fiebelman

Sagie Benaim

arXiv:2605.30268v1 [cs.CV] 28 May 2026

Hebrew University of Jerusalem

Abstract We address the task of generating physically accurate and visually faithful 4D Human-Object Interaction (HOI). Given a static 3D human and target object represented as 3D Gaussian Splats (3DGS), our goal is to synthesize dynamic scenes where the human actively engages with the object through actions, such as punching or kicking, in accordance with a given input text. To this end, we introduce PhyGenHOI, a novel framework that couples generative human motion with an explicit physical object simulation. We model the human as a semantic agent driven by a Motion Diffusion Model (MDM) and the object as a physical agent simulated via the Material Point Method (MPM), utilizing 3D Gaussians as a unified, differentiable representation. We supervise their interaction through three coupled mechanisms: (1) A Windowed Attraction Loss that temporally synchronizes generative motion to intercept the object; (2) A Contact-Driven Re-simulation step that triggers physically consistent momentum transfer upon impact; and (3) A Masked Video-SDS objective that injects video-based priors to enhance contact fidelity. Experiments show PhyGenHOI generates physically consistent 4D HOI across diverse actions, humans, and objects, outperforming baselines. Project page and videos: https://omerbenishu.github.io/PhyGenHOI/

1

Introduction

Synthesizing dynamic human-object interactions that are both visually faithful and physically plausible is a fundamental challenge in computer graphics, with critical applications in animation, gaming, and immersive virtual reality. To this end, we consider the task of generating physically accurate and visually faithful 4D Human-Object Interaction (HOI). Specifically, given a static 3D human and a static target object, both represented as 3D Gaussian Splats (3DGS) [1], our goal is to synthesize a dynamic 4D scene where the human actively engages with a dynamic object, such as kicking a soccer ball or pushing a file cabinet, in accordance with an input text. We aim to produce human and object motion that is both visually faithful and physically plausible, capturing the causal interplay of forces and collisions. By leveraging the explicit 3D Gaussians, we ensure that the resulting 4D content not only respects the laws of physics but also supports efficient rendering from novel viewpoints. Despite the rapid evolution of text-to-4D generation approaches [2–4], a critical dichotomy persists between semantic coherence and physical fidelity. On one hand, purely generative approaches such as 4DFY [2] distill motion directly from large-scale video priors. While these methods excel at synthesizing diverse open-world scenarios, they fundamentally lack an underlying model of physics, frequently producing causal anomalies like “ghosting” artifacts where objects react before contact. On the other hand, kinematic frameworks like AvatarGO [5] and InterDreamer [6] introduce structured human priors (e.g., SMPL [7]) to ensure anatomical consistency. However, these methods typically reduce interaction to a geometric constraint, treating the target object as a “static prop” or a rigid Preprint.

“A man hits the ball overhead in a volleyball set.” “A in aa volleyball volleyballset.” set.” “Aman manhits hits the the ball ball overhead overhead in

“A man punches the ball with his left hand.” “Aman man punches punches the the ball “A ball with with his his left lefthand.” hand.”

“A man pushes a file cabinet.” “A man pushes a file cabinet.” “A man pushes a file cabinet.”

Figure 1: PhyGenHOI generates physically plausible 4D human-object interactions. Given static 3D Gaussian Splats of a human and a target object, our framework synthesizes a dynamic scene by coupling a generative “semantic agent” (human) with a simulated “physical agent” (object) aligned with a text prompt. We demonstrate here a single view across different timesteps for the actions overhead pass, punch, and push (top to bottom).

accessory, failing to capture dynamic forces like ballistic momentum transfer. Similarly, recent 3D asset animation methods [8, 9] animate individual entities but lack the coupled interaction logic required for human-object contact. To bridge this gap, we introduce PhyGenHOI, generating 4D human-object interactions that are both semantically responsive and physically grounded. We devise a unified framework where 3D Gaussian Splatting serves as the common substrate for coupling semantic generation with physical simulation. To ensure kinematic fidelity, we model the human as an active agent driven by an SMPL-constrained Motion Diffusion Model (MDM), which provides a robust semantic prior for generating diverse, text-aligned actions. Conversely, we treat the object as a reactive physical agent by mapping its Gaussian kernels directly to particles in a differentiable Material Point Method (MPM) simulator, enforcing physically consistent object trajectories and deformations. To coordinate these distinct agents into a cohesive interaction, we leverage three targeted mechanisms. First, to synchronize the human’s semantic intent with the object’s position, we propose a Windowed Attraction Loss that spatially and temporally guides the generative motion to intercept the target. Second, to ensure physical causality, we implement Contact Detection and MPM Re-simulation; upon detecting collision, the object’s trajectory is explicitly updated to reflect realistic momentum transfer and material deformation. Finally, we apply a Temporally-Masked Video-SDS that injects rich visual priors specifically around the contact frames, enhancing interaction fidelity without disrupting the physically grounded motion. Our framework targets actions involving discrete momentum transfer upon contact, such as kicking, punching, and pushing. We validate our framework against state-of-the-art generative (4DFY [2]) and animation (AnimateAnyMesh [8]) baselines across a suite of dynamic interaction scenarios. Our method eliminates the ghosting and interpenetration artifacts of purely generative models while producing dynamic object responses that animation methods cannot capture, achieving superior performance in text alignment, physical plausibility, contact quality, and visual fidelity. 2

Figure 2: Overview of PhyGenHOI. (a) Scene Representation + Agent Motion Synthesis (Sec. 3.1+3.2): Given a 3DGS human and 3DGS object, we treat the human as a semantic agent and synthesize motion via Human Motion Score Distillation (HMSD) (LHMSD ) from a pretrained motion diffusion model, producing natural text-aligned motion. The object is treated as a physical agent, with its trajectory computed via MPM simulation. At this stage, both agents move independently. (b) Physically-Aware Interaction Synthesis (Sec. 3.3): While continuing to optimize LHMSD from (A), we coordinate the agents through a Windowed Attraction Loss (Lattr ) that guides the human toward the object. Upon Contact Detection, we trigger object re-simulation with physically consistent momentum transfer. Finally, we render the composed 4D HOI scene and apply Video-SDS (LV-SDS ) to enhance contact fidelity.

2

Related Work

Text-to-4D Generation. Early text-to-4D methods primarily extended 2D diffusion priors to 3D representations via Score Distillation Sampling (SDS). DreamFusion [10] established the baseline using 2D priors, while subsequent approaches like DreamGaussian [11] and GaussianDreamer [12] adopted 3D Gaussian Splatting (3DGS) for efficiency. To handle temporal consistency, 4D-fy [2] and Consistent4D [13] introduced temporal attention, while recent work like CHORD [14] extends these priors to multi-object choreography. Regardless, these methods rely purely on visual priors, resulting in inconsistent motions that ignore collisions. Human–Object Interaction (HOI) Generation. OMOMO [15] generates motion from object trajectories, paving the way for text-driven works like AvatarGO [5] and InterDreamer [6], which utilize contact retargeting and 2D priors. To improve synchrony, SyncDiff [16] and HOIDiNi [17] explicitly optimize geometric alignment. However, these purely kinematic methods lack physical modeling (mass, elasticity), treating objects as rigid props and failing to capture realistic deformations or prevent interpenetration. Generative 3D Animation. Recent approaches focus on animating static 3D assets. To this end, Animate3D [18] and AKD [19] utilize video diffusion models. AnimateAnyMesh [8] performs feed-forward 3D asset animation while Animus3D [9] introduces “Motion Score Distillation”. However, these methods operate on individual entities in non-physical environments. They fail to model the coupled physics of human-object interaction, frequently leading to scenes where contact is physically implausible or entirely absent. Physics-Based MPM & Gaussian Splatting. Existing works [20–22] combine MPM with 3DGS to optimize physical properties but are restricted to single-object dynamics. In contrast, we apply this “Neuro-Physical” approach to a coupled system, utilizing simulation to enforce causal interaction between an articulated human and a deformable object.

3

Method

Given a static 3D human and object represented as 3D Gaussian Splats (3DGS), along with a text prompt describing the desired human motion and a prompt describing the scene interaction, our goal is to synthesize a dynamic 4D scene where the human actively engages with the object in a physically plausible manner. As illustrated in Fig. 2, our framework couples generative human motion with explicit physical simulation under a unified 3DGS representation (Sec. 3.1). We synthesize motion independently for each agent (Sec. 3.2), then coordinate them through attraction-based guidance, contact-driven re-simulation, and video prior distillation (Sec. 3.3). Implementation details are in the appendix and code will be made fully available. 3

3.1

Scene Representation

We adopt 3D Gaussian Splatting [1] as a shared representation for both agents, enabling joint rendering and optimization in a unified differentiable pipeline. 3D Gaussian Splatting. 3DGS represents scenes using a set of anisotropic Gaussians. Each Gaussian Gi is defined by position xi , covariance Σi , opacity σi , and spherical harmonics ci for view-dependent appearance. The color C of a pixel is computed by alpha-blending these 3D Gaussians when projected PN Qi−1 to the image plane: C = i=1 Ti αi Ci , with Ti = j=1 (1 − αj ), where N is the set of depth-sorted Gaussian kernels affecting the pixel, and Ci and αi represents the color and density of this point computed by a 3D Gaussian G with covariance Σ and opacity σ. Human as Semantic Agent. We represent the human using 3D Gaussians bound to the SMPL parametric body model [7], following HUGS [23]. Each Gaussian is defined in an initial pose and deformed via Linear Blend Skinning (LBS). Given pose parameters θ andjoint transformations P K K ′ {Gk }k=1 , the position µi of Gaussian i transforms as µi = k=1 wi,k Gk µi , where wi,k are skinning weights associating Gaussian i with joint k, allowing direct optimization of pose parameters. Object as Physical Agent. The object must respond to physical forces rather than learned priors.We treat its Gaussians as particles in a Material Point Method (MPM) simulation [24, 25], following PhysGaussian [20], evolving positions xi (t) according to continuum mechanics. Unlike the human, the object’s motion is determined entirely by simulation, ensuring physical plausibility. 3.2

Agent Motion Synthesis

Having established the scene representation, we now synthesize motion for each agent, the physical agent via physical simulation, and the semantic agent via learned motion priors. Object Motion Simulation. The object’s initial trajectory is computed via forward MPM simulation from t = 0 to T , producing a physically consistent free-motion trajectory. This trajectory is updated once contact with the human is established (Sec. 3.3). Human Motion Score Distillation. We parameterize human motion as a sequence X = {xt }Tt=0 , where each frame xt = (rt , ω t , θ t ) ∈ RD consists of root translation rt ∈ R3 , global orientation ω t ∈ R6 in 6D rotation, and per-joint pose parameters θ t ∈ RJ×3 for J joints. Given a pretrained Human Motion Diffusion Model (MDM) [26] and a text prompt pmotion describing the desired human motion, we define Human Motion Score Distillation (HMSD): h  i ∇X LHMSD = Et,ϵ wHM SD (t) X̂0 (Xt , t, pmotion ) − X , (1) where Xt is the motion X corrupted with Gaussian noise ϵ ∼ N (0, I) at diffusion timestep t, X̂0 is the MDM’s prediction of the clean motion conditioned on Xt and the text prompt pmotion , and wHM SD (t) is a timestep-dependent weighting function. This objective pulls the optimized motion toward the manifold of natural human movements described by the text prompt. We optimize the human pose parameters using LHMSD alone for Ninit iterations, producing natural human motion. However, at this stage, the motion is generated independently of the object’s position and may not result in contact. 3.3

Physically-Aware Interaction Synthesis

Given both agents’ initial motions, the central challenge becomes coordinating them into a coherent interaction. We address this through three coupled mechanisms: (1) a windowed attraction loss for human-object coordination, (2) contact-driven re-simulation for physical response, and (3) distilling video priors for contact fidelity. Windowed Attraction Loss. To coordinate the generated motion with the object, we introduce a mechanism that identifies when and where contact should occur, then guides the relevant body part toward the object. This requires determining two quantities: the contact joint j ∗ , i.e., which body part will make contact, and the contact frame t∗ , i.e., when impact should occur. We estimate both by 4

analyzing the velocity profile of the initial motion. Intuitively, the joint most involved in the action exhibits the highest cumulative motion throughout the sequence, e.g. for a kick, this is the foot and for a punch, the hand. Contact should occur at the moment of peak velocity, as this is when the striking limb is maximally extended toward the target, transitioning from the acceleration phase to deceleration or follow-through. We demonstrate this intuition in Fig. 3, where for a kicking motion, the foot joint exhibits both the highest cumulative velocity and a clear peak at the natural contact moment. We first identify the contact joint by selecting the joint with highest cumulative velocity across all frames, then determine the contact frame as the moment of peak velocity for that joint: Figure 3: Contact Joint and Frame Selection. T −1 X j ∗ = argmax vj (t), t∗ = argmax vj ∗ (t), Per-joint velocity profiles for a kicking motion. t j Each curve represents a different SMPL joint, with t=0 (2) the left foot (⋆) and right knee (■) highlighted. where pj (t) is the world-space position of joint The left foot exhibits the highest cumulative vej at frame t obtained from SMPL forward kine- locity and is automatically selected as the contact matics, and vj (t) = ∥pj (t + 1) − pj (t)∥ is its joint j ∗ , with the contact frame t∗ identified at its per-frame velocity. We then apply a Gaussian- peak. In contrast, the right knee (blue) maintains weighted attraction loss that pulls the contact low velocity throughout the sequence, illustrating joint j ∗ toward the object, with guidance con- why it is not selected. We visualize human poses at centrated around the contact frame t∗ while al- two frames where peak motion occurs, illustrating the motion progression. lowing natural motion elsewhere: P   2 g(t)∥pj ∗ (t) − cobj (t)∥ (t − t∗ )2 P Lattr = t , g(t) = exp − , 2σ 2 t g(t) (3) where pj ∗ (t) is the position of the contact joint at frame t, cobj (t) is the object’s center of mass, and g(t) is a Gaussian weighting function within a window, [t∗ − ∆t∗ , t∗ + ∆t∗ ], of the contact frame t∗ , with standard deviation σ. The Gaussian weighting concentrates guidance around the predicted contact moment while allowing the motion prior to govern the natural wind-up and follow-through phases without interference. We continue optimization for Nsync iterations with the objective Lhuman = λHMSD LHMSD + λattr Lattr . This couples the motion prior with scene awareness, yielding coordinated human-object motion. We optimize the underlying SMPL parameters θ throughout, not joint positions directly. Contact Detection and Re-simulation. While the attraction loss ensures the human motion is coordinated with the object, the object itself is not yet affected by this interaction and continues to follow its initial free-motion trajectory. To achieve physically plausible dynamics and contact, we detect the contact event and re-simulate the object’s response to the applied force. After Nsync iterations, we identify the contact frame and recompute the object trajectory accordingly, optimization then proceeds with this updated motion. To detect contact, we first assign each human Gaussian Gi to its dominant joint based on skinning weights, where Gi is associated with joint j if j = argmaxk wi,k . For each joint j, we compute its axis-aligned bounding box Bj (t) from the positions of its associated Gaussians at frame t, and similarly compute the object’s bounding box Bobj (t). We identify contact at frame tc with joint jc when: (1) Bjc (tc ) ∩ Bobj (tc ) ̸= ∅, and (2) at least τcontact fraction of joint jc ’s Gaussians lie within distance dcontact of the nearest object Gaussian. Once contact is detected, we compute the momentum transfer and update the object’s velocity. We estimate the human velocity Vhuman from the contact joint’s displacement. The contact normal n is defined as the direction from the mean position of contacting object Gaussians toward the object’s center of mass. The post-impact velocity is then: vin = (Vhuman − Vobj ) · n, Vpost = Vobj + (1 + e) vin n, (4) where e is the coefficient of restitution. We perform a single forward MPM simulation from tc to T with the post-impact velocity, producing a physically consistent trajectory that respects momentum 5

Stand Still Punch

Step Punch

High Punch Low Punch

Figure 4: In-Scene Variations. We demonstrate controllability by varying human/object movements. Top & Second Rows: Changing object position (High vs. Low) forces trajectory adaptation. Third & Bottom Rows: Altering intensity (Step vs. Stand Still) yields distinct impact velocities. transfer and material properties. This simulated trajectory is then held fixed, such that subsequent optimization adjusts only human pose parameters, ensuring the object’s response remains physically consistent. Additional details are provided in the appendix. Video-SDS for Contact Fidelity. The contact region may still exhibit artifacts due to the discrete nature of contact detection and the independent optimization of human and object. Since both agents share a 3DGS representation, we can render the composed scene and apply Video Score Distillation Sampling [2] to enhance contact fidelity. Utilizing the v-prediction formulation from [19], given rendered frames V = {I t }Tt=1 from sampled viewpoints, we encode them into latent space z = E(V ), where E is the pretrained VAE encoder, and define the diffusion loss as: h i 2 LDiff (z, pscene ) = Et,ϵ wSDS (t) ∥z − ẑ∥2 , (5) √ where ẑ = αt zt − vϕ (zt ; t, pscene ) is the reconstruction based on the predicted velocity vϕ from the pretrained video diffusion model, pscene is a text prompt describing the interaction, and wSDS (t) is a timestep-dependent weighting function. Omitting the gradient through the velocity-predicting transformer, we optimize human pose parameters θ via:   ∂z ∇θ LV-SDS = Et,ϵ wSDS (t) (z − ẑ) . (6) ∂θ We apply temporal masking, optimizing only frames within a window [tc − ∆t, tc + ∆t] around the contact frame, focusing optimization on contact frames while preserving the motion prior’s influence elsewhere. Additional Video-SDS details are in the appendix. Optimization. Our optimization proceeds in three stages: (1) Ninit iterations of LHMSD to establish natural motion, (2) Nsync iterations of Lhuman = λHMSD LHMSD + λattr Lattr to coordinate with the object, followed by contact detection and MPM re-simulation, and (3) temporally-masked Video-SDS around contact frames to enhance contact fidelity. Additional details are in the supplementary.

4

Experiments

We evaluate PhyGenHOI on diverse human-object interaction scenarios. We present qualitative results demonstrating the range of supported actions, humans, and objects in Sec. 4.1, compare against state-of-the-art baselines in Sec. 4.2, and provide ablation studies in Sec. 4.3. We discuss limitations in the appendix. 4.1

Interaction Generation

Fig. 1 demonstrates our method’s ability to generate physically plausible 4D human-object interactions across a variety of scenarios. We showcase multiple action types including punching, kicking, and pushing, paired with different objects such as basketballs, soccer balls, file cabinets, etc. For each scenario, our framework successfully coordinates the human motion with the object trajectory, 6

Animate AnyMesh 4DFY Ours Ours

4DFY

Animate AnyMesh

“A man hits the ball overhead in a volleyball set.”

“A man kicks the ball with his right leg.”

Figure 5: Baseline Comparison. We show a single view (see more views in appendix). While baselines exhibit missing contact (top) or ghosting artifacts (middle), our method (bottom) produces coherent interactions with causal momentum transfer and accurate physical response. producing realistic interactions where the object responds according to its material properties. Across all examples, our method eliminates the ghosting and interpenetration artifacts common in purely generative approaches, while capturing dynamic object responses that kinematic methods cannot achieve. To further demonstrate controllability and physical consistency, we show in-scene variations in Fig. 4, including different initial object velocities, positions, and contact intensities. These variations highlight that our framework produces coherent, physically plausible results across a range of initial conditions. Additional visualizations are provided in the supplementary material. 4.2

Quantitative and Qualitative Evaluation

We assemble a benchmark of 10 distinct human-object interaction scenarios spanning different humans, objects, and interactions. For each combination, we generate 4D interactions and evaluate physical plausibility, semantic alignment, and visual quality. Baselines. We compare against 4D-fy [2] and AnimateAnyMesh [8], representing the most relevant baselines with available implementations. 4D-fy lacks explicit physics, leading to ghosting artifacts, while AnimateAnyMesh lacks coordination, frequently missing contact. We note that directly relevant HOI 7

and 4D generation methods (AvatarGO [5], InterDreamer [6], CHORD [14]) lack publicly available code, so we compare against the strongest available methods spanning generative and animation paradigms. Metrics. We employ metrics that assess both semantic alignment and temporal quality of the generated interactions. ViCLIP [27] measures semantic alignment between rendered videos and text prompts via cosine similarity in the joint video-text embedding space, providing a measure of how well the generated interaction matches the intended action. To evaluate physical realism, we employ a VQA Physics Score [28], where using Table 1: Quantitative Evaluation. Comparison on a VLM (Qwen-VL-7B), one queries: “Is VQA Phys., ViCLIP, and User Study MOS (Q1-Q4). the interaction physically plausible overVQA ViCLIP Q1 Q2 Q3 Q4 all?” and reports the probability of the to- Method ken “Yes”. 4DFY 0.15 0.26 1.42 1.44 1.85 1.76 AnimateAnyMesh 0.19 0.24 1.61 1.51 2.40 2.11 In addition, we conduct a user study, evaluOurs 0.25 0.30 4.33 4.29 4.21 4.04 ating the perceptual quality of our method against baselines. Participants were presented with videos and asked to rate each method on a scale of 1 (worst) to 5 (best) based on four criteria: (Q1) Physical Plausibility of the object’s response to physics; (Q2) Contact Quality, assessing the accuracy and realism of the interaction; (Q3) Motion Naturalness of the human agent; and (Q4) Photorealism of the visual appearance. We collected responses from 23 participants and report MOS scores. Qualitative Evaluation. A qualitative comparison is shown in Fig. 5. 4D-fy struggles to maintain object consistency, often hallucinating multiple instances of the object throughout the sequence, while producing minimal human motion that fails to convey the intended action. AnimateAnyMesh generates limited motion for both human and object, with no meaningful contact occurring between them. In contrast, our method produces dynamic human motion that coordinates with the object, achieving proper contact where the object responds with physically plausible trajectories and materialappropriate dynamics. Quantitative Evaluation. Tab. 1 presents the quantitative comparisons to baselines. Our method achieves the highest scores on all metrics, significantly outperforming baselines on VQA Physics (0.253 vs. 0.196), ViCLIP (0.295 vs. 0.256) and in perceptual studies. 4.3

Ablation Study

We validate the necessity of individual components of our method in Tab. 2, considering both automated metrics and a perceptual user study as noted above. We also visualize their effect for a single example in Fig. 6. Removing Video-SDS (w/o Video-SDS) preserves global physics but leaves local penetration artifacts due to discrete contact detection. Removing the windowed attraction loss Table 2: Ablation Study. Impact of components on (w/o Attraction) decouples the agent from automated metrics and User Study MOS scores (Q1– the scene, causing the otherwise natural Q4). human motion to miss the target entirely. Method VQA ViCLIP Q1 Q2 Q3 Q4 Replacing the motion diffusion prior with w/o Attraction 0.24 0.23 2.46 1.85 2.97 2.83 direct pose optimization (w/o MDM) prow/o Contact 0.20 0.23 1.92 1.61 2.83 2.55 duces unnatural, anatomically implausible w/o MDM 0.20 0.22 3.26 2.82 1.85 2.65 motion to satisfy attraction constraints. Disw/o Video-SDS 0.25 0.27 4.13 4.15 4.11 3.75 abling contact detection and re-simulation w/o MPM 0.21 0.23 3.04 3.58 3.65 3.56 (w/o Contact) breaks causality; the human reaches the object, but the object ignores Ours (Full) 0.25 0.30 4.41 4.44 4.25 4.15 the collision and continues its trajectory. Finally, removing MPM simulation entirely (w/o MPM) reduces object dynamics to constant velocity, losing material-aware physical fidelity. Notably, the metrics reflect the ablations’ failure modes. The w/o Attraction variant retains high VQA Physics as motion remains natural, but drops in ViCLIP due to absent contact; w/o Contact scores lowest on physics as ignored collisions are the most salient violation, and w/o MDM suffers in prompt alignment since only frames within the optimization window exhibit motion. 8

w/o Attraction w/o MDM w/o Video-SDS w/o Contact w/o MPM Ours

Figure 6: Qualitative Ablation. We highlight failure cases when removing components of our method (see highlighted boxes emphasizing the failure). w/o Attraction: The agent fails to hit the object. w/o MDM: The human mesh deforms unnaturally. w/o Video-SDS: Severe penetration occurs. w/o Contact: The hand passes through the object. w/o MPM: The object moves via velocity transfer, lacking physical realism.

5

Conclusion

We presented PhyGenHOI, a framework that couples generative human motion with MPM-based physical simulation under a shared 3DGS representation to produce physically plausible 4D humanobject interactions. Experiments show that this neuro-physical coupling eliminates ghosting and interpenetration artifacts while enabling dynamic post-contact object responses, outperforming existing baselines in text alignment, physical plausibility, and contact quality. We believe bridging data-driven generation with physics-based simulation opens promising avenues for realistic 4D content creation.

9

References [1] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139–1, 2023. [2] Sherwin Bahmani, Ivan Skorokhodov, Victor Rong, Gordon Wetzstein, Leonidas Guibas, Peter Wonka, Sergey Tulyakov, Jeong Joon Park, Andrea Tagliasacchi, and David B Lindell. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7996–8006, 2024. [3] Yuyang Zhao, Zhiwen Yan, Enze Xie, Lanqing Hong, Zhenguo Li, and Gim Hee Lee. Animate124: Animating one image to 4d dynamic scene. arXiv preprint arXiv:2311.14603, 2023. [4] Jiawei Ren, Liang Pan, Jiaxiang Tang, Chi Zhang, Ang Cao, Gang Zeng, and Ziwei Liu. Dreamgaussian4d: Generative 4d gaussian splatting. arXiv preprint arXiv:2312.17142, 2023. [5] Yukang Cao, Liang Pan, Kai Han, Kwan-Yee K Wong, and Ziwei Liu. Avatargo: Zero-shot 4d human-object interaction generation and animation. arXiv preprint arXiv:2410.07164, 2024. [6] Sirui Xu, Yu-Xiong Wang, Liangyan Gui, et al. Interdreamer: Zero-shot text to 3d dynamic human-object interaction. Advances in Neural Information Processing Systems, 37:52858– 52890, 2024. [7] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851–866. 2023. [8] Zijie Wu, Chaohui Yu, Fan Wang, and Xiang Bai. Animateanymesh: A feed-forward 4d foundation model for text-driven universal mesh animation. arXiv preprint arXiv:2506.09982, 2025. [9] Qi Sun, Can Wang, Jiaxiang Shang, Wensen Feng, and Jing Liao. Animus3d: Text-driven 3d animation via motion score distillation. In Proceedings of the SIGGRAPH Asia 2025 Conference Papers, pages 1–11, 2025. [10] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022. [11] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023. [12] Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. Gaussiandreamer: Fast generation from text to 3d gaussians by bridging 2d and 3d diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6796–6807, 2024. [13] Yanqin Jiang, Li Zhang, Jin Gao, Weimin Hu, and Yao Yao. Consistent4d: Consistent 360 {\deg} dynamic object generation from monocular video. arXiv preprint arXiv:2311.02848, 2023. [14] Yanzhe Lyu, Chen Geng, Karthik Dharmarajan, Yunzhi Zhang, Hadi AlZayer, Shangzhe Wu, and Jiajun Wu. Choreographing a world of dynamic objects, 2026. URL https://arxiv. org/abs/2601.04194. [15] Jiaman Li, Jiajun Wu, and C Karen Liu. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG), 42(6):1–11, 2023. [16] Wenkun He, Yun Liu, Ruitao Liu, and Li Yi. Syncdiff: Synchronized motion diffusion for multi-body human-object interaction synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11731–11743, 2025. [17] Roey Ron, Guy Tevet, Haim Sawdayee, and Amit H Bermano. Hoidini: Human-object interaction through diffusion noise optimization. arXiv preprint arXiv:2506.15625, 2025. 10

[18] Yanqin Jiang, Chaohui Yu, Chenjie Cao, Fan Wang, Weiming Hu, and Jin Gao. Animate3d: Animating any 3d model with multi-view video diffusion. Advances in Neural Information Processing Systems, 37:125879–125906, 2024. [19] Xuan Li, Qianli Ma, Tsung-Yi Lin, Yongxin Chen, Chenfanfu Jiang, Ming-Yu Liu, and Donglai Xiang. Articulated kinematics distillation from video diffusion models. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 17571–17581, 2025. [20] Tianyi Xie, Zeshun Zong, Yuxing Qiu, Xuan Li, Yutao Feng, Yin Yang, and Chenfanfu Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4389–4398, 2024. [21] Tianyu Huang, Haoze Zhang, Yihan Zeng, Zhilu Zhang, Hui Li, Wangmeng Zuo, and Rynson WH Lau. Dreamphysics: Learning physics-based 3d dynamics with video diffusion priors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 3733–3741, 2025. [22] Haoyu Zhao, Hao Wang, Xingyue Zhao, Hao Fei, Hongqiu Wang, Chengjiang Long, and Hua Zou. Efficient physics simulation for 3d scenes via mllm-guided gaussian splatting. arXiv preprint arXiv:2411.12789, 2024. [23] Muhammed Kocabas, Jen-Hao Rick Chang, James Gabriel, Oncel Tuzel, and Anurag Ranjan. Hugs: Human gaussian splats. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 505–515, 2024. [24] Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. A material point method for snow simulation. ACM Transactions on Graphics (TOG), 32(4):1–10, 2013. [25] Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The material point method for simulating continuum materials. In Acm siggraph 2016 courses, pages 1–52. 2016. [26] Guy Tevet, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209.14916, 2022. [27] Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, et al. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942, 2023. [28] Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pages 366–384. Springer, 2024. [29] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01506, 2024. [30] Mathis Petrovich, Or Litany, Umar Iqbal, Michael J. Black, Gül Varol, Xue Bin Peng, and Davis Rempe. Multi-track timeline control for text-driven 3d human motion generation. In CVPR Workshop on Human Motion Generation, 2024. [31] Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. Taichi: a language for high-performance computation on spatially sparse data structures. ACM Transactions on Graphics (TOG), 38(6):1–16, 2019. [32] Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiaohan Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024.

11

A

Interactive Visualizations

We refer readers to the interactive visualizations on our project page at https://omerbenishu. github.io/PhyGenHOI/ for full temporal sequences of generated 4D human-object interactions, comparisons with baselines, and ablation studies across diverse action types.

B

Additional Details

B.1

Implementation Details

Hardware and Runtime. All experiments are conducted on a single NVIDIA H200 GPU. The full pipeline runtime per scene is approximately 74 minutes, broken down as follows: human motion optimization takes about 10 minutes, MPM simulation takes 4 minutes, and Video-SDS refinement takes approximately 1 hour. Rendering the final 4D sequence achieves 20 FPS. Human Representation. We represent the human using 3D Gaussians bound to the SMPL parametric body model [7], following HUGS [23]. We use the default hyperparameters provided by the authors https://github.com/apple/ml-hugs. Each human is initialized from a pre-trained HUGS model, in which the Gaussian representation is already learned and coupled with the SMPL parameters. Object Representation. Object 3DGS representations are obtained from from two sources. The blue ball object is taken directly from the DreamPhysics [21] dataset hosted on Hugging Face, while all other objects are reconstructed from single images using Trellis [29] image-to-3D pipeline. We use the standard 3D Gaussian Splatting [1] representation with default parameters from the code provided by the authors https://github.com/graphdeco-inria/gaussian-splatting. B.2

Human Motion Score Distillation Details

We employ the Motion Diffusion Model (MDM) [26] as our human motion prior, and we use the pretrained model from STMC [30] (https://github.com/nv-tlabs/stmc), which operates directly in SMPL pose space, enabling seamless integration with our 3DGS human representation. Motion Representation. Human motion is parameterized as a sequence X = {xt }Tt=0 , where each frame xt = (rt , ω t , θ t ) consists of root translation rt ∈ R3 , global orientation ω t ∈ R6 in 6D rotation representation, and per-joint pose parameters θ t ∈ RJ×3 for J = 24 joints. We generate sequences of T = 40 frames at 20 FPS. Score Distillation. For HMSD, we sample diffusion timesteps uniformly from [tmin , tmax ], where tmin = 0 and tmax = 100. The weighting function is defined as w(t) = 1 − ᾱt . We use classifier-free guidance with a scale of 7.5. B.3

Contact Detection and Re-simulation Details

Contact Detection. As described in Sec. 3.3 of the main paper, we detect contact by first assigning each human Gaussian to its dominant joint based on skinning weights. Contact at frame tc with joint jc is identified when two conditions are satisfied: 1. The axis-aligned bounding boxes overlap: Bjc (tc ) ∩ Bobj (tc ) ̸= ∅ 2. At least τcontact = 0.05 fraction of joint jc ’s Gaussians lie within distance dcontact = 0.01 of the nearest object Gaussian. Velocity Update. Upon detecting contact, we compute the momentum transfer as follows. The human velocity Vhuman is estimated from the contact joint’s displacement: Vhuman =

pjc (tc ) − pjc (tc − 1) , ∆t 12

(7)

where ∆t = 1. The contact normal n is computed as the normalized direction from the mean position of contacting object Gaussians toward the object’s center of mass. The post-impact velocity applied to the object is: Vpost = Vobj + (1 + e) · vin · n, (8) where vin = (Vhuman − Vobj ) · n is the relative velocity along the contact normal, and e is the coefficient of restitution. MPM Simulation Parameters. We base our MPM simulation on PhysGaussian [20], using the Taichi [31] framework. We use a grid resolution of 64 with simulation timestep 4 · 10−5 and run for 1250 total steps per frame. Material properties are set with Young’s modulus 107 and Poisson ratio 0.45. The coefficient of restitution e is 0.6. After contact detection at frame tc , we perform a single forward MPM simulation from tc to T with the computed post-impact velocity, producing the final object trajectory. B.4

Video-SDS Details

We employ Video Score Distillation Sampling using CogVideoX-5B [32] as the video diffusion model. Rendering and Sampling. During optimization, we render video clips at 480x720 resolution with a length of 49 frames from randomly sampled training camera viewpoints. Camera viewpoints are sampled uniformly from a circular trajectory around the scene, maintaining a fixed elevation for all cameras. A total of 100 viewpoints are used, evenly spaced along the circle to ensure uniform coverage. Temporal Masking. We apply temporal masking to focus optimization on contact frames while preserving the motion prior’s influence elsewhere. Specifically, we optimize only frames within a window [tc − ∆t, tc + ∆t] around the contact frame tc , where ∆t = 1 frames. Diffusion Parameters. We apply a classifier-free guidance (CFG) scale of 100. For timestep sampling, we sample uniformly from [tmin , tmax ] where tmin = 100 and tmax = 980, and the maximum timestep decreases linearly, reaching tmax = 300 at iteration 1000. The text prompt pscene describes the interaction. The prompt is rich, serving as the standard for the video models, with particular attention in the negative prompt to ensure realistic contact and avoid any penetration. B.5

Optimization Details

Our optimization proceeds in three stages as described in Sec. 3.3 of the main paper. Stage 1: Motion Initialization. We optimize human pose parameters using LHMSD alone for Ninit = 100 iterations. We use the Adam optimizer with learning rate 0.005. Stage 2: Human-Object Coordination. We continue optimization for Nsync = 200 iterations with the combined objective: Lhuman = λHMSD LHMSD + λattr Lattr , (9) where λHMSD = 10.0 and λattr = 1.0. The Gaussian window standard deviation for the attraction loss is set to σ = 2 frames. Learning rate remains 0.005. Following this stage, we perform contact detection and MPM re-simulation as described in Sec. B.3. The object trajectory is then fixed for subsequent optimization. Stage 3: Video-SDS for Contact Fidelity. We optimize human pose parameters using temporallymasked Video-SDS for 3000 iterations with learning rate 0.001. Video diffusion model details, rendering parameters, temporal masking window, and CFG scale are provided in Sec. B.4. B.6

Comparisons and Ablations

Below we provide details needed to reproduce the comparisons and ablations shown in the paper. 13

Animate AnyMesh 4DFY Ours Ours

4DFY

Animate AnyMesh

“A man punches the ball with his left hand.”

“A man punches the ball with his left hand.”

Figure 7: Additional Comparisons. Extended evaluation across diverse actions. Our framework consistently maintains physical causality and contact fidelity, whereas baselines fail to coordinate the human agent with the dynamic object. 4D-fy [2] We use the code provided by the authors https://github.com/sherwinbahmani/4dfy. We follow the original configurations used in the paper, while additionally applying the authors’ recommendation to increase motion by setting system.loss.lambda_sds_video = 0.5. AnimateAnyMesh [8] We use the code provided by the authors https://github.com/ JarrentWu1031/AnimateAnyMesh. Since the method takes meshes as input and we have 3DGS objects, we first rendered the scenes using https://meshy.ai/ and exported them to GLB format. However, due to one of their limitations, the mesh face counts are relatively low. In addition, to simplify the setup and improve motion, we positioned the objects closer than in the original scenes, as objects placed further away resulted in barely any scene movement. Ablation Variants. We evaluate five ablation variants. w/o Video-SDS skips optimization stage 3 entirely (Video-SDS), using the output from optimization stage 2 directly. w/o Attraction sets λattr = 0 during optimizationtage 2, optimizing only with LHMSD . w/o MDM replaces MDM-based optimization with direct pose parameter optimization using only Lattr and Video-SDS, initialized from the given initial position. w/o Contact skips contact detection and MPM re-simulation, allowing the object to follow its initial free-motion trajectory throughout. w/o MPM replaces MPM simulation with constant-velocity linear trajectories for the object.

14

C

Additional Qualitative Results

Detailed visual comparisons across our full benchmark are provided in Fig. 7. As observed, our method successfully coordinates the human agent with the dynamic object across diverse action types, whereas baselines often struggle with physical causality or contact fidelity.

D

Limitations

Our framework is designed for impulsive interactions such as kicking, punching, and pushing where contact triggers a discrete momentum transfer. We note that continuous contact scenarios fall outside the current scope, as these require sustained force modeling rather than instantaneous impact. Extending our formulation to handle such interactions is an interesting direction for future work. Our underlying formulation, however, is general, the windowed attraction and re-simulation components can naturally extend to multiple sequential contacts or multi-object scenes. Second, our attraction loss targets the object’s center of mass, which is effective for convex objects, but may be suboptimal for complex geometries requiring contact at specific surface regions. Finally, while the object exhibits physical deformation via MPM, the human agent remains kinematic (SMPL), and thus does not respond to reaction forces or secondary collisions. Coupling soft-body simulation and two-way physical feedback to model bidirectional tissue deformation is a promising avenue for future work.

E

Broader Impacts

Our work on PhyGenHOI focuses on foundational research in physical simulation and 3D/4D generative methodologies. By enabling the synthesis of dynamic human-object interactions that are both visually faithful and physically plausible, our framework offers significant positive societal impacts for applications in animation, gaming, and immersive virtual reality. However, as with many advancements in generative modeling, we must acknowledge potential negative societal impacts. Specifically, improving the physical realism and causal accuracy of human actions, such as realistically depicting a person punching or kicking an object, could theoretically be misused by bad actors to generate highly convincing deepfakes for disinformation or malicious narratives. While our current method is not tied to a specific real-world deployment, mitigating these potential risks in future downstream applications will be important. Possible mitigation strategies include the gated release of interaction models, the integration of robust forensic watermarking on generated 4D assets, and the parallel development of advanced deepfake detection mechanisms to monitor potential misuse.

15