ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecasting

ExPhy: A Benchmark for Explicit Physical Property Learning in Multi-Object Trajectory Forecasting Rui Wang1 , Yeteng Wu1 , Xianlin Zhang2 , Mengshi Qi∗1

arXiv:2608.20009v1 [cs.AI] 20 Aug 2026

1

State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications 2 School of Digital Media & Design Art, Beijing University of Posts and Telecommunications Beijing, China wr@bupt.edu.cn, wuyeteng@bupt.edu.cn, zxlin@bupt.edu.cn, qms@bupt.edu.cn Abstract

Understanding object dynamics requires not only predicting future trajectories but also examining whether a model captures the physical properties that govern motion. However, existing benchmarks rarely expose object-level physical properties as explicit evaluation targets alongside trajectory forecasting. To address this gap, we introduce ExPhy, a multiobject trajectory forecasting benchmark containing 24,000 simulated physical scenes with explicit object-level labels for mass, friction, and restitution. ExPhy provides observed and future trajectories together with an in-distribution (ID) split and two out-of-distribution (OOD) splits over physical parameters (OOD-Parameter) and initial states (OOD-Initial) for jointly evaluating trajectory forecasting and physical property estimation. We further instantiate PhyODE, a physicsguided model with an explicit property interface that estimates physical properties from observed trajectories and uses them for differentiable future rollout. On the long-horizon OODInitial setting, PhyODE reduces ADE and FDE by 33.1% and 31.0%, respectively, compared with the strongest baseline. Zero-shot evaluation on ComPhy further assesses crossbenchmark transfer. Property-level analyses reveal that accurate trajectory forecasting does not necessarily imply accurate recovery of the underlying physical properties. Code and data are available at https://github.com/Zest86/ExPhy.

Introduction Understanding object dynamics is a central problem in physical reasoning, as predicting future motion from observed trajectories requires accounting for the physical properties that shape how objects move and interact (Battaglia et al. 2016; Watters et al. 2017). Humans exhibit intuitive physical reasoning from early development, such as anticipating motion, collision outcomes, and material responses (Davis 2008; Spelke and Kinzler 2007; Wu 2024). In dynamic multiobject interactions, properties such as mass, friction, and restitution govern inertial response, tangential contact behavior, and collision rebound, respectively. Existing evaluations of object-centric physical dynamics commonly focus on future outcomes or predicted trajectories, while estimation of the underlying object-level properties is often assessed separately or not at all. This distinction matters because low trajectory error does not necessarily imply accurate physical ∗

Corresponding author.

Figure 1: Comparison between existing benchmarks and the proposed ExPhy benchmark. (a) Existing benchmarks primarily supervise future outcomes, while object-level physical properties are often represented implicitly or indirectly and are unavailable as dedicated evaluation targets. (b) ExPhy provides explicit continuous-valued labels for mass, friction, and restitution, enabling joint evaluation of trajectory forecasting and physical property estimation. property estimation. It therefore motivates evaluating trajectory forecasting alongside explicit object-level property learning. Despite recent progress in physical reasoning and dynamics prediction, existing benchmarks still largely emphasize future-outcome prediction or task-specific reasoning, including future-state prediction, physical question answering, and event plausibility judgment (Yi et al. 2020; Wang et al. 2023; Bear et al. 2021; Chen et al. 2025). As illustrated in Fig. 1 and summarized in Table 1, object-level physical factors are often represented implicitly or indirectly, rather than exposed as dedicated object-level evaluation targets. To address this gap, we introduce ExPhy, a benchmark for joint trajectory and physical-property evaluation in multi-object trajectory forecasting. ExPhy comprises 24k simulated dynamic scenes with observed and future trajectories and explicit objectlevel labels for mass, friction, and restitution. Together with dedicated in-distribution (ID) and out-of-distribution (OOD) splits, it enables unified evaluation of trajectory forecasting, physical property estimation, and generalization under controlled distribution shifts. With ExPhy, we distinguish two complementary questions: whether a model forecasts future trajectories accu-

Table 1: Comparison of representative physical reasoning benchmarks. ◦ denotes property-dependent evaluation without direct property targets. Obj. denotes direct object-level property evaluation; Cont. denotes continuous parameter regression; Traj. denotes trajectory-level forecasting; and OOD denotes a controlled distribution-shift protocol. Dataset

Phys. Prop. Eval.

Traj. Eval.

OOD Eval.

× × × × × ✓

× × × × × ×

× ✓ × × × ×

✓

✓

✓

Obj-level

Cont.

CLEVRER [ICLR20] Super-CLEVR [CVPR23] Physion++ [NeurIPS23] ComPhy [TPAMI25] PhysBench [ICLR25] PhysInOne [CVPR26]

× × ◦ ✓ ✓ ✓

ExPhy (Ours)

✓

rately and whether its estimated physical properties agree with the underlying simulator parameters. Existing trajectory forecasting models typically learn future states directly from observed trajectories through graph interactions, Transformers, or latent dynamics models (Li et al. 2020; Han et al. 2022; Huang et al. 2022; Liu et al. 2023; Wen, Wang, and Metaxas 2022; Fang, Hsu, and Lee 2025; Fu et al. 2025). Although effective under ADE/FDE, their internal variables need not correspond to physically meaningful properties. To instantiate the ExPhy evaluation, we develop PhyODE, a physics-guided hybrid model that estimates object-level physical properties from observed trajectories and uses them for differentiable future rollout. Property labels are used as training supervision but are never provided as inference inputs. We evaluate PhyODE under ID and OOD settings, conduct zero-shot transfer to ComPhy, and analyze object-level property estimation. Together, these experiments expose the distinction between accurate trajectory forecasting and accurate physical property estimation. The main contributions are summarized as follows: (1) We introduce ExPhy, a benchmark comprising 24k multi-object scenes with explicit object-level labels for mass, friction, and restitution, together with ID, OOD-Parameter, and OOD-Initial evaluation protocols. (2) We develop PhyODE, a physics-guided hybrid model with an explicit property interface that estimates object-level physical properties from observed trajectories and uses them for differentiable future rollout. (3) Extensive experiments demonstrate competitive ID/OOD and cross-benchmark forecasting, and reveal that trajectory accuracy and physical property accuracy are related but distinct evaluation dimensions.

Related Work Physical Reasoning Benchmarks. Existing benchmarks evaluate complementary aspects of physical understanding. CLEVRER (Yi et al. 2020) focuses on causal and futureevent reasoning, while Super-CLEVR (Li et al. 2023b) introduces controlled domain shifts for compositional visual reasoning. Physion and Physion++ (Bear et al. 2021; Tung et al. 2023) primarily evaluate future-contact or outcome predic-

tion, which require inferring latent mechanical properties. ComPhy (Chen et al. 2025) directly evaluates object-level mass and charge through categorical targets, whereas PhysBench (Chow et al. 2025) assesses broader physical properties through multiple-choice questions. PhysInOne (Zhou et al. 2026b) further supports continuous parameter estimation and physics-based resimulation. Overall, prior benchmarks address property reasoning, future prediction, and distribution shifts, but largely through separate protocols. ExPhy instead unifies direct continuous object-level property evaluation, future trajectory forecasting, and controlled extrapolation over physical parameters and initial states. Trajectory and Dynamics Forecasting. Trajectory forecasting methods predict future motion using recurrent networks (Alahi et al. 2016; Qi et al. 2020), Transformers (Zhang, Ding, and Tian 2024; Zhou, Dong, and Du 2025), interaction models, or generative frameworks (Fu et al. 2025; Fang, Hsu, and Lee 2025). Despite strong ADE/FDE performance, their representations need not correspond to physically meaningful object properties. Physics-informed methods introduce analytical dynamics, differentiable simulators, or structural constraints (Tang et al. 2023; Le Cleac’h et al. 2023; Xu, Chen, and Xu 2024), while Neural ODE approaches (Huang, Sun, and Wang 2020; Wen, Wang, and Metaxas 2022; Luo et al. 2023; Yuan et al. 2024) model continuous-time evolution. However, these models often use entangled latent states without direct supervision or evaluation of named physical properties. Physical Property Learning. Related studies infer hidden physical properties from visual observations or interactions. Latent dynamics approaches (Battaglia et al. 2016; Watters et al. 2017; Zhu et al. 2024) encode physical information without requiring interpretable variables. Direct estimators predict quantities such as mass, material, or interaction parameters, but may rely on appearance cues (Wu et al. 2015; Standley et al. 2017), semantic priors (Zhai et al. 2024), multi-view observations (Li et al. 2023a), or foundation models (Zhan et al. 2026). Property estimation is also commonly evaluated separately from future trajectory forecasting. PhyODE instead estimates mass, friction, and restitution from observed trajectories and integrates them into differentiable rollout, enabling trajectory and property evaluation within a single model.

ExPhy benchmark We introduce ExPhy, a benchmark for jointly evaluating multi-object trajectory forecasting and explicit physical property estimation. ExPhy provides observed and future trajectories together with object-level mass, friction, and restitution annotations, as well as controlled distribution shifts over physical parameters and initial states. Problem Formulation. Each ExPhy instance contains the observed trajectories of N interacting objects. For each object i ∈ {1, . . . , N }, let xti = [xti , yit , zit ]⊤ ∈ R3 denote its 3D position at time step t, and let Xi,obs = (x1i , . . . , xTi obs ) denote its observed trajectory. We collect all object trajectories as Xobs = (X1,obs , . . . , XN,obs ) ∈ RN ×Tobs ×3 .

Table 2: Sampling ranges and supports for the ExPhy splits. For friction and restitution, one of the two listed intervals is selected uniformly at random before sampling within it. ExPhy-B shifts the physical-property distributions, whereas ExPhy-C shifts the initial-state distributions. “Same” denotes the corresponding ExPhy-A setting. Variable

Figure 2: Overview of the ExPhy benchmark. (a) Benchmark construction and annotation pipeline. (b) ID, OODParameter, and OOD-Initial evaluation protocols. Blue and red denote ID and OOD variables, respectively; the blue dashed box and red-shaded outer region mark their location ranges, while arrows depict initial velocities. Given Xobs , the primary task is to predict the future trajectories Xbpred ∈ RN ×Tpred ×3 over the following Tpred steps. ExPhy additionally provides an explicit property vector pi = [mi , µi , ei ]⊤ for each object, corresponding to mass, friction, and restitution. These labels are not provided as inference inputs, but support property-supervised training and property-level evaluation.

Dataset Construction ExPhy is constructed using the PyBullet physics engine (Coumans and Bai 2016) to generate controllable multiobject rigid-body interactions. Each scene contains 3–8 objects with diverse geometric shapes, including cubes, cylinders, and spheres. Their initial states and physical properties are sampled from predefined distributions to produce diverse motion patterns and collision events. ExPhy contains 24k dynamic scenes. ExPhy-A comprises 20k scenes, split into 16k/2k/2k training, validation, and test sets, while ExPhyB and ExPhy-C each contain 2k held-out OOD test scenes. Object trajectories and object-level mass, friction, and restitution labels are recorded directly from the simulator. Figure 2 illustrates the construction and evaluation protocols, and Table 2 summarizes the sampling ranges.

Explicit Physical Property Labels For each object i, ExPhy provides an explicit physical property vector pi = [mi , µi , ei ]⊤ , comprising mass, friction, and restitution. These labels correspond to the simulator mass, lateral friction coefficient, and restitution parameters, which affect inertial response, tangential contact behavior, and collision rebound, respectively. The property labels are

ExPhy-A

ExPhy-B

ExPhy-C

Physical properties Mass m [0.1, 10]

[10.01, 15]

Same

Friction µ

[0.35, 0.60] [0.25, 0.34] [0.70, 0.95] [0.96, 1.00]

Same

Restitution e

[0.15, 0.40] [0.05, 0.14] [0.55, 0.85] [0.86, 0.95]

Same

Initial state Location x1i,xy 1 Velocity vi,xy

[−7, 7]2 [−3, 3]2

Same Same

[−10, 10]2 \ [−7, 7]2
2 [−5, −3] ∪ [3, 5]

recorded directly from the simulation configuration and remain constant for each object throughout a scene. Because these properties are not directly observable from a single 3D position, their estimation relies on temporal motion and interobject interaction cues. The explicit annotations support supervised property learning and direct object-level property evaluation. The sampling ranges across ExPhy-A/B/C are summarized in Table 2.

Evaluation Protocols To evaluate generalization beyond the training distribution, ExPhy provides three complementary splits. ExPhy-A (InDistribution) follows the same physical-property and spatial distributions across training, validation, and test sets, and measures standard interpolation performance. ExPhy-B (OOD-Parameter) evaluates extrapolation to unseen physical properties by sampling mass, friction, and restitution outside the training ranges while retaining the same spatial distribution. ExPhy-C (OOD-Initial) evaluates extrapolation to unseen initial states by shifting both the initiallocation and initial-velocity distributions while preserving the ExPhy-A physical-property ranges. All models are trained on ExPhy-A and directly evaluated on ExPhy-B/C without fine-tuning. We further define three observation-prediction horizons: Short (10–10), Mid (20–40), and Long (30–60), covering increasingly challenging forecasting durations.

Methodology Overview As shown in Figure 3, PhyODE is a physics-guided hybrid model that couples an explicit property estimator with a differentiable trajectory solver. Given Xobs , Fψ estimates object-level mass, friction, and restitution, which condition Gϕ to combine physics-based dynamics with a residual Neural ODE and produce X̂pred . The model is trained end-toend with trajectory and property supervision, while inference uses only the observed trajectories without ground-truth property labels.

Figure 3: Overview of PhyODE. The physical property estimator Fψ combines raw motion, invariant trajectory, and pairwise relation features to estimate object-level mass, friction, and restitution. Conditioned on these properties, the differentiable trajectory solver Gϕ combines frictional dissipation and discrete collision updates with a residual Neural ODE, and uses RK4 integration to forecast future trajectories.

Explicit Physical Property Estimator Fψ

Physics-Based Hybrid Trajectory Solver Gϕ

Given the observed scene trajectories Xobs = {Xi,obs }N i=1 ∈ RN ×Tobs ×3 , the estimator extracts complementary cues through three encoding branches. The raw motion encoder captures coordinate-level temporal evolution from positions and their first- and second-order differences:  
hraw = fraw Xi,obs ; ∆Xi,obs ; ∆2 Xi,obs , (1) i

Hybrid dynamics formulation. Given the estimated object-level physical properties p̂i = [m̂i , µ̂i , êi ]⊤ , the trajectory solver rolls out future object states in continuous time. We use xi (t) to denote the continuoustime counterpart of the discretely observed coordinates xti . For each object i, we define an eight-dimensional state zi (t) = [xi (t), vi (t), θi (t), ωi (t)], where xi (t) = [xi (t), yi (t), zi (t)] contains the 3D coordinates, vi (t) = [vi,x (t), vi,y (t), vi,z (t)] denotes the corresponding velocity, and θi (t) and ωi (t) represent scalar planar orientation and angular velocity, respectively. Let Z(t) = [z1 (t), . . . , zN (t)] and P̂ = [p̂1 , . . . , p̂N ]. The hybrid dynamics combine a property-conditioned physics branch with a learnable residual vector field:

dZ(t) = fDPE Z(t), P̂ + λres fϕres Z(t), t . (5) dt Here, fDPE models the continuous dynamics under kinetic friction, while fϕres provides learned corrections to the translational and angular derivatives. The DPE additionally handles collision detection and impulse-based state updates during rollout.

where ∆Xi,obs and ∆2 Xi,obs represent the finite-difference velocities and accelerations, respectively. Complementarily, the invariant descriptor encoder summarizes trajectory geometry independently of absolute coordinates: hinv = finv (Φinv (Xi,obs )) , (2) i where Φinv concatenates relative temporal changes and motion-magnitude statistics, with the complete descriptor definition provided in the supplementary material. The pairwise relation encoder captures interactiondependent physical cues. For each object pair (i, j), relation features are constructed from relative positions, relative velocities, and pairwise distances, and then aggregated using learned attention weights: X rij = Φpair (Xi,obs , Xj,obs ) , hpair = αij fpair (rij ). i j̸=i

(3) This aggregation is permutation equivariant with respect to object ordering and allows the estimator to identify physical cues revealed through inter-object interactions. The three branch representations are fused into an objectlevel latent feature: i h pair inv . (4) hi = ffusion hraw ; h ; h i i i The fused feature is decoded by three property-specific regression heads to produce p̂i = [m̂i , µ̂i , êi ]⊤ = Fψ (Xobs )i . We use a Softplus output for positive mass prediction and sigmoid outputs to constrain friction and restitution to [0, 1].

Physics-based dynamics and numerical rollout. During each collision-free interval, the continuous component of the DPE advances the translational and angular states according to vi,∥ (t) dxi (t) dvi (t) = vi (t), = −µ̂i g , dt dt ∥vi,∥ (t)∥2 + ϵ (6) dθi (t) dωi (t) = ωi (t), = 0, dt dt where g is the gravitational acceleration, vi,∥ (t) is the velocity tangent to the supporting surface, and ϵ ensures numerical stability. This defines the continuous DPE step, followed by impulse-based collision updates to the linear and angular velocities. After continuous integration, the DPE applies an impulsebased update to each detected collision. Superscripts − and

+ denote the states immediately before and after the impulse update, respectively. For an approaching pair (i, j), let nij and tij denote the contact normal and tangent, and define the n corresponding relative velocities as vij = (vi− − vj− )⊤ nij − − t and vij = (vi − vj )⊤ tij . The normal and tangential im−1 n n /(m̂−1 = −(1 + êij )vij pulse components are Jij i + m̂j ) t n t and Jij = −µ̂ij |Jij | sign(vij ), respectively. Defining the ton t tal impulse as Jij = Jij nij + Jij tij , the linear and angular velocities are updated by vi+ = vi− + m̂−1 i Jij , ωi+ = ωi− +

t Jij ri , Ii

vj+ = vj− − m̂−1 j Jij , ωj+ = ωj− −

t Jij rj . Ij

(7)

Here, the DPE uses the symmetric pairwise coefficients êij = (êi +êj )/2 and µ̂ij = (µ̂i +µ̂j )/2. The effective planar object scale ri is obtained from the observed state and defines the corresponding effective moment of inertia Ii = 12 m̂i ri2 . The instantaneous impulse update changes the linear and angular velocities while leaving xi and θi unchanged. Let Zk = Z(t0 + k∆t) denote the rollout state at the k-th prediction step. At each step, the DPE computes the propertyconditioned friction and collision responses from the current state. These physics-based dynamics are combined with the residual vector field and integrated using RK4:
Zk+1 = RK4 Zk , fDPE + λres fϕres , ∆t . (8) After Tpred steps, the predicted trajectories are   X̂pred = Πx (Z1 ), . . . , Πx (ZTpred ) ∈ RN ×Tpred ×3 , (9) where Πx (Zk ) ∈ RN ×3 extracts the 3D coordinates of all objects at prediction step k.

Training Objective PhyODE is trained end-to-end using both trajectory supervision and physical-property supervision. The trajectory loss is defined over all objects and future time steps: Ltraj =

pred N TX X

1 2 x̂ti − xti 2 , N Tpred i=1 t=1

(10)

where N denotes the number of objects in the scene. For physical-property supervision, prediction errors are normalized using fixed property-specific scales rp , shared across training and evaluation. The property loss is then given by N

Lprop =

1 X 3N i=1

X

 SmoothL1

p∈{m,µ,e}

p̂i − pi rp

 . (11)

The overall training objective is Ltotal = Ltraj + λprop Lprop ,

(12)

where λprop balances trajectory forecasting and physical property estimation.

Experiments Settings Datasets. We evaluate all methods on the three ExPhy splits. ExPhy-A contains 20k in-distribution scenes, divided into 16k/2k/2k training, validation, and test sets. ExPhy-B and ExPhy-C each contain 2k held-out test scenes. ExPhy-B (OOD-Parameter) shifts the object-level physical-property distributions, whereas ExPhy-C (OOD-Initial) shifts both initial locations and velocities. All models are trained on the ExPhy-A training set, with checkpoints selected on its validation set, evaluated on the ExPhy-A test set and ExPhyB/C without fine-tuning. We additionally evaluate crossbenchmark transfer on ComPhy (Chen et al. 2025), an independently constructed video reasoning benchmark centered on hidden mass and charge. We repurpose its object trajectories for forecasting and evaluate the ExPhy-trained models without fine-tuning. Compared Methods. We compare PhyODE with representative baselines for trajectory forecasting and physical property estimation. For trajectory forecasting, physical reasoning baselines include VRDP (Ding et al. 2021), PHYCINE (Tang et al. 2023), and PCR (Chen et al. 2025), whose visual frontends are replaced with trajectory encoders. Geometric dynamics baselines include PAINET (Yang et al. 2026) and GSE-Flow (Wu et al. 2026), while generalpurpose forecasting baselines include MoFlow (Fu et al. 2025), Neuralized MRF (Fang, Hsu, and Lee 2025), and PRF (Zhou et al. 2026a). All methods use the same observed trajectories and prediction horizons. For property estimation, Mean and Uniform Random are input-free baselines. Mean uses the empirical ExPhy-A training means, while Uniform Random independently samples mass from U [0.1, 10] and friction and restitution from U [0, 1] for each test object. Temporal MLP, Transformer, and Object-GNN are supervised property predictors trained on ExPhy-A labels. Ground-truth properties are never provided at inference. Metrics. We evaluate trajectory forecasting using Average Displacement Error (ADE) and Final Displacement Error (FDE), which measure the average prediction error over all future steps and the error at the final step, respectively. Physical property estimation is evaluated using normalized mean absolute error (NMAE) for mass, friction, and restitution. We divide the corresponding MAEs by the fixed scales (rm , rµ , re ) = (9.9, 1, 1), where rm is the ExPhy-A mass span and rµ , re are the spans of the admissible coefficient domains [0, 1]. The same scales are used for all splits, and the average NMAE is the unweighted mean across the three properties. Lower is better for all metrics. Implementation Details. All models are implemented in PyTorch and trained and evaluated on a single NVIDIA RTX 3090 GPU. Unless otherwise specified, baseline models follow their original implementations and are trained under the same observation-prediction horizons. We train PhyODE with AdamW using a learning rate of 1 × 10−4 , weight decay of 1 × 10−5 , batch size 64, and 50 epochs.

Table 3: Quantitative comparison of trajectory forecasting error (ADE/FDE ↓) on ExPhy-A, ExPhy-B and ExPhy-C. Lower is better. The prediction horizons are explicitly defined based on observation-prediction steps (Tobs -Tpred ): Short (10-10), Mid (2040), and Long (30-60). † indicates trajectory-only adaptations of physical reasoning baselines, where visual/perceptual frontends are replaced with trajectory encoders while preserving their original reasoning mechanisms. The baselines are grouped according to their primary inductive biases. Bold and underlined indicate the best and second-best results, respectively. Methods

ExPhy-A (In-Distribution) Short

ExPhy-B (OOD-Parameter)

ExPhy-C (OOD-Initial)

Mid

Long

Short

Mid

Long

Short

Mid

Long

Physical reasoning baselines VRDP† [NeurIPS21] 0.04/0.08 PHYCINE† [CVPR23] 0.04/0.08 PCR† [TPAMI25] 0.05/0.10

0.28/0.58 0.34/0.66 0.28/0.57

0.41/0.83 0.46/0.90 0.48/0.94

0.04/0.08 0.04/0.08 0.05/0.10

0.29/0.61 0.36/0.70 0.28/0.58

0.45/0.92 0.51/1.01 0.52/1.04

0.12/0.23 0.12/0.21 0.14/0.28

0.93/1.93 1.12/2.15 0.67/1.47

2.13/4.23 1.95/3.86 1.58/3.27

Geometric dynamics baselines PAINET [ICLR26] 0.05/0.10 GSE-Flow [ICML26] 0.13/0.24

0.27/0.57 0.28/0.59

0.40/0.81 0.52/0.99

0.06/0.11 0.12/0.24

0.29/0.60 0.29/0.64

0.43/0.90 0.53/1.07

1.32/1.33 0.47/0.89

2.16/3.65 1.17/2.07

2.46/5.14 2.66/4.60

General-purpose trajectory forecasting baselines MoFlow [CVPR25] 0.07/0.11 0.27/0.53 0.40/0.76 Neuralized MRF [ICLR25] 0.09/0.18 0.68/1.33 0.90/1.73 PRF [CVPR26] 0.04/0.09 0.29/0.60 0.42/0.85

0.07/0.11 0.10/0.20 0.05/0.10

0.28/0.56 0.73/1.47 0.30/0.63

0.41/0.81 1.07/2.02 0.49/1.02

0.44/0.58 0.51/1.01 0.12/0.25

1.04/1.97 2.61/5.40 0.91/1.98

1.85/3.47 4.07/7.98 1.45/2.90

Physics-guided dynamics PhyODE 0.03/0.07

0.04/0.07

0.25/0.51

0.40/0.85

0.07/0.13

0.48/1.08

0.97/2.00

0.26/0.51

0.36/0.75

Table 4: Zero-shot transfer results on ComPhy. All models are trained on ExPhy-A and directly evaluated on ComPhy without fine-tuning. We report ADE/FDE (↓). † indicates trajectory-only adaptations of physical reasoning baselines. Best and second-best results are shown in bold and underlined, respectively.

Table 5: Object-level property estimation under the Long setting (Tobs = 30). Entries report NMAE on ExPhy-A/ExPhyB (ID/OOD-Parameter). All learned models are trained on ExPhy-A and evaluated zero-shot on ExPhy-B. “Prop. only”, “Traj. only”, and “Full” use Lprop , Ltraj , and their joint objective, respectively. Lower is better; bold denotes the best results, including ties.

ComPhy [TPAMI25]

Methods

Mass ↓ A/B

Fric. ↓ A/B

Rest. ↓ A/B

Avg. ↓ A/B

Non-learned baselines Mean 0.25/0.77 Random 0.33/0.75

0.17/0.34 0.31/0.39

0.22/0.41 0.30/0.41

0.21/0.51 0.31/0.52

1.56/2.78 1.42/1.95

Supervised property predictors Temporal MLP 0.24/0.77 Transformer 0.25/0.77 Object-GNN 0.22/0.79

0.13/0.28 0.11/0.24 0.09/0.21

0.17/0.33 0.14/0.27 0.13/0.27

0.18/0.46 0.17/0.43 0.15/0.42

General-purpose trajectory forecasting baselines MoFlow [CVPR25] 0.24/0.37 0.89/1.50 Neuralized MRF [ICLR25] 0.13/0.20 0.56/0.93 PRF [CVPR26] 0.24/0.44 1.48/2.70

1.20/1.93 0.63/1.00 3.09/5.52

PhyODE variants PhyODE (Prop. only) PhyODE (Traj. only) PhyODE (Full)

0.09/0.21 0.63/0.63 0.17/0.34

0.13/0.27 0.26/0.41 0.22/0.40

0.15/0.42 0.40/0.68 0.21/0.50

Physics-guided dynamics PhyODE

0.51/0.82

Quantitative Results

the pronounced gains suggest that the structured dynamics design of PhyODE is effective for long-horizon extrapolation to unseen initial states.

Trajectory forecasting. Table 3 reports ADE/FDE on ExPhy-A/B/C across three horizons. PhyODE achieves competitive ID and OOD performance, including the best Longhorizon result of 0.36/0.75 on ExPhy-A and the best ADE on ExPhy-B. Its advantage is most pronounced on ExPhyC, reducing Long-horizon ADE/FDE from the second-best 1.45/2.90 to 0.97/2.00. Since ExPhy-C shifts both initial locations and velocities while preserving the property ranges,

Cross-benchmark transfer. Table 4 reports zero-shot forecasting results on ComPhy, where all models are trained only on ExPhy-A and evaluated without fine-tuning. PhyODE achieves the best performance across all horizons, with ADE/FDE of 0.12/0.20, 0.37/0.65, and 0.51/0.82 from Short to Long. Its gains over Neuralized MRF become more pronounced at the Mid and Long horizons. These results provide external validation that dynamics learned from ExPhy remain

Short

Mid

Long

Physical reasoning baselines VRDP† [NeurIPS21] 0.13/0.23 PHYCINE† [CVPR23] 0.13/0.24 PCR† [TPAMI25] 0.19/0.34

0.51/0.92 0.56/0.99 0.77/1.41

0.79/1.32 0.84/1.41 1.15/2.00

Geometric dynamics baselines PAINET [ICLR26] 0.27/0.36 GSE-Flow [ICML26] 0.21/0.32

1.70/3.52 0.82/1.18

0.12/0.20

0.37/0.65

Method

0.22/0.78 0.30/0.99 0.25/0.75

useful beyond its native scene distribution, while the sustained Mid- and Long-horizon advantages suggest that the structured dynamics design remains effective under crossbenchmark transfer. Physical property estimation. On ExPhy-A, dedicated property predictors outperform the non-learned baselines, with Object-GNN and PhyODE (Prop. only) achieving the best average NMAE of 0.15. On ExPhy-B, errors increase across all methods, especially for mass, while friction and restitution generalize more reliably; the same two models remain the strongest overall with an average NMAE of 0.42. Among the PhyODE variants, property-only supervision achieves 0.15/0.42 on ExPhy-A/B, trajectory-only supervision degrades to 0.40/0.68, and joint training improves the results to 0.21/0.50 but remains less accurate than direct property supervision. Together with the forecasting results, these findings show that low trajectory error does not necessarily imply accurate physical property estimation.

Figure 5: Qualitative property interventions under the Long setting. Red, blue, and green denote predicted, observed, and ground-truth trajectories, respectively. Table 6: Component ablation of PhyODE under the Long horizon setting (Tobs -Tpred = 30-60). We report trajectory forecasting errors as ADE/FDE (↓) on both ID and OOD splits. Best results are shown in bold. ExPhy-A

Variant

ExPhy-B

ExPhy-C

ADE/FDE ↓ ADE/FDE ↓ ADE/FDE ↓ w/o explicit physics w/o Neural ODE

0.42/0.86 0.38/0.79

0.48/0.99 0.41/0.86

1.80/3.41 1.12/2.30

PhyODE

0.36/0.75

0.40/0.85

0.97/2.00

Qualitative Results Figure 4 compares long-horizon forecasts on ExPhy-A/B/C, while Figure 5 qualitatively examines inference-time property interventions. PhyODE produces stable predictions under both ID and OOD settings. Compared with the original rollout, fixed perturbations of ±30% of each property range and random property replacement produce visible trajectory changes, providing qualitative evidence that the estimated properties actively influence trajectory rollout.

Ablation Study Table 6 evaluates the main rollout components of PhyODE. Removing explicit physics causes the largest degradation, especially on ExPhy-C, indicating the importance of propertyconditioned physical rollout under OOD initial states. Removing the Neural ODE component also degrades longhorizon forecasting, suggesting that the learnable residual dynamics complement the structured physical module.

Conclusion

Figure 4: Visualization of long-horizon trajectory forecasting on ExPhy-A, ExPhy-B, and ExPhy-C. Rows show representative methods from different model families, and columns correspond to ID/OOD splits. Red, blue, and green dots denote predicted, observed, and ground-truth trajectories, respectively.

In this paper, we presented ExPhy, a benchmark for joint trajectory and physical-property evaluation in multi-object trajectory forecasting. ExPhy contains 24,000 dynamic scenes with trajectories, object-level mass, friction, restitution annotations, and controlled ID/OOD protocols. We also introduced PhyODE, a physics-guided hybrid model that explicitly estimates these properties and uses them for differentiable rollout. Experiments on ExPhy and zero-shot transfer to ComPhy show that low trajectory error does not necessarily imply accurate physical property estimation.

References Alahi, A.; Goel, K.; Ramanathan, V.; Robicquet, A.; Fei-Fei, L.; and Savarese, S. 2016. Social LSTM: Human trajectory prediction in crowded spaces. In CVPR. Battaglia, P.; Pascanu, R.; Lai, M.; Rezende, D. J.; et al. 2016. Interaction networks for learning about objects, relations and physics. In NeurIPS. Bear, D. M.; Wang, E.; Mrowca, D.; Binder, F. J.; Tung, H.Y. F.; Pramod, R.; Holdaway, C.; Tao, S.; Smith, K.; Sun, F.-Y.; et al. 2021. Physion: Evaluating physical prediction from vision in humans and machines. In NeurIPS. Chen, Z.; Dong, S.; Yi, K.; Li, Y.; Ding, M.; Torralba, A.; Tenenbaum, J. B.; and Gan, C. 2025. Compositional physical reasoning of objects and events from videos. IEEE Transactions on Pattern Analysis and Machine Intelligence, 47(9): 7689–7703. Chow, W.; Mao, J.; Li, B.; Seita, D.; Guizilini, V.; and Wang, Y. 2025. PhysBench: Benchmarking and Enhancing Vision Language Models for Physical World Understanding. In ICLR. Coumans, E.; and Bai, Y. 2016. Pybullet, a python module for physics simulation for games, robotics and machine learning. Davis, E. 2008. Physical reasoning. Foundations of Artificial Intelligence, 3: 597–620. Ding, M.; Chen, Z.; Du, T.; and Luo, P. 2021. Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language. In NeurIPS. Fang, Z.; Hsu, D.; and Lee, G. H. 2025. Neuralized Markov Random Field for Interaction-Aware Stochastic Human Trajectory Prediction. In ICLR. Fu, Y.; Yan, Q.; Wang, L.; Li, K.; and Liao, R. 2025. MoFlow: One-Step Flow Matching for Human Trajectory Forecasting via Implicit Maximum Likelihood Estimation based Distillation. In CVPR. Han, J.; Huang, W.; Ma, H.; Li, J.; Tenenbaum, J.; and Gan, C. 2022. Learning physical dynamics with subequivariant graph neural networks. In NeurIPS. Huang, W.; Han, J.; Rong, Y.; Xu, T.; Sun, F.; and Huang, J. 2022. Equivariant graph mechanics networks with constraints. In ICLR. Huang, Z.; Sun, Y.; and Wang, W. 2020. Learning continuous system dynamics from irregularly-sampled partial observations. In NeurIPS. Le Cleac’h, S.; Yu, H.-X.; Guo, M.; Howell, T.; Gao, R.; Wu, J.; Manchester, Z.; and Schwager, M. 2023. Differentiable Physics Simulation of Dynamics-Augmented Neural Objects. IEEE Robotics and Automation Letters, 8(5): 2780– 2787. Li, X.; Qiao, Y.-L.; Chen, P. Y.; Jatavallabhula, K. M.; Lin, M.; Jiang, C.; and Gan, C. 2023a. PAC-NeRF: Physics augmented continuum neural radiance fields for geometryagnostic system identification. In ICLR. Li, Y.; Lin, T.; Yi, K.; Bear, D.; Yamins, D.; Wu, J.; Tenenbaum, J.; and Torralba, A. 2020. Visual grounding of learned physical models. In ICML.

Li, Z.; Wang, X.; Stengel-Eskin, E.; Kortylewski, A.; Ma, W.; Van Durme, B.; and Yuille, A. L. 2023b. Super-CLEVR: A virtual benchmark to diagnose domain robustness in visual reasoning. In CVPR. Liu, Y.; Cheng, J.; Zhao, H.; Xu, T.; Zhao, P.; Tsung, F.; Li, J.; and Rong, Y. 2023. SEGNO: Generalizing equivariant graph neural networks with physical inductive biases. In ICLR. Luo, X.; Yuan, J.; Huang, Z.; Jiang, H.; Qin, Y.; Ju, W.; Zhang, M.; and Sun, Y. 2023. HOPE: High-order graph ode for modeling interacting dynamics. In ICML. Qi, M.; Qin, J.; Wu, Y.; and Yang, Y. 2020. Imitative NonAutoregressive Modeling for Trajectory Forecasting and Imputation. In CVPR. Spelke, E. S.; and Kinzler, K. D. 2007. Core knowledge. Am. Psychol., 10(1): 89–96. Standley, T.; Sener, O.; Chen, D.; and Savarese, S. 2017. image2mass: Estimating the mass of an object from its image. In CoRL. Tang, Q.; Zhu, X.; Lei, Z.; and Zhang, Z. 2023. Intrinsic Physical Concepts Discovery With Object-Centric Predictive Models. In CVPR. Tung, H.-Y.; Ding, M.; Chen, Z.; Bear, D. M.; Gan, C.; Tenenbaum, J. B.; Yamins, D. L. K.; Fan, J.; and Smith, K. A. 2023. Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Properties. In NeurIPS. Wang, X.; Ma, W.; Li, Z.; Kortylewski, A.; and Yuille, A. 2023. 3D-Aware Visual Question Answering about Parts, Poses and Occlusions. In NeurIPS. Watters, N.; Zoran, D.; Weber, T.; Battaglia, P.; Pascanu, R.; and Tacchetti, A. 2017. Visual interaction networks: Learning a physics simulator from video. In NeurIPS. Wen, S.; Wang, H.; and Metaxas, D. 2022. Social ODE: Multi-agent trajectory forecasting with neural ordinary differential equations. In ECCV. Wu, J. 2024. Physical scene understanding. AI Magazine, 45(1): 156–164. Wu, J.; Liu, Y.; Yu, R.; and Sun, J. 2026. Flow for Future: Geometric SE (3)-Equivariant Flow Matching for 3D Trajectory Prediction. In ICML. Wu, J.; Yildirim, I.; Lim, J. J.; Freeman, B.; and Tenenbaum, J. 2015. Galileo: Perceiving physical object properties by integrating a physics engine with deep learning. In NeurIPS. Xu, H.; Chen, T.; and Xu, F. 2024. Learning Physical Dynamics for Object-centric Visual Prediction. arXiv preprint arXiv:2403.10079. Yang, K.; Huang, Y.; Tao, J.; Wang, W.; and Wu, Q. 2026. PAINET: A Principled Efficient Transformer for 3D Dynamics Modeling. In ICLR. Yi, K.; Gan, C.; Li, Y.; Kohli, P.; Wu, J.; Torralba, A.; and Tenenbaum, J. B. 2020. CLEVRER: CoLlision Events for Video REpresentation and Reasoning. In ICLR. Yuan, J.; Sun, G.; Xiao, Z.; Zhou, H.; Luo, X.; Luo, J.; Zhao, Y.; Ju, W.; and Zhang, M. 2024. EGODE: An event-attended graph ODE framework for modeling rigid dynamics. In NeurIPS.

Zhai, A. J.; Shen, Y.; Chen, E. Y.; Wang, G. X.; Wang, X.; Wang, S.; Guan, K.; and Wang, S. 2024. Physical property understanding from language-embedded feature fields. In CVPR. Zhan, G.; Ma, X.; Xie, W.; and Zisserman, A. 2026. Inferring Dynamic Physical Properties from Video Foundation Models. In CVPR Workshops. Zhang, Z.; Ding, Z.; and Tian, R. 2024. Decouple ego-view motions for predicting pedestrian trajectory and intention. IEEE Transactions on Image Processing, 33: 4716–4727. Zhou, H.; Qi, L.; Li, X.; Zhang, J.; Liu, Y.; Yang, X.; Fan, M.; and Luo, F. 2026a. Recover to Predict: Progressive Retrospective Learning for Variable-Length Trajectory Prediction. In CVPR. Zhou, J.; Dong, Y.; and Du, B. 2025. Siam TITP: Incorporating Temporal Information and Trajectory Prediction Siamese Network for Satellite Video Object Tracking. IEEE Transactions on Image Processing, 34: 4120–4133. Zhou, S.; Wang, H.; Cheng, H.; Li, J.; Wang, D.; Jiang, J.; Jin, Y.; Huang, J.; Mao, S.; Liu, S.; Yang, Y.; Song, H.; Wei, S.; Zhang, Z.; Huang, P.; Liu, S.; Hao, Z.; Li, H.; Li, Y.; Zhou, W.; Zhao, Z.; He, Z.; Wen, H.; Huang, S.; Yun, P.; Cheng, B.; Fu, P. K.; Lai, W. K.; Chen, J.; Wang, K.; Sun, Z.; Li, Z.; Hu, H.; Zhang, D.; Yuen, C. H.; Wang, B.; Wang, Z.; Zou, C.; and Yang, B. 2026b. PhysInOne: Visual Physics Learning and Reasoning in One Suite. In CVPR. Zhu, X.; Deng, H.; Yuan, H.; Wang, Y.; and Yang, X. 2024. Latent Intuitive Physics: Learning to Transfer Hidden Physics from A 3D Video. In ICLR.

Related documents

• NaviScale: Generating Large-Scale Semantic Map Datasets for Object Navigation#1063429
• Artificial Intelligence in Physical Sciences: Symbolic Regression Trends and Perspectives#686397
• S2Planner: Multi-Scale Semantic Planner for End-to-End Autonomous Driving#1077740
• RoboHarn-Evo: Evolving Hierarchical Physical Knowledge for Self-Improving Robotic Manipulation#1143898
• MoveBench: A Benchmark for Global-Scale Wildlife Movement Forecasting#1109739