CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?

CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated? Jonathan Sadeghi Jenny Seidenschwarz Jesse Allardice Sirish Srinivasan Benjamin Graham Jeffrey Hawke

jon@odyssey.systems jenny@odyssey.systems jesse@odyssey.systems sirish@odyssey.systems ben@odyssey.systems jeff@odyssey.systems

Odyssey

Abstract

arXiv:2608.16829v1 [cs.LG] 17 Aug 2026

Video world models are designed to approximate the stochastic distribution of physical outcomes through generative sampling, but existing benchmarks evaluate properties of each individual generation or aim to compare output distributions in a coarse grained way over an entire dataset, and do not investigate the fine grained aleatoric uncertainty of specific physical phenomena. We introduce CaliBench, a benchmark that tests this directly by scoring outcomes in a physically interpretable discrete space — a bin index, a die face, a suit, a colour — rather than in a learned feature space (e.g. that of FID), where we are able to directly calculate the distributional distance of the model outputs from a known reference distribution. This is possible because we carefully curate outcome spaces where the reference distribution is known in closed form (binomial Galton boards, Bernoulli forks, uniform dice/cards/lottery, a known-skewed European-roulette colour), which enables an exact calibration test which is not dependent on an empirical proxy. We decompose performance into two orthogonal axes that a single accuracy metric would conflate: scorability (the fraction of video generations producing a scoreable outcome) and calibration (the total variation distance from the reference distribution on the scoreable sample). We also test the statistical significance of miscalibration with a χ2 test; because calibration is its null hypothesis, the test can only evidence miscalibration, never calibration, and at N =32 generations per cell it reliably detects only large miscalibrations. We apply the protocol to nine scenes and six state-of-the-art image-to-video models (WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, Runway Gen-4.5, Cosmos3-Super) on 32 video generations each. Results reveal a consistent pattern: models concentrate output probability mass on a small subset of outcomes rather than reproducing the reference distribution. Most scene–model combinations are significantly miscalibrated, in the most extreme case collapsing entirely to a single outcome, for example Veo 3.1 on dice. On roulette, generated videos often leave the ball ambiguously placed, so several models also have low scorability. Performance varies sharply by scene rather than by model: no single model dominates across all nine scenes. We release the protocol and a metric (mean normalised total variation, mnTV) to enable comparison of new models with the results in this paper.

1

Introduction

Simulation theory fundamentally questions whether a computed environment can be statistically distinguished from reality (Bostrom, 2003). In generative modelling, a parallel problem emerges: a simulation often betrays itself not through catastrophic visual failures, but through subtle statistical anomalies—glitches or regularities that deviate from genuine randomness. If stochastic macroscopic events generated by a model, such as coin flips or roulette spins, follow distributions that are artificially peaked or uniformly flattened, the model reveals its underlying miscalibration while potentially still yielding physically plausible generations. 1

1

Scene + known reference

2

Video model generation

3

VLM outcome extraction

4

Score vs. reference

Gemini 3.1 Pro reads each final frame:

6

reference observed

× 32 conditioning frame + text prompt

5 1

“a single ball falls through the pegs . . . ” input

Reference: Binomial(10, 1/2) known in closed form · peak ≈ 24.6%

same input; vary only the seed

videos

null

N = 32 generations per scene

invalid: multiple balls

repeated across all 9 scenes

majority vote of 3 VLM queries null fraction defines scorability ρ

any image-to-video model

6

11

outcomes Scorability ρ — fraction scoreable Calibration TV(p̂, p0 ) — distance Significance Monte-Carlo χ2 test scored per (scene, model) cell

outcome space = bin index (discrete, interpretable)

Figure 1: Overview of the CaliBench protocol. (1) Each scene pairs a conditioning frame and text prompt with an analytically known reference distribution over a discrete, interpretable outcome space (e.g. Binomial(10, 1/2) over Galton-board bins). (2) A video model generates N =32 videos from identical inputs, varying only the random seed. (3) A vision–language model (Gemini 3.1 Pro, majority vote of three queries) extracts each video’s discrete outcome from its final frame. (4) We compare the empirical outcome distribution to the reference along two orthogonal axes: scorability ρ (fraction of scoreable generations) and calibration (total variation distance TV(p̂, p0 ) from the reference).

This thought experiment exposes a critical gap in how we evaluate video world models. At macroscopic scales, physical dynamics are effectively deterministic; apparent randomness arises from a sensitive dependence on initial conditions that cannot be fully resolved under sensory constraints (Strogatz, 2024). For a video model, the initial conditioning frame is inherently low-resolution and quantised, failing to capture the micro-states which dictate chaotic trajectories. To simulate these physics correctly, the model must map a single deterministic macro-state input to a distribution of diverse, valid trajectories via generative sampling. A model is calibrated if its predicted distribution over outcomes matches the true distribution of those outcomes. The true calibration target is thus the marginal distribution of physical outcomes, integrated over the unresolved microscopic variations. For example, while the exact path of a ball through a peg array cannot be deterministically inferred from a starting frame, a physically grounded world model must reproduce the correct binomial distribution across landing bins. A model that collapses its generations entirely onto the central bin has failed to simulate the physics; it has erroneously learned an average behaviour rather than the distribution it averages over. Existing generative video evaluation frameworks predominantly operate by comparing video samples within abstract, learned embedding spaces, e.g. Fréchet Video Distance (FVD) or by assessing qualitative persample physical plausibility. While effective for auditing coarse perceptual quality, these methodologies yield uninterpretable, scalar-valued distances whose latent coordinates do not map to physical conservation laws, and they fundamentally ignore the dimension of statistical calibration. To bridge this gap, we introduce CaliBench (Figure 1), an evaluation benchmark that profiles generative world models directly within discrete, low-dimensional, and physically meaningful outcome spaces, such as a Galton board bin index, a die face, or a roulette colour pocket. Because these environments encapsulate well-characterised random phenomena, their target reference distributions are analytically tractable and known in closed form, e.g. uniform for fair dice, binomial for ideal Galton boards. Consequently, CaliBench enables the exact computation of distributional divergences against an absolute analytical ground truth, bypassing the need for biased empirical dataset proxies. Our evaluation protocol decomposes world model performance into two orthogonal axes of failure: (i) Scorability: The fraction of video generations that produce a physically interpretable, completed outcome (e.g., a ball cleanly settling in a single bin without dissolving into motion blur, structural deformation, or artifacting); (ii) Calibration: The Total Variation Distance (TVD) between the empirical distribution of scoreable outcomes and the analytical reference distribution. These metrics diagnose distinct model deficiencies. A model can exhibit low scorability but high calibration (generating few valid videos, but distributing them accurately), or high scorability alongside severe modecollapse. Both dimensions are necessary to achieve a faithful physical simulation. We apply this protocol across 2

nine distinct scenes spanning five distribution families where the ground-truth distribution is analytically known, visual outcomes are unambiguous, and bin assignments can be automatically extracted using a vision-language model (VLM). Contributions. 1. CaliBench: We introduce a novel benchmark comprising nine physical simulation environments where outcomes map to interpretable, discrete spaces (e.g., bin indices, dice faces, suit/colour identifiers). By design, these environments possess analytically known reference distributions (binomial, Bernoulli, uniform, or skewed), enabling exact distributional calibration testing without relying on empirical dataset proxies or uninterpretable learned feature spaces. 2. Orthogonal Evaluation Protocol: We disentangle generative performance into two diagnostic axes: scorability (ρ), which quantifies the proportion of structurally valid generations, and calibration (Total Variation Distance), which measures distributional fidelity. We demonstrate that this decomposition is essential to isolate distinct failure modes—specifically, distinguishing between structural instability (low ρ) and mode collapse (high ρ but high TVD)—which are otherwise conflated by aggregate accuracy metrics. 3. Systematic Benchmarking: We conduct the first large-scale calibration study of six state-of-theart image-to-video models across these nine scenes. Our results utilise χ2 significance testing to demonstrate that most contemporary models are significantly miscalibrated, often collapsing probability mass despite producing individually "plausible" frames. We additionally probe (Appendix J) how classifier-free guidance affects this behaviour on the one model that exposes it. As our χ2 test takes calibration as the null hypothesis, it can only evidence miscalibration, never calibration; non-significant cells are inconclusive rather than calibrated. In addition, at N =32 generations per cell the test reliably detects only large miscalibrations (per-scene detectability floors in Appendix C). 4. Reproducibility and Standardisation: We release our comprehensive evaluation protocol, including the curated conditioning imagery, standardised VLM extraction prompts, and analysis codebase. We introduce the Mean Normalised Total Variation (mnTV) as a unified metric to track model progress toward physically grounded stochastic simulation.

2

Related Work

Per-Sample Quality and Physical Plausibility. Existing evaluation suites primarily focus on assessing individual video quality or per-sample physical adherence. VBench (Huang et al., 2024; Zheng et al., 2025) evaluates video rollouts along 16-plus dimensions; while it includes a diversity metric to detect near-identical generations, sample diversity is fundamentally distinct from calibration, as a diverse model can still be heavily mis-distributed. Other frameworks scrutinise specific structural properties of individual samples: FVMD (Liu et al., 2024) measures per-video motion consistency, the World Consistency Score (WCS) (Rakheja et al., 2025) tests causal compliance, WBench (Ying et al., 2026) evaluates interactive world models across physics and consistency axes, and PhysicsIQ (Motamed et al., 2025) tracks physical trajectories against specific ground-truth continuations. Similarly, automated auditing frameworks like PhyGenBench (Meng et al., 2024) leverage vision-language models (VLMs) to judge per-prompt physical common sense. These metrics are entirely orthogonal to our framework; they evaluate whether each individual video is internally consistent, whereas we interrogate whether the ensemble of video generations accurately recovers a known physical distribution. Feature-Space Distributional Metrics. Standard distributional evaluation in generative video relies on metrics like FVD (Unterthiner et al., 2019) and JEDi (Luo et al., 2025). While these approaches capture appearance-level divergence by computing distances within learned perceptual feature spaces, they aggregate statistics across an entire dataset. This global aggregation renders them ill-suited for physical interpretability, as they cannot isolate the specific aleatoric uncertainty of a given scene. In contrast, our evaluation stratifies by scene, testing whether a specific analytical distribution is faithfully reproduced under a tightly controlled conditioning input. 3

Distributional Calibration without a Known Reference. A parallel line of work addresses image generation calibration against empirical datasets where no closed-form distribution is available. These methods typically implement two-sample tests within learned feature spaces, e.g. by partitioning sample space into Voronoi cells (Richardson & Weiss, 2018), computing precision and recall on density estimates (Sajjadi et al., 2018; Kynkäänniemi et al., 2019), or bounding the generator’s effective support size via birthday paradox estimators (Arora et al., 2018). We side-step the inherent ambiguities of feature-space representations and empirical reference approximations. By designing physics experiments that project onto highly interpretable, discrete outcome spaces, we leverage precise, analytically known reference distributions. Uncertainty Estimation in Video Generation. Quantifying uncertainty in video synthesis remains a nascent area. S-QUBED (Mei et al., 2025a) constructs predictive models to separate aleatoric and epistemic uncertainty in video generation, while C3 (Mei et al., 2025b) generates per-pixel confidence maps for action-conditioned robotics models. However, neither framework evaluates model calibration against established physical distributions. CaliBench explicitly addresses this gap by auditing how accurately generative probabilities match physical reality Benchmarking Video World Models. While recent literature increasingly frames video models as physical simulators, existing benchmarks remain confined to assessing per-sample physical plausibility. For instance, WorldModelBench (Li et al., 2025) assesses per-sample physics adherence; STEVO-Bench (Ma et al., 2026) evaluates the object permanence of occluded structures; TiViBench (Chen et al., 2025) evaluates per-sample reasoning; and PAI-Bench (Zhou et al., 2025) scores visual plausibility across video understanding tasks. Similarly, PhysicsIQ (Motamed et al., 2025) measures whether a generated continuation matches a single ground-truth physical trajectory, while PhyGenBench (Meng et al., 2024) leverages a VLM judge to evaluate per-prompt adherence to physical commonsense laws. All of these frameworks evaluate physical fidelity strictly at the per-sample level—checking whether each individual video is internally coherent. Although Yue et al. (2025) identify stochastic calibration as a defining capability for next-generation video world models, they do not formalise an experimental measurement protocol. CaliBench addresses this critical blind spot by testing the complementary distributional question: whether an ensemble of video generations accurately recovers a known physical distribution. A model can successfully pass existing per-sample checks by producing plausible individual videos, yet be miscalibrated and completely fail our population-level audit due to systemic mode collapse.

3

CaliBench

To evaluate models on their physical plausibility as well as their distributional calibration, CaliBench evaluates generative video world models across nine distinct statistical, physical scenes under a tightly controlled stochastic generation protocol. 3.1

Scene Definition

Each evaluation environment is formalised as a paired conditioning input (x0 , c), where x0 denotes the initial frame and c represents the accompanying natural language prompt. For every model-scene pair, we sample an ensemble of N = 32 video trajectories to evaluate the empirical distribution of discrete macroscopic outcomes. The nine scenes are categorised into four distinct groups based on the analytical properties of their underlying ground-truth reference distributions (Figure 2). Binomial — Galton Boards. A Galton board consists of a triangular lattice of deflecting pegs. A falling ball hits each peg and undergoes a sequence of independent Bernoulli trials, deflecting left or right with equal probability. For a lattice with n rows and n+1 landing bins, the probability P (k) of the ball terminating in bin k follows the binomial mass function:  
n 1 n , k = 1, . . . , n+1. (1) P (k) = 2 k−1 4

(a) Phys. board

(f) Dice

(b) Anim. board

(c) Ball fork

(g) Cards

(d) Walking

(h) Lottery

(e) Pendulum

(i) Roulette

Figure 2: Conditioning frames for all nine scenes. Top row (left to right): physical board, animated board [Binomial(10, 1/2)]; ball fork, walking, double pendulum [Bernoulli(1/2)]. Bottom row: dice, cards, lottery [Uniform]; roulette colour [known non-uniform].

We instantiate two variants with n = 10 rows (resulting in 11 discrete bins with a central mode at bin 6, where P (6) ≈ 24.6%): an animated board utilising a 2D vector graphic illustration with a green ball, and a photorealistic physical board captured via high-fidelity imagery with a metallic ball. While both setups share an identical analytical target distribution, their visual domains differ significantly, allowing us to evaluate whether a model’s calibration is robust to low-level conditioning styles. Bernoulli(1/2) — Symmetric Binary Environments. Three scenes isolate binary symmetric outcomes with equiprobable distributions (P = 0.5): (i) Ball fork: A ball rolls down a symmetric, bifurcated Y-shaped ramp, exiting via either the left or right channel; (ii) Walking: A human subject approaches a symmetric T-junction and executes either a left or right turn; (iii) Double pendulum: The horizontal orientation (left or right of the central vertical axis) of the outer mass is recorded at the final frame of the video. The latter frequently induces structural artefacts; rollouts are flagged as invalid if the model deforms the rigid, dual-rod mechanical geometry. Uniform Discrete — Dice, Cards, and Lottery. Three scenes evaluate uniform calibration over a discrete sample space of cardinality k: (i) Dice: A standard six-sided die is tossed into a bounded wooden box (k = 6); (ii) Cards: The top card is drawn from a face-down deck and scored strictly by its suit (k = 4); (iii) Lottery: A numbered ball is randomly ejected from a rotating tumbler into a transparent collection tube (k = 20). Cumulatively, these three setups probe calibration capabilities across scaling cardinalities. Known non-uniform — Roulette. We model an asymmetric multi-outcome distribution using a standard European roulette wheel. Rollouts are scored based on the final colour pocket where the ball settles, yielding an analytical reference distribution of P (red) = P (black) = 18/37 and P (green) = 1/37. This environment evaluates whether a model can accurately synthesise a known, skewed categorical distribution containing rare events. The conditioning prompt explicitly mandates that the ball must settle into a single numbered pocket; rollouts where the ball remains in motion or fails to settle represent a failure of text-prompt alignment and are categorised as unscorable. Reference Distributions and Dynamical Regime. Two of our scenes, the double pendulum and roulette, invoke reference distributions that presuppose chaotic operation. The double pendulum’s marginal spatial distribution approaches uniformity only at energy thresholds sufficient to enter chaotic trajectories (Levien & Tan, 1993; Shinbrot et al., 1992). Within this regime, the system’s short Lyapunov time ensures that unresolved microscopic variations in the initial frames are macroscopically amplified over the course of a multi-second video sequence. Similarly, the roulette reference 5

Table 1: Per-model video-generation settings. All models receive the same conditioning frames, text prompts, and 32 fixed seeds; only the settings below differ. Italic marks a value equal to the provider’s API default (whether we set it explicitly or left it unset); N/A means the model’s API does not expose that parameter. Per-model durations are the shortest length each provider offers; WAN-2.7 (3 s) and SeeDance-2.0 (4 s) accept an arbitrary integer duration and were fixed slightly above the API minimum. Model

ID

Duration

Resolution

Prompt exp.

Neg. prompt

Audio

WAN-2.7 SeeDance-2.0 HappyHorse-1.0 Veo 3.1 Runway Gen-4.5 Cosmos3-Super

wan-video/wan-2.7-i2v bytedance/seedance-2.0 alibaba/happyhorse-1.0 google/veo-3.1 runwayml/gen-4.5 nvidia/Cosmos3-Super (local)

3s 4s 3s 4s 5s 5.04 s†

720p 480p 720p 720p N/A 1280×704

yes N/A N/A N/A N/A N/A

"" N/A N/A unset N/A set

N/A off N/A on N/A N/A

† Cosmos3-Super: 121 frames @ 24 fps; steps=35, guidance_scale=6.0.

distribution assumes multiple erratic deflections off the boundary frets, which requires high initial velocity. We induce these chaotic operational regimes directly through the initial conditioning frame configurations and text prompts (e.g. starting the pendulum from an elevated potential energy state or appending “spinning rapidly” to the roulette prompt), though we do not explicitly compute the exact underlying energy thresholds. Chaotic operation alone, however, does not guarantee the symmetric Bernoulli(1/2) marginal we assign to the pendulum: a chaotic trajectory could still be biased towards a particular side of the vertical axis at a particular time. Therefore we verify the Bernoulli(1/2) marginal for our system by simulation. Integrating the full nonlinear double-pendulum equations of motion from the conditioning frame’s measured release state (θ1 ≈ 146◦ , θ2 ≈ 215◦ , length ratio l2 /l1 = 0.82, equal masses, released from rest), we propagate a dense ensemble of micro-perturbations (independent Gaussian noise of standard deviation 0.5◦ on each release angle) about that state—the sub-frame micro-state a video model cannot resolve from a single frame—with an adaptive RK45 integrator. The simulated marginal P (left) of the outer mass converges towards 0.50 as the trajectory mixes, staying within 0.05 of 0.50 across the range of model video lengths (t from 3 to 5 s); below roughly one second the outcome is still near-deterministic and has not yet mixed. The same construction—selecting a release configuration and reading off its simulated marginal—could equally yield scenes with analytically characterised but deliberately skewed references; we do not pursue this here, but note it as an exciting avenue for future work.

3.2

Generation Protocol

We evaluate six state-of-the-art image-to-video architectures: WAN-2.7 (Wan Team, Alibaba, 2025), SeeDance2.0 (ByteDance Seed, 2026), HappyHorse-1.0 (Alibaba Group, 2026), Veo 3.1 (Google DeepMind, 2025), Runway Gen-4.5 (Runway, 2025), Cosmos3-Super (Agarwal et al., 2026). For each scene–model combination, we produce 32 distinct video generations from identical inputs modulated only by the integer random seed passed to the respective model APIs, drawn from a fixed set of 32 unique seed integers representing the source of stochastic variance. All other configuration variables—including the set of conditioning frames and text prompts (detailed in Appendix G), generation duration, and target output resolution are held static per model (Table 1). Default sampling parameters (e.g. classifier-free guidance scales, internal prompt expansion, temperature, etc.) are maintained at the respective model defaults. We deliberately choose an ensemble size of N = 32 generations to balance statistical validity with computational feasibility. Evaluating a single architecture across all nine scenes requires 9 × 32 = 288 video generations, keeping the computational overhead on par with related video physics benchmarks such as PhysicsIQ (Motamed et al., 2025) and PhyGenBench (Meng et al., 2024). Figure 3 provides illustrative frame strips comparing a valid, scorable rollout against a structural simulation failure (spurious multi-object generation). 6

t = 0.0 s

t = 1.0 s

t = 2.0 s

t = 2.9 s

Scoreable HappyHorse, seed 0 (lands in bin 6)

Invalid Veo 3.1, seed 29 (multiple balls; unscoreable)

Figure 3: Frame strips from two physical Galton-board video generations illustrating the two outcome categories. Top: a scoreable video generation (HappyHorse): a single ball is released, falls through the peg array, and comes to rest in a single bin. Bottom: an invalid video generation (Veo 3.1): the model generates multiple balls during the simulation, leaving the outcome unscoreable. Failures of this type drive the scorability ρ metric below 1.

3.3

Outcome Extraction

Discrete outcomes are programmatically extracted from the synthesised videos using Gemini 3.1 Pro (Google, 2026). We deploy scene-specific prompts that instruct the vision-language model (VLM) to map the visual trajectory to a natural language token matching the target outcome space (e.g. “left”, “right”, an explicit bin index, a suit, or a colour pocket). If the terminal state is structurally ambiguous or unresolvable, the VLM is explicitly directed to return a null value. The comprehensive prompt suite is provided in Appendix H. The null assignment reflects failure modes observed during prompt development: • Roulette: The ball must visually come to rest entirely within a single pocket boundary, if the ball disappears, floats outside the track, or balances statically on an intersecting fret are marked null. • Lottery: Exactly one distinct ball must occupy the transparent exit tube. • Dice: A single, identifiable die face must be oriented parallel to the upper perspective plane. • Cards: The top card must be visually drawn and resolved as a standard, recognizable suit. • Double Pendulum: The structural integrity of the rigid dual-rod mechanism must be preserved without topological fractures or geometric morphing throughout the video duration. To reduce variance from VLM imprecision, we run each evaluation query three times at the model’s default sampling temperature and aggregate by majority vote (ties resolved to null). A video generation is classified as valid if the majority vote returns an unambiguous outcome and invalid otherwise. We validate the extraction pipeline by comparing against human annotation on a stratified sample of 18 video generations per scene (162 videos total), achieving 152/162 = 93.8% overall agreement with a low 3.7% false-null rate. Appendix F reports the per-scene breakdown. 3.4

Metrics

We evaluate each model along two orthogonal axes (Section 3.4.1), test the statistical significance of the resulting metrics, and aggregate them into a single score via the algorithm in Section 3.4.2. 3.4.1

Evaluation Axes

To comprehensively evaluate video world models, we decouple performance into two orthogonal axes: scorability (ρ) and (Total Variation Distance). 7

Scorability ρ. The scorability metric ρ quantifies the proportion of scorable generations within an ensemble: ρ=

nvalid , N

(2)

where N = 32 denotes the total number of generated video rollouts. It is a necessary but insufficient condition for a faithful physical simulation. A model can achieve a perfect score of ρ = 1 while exhibiting severe calibration failures. Furthermore, a scorable generation does not inherently guarantee that the generated video is physically plausible. Effect size: total variation distance (TV). Let p̂valid denote the empirical categorical distribution computed over the nvalid successful video generations for a given scene–model combination. We quantify the calibration error as the Total Variation Distance (TVD) between the empirical distribution p̂valid and the analytical reference distribution p0 : TV(p̂valid , p0 ) = 21

k X

|p̂valid,i − p0,i | ∈ [0, 1],

(3)

i=1

reading as the fraction of probability mass misallocated (Levin & Peres, 2017). We select TVD rather than an Earth Mover’s (Wasserstein) Distance because many evaluation scenes feature unordered, categorical sample spaces, such as card suits or roulette colours, where an inter-outcome distance metric cannot be naturally defined. Although TV ignores the ordinal spatial proximity present in the Galton board, it provides a unified, cross-scene metric that safely accommodates our purely nominal categories. At finite sample sizes, e.g. nvalid generations, the plug-in estimator for TVD is inherently biased upward: a perfectly calibrated model produces non-zero TV from sampling noise alone, and this null floor grows with the number of outcomes k. We do not correct TV per scene–model combination. Therefore, raw per-sample TV values are comparable within one scene only and not across scenes. The benchmark-aggregate score below makes the cross-scene comparison explicit via floor subtraction. Because this estimate is evaluated over the subset of scorable generations, it represents the outcome distribution conditional on validity, a framework aligning with the true target distribution under a Missing Completely At Random (MCAR) assumption (see Section 5 for an extended justification). 3.4.2

Statistical Inference and Aggregation

Significance. While TV provides an interpretable effect size, verifying whether an observed divergence is statistically distinguishable from pure sampling noise requires an independent inferential framework. We evaluate the null hypothesis H0 , that the generated outcomes are sampled directly from the true physical reference distribution, using a Monte Carlo exact χ2 test applied to the valid rollout subset. We deliberately treat both as distinct objects: TV describes the size of the miscalibration and never carries a significance marker, while the χ2 test is reported as a verdict (and a p-value) and is never read as a magnitude in the main text. The χ2 test exhibits higher sensitivity than TV to deviations on low-probability bins, e.g. the rare green roulette pocket. Hence, the two quantities are complementary rather than redundant. For each scene–model combination we compute Pearson’s χ2 statistic on the valid generations against the reference distribution: k X (Oi − Ei )2 χ2 = , (4) Ei i=1 where Oi and Ei represent the observed and expected counts, respectively. We obtain the p-value via Monte Carlo: 50,000 independent samples of size nvalid are drawn from the reference distribution and the p-value is defined as the fraction of replicates whose χ2 statistic equals or exceeds the observed value. Utilising an asymptotic χ2k−1 reference would be unreliable at N = 32 because Cochran’s expected-count rule is violated on four scenes (Cochran, 1954). Specifically, the Galton boards yield Ei < 1 in the tail bins, the green roulette bin yields Egreen ≈ 0.86, and the uniform lottery bounds all bins at Ei ≈ 1.6. Constructing an exact empirical distribution via Monte Carlo (Hope, 1968) circumvents that issue. Each compound Monte Carlo p-value is the add-one estimator p̂ = (b + 1)/(M + 1) of Hope (1968), a binomial proportion over M = 50,000 8

i.i.d. draws under H0 . Its corresponding standard error is formalised as: r p(1 − p) SE(p̂) = . (5) M p p It is bounded at its maximum p = 0.5, where SE = 0.25/50,000 ≈ 0.0022, and falls to 0.0475/50,000 ≈ 0.00097 near p = 0.05. Therefore, it resolves p-values far more finely than the α = 0.05 decision threshold requires. Because each of the scene–model cells is tested separately, we additionally control the false discovery rate across the family of testable cells using the Benjamini–Hochberg procedure at α = 0.05 (Benjamini & Hochberg, 1995). The verdicts for each scene–model combination are reported after this correction, and Appendix D lists the raw, Benjamini–Hochberg, and Bonferroni p-values for every cell. Benchmark score: floor-referenced aggregate. To establish an aggregate leaderboard ranking, the per-scene–model TV does not suffice: TV alone neither goes to zero at calibration nor mixes commensurably across scenes of different k. To resolve this, we define a per-scene–model excess by subtracting the expected sampling noise floor: TVD − EH0 [TVD] esm = , (6) 1 − EH0 [TVD] where the per-scene–model null floor EH0 [TV] has a closed form derived in Appendix A. The floor is therefore a deterministic function of (p0 , nvalid ) that the released code evaluates at each scene–model’s nvalid . We define the per-scene–model score as ssm : ssm = (1 − ρ) + ρ · esm . (7) Our final benchmark metric mean normalised Total Variation (mnTV) is given by the mean of ssm across all nine scenes, bounded above by 1, and approaches zero when every scene is both fully scoreable and calibrated up to sampling noise. We compute bootstrap confidence intervals for mnTV to enable fair comparison of models. We emphasise that mnTV is a one-number summary intended to sit alongside the per-scene grid, not in place of it: averaging over heterogeneous scenes masks per-scene reversals, e.g. a model that is best on the ball fork can be most collapsed on lottery, and the score conflates the two axes (scorability and calibration) that the rest of the paper treats as orthogonal. A model with mnTV near zero would be both highly scorable and calibrated — producing scoreable videos whose outcome distribution matches the reference up to sampling noise.

4

Results

We evaluate six frontier architectures on CaliBench. Table 2 reports the per-scene–model scorability ρ and Total Variation Distance (TVD) across all nine environments as well as the aggregate Mean Normalised Total Variation (mnTV). While SeeDance-2.0 yields the lowest overall calibration penalty (mnTV = 0.39), overlapping bootstrap intervals indicate that models resolve into broad performance tiers rather than a strict ranking. Crucially, all empirical aggregates sit far above the baseline of a perfectly calibrated operator (mnTV ≈ 0). Table 3 reports p-values from our Monte Carlo χ2 test. While TVD measures how far the empirical distribution sits from the reference, p < 0.05 in the χ2 test means that sampling noise alone is an implausible explanation for that gap at a 95% confidence level. These metrics are complementary: because the χ2 statistic normalises deviations by expected counts, it exhibits high sensitivity to anomalies in low-probability bins that contribute minimally to the absolute TVD sum. Neither the differing output resolutions nor video durations account for these deficits: regenerating SeeDance-2.0 at 720p, and re-running the applicable models at a uniform 5 s, both leave the calibration results essentially unchanged (Appendix I). The main finding is heterogeneity: across the eight informative scenes no model dominates, and the per-scene best swaps between models (Runway Gen-4.5 leads on the animated Galton board, SeeDance-2.0 on the physical board and lottery, HappyHorse on the ball fork, Cosmos3-Super on walking and dice, and Veo 3.1 on roulette; cards is a three-way tie in TV). Despite these localised strengths, within most scenes every model is significantly miscalibrated; the few unrejected cells lie inside or close to their null band and should be read together with each scene’s detectability floor (Appendix C). 9

Table 2: Per-scene–model descriptive results across all nine evaluation environments. We evaluate the Total Variation Distance TV from the scene’s reference distribution on the valid sample (lower is better, range [0, 1]). †: small nvalid (≤3), so per-scene–model distributional statements are unreliable. —: 0 valid video generations. Bold: lowest TV per scene among non-† cells. mnTV footer: [2.5%, 97.5%] bracketed values are the bootstrap interval from 50,000 resamples within each scene–model combination (Appendix A). Mean Normalised Total Variation (mnTV): mean of ssm across all nine scenes, scene–model decomposition in Appendix A. For reference, a perfectly calibrated, fully scoreable model has mnTV ≈ 0 (central-95% range ≈ [−0.04, 0.04] over 50,000 synthetic draws); the observed mnTVs in the footer all sit far above this null range. Pendulum is daggered for all six models — none yields more than three valid generations (Cosmos3-Super three, SeeDance-2.0 two, the rest at most one): a universal structural failure. WAN-2.7

SeeDance-2.0 HappyHorse

ρ↑

TV ↓

ρ↑

TV ↓

ρ↑

TV ↓

ρ↑

TV ↓

ρ↑

TV ↓

ρ↑

TV ↓

Binomial(10, 1/2) — 11 bins Physical board 0.69 0.71 Animated board 0.91 0.75

0.59 0.88

0.27 0.66

0.88 0.91

0.58 0.68

0.03 0.97

0.99† 0.49

0.56 0.88

0.37 0.20

0.19 0.03

0.50 0.99†

Bernoulli(1/2) — 2 outcomes Ball fork 1.00 0.25 Walking 0.97 0.21 Pendulum 0.00 —

1.00 1.00 0.06

0.19 0.19 0.00†

0.94 1.00 0.00

0.00 0.12 —

0.97 1.00 0.03

0.24 0.50 0.50†

0.91 1.00 0.00

0.12 0.03 —

0.84 0.84 0.09

0.06 0.02 0.50†

Uniform{1, . . . , k} Dice (k = 6) 1.00 Cards (k = 4) 0.56 Lottery (k = 20) 0.88

1.00 0.91 0.75

0.42 0.50 0.40

0.97 0.38 0.78

0.80 0.50 0.95

1.00 0.34 0.59

0.83 0.50 0.90

0.94 0.03 0.62

0.40 0.75† 0.50

1.00 0.03 0.22

0.28 0.75† 0.70

Known non-uniform: P (red) = P (black) = 18/37, P (green) = 1/37 Roulette 0.69 0.26 0.84 0.12 0.97 0.19 0.66

0.11

0.22

0.12

0.38

0.47

Scene

mnTV ↓

0.68 0.58 0.69

0.58 [0.54,0.63] 0.39 [0.38,0.48]

Veo 3.1

Runway 4.5 Cosmos3-Super

0.54 [0.51,0.59] 0.65 [0.63,0.69] 0.48 [0.48,0.57]

0.63 [0.62,0.70]

Table 3: Monte Carlo p-values from the χ2 test against the reference distribution (50,000 null hypothesis (H0 ) replicates per scene–model combination, seed = 0). H0 : outputs are i.i.d. from the reference (calibrated). Values shown are raw p-values; we control the false discovery rate across the 44 computed tests with the Benjamini–Hochberg procedure at α = 0.05 (BH critical threshold p ≤ 0.0297; full raw and adjusted values in Appendix D). †: ≤3 valid video generations; —: 0 valid video generations. ‡: significant at raw α = 0.05 but not after Benjamini–Hochberg correction (q > 0.05). Scene

WAN-2.7

SeeDance

HappyHorse

Veo 3.1

Runway

Cosmos3-Super

Physical board 0.005 Animated board <0.001 Bernoulli(1/2) Ball fork 0.007 Walking 0.0297 Pendulum — Uniform discrete Dice (k = 6) <0.001 Cards (k = 4) <0.001 Lottery (k = 20) <0.001 Known non-uniform Roulette 0.004

0.088 <0.001

0.004 0.001

† 0.057

0.038‡ 0.002

0.013 †

0.051 0.051 †

1.000 0.214 —

0.011 <0.001 †

0.264 0.859 —

0.699 1.000 †

<0.001 <0.001 0.049‡

<0.001 0.008 <0.001

<0.001 <0.001 <0.001

<0.001 † 0.010

0.011 † 0.696

0.004

0.096

0.490

0.279

<0.001

Binomial(10, 1/2)

Applying the multiple-comparison correction (Section 3.4.2) leaves 28 of the 44 testable cells significant. The severe mode collapses (p < 0.001) survive any correction, including the far stricter Bonferroni bound (p ≤ 0.00114); two cells change verdict relative to an uncorrected threshold. Runway Gen-4.5 (p = 0.038 → q = 0.057) on the physical board and SeeDance-2.0 on lottery (p = 0.049 → q = 0.070) are no longer rejected, joining the small set of cells whose deviation from calibration is indistinguishable from sampling noise. Borderline cells elsewhere are unaffected — Cosmos3-Super on the dice, for instance, remains significant after correction (p = 0.011 → q = 0.019). The full raw and adjusted p-values are in Appendix D. To provide further intuition for the results presented in this section, Figure 4 shows empirical bin distributions for the Galton board scenes, overlaid on the reference binomial(10, 1/2) (for remaining distribution plots refer 10

Proportion

Proportion

Reference

1.00 0.75 0.50 0.25 0.00 Reference 1.00 1 3 5 7 9 11 Outcome 0.75 0.50 0.25 0.00 1 3 5 7 9 11

Outcome

WAN-2.7

WAN-2.7

SeeDance-2.0

SeeDance-2.0

HappyHorse

HappyHorse

Veo 3.1

Runway Gen-4.5 Cosmos3-Super

Veo 3.1

Runway Gen-4.5 Cosmos3-Super

1 3 5 7 9 11

1 3 5 7 9 11

1 3 5 7 9 11

1 3 5 7 9 11

1 3 5 7 9 11

1 3 5 7 9 11

1 3 5 7 9 11

1 3 5 7 9 11

Outcome

Outcome

Outcome

Outcome

Outcome

Outcome

Outcome

Outcome

1 3 5 7 9 11

1 3 5 7 9 11

1 3 5 7 9 11

1 3 5 7 9 11

Outcome

Outcome

Outcome

Outcome

Figure 4: Empirical bin distributions for each model (blue) vs. ground-truth Binomial(10, 1/2) (orange). Top: physical board. Bottom: animated board. Most models over-concentrate near the central bin, far exceeding the expected peak probability of 24.6%; on the animated board SeeDance-2.0 instead places excessive probability mass on the left tail (bin 1 and 2). to Appendix B). Figure 5 maps the evaluation space by plotting scorability ρ against TVD alongside a shaded envelope tracking the expected 95th percentile of null sampling noise. Because this visual boundary reflects unweighted TVD thresholds, it functions as a purely descriptive baseline. Consequently, individual cells with generative errors concentrated entirely on low-probability bins can trigger an inferential rejection via the χ2 test while visually remaining within the shaded geometric envelope.

5

Discussion, Limitations, and Future Work

Discussion. Our evaluation reveals that the dominant failure mode in frontier world models is systemic probability mass over-concentration, often manifesting as total mode collapse despite high individual sample realism. This behaviour stems from a mixture of shared, a priori biases (e.g. over-representing the central modal bin of Galton boards) and localised conditioning cues present in the initial image (Appendix K). Crucially, performance fails to generalise across distribution families: models nearing perfect calibration on binary symmetric environments degrade sharply on high-cardinality or highly skewed targets, proving that physical calibration is governed by factors deeper than the mathematical structure of the target reference distribution. These findings expose severe risks for utilising world models to stress-test autonomous systems in safety-critical domains (e.g. autonomous driving or robotics), where simulating the full variance of plausible futures is mandatory. Relying on current uncalibrated priors yields overly deterministic rollouts that mask critical tail risks. Mechanistically, classifier-free guidance (CFG) acts as a critical dial; our sweeps (Appendix J) show that lowering CFG strength mitigates over-concentration but directly trades away structural scorability (ρ), uncovering a fundamental optimisation tension between rendering validity and distributional calibration. Limitations. Several limitations apply. We evaluate image-to-video pipelines exclusively, leaving text-tovideo stochasticity for future work. Additionally, our automated VLM outcome extraction introduces a minor noise floor, though human validation verifies a 93.8% annotation agreement. Mechanistically, our TVD metric conditions on scorable rollouts; while localised rendering pathologies (e.g. object merging) drive unscorability, validation suggests these failures are independent of the physical trajectories the dynamics resolve toward. Finally, CaliBench evaluates marginal outcome distributions rather than full continuous trajectory spaces. Because several evaluated models are closed-weight commercial systems, we cannot audit their training corpora for depictions of our scene families; we note, however, that memorised process statistics for an unbiased dataset would be expected to improve calibration on these canonical textbook processes. Future Work. A natural path forward is integrating distributional verification directly into model pre-training or alignment phases (Yuan et al., 2026). While multi-sample evaluation is computationally intensive—requiring 1,728 video rollouts and 5,184 VLM queries across our current matrix—it remains highly viable for low-step or autoregressive architectures, providing a critical engineering pathway toward calibrated physical simulators. The protocol described in this paper could be used to create larger benchmarks in the future, with a 11

Physical Galton board

TV (lower = better)

1.0

0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

Walking T-junction

TV (lower = better)

1.0

0.0

Double pendulum

1.0 0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

Cards

1.0

0.0

Lottery

1.0 0.8

0.8

0.6

0.6

0.6

0.4

0.4

0.4

0.2

0.2

0.2

0.0

0.2

0.4

0.6

Scorability

0.8

WAN-2.7 SeeDance-2.0

1.0

0.0

0.0

HappyHorse

0.2

0.4

0.6

Scorability Veo 3.1

Roulette

1.0

0.8

0.0

Dice

1.0

0.8

0.0

Ball fork

1.0

0.8

0.0

TV (lower = better)

Animated Galton board

1.0

0.8

1.0

0.0

Runway 4.5

0.0

0.2

0.4

0.6

Scorability

0.8

1.0

Cosmos3-Super

Figure 5: Scorability vs. total variation distance TV for all nine scenes. The bottom-right corner of each panel (high scorability, low TV) is the calibration target; no evaluated model approaches it consistently across scenes. The shaded band per panel is the central-95% envelope of TV under perfect calibration at nvalid = N , shown as a descriptive aid for interpreting the magnitude of TV relative to sampling noise — it is not a rejection region, and a marker’s position relative to the band is not equivalent to the calibration verdict (which comes from the Monte Carlo exact χ2 test). The band is drawn at nvalid = N = 32 per panel and does not directly characterise sampling noise for cells with ρ < 1 (those cells have a wider envelope, not shown). Because of the upward bias of the TV estimator, the shaded null band does not necessarily include TV = 0. larger number of reference scenes — we believe the current number of scenes provides a pragmatic balance between computational expense and evaluation fidelity. A natural extension is testing whether calibration on CaliBench’s closed-form scenes predicts distributional fidelity on open-world footage, where reference distributions must instead be estimated empirically from large observational samples.

6

Conclusion

In this work, we introduced CaliBench, an evaluation framework designed to benchmark the statistical calibration of generative video world models within discrete, physically meaningful outcome spaces. By evaluating six state-of-the-art models across nine classic stochastic environments, we demonstrated that 12

current architectures routinely fail to function as reliable stochastic simulators. Even when generating highly realistic, physically plausible individual trajectories, these models systematically suffer from severe probability mass over-concentration and mode collapse. By formalising the distinct properties of scorability and distributional calibration, CaliBench exposes critical vulnerabilities in the stochastic dynamics of frontier world models, establishing a rigorous mathematical baseline necessary for developing reliable, safe, and truly calibrated physical simulation pipelines.

Acknowledgements The authors gratefully acknowledge the support of the entire Odyssey team.

References Niket Agarwal, Arslan Ali, Jon Allen, Martin Antolini, Adeline Aubame, Alisson Azzolini, Junjie Bai, Maciej Bala, Yogesh Balaji, Josh Bapst, et al. Cosmos 3: Omnimodal world models for physical ai, 2026. URL https://arxiv.org/abs/2606.02800. Alibaba Group. Alibaba rolls out HappyHorse 1.0 in limited beta. alibaba-rolls-out-happyhorse-1-0-in-limited-beta/, 2026.

https://www.alizila.com/

Sanjeev Arora, Andrej Risteski, and Yi Zhang. Do GANs learn the distribution? Some theory and empirics. In International Conference on Learning Representations (ICLR), 2018. arXiv:1706.08224. Pietro Astolfi, Marlene Careil, Melissa Hall, Oscar Mañas, Matthew Muckley, Jakob Verbeek, Adriana Romero Soriano, and Michal Drozdzal. Consistency-diversity-realism pareto fronts of conditional image generative models, 2024. URL https://arxiv.org/abs/2406.10429. Yoav Benjamini and Yosef Hochberg. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society: Series B (Methodological), 57(1): 289–300, January 1995. ISSN 0035-9246. doi: 10.1111/j.2517-6161.1995.tb02031.x. Nick Bostrom. Are you living in a computer simulation? Philosophical Quarterly, 53(211):243–255, 2003. ByteDance Seed. Seedance 2.0: Advancing video generation for world complexity, 2026. URL https: //arxiv.org/abs/2604.14148. Harold Haodong Chen, Disen Lan, Wen-Jie Shu, Qingyang Liu, Zihan Wang, Sirui Chen, Wenkai Cheng, Kanghao Chen, Hongfei Zhang, Zixin Zhang, Rongjin Guo, Yu Cheng, and Ying-Cong Chen. TiViBench: Benchmarking think-in-video reasoning for video generative models, 2025. URL https://arxiv.org/abs/ 2511.13704. William G. Cochran. Some methods for strengthening the common χ2 tests. Biometrics, 10(4):417–451, 1954. Google. Gemini 3.1 Pro. https://blog.google/innovation-and-ai/models-and-research/ gemini-models/gemini-3-1-pro/, February 2026. Google DeepMind. Veo 3: A text-to-video generation system. Technical report, Google DeepMind, 2025. URL https://storage.googleapis.com/deepmind-media/veo/Veo-3-Tech-Report.pdf. Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS Workshop on Deep Generative Models and Downstream Applications, 2021. arXiv:2207.12598. A. C. A. Hope. A simplified Monte Carlo significance test procedure. Journal of the Royal Statistical Society, Series B, 30(3):582–598, 1968. Ziqi Huang, Yinan He, Jiashuo Yu, et al. VBench: Comprehensive benchmark suite for video generative models, 2024. URL https://arxiv.org/abs/2311.17982. 13

Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems (NeurIPS), 2019. arXiv:1904.06991. R. B. Levien and S. M. Tan. Double pendulum: An experiment in chaos. American Journal of Physics, 61 (11):1038–1044, 1993. doi: 10.1119/1.17335. David A. Levin and Yuval Peres. Markov Chains and Mixing Times. American Mathematical Society, 2nd edition, 2017. Dacheng Li, Yunhao Fang, Yukang Chen, Shuo Yang, Shiyi Cao, Justin Wong, Michael Luo, Xiaolong Wang, Hongxu Yin, Joseph E. Gonzalez, Ion Stoica, Song Han, and Yao Lu. WorldModelBench: Judging video generation models as world models, 2025. URL https://arxiv.org/abs/2502.20694. Jiahe Liu, Youran Qu, Qi Yan, Xiaohui Zeng, Lele Wang, and Renjie Liao. Fréchet video motion distance: A metric for evaluating motion consistency in videos. In International Conference on Machine Learning (ICML), 2024. arXiv:2407.16124. Ge Ya Luo, Gian Mario Favero, Zhi Hao Luo, Alexia Jolicoeur-Martineau, and Christopher Pal. Beyond FVD: Enhanced evaluation metrics for video generation quality. In International Conference on Learning Representations (ICLR), 2025. arXiv:2410.05203. Ziqi Ma, Mengzhan Liufu, and Georgia Gkioxari. Out of sight, out of mind? Evaluating state evolution in video world models, 2026. URL https://arxiv.org/abs/2603.13215. Zhiting Mei, Ola Shorinwa, and Anirudha Majumdar. How confident are video models? Empowering video models to express their uncertainty, 2025a. URL https://arxiv.org/abs/2510.02571. Zhiting Mei, Tenny Yin, Micah Baker, Ola Shorinwa, and Anirudha Majumdar. World models that know when they don’t know: Controllable video generation with calibrated uncertainty, 2025b. URL https://arxiv.org/abs/2512.05927. Fanqing Meng, Jiaqi Liao, Xinyu Tan, Wenqi Shao, Quanfeng Lu, Kaipeng Zhang, Yu Cheng, Dianqi Li, Yu Qiao, and Ping Luo. Towards world simulator: Crafting physical commonsense-based benchmark for video generation, 2024. URL https://arxiv.org/abs/2410.05363. Saman Motamed, Laura Culp, Kevin Swersky, Priyank Jaini, and Robert Geirhos. Do generative video models understand physical principles?, 2025. URL https://arxiv.org/abs/2501.09038. Akshat Rakheja, Aarsh Ashdhir, Aryan Bhattacharjee, and Vanshika Sharma. World consistency score: A unified metric for video generation quality, 2025. URL https://arxiv.org/abs/2508.00144. Eitan Richardson and Yair Weiss. On GANs and GMMs. In Advances in Neural Information Processing Systems (NeurIPS), 2018. arXiv:1805.12462. Runway. Introducing Runway Gen-4.5. introducing-runway-gen-4.5, 2025.

Runway Research, https://runwayml.com/research/

Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems (NeurIPS), 2018. arXiv:1806.00035. Troy Shinbrot, Celso Grebogi, Jack Wisdom, and James A. Yorke. Chaos in a double pendulum. American Journal of Physics, 60(6):491–499, 1992. S.H. Strogatz. Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry, and Engineering. CRC Press, 2024. ISBN 9780429676284. Thomas Unterthiner, Sjoerd van Steenkiste, Karol Kurach, et al. Towards accurate generative models of video: A new metric & challenges, 2019. URL https://arxiv.org/abs/1812.01717. 14

Wan Team, Alibaba. WAN: Open and advanced large-scale video generative models, 2025. URL https: //arxiv.org/abs/2503.20314. Kaining Ying, Hengrui Hu, Siyu Ren, Jiamu Li, Fengjiao Chen, Ziwen Wang, Xuezhi Cao, Xunliang Cai, and Henghui Ding. Wbench: A comprehensive multi-turn benchmark for interactive video world model evaluation, 2026. URL https://arxiv.org/abs/2605.25874. Jianhao Yuan, Xiaofeng Zhang, Felix Friedrich, Nicolas Beltran-Velez, Melissa Hall, Reyhane Askari-Hemmat, Xiaochuang Han, Nicolas Ballas, Michal Drozdzal, and Adriana Romero-Soriano. Inference-time physics alignment of video generative models with latent world models, 2026. URL https://arxiv.org/abs/ 2601.10553. Jingtong Yue, Ziqi Huang, Zhaoxi Chen, Xintao Wang, Pengfei Wan, and Ziwei Liu. Simulating the visual world with artificial intelligence: A roadmap, 2025. URL https://arxiv.org/abs/2511.08585. Dian Zheng, Ziqi Huang, Hongbo Liu, Kai Zou, Yinan He, Fan Zhang, Lulu Gu, Yuanhan Zhang, Jingwen He, Wei-Shi Zheng, Yu Qiao, and Ziwei Liu. Vbench-2.0: Advancing video generation benchmark suite for intrinsic faithfulness, 2025. URL https://arxiv.org/abs/2503.21755. Fengzhe Zhou, Jiannan Huang, Jialuo Li, Deva Ramanan, and Humphrey Shi. Pai-bench: A comprehensive benchmark for physical ai, 2025. URL https://arxiv.org/abs/2512.01989.

A

mnTV Definition and Per-Cell Decomposition

Our metric mnTV aggregates per-scene calibration into a single, noise-corrected scalar. The underlying scene–model components are defined as: TV(p̂valid , p0 ) − EH0 [TV] , 1 − EH0 [TV] ssm = (1 − ρ) + ρ · esm , esm =

(8) (9)

where the expected null term EH0 [TV] (Equation 14) accounts for finite-sample noise. Consequently, esm isolates the true excess calibration error attributable to model bias, with esm = 0 at perfect calibration (up to sampling noise) and unity representing total mass concentration on a single outcome that has zero probability under the reference distribution. Because the excess is signed, esm can be marginally negative for cells better-calibrated than the noise floor. Unscorable generations are penalised via the (1−ρ) term which reflects the maximum distributional divergence since the physical reference distribution places zero probability mass on the null outcome. Because this term dominates the score on the null-heavy scenes (cards, roulette, lottery, pendulum), it is possible that if the models’ scorability were improved our conclusion about the models’ calibration could significantly change. Therefore we show in Appendix E the effect of bounding each cell’s TV under best and worst-case null imputation. The final benchmark scalar is computed as the arithmetic mean across all evaluated environments: mnTV =

1

NX scenes

Nscenes

c=1

ssm ≤ 1.

(10)

This formulation explicitly includes the chaotic double pendulum environment, which contributes approximately 1 − ρ for every model because the ρ esm term is negligible: no model produces more than three valid samples, too few to yield an informative esm . Crucially, mnTV values are inherently conditioned on the specific sample size N and the chosen VLM outcome extraction protocol. To preserve direct baseline comparability with the leaderboard tracking established in this work, these experimental hyperparameters must remain invariant. We note that mnTV aggregates nine heterogeneous scenes characterised by distinct reference distributions and cardinalities; per-scene performance reversals are expected, i.e. a model achieving strong calibration in 15

one environment may exhibit total mode collapse in another. Because the score conflates the two orthogonal axes of scorability and calibration, we report mnTV strictly as a convenient one-number ordering tool for benchmark users, rather than a substitute for the comprehensive per-scene evaluation grid detailed in Table 2. Null mnTV Baseline. To ground the empirical mnTV scores presented in Table 2, we establish an analytical baseline for an idealised, perfectly calibrated, and fully scorable operator. A Monte Carlo simulation of 50,000 such configurations—setting ρ = 1 across all environments and drawing N = 32 outcomes per scene–model cell i.i.d. from the true reference distribution p0 —yields a null distribution centred on zero, with mean ≈ 0, median ≈ 0, and a central 95% interval of ≈ [−0.04, 0.04]. Because the excess term esm is signed and the null floor EH0 [TV] is computed exactly in closed form, TV − EH0 [TV] is exactly mean-zero under calibration, so a perfectly calibrated model scores zero in expectation. All six evaluated frontier models exhibit an mnTV that sits far above this null range. The lowest empirical error observed is SeeDance-2.0 at 0.39, well above the sampling-noise floor, confirming that the observed calibration deficits cannot be explained by finite-sample fluctuations. This null distribution serves as the reference baseline reported in the caption of Table 2. Table 4 reports the resulting per-scene–model scores ssm , with the aggregate mnTV in the bottom row. Bootstrap Uncertainty Intervals. The bracketed values reported in the mnTV row of Table 2 denote central 95% percentile intervals computed via 50,000 bootstrap iterations. Because the specific suite of nine scenes defines the benchmark itself rather than a random draw from an environment population, the scene composition is held static across resamples. Instead, the bootstrap scheme isolates finite-sample variance within each scene–model configuration by resampling all N = 32 video generations with replacement from the cell’s empirical multinomial over the valid outcome bins plus a null (unscorable) category, so that both the scorability ratio ρ and nvalid vary across replications. For each bootstrap replication, we recompute ρ, TV, and the corresponding excess error esm using the closed-form null floor EH0 [TV] re-evaluated at the resampled nvalid . This uncertainty interval thus quantifies the sensitivity of mnTV purely to the finite allocation of N = 32 video generations per cell. We report these empirical quantiles directly. Table 4: Per-scene–model score ssm (Eq. 9), with mnTV (mean across nine scenes) in the bottom row. Lower is better; bounded above by 1, with 0 the calibrated expectation (small negatives occur for cells better-calibrated than the sampling-noise floor). Bold: lowest per scene; mnTV: lowest overall. Scene

WAN-2.7

Binomial(10, 1/2) Physical board 0.75 Animated board 0.73 Bernoulli(1/2) Ball fork 0.19 Walking 0.17 Pendulum 1.00 Uniform discrete Dice (k=6) 0.62 Cards (k=4) 0.72 Lottery (k=20) 0.59 Known non-uniform Roulette 0.43 mnTV ↓

0.58

SeeDance

HappyHorse

Veo 3.1

Runway

Cosmos3-Super

0.44 0.63

0.54 0.65

1.00 0.39

0.54 0.13

0.84 1.00

0.13 0.13 0.92

-0.01 0.06 1.00

0.21 0.46 0.97

0.14 -0.04 1.00

0.14 0.10 0.94

0.31 0.48 0.30

0.77 0.77 0.94

0.80 0.78 0.90

0.33 0.97 0.51

0.15 0.97 0.78

0.19

0.15

0.35

0.76

0.77

0.39

0.54

0.65

0.48

0.63

Null floor: a closed-form, deterministic function. The per-scene–model excess error esm requires subtracting the expected null variation EH0 [TV]. This baseline depends exclusively on the scene’s true reference distribution p0 and the number of valid samples nvalid . Crucially, because a model’s specific nvalid is dynamically determined by the VLM’s automated filtration process, this baseline cannot be pre-computed as a static scalar constant. Evaluating the baseline requires a flexible, closed-form analytic expression parameterised by the realised value of nvalid . Let Xi denote the number of valid video generations whose extracted outcomes fall within bin i, such that the empirical probability is p̂valid,i = Xi /nvalid . Substituting this expression into the definition of Total Variation Distance (TV) and factoring the scalar sample count out of the absolute value yields: X X Xi 1 TV(p̂valid , p0 ) = 12 Xi − nvalid p0,i . (11) nvalid − p0,i = 2nvalid i

i

16

By the linearity of expectation, this summation decomposes regardless of any joint dependencies between the individual bin counts Xi : X   1 EH0 |Xi − nvalid p0,i | . (12) EH0 [TV] = 2nvalid i

Under the null hypothesis H0 , the joint distribution (X1 , . . . , Xk ) follows a multinomial distribution parameterised by (nvalid , p0 ). Consequently, the marginal distribution of any single bin simplifies to a binomial distribution, Xi ∼ Binomial(nvalid , p0,i ), with an expected mean of nvalid · p0,i . Each individual term in Equation (12) therefore represents the mean absolute deviation about the mean of a binomial random variable. This deviation possesses an exact, deterministic closed form:   n E[|X − np|] = 2 (1 − p)n−⌊np⌋ p⌊np⌋+1 (⌊np⌋ + 1) , (13) ⌊np⌋ + 1 where X ∼ Binomial(n, p) (De Moivre’s formula). Substituting this identity back into Equation (12) and simplifying yields the final expression for the analytical noise floor:   nvalid 1 X EH0 [TV] = (mi + 1) p mi +1 (1 − p0,i ) nvalid −mi , mi := ⌊nvalid p0,i ⌋, (14) nvalid i mi + 1 0,i providing a completely deterministic evaluation framework defined purely by (p0 , nvalid ). In contrast, the null fluctuation envelopes illustrated in Figure 5 represent the specific quantiles of the sampling distribution of TV(p̂, p0 ) under H0 . Because these high-dimensional boundary conditions lack a comparably clean analytic closed form when the cardinality exceeds two (k > 2), we estimate and plot them via 50,000 Monte Carlo simulation trajectories using a fixed pseudo-random seed strictly for visualisation purposes. The core benchmark metric mnTV relies exclusively on the exact analytic formulation in Equation (14) and remains completely independent of stochastic sampling noise.

B

Empirical Distributions for All Scenes

We visualise the empirical outcome distributions for each model (blue) against the reference distribution (orange) in Figures 6–12, analogous to Figure 4 for the Galton board scenes.

Proportion

Reference 0.6 0.4 0.2 0.0

left

t

righ

Outcome

WAN-2.7

left

t righ

Outcome

SeeDance-2.0

left

t

righ

Outcome

HappyHorse

left

righ

t

Outcome

Veo 3.1

left

righ

t

Outcome

Runway Gen-4.5 Cosmos3-Super

left

righ

t

Outcome

left

t

righ

Outcome

Figure 6: Ball fork scene. Ground truth: Bernoulli(1/2) (uniform over left/right). Most models are close to calibrated; small TV values visible as mild left/right imbalance.

C

Statistical Power of the χ2 Test

At a baseline allocation of N = 32 video generations, a failure to reject the null hypothesis at a significance level of α = 0.05 should not be interpreted as definitive evidence of physical calibration if the underlying statistical test lacks sufficient power to detect moderate distributional deviations. To quantify this boundary, we conduct a statistical power analysis to establish the minimum detectable discrepancy each scene configuration can reliably identify. 17

Proportion

1.0

Reference

WAN-2.7

SeeDance-2.0

HappyHorse

Veo 3.1

Runway Gen-4.5 Cosmos3-Super

0.5 0.0

left

t

left

righ

Outcome

t righ

Outcome

left

t

left

righ

Outcome

righ

t

left

Outcome

righ

t

left

Outcome

righ

t

left

Outcome

t

righ

Outcome

Figure 7: Walking (T-junction) scene. Ground truth: Bernoulli(1/2). Similar to the ball fork: most models show mild directional imbalance.

Proportion

1.0

Reference

SeeDance-2.0

Veo 3.1

Cosmos3-Super

0.5 0.0

left

righ

t

Outcome

left

t

left

righ

Outcome

t

righ

Outcome

left

t

righ

Outcome

Figure 8: Double pendulum scene. Ground truth: Bernoulli(1/2). Only models with at least one valid video generation appear (SeeDance-2.0 with two, Veo 3.1 with one, and Cosmos3-Super with three; WAN, HappyHorse, and Runway produced none), so panels reflect very few samples. For each scene, we determine the minimum contamination magnitude required for the χ2 goodness-of-fit test to reject the null hypothesis with a probability of at least 80%. We define a single-parameter contamination family over the discrete probability distributions as: pϵ = (1 − ϵ)p0 + ϵ δj ∗ ,

(15)

which mixes the true analytical reference distribution p0 with a Dirac delta point mass δj ∗ concentrated entirely on a single outcome j ∗ . The contaminated target index j ∗ is chosen to reflect each scene’s empirically dominant failure mode, e.g. bin 6 for the Galton boards, green for the roulette wheel, and an arbitrary category for symmetric binary environments. For a given contamination intensity ϵ, we sample N outcomes from pϵ , evaluate the empirical rejection rate of the χ2 test against p0 across simulated trials, and isolate the smallest ϵ at which the power reaches 80%. We report these boundaries as the minimum detectable Total Variation Distance (TVD). Table 5 crossreferences these thresholds at N = 32 (the nominal sample size when ρ = 1) and N = 16 approximating sample degradation when the scorability ratio ρ ≈ 0.5. The complete sensitivity profiles across a continuous range of sample allocations are illustrated via the power curves in Figure 13. Implication. Every non-rejection in Table 3 must be read against its corresponding scene-specific sensitivity threshold in Table 5. In environments characterised by low detection thresholds e.g. roulette and binary forks, a non-rejection is highly informative, indicating that the generative trajectory matches the target distribution within a narrow margin. Conversely, in high-cardinality or highly structured spaces e.g. Galton boards, dice, cards, and lotteries, moderate miscalibration can remain undetected at N = 32. In these regimes, a non-rejection provides weaker evidence of physical calibration. In practice, however, our framework successfully rejects model calibration across a wide array of environments, demonstrating that the benchmark 18

Proportion

1.00 0.75 0.50 0.25 0.00

Reference

WAN-2.7

SeeDance-2.0

HappyHorse

Veo 3.1

1 2 3 4 5 6

1 2 3 4 5 6

1 2 3 4 5 6

1 2 3 4 5 6

1 2 3 4 5 6

Outcome

Outcome

Outcome

Outcome

Outcome

Runway Gen-4.5 Cosmos3-Super

1 2 3 4 5 6

Outcome

1 2 3 4 5 6

Outcome

Proportion

Figure 9: Dice scene. Ground truth: Uniform{1, . . . , 6}. Mode collapse is severe for several models, with Veo 3.1 always producing the same face.

Reference

WAN-2.7

0.0 rts ds bs es heaiamon cluspad d

rts ds bs es heaiamon cluspad d

1.0

SeeDance-2.0

HappyHorse

Veo 3.1

rts ds bs es heaiamon cluspad d

rts ds bs es heaiamon cluspad d

Runway Gen-4.5

Cosmos3-Super

rts ds bs es heaiamon cluspad d

rts ds bs es heaiamon cluspad d

0.5

Outcome

Outcome

rts ds bs es heaiamon cluspad d

Outcome

Outcome

Outcome

Outcome

Outcome

Figure 10: Cards scene. Ground truth: Uniform over 4 suits. Scorability failures dominate: five of six models have ρ < 0.6. SeeDance-2.0 retains the highest scorability (ρ = 0.91) and has the lowest TV among models with ρ > 0.5 (TV = 0.50, still significantly miscalibrated). possesses sufficient statistical power to expose the calibration deficits characteristic of current state-of-the-art video generation architectures.

D

Multiple-Comparison Correction

Table 3 evaluates 54 distinct scene–model configurations, 44 of which yield a sufficient number of valid video generations to compute a finite χ2 goodness-of-fit p-value. The remaining ten excluded configurations consist of the six near-empty chaotic pendulum cells, the single-valid physical-board cell (Veo 3.1), the intersection of the animated-board environment with Cosmos3-Super, and the two single-valid cards cells (Runway Gen-4.5 and Cosmos3-Super). To account for multiple comparisons across this family of 44 simultaneous statistical tests, we report two distinct adjustments in Table 6: Benjamini–Hochberg (BH) adjusted p-values (q-values) to control the False Discovery Rate (FDR), and Bonferroni-corrected p-values to control the Family-Wise Error Rate (FWER). At a nominal significance level of α = 0.05, the BH procedure rejects 28 of the 44 null hypotheses, corresponding to a critical raw significance threshold of p ≤ 0.0297. Conversely, the more conservative Bonferroni correction rejects only those configurations exhibiting a raw p ≤ 0.00114. These two adjustments effectively bracket our core conclusions: every environment characterised by severe mode collapse—where the raw p < 0.001—is rejected under both correction paradigms. The only statistical outcomes sensitive to the choice of multipletesting correction are marginal cases where the raw p ≈ 0.05 (Runway Gen-4.5 on the physical board and SeeDance-2.0 on lottery, denoted by ‡ in Table 3), which fail to be rejected under the BH procedure. Because these marginal cases lie within or immediately adjacent to their respective empirical detectability thresholds (see Appendix C), we do not interpret them as definitive evidence of physical miscalibration.

E

MCAR Null-Sensitivity Analysis

For the four null-heavy scenes (cards, roulette, lottery, pendulum) a large fraction of generations are unscoreable, so the null-extended term (1 − ρ) dominates the reported TV. To understand how the models’ calibration could change if the scorability of the models were improved we bound each cell’s TV under the 19

Proportion

1.00 0.75 0.50 0.25 0.00

Reference

WAN-2.7

SeeDance-2.0

HappyHorse

Veo 3.1

1 4 7 10131619

1 4 7 10131619

1 4 7 10131619

1 4 7 10131619

1 4 7 10131619

Outcome

Outcome

Outcome

Outcome

Outcome

Runway Gen-4.5 Cosmos3-Super

1 4 7 10131619

Outcome

1 4 7 10131619

Outcome

Proportion

Figure 11: Lottery scene. Ground truth: Uniform{1, . . . , 20}. High cardinality exposes severe collapse: probability mass concentrates on a small number of ball numbers across all models.

Reference

WAN-2.7

SeeDance-2.0

HappyHorse

Veo 3.1

red black green

red black green

red black green

red black green

red black green

Runway Gen-4.5 Cosmos3-Super

0.6 0.4 0.2 0.0

Outcome

Outcome

Outcome

Outcome

Outcome

red black green

Outcome

red black green

Outcome

Figure 12: Roulette scene. Ground truth: P (red) = P (black) = 18/37, P (green) = 1/37. Among models with non-trivial scorability, green is over-represented relative to the reference 2.7% in five of six (WAN: 14%, SeeDance: 15%, Veo 3.1: 5%, Runway: 14%, Cosmos3-Super: 50% on 6/12 valid); HappyHorse is the exception, producing no green outcomes (0/31). two extreme missing-completely-at-random (MCAR) imputations: assigning the N − nvalid missing outcomes so as to minimise (best) and to maximise (worst) the full-N total variation from the reference. Table 7 reports both bounds alongside the reported charge TVrep . Our analysis shows that reducing the number of unscorable outcomes could significantly alter the reported mnTV for the models — depending upon how the previously unscorable outcomes would be scored. That said, even under the most favourable (best-case) imputation every model’s mnTV stays above the calibration noise floor (minimum 0.08 vs. the ≈ 0.04 null band), so the finding that all six models are miscalibrated is itself robust; what is sensitive is their magnitude and relative ordering.

F

VLM Extraction Validation

To quantify the reliability of the automated VLM outcome extraction pipeline, we evaluate its performance against manual human annotation on a stratified sample of generated videos. For each of the nine environments, we hand-label 18 video generations, yielding an evaluation corpus of 162 videos spanning the six architectures. The human annotator records either the specific physical outcome category or a null token for unscorable sequences. Using this ground-truth subset, we report three validation metrics per scene: • Exact Agreement: The proportion of sequences where the automated VLM label matches the human annotation identically, treating the null token as a distinct discrete state. • False-Null Rate (Type-I Error Ratio): The proportion of sequences annotated as valid by the human observer that the VLM incorrectly classified as null. This represents conservative pipeline invalidation that artificially deflates the observed scorability ratio ρ. • Missed-Null Rate (Type-II Error Ratio): The proportion of sequences annotated as unscorable (null) by the human observer that the VLM incorrectly assigned a definite physical outcome. This represents false positive extraction that introduces structural noise into the empirical distribution p̂valid . 20

Table 5: Minimum Total Variation Distance (TVD) required for the Monte Carlo χ2 test to reject the null hypothesis of perfect calibration with ≥ 80% power under an extreme-mode contamination alternative. Lower thresholds indicate higher test sensitivity within that environment. The N = 16 column models performance degradation in cells exhibiting a low scorability ratio (ρ ≈ 0.5). Scene

Min TV (N = 32)

Min TV (N = 16)

0.24 0.12 0.26 0.26 0.49 0.22

0.30 0.20 0.36 0.37 0.59 0.31

Ball / Walking / Pendulum Roulette Cards Dice Galton board Lottery

Galton board

0.8

0.6

0.6

0.6

N=8 N=16 N=24 N=32 80% power

0.0 0.0

0.2

0.4

TV

0.6

0.8

0.4 N=8 N=16 N=24 N=32 80% power

0.2 0.0 0.0

1.0

Cards

1.0

Power

0.8

0.4

0.2

0.4

TV

0.6

0.8

0.4

0.0 0.0

1.0

0.6

0.6

0.6

0.2 0.0 0.0

0.2

0.4

TV

0.6

0.8

1.0

Power

0.8

Power

0.8

N=8 N=16 N=24 N=32 80% power

0.4 N=8 N=16 N=24 N=32 80% power

0.2 0.0 0.0

0.2

0.4

TV

0.6

0.2

0.8

1.0

0.4

TV

0.6

0.8

1.0

Roulette

1.0

0.8

0.4

N=8 N=16 N=24 N=32 80% power

0.2

Lottery

1.0

Dice

1.0

0.8

0.2

Power

Ball / Walking / Pendulum

1.0

Power

Power

1.0

0.4 N=8 N=16 N=24 N=32 80% power

0.2 0.0 0.0

0.2

0.4

TV

0.6

0.8

1.0

Figure 13: Statistical power curves of the Monte Carlo χ2 test against total variation distance TV for each scene’s reference distribution, at four sample sizes. Curves use the contamination alternative described above; dashedline marks 80% power.

Results. We report the per-scene verification statistics in Table 8. The aggregate agreement between the automated VLM extraction pipeline and manual human annotation is 152/162 (≈ 93.8%), with two environments achieving perfect alignment (100%): the physical Galton board and dice. The lowest exact agreement occurs in cards (83%). The aggregate false-null rate is bounded at 4/107 (≈ 3.7%) of human-labelled valid outcomes, occurring in pendulum (two sequences where the model generated a scorable trajectory that the VLM conservatively rejected as null), cards (a card the VLM left unscored at the final frame), and lottery (a ball whose number the VLM could not read). Two classification errors occurred: in walking (a Cosmos3 sequence the human read as leftward but the VLM as rightward) and the baseline tracking error noted in ball-fork. Conversely, the missed-null rate constitutes 4/55 (≈ 7.3%) of human-annotated unscorable sequences, driven by unflagged 21

Table 6: Raw Monte Carlo χ2 p-values for all 44 tested scene–model cells (sorted ascending), with Benjamini– Hochberg adjusted p-values (qBH , FDR control) and Bonferroni-adjusted p-values (pBonf , FWER control) over the family of 44 tests. ‡ marks the cells significant at raw α = 0.05 but not after BH correction. Scene

Model

Animated board Walking Dice Dice Dice Dice Cards Cards Lottery Lottery Lottery Roulette Dice Cards Animated board Animated board Animated board Physical board Roulette Roulette Physical board Ball fork Cards Lottery Dice Ball fork Physical board Walking Physical board Lottery Ball fork Walking Animated board Physical board Roulette Walking Ball fork Roulette Roulette Lottery Ball fork Walking Ball fork Walking

SeeDance Veo 3.1 WAN-2.7 SeeDance HappyHorse Veo 3.1 WAN-2.7 SeeDance WAN-2.7 HappyHorse Veo 3.1 Cosmos3-Super Runway Veo 3.1 WAN-2.7 HappyHorse Runway HappyHorse WAN-2.7 SeeDance WAN-2.7 WAN-2.7 HappyHorse Runway Cosmos3-Super Veo 3.1 Cosmos3-Super WAN-2.7 Runway SeeDance SeeDance SeeDance Veo 3.1 SeeDance HappyHorse HappyHorse Runway Runway Veo 3.1 Cosmos3-Super Cosmos3-Super Runway HappyHorse Cosmos3-Super

raw p

qBH

pBonf

<0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 0.001 0.002 0.004 0.004 0.004 0.005 0.007 0.008 0.010 0.011 0.011 0.013 0.0297 0.038‡ 0.049‡ 0.051 0.051 0.057 0.088 0.096 0.214 0.264 0.279 0.490 0.696 0.699 0.859 1.000 1.000

<0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 0.003 0.003 0.003 0.004 0.009 0.009 0.009 0.009 0.014 0.015 0.018 0.019 0.019 0.021 0.047 0.057 0.070 0.070 0.070 0.076 0.114 0.121 0.262 0.314 0.323 0.552 0.751 0.751 0.900 1.000 1.000

<0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 <0.001 0.002 0.036 0.043 0.055 0.075 0.157 0.171 0.182 0.199 0.313 0.342 0.434 0.489 0.505 0.566 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000 1.000

rendering artefacts: two cards displaying conflicting suit symbols, a roulette pocket with discontinuous colouring, and a Galton ball suspended directly on a slot divider. Impact on results. Since we conduct human validation on a stratified subset, we cannot report the exact χ2 change every scene–model cell would undergo under full manual re-annotation. The effect of correcting a few labels on χ2 is small, so it can change a cell’s reject/non-reject verdict only when the cell’s p-value already lies close to the 0.05 threshold; cells with p far from 0.05 are unaffected. Given the overall 6.2% disagreement rate (Table 8), most scenes have at most one label disagreement, with 3 on the worst-case scene (cards). At this scale the headline findings—the severe mode-collapse cells at p < 0.001—are unaffected; the only fragile cells are Runway Gen-4.5 on the physical board and SeeDance-2.0 on lottery (marked ‡ in Table 3), which we already decline to interpret as evidence of miscalibration regardless of annotation accuracy.

G

Conditioning Text Prompts

The conditioning frames are shown in Figure 2. The text prompt passed alongside each conditioning frame is fixed per scene and reproduced below verbatim. 22

Table 7: MCAR sensitivity for the null-heavy scenes. For each cell we impute the N − nvalid nulls to minimise (best) and maximise (worst) total variation over the full N ; TVrep = (1 − ρ) + ρ TV(p̂, p0 ) is the reported null-extended charge. Scene

Model

ρ

TVbest

TVworst

TVrep

Cards Cards Cards Cards Cards Cards

WAN-2.7 SeeDance-2.0 HappyHorse-1.0 Veo 3.1 Runway Gen-4.5 Cosmos3-Super

0.56 0.91 0.38 0.34 0.03 0.03

0.219 0.406 0.000 0.000 0.000 0.000

0.656 0.500 0.562 0.656 0.750 0.750

0.766 0.547 0.812 0.828 0.992 0.992

Roulette Roulette Roulette Roulette Roulette Roulette

WAN-2.7 SeeDance-2.0 HappyHorse-1.0 Veo 3.1 Runway Gen-4.5 Cosmos3-Super

0.69 0.84 0.97 0.66 0.22 0.38

0.067 0.098 0.170 0.018 0.018 0.160

0.379 0.254 0.201 0.348 0.785 0.785

0.491 0.258 0.216 0.413 0.807 0.802

Lottery Lottery Lottery Lottery Lottery Lottery

WAN-2.7 SeeDance-2.0 HappyHorse-1.0 Veo 3.1 Runway Gen-4.5 Cosmos3-Super

0.88 0.75 0.78 0.59 0.62 0.22

0.581 0.225 0.731 0.494 0.244 0.150

0.706 0.463 0.950 0.900 0.594 0.794

0.731 0.550 0.961 0.941 0.688 0.934

Pendulum Pendulum Pendulum Pendulum Pendulum Pendulum

WAN-2.7 SeeDance-2.0 HappyHorse-1.0 Veo 3.1 Runway Gen-4.5 Cosmos3-Super

0.00 0.06 0.00 0.03 0.00 0.09

0.000 0.000 0.000 0.000 0.000 0.000

0.500 0.469 0.500 0.500 0.500 0.500

1.000 0.938 1.000 0.984 1.000 0.953

Model WAN-2.7 SeeDance-2.0 HappyHorse-1.0 Veo 3.1 Runway Gen-4.5 Cosmos3-Super

mnTVbest

mnTVworst

mnTV (reported)

0.318 0.196 0.278 0.315 0.079 0.206

0.491 0.321 0.448 0.564 0.383 0.541

0.578 0.391 0.540 0.651 0.483 0.631

Physical Galton board. A clean physics animation of a Galton board: a vertical triangular peg array above a row of collection bins. A single metallic ball is released from the top funnel and falls through the pegs, bouncing left or right at each, until it lands in one of the bins. The animation is smooth and physically accurate. Animated Galton board. A clean 2D physics animation of a Galton board: a vertical triangular peg array above a row of collection bins. A single green ball is released from the top funnel and falls through the pegs, bouncing left or right at each, until it lands in one of the bins. The animation is smooth and physically accurate. Ball fork. A clean 2D physics simulation. A single ball rolls down a straight ramp and enters the top of a perfectly symmetric Y-shaped fork. The fork splits into two equal channels. The ball travels through one of the two channels and exits at the bottom. The simulation is smooth and physically accurate, viewed from the front. 23

Table 8: Validation of the VLM outcome extraction pipeline against human annotation on a stratified sample (N = 18 per environment; 162 total). FN: False-Null rate (VLM marked null, human marked valid) over human-labelled outcomes. MN: Missed-Null rate (VLM extracted an outcome, human marked null) over human-null cases. WL: Wrong-Label cases where both annotators assigned definite outcomes but disagreed on the category. Dashes indicate a zero denominator. Scene

n

Agreement

FN

MN

WL

Physical board Animated board Ball fork Walking Pendulum Dice Cards Lottery Roulette

18 18 18 18 18 18 18 18 18

18/18 (100%) 17/18 (94%) 17/18 (94%) 17/18 (94%) 16/18 (89%) 18/18 (100%) 15/18 (83%) 17/18 (94%) 17/18 (94%)

0/13 0/13 0/17 0/17 2/2 0/18 1/6 1/14 0/7

0/5 1/5 0/1 0/1 0/16 — 2/12 0/4 1/11

0 0 1 1 0 0 0 0 0

Total

162

152/162 (93.8%)

4/107 (3.7%)

4/55 (7.3%)

2

Walking. A video of a person walking down a long empty corridor who reaches a perfectly symmetric T-junction. The left and right paths are identical in appearance, lighting, and length. The person chooses one direction and walks off screen. The scene is shot from behind at ground level. Double pendulum. A clean physics simulation of a double pendulum. Two thin brass-coloured rods connect three points: a fixed pivot at the top of the post, a brass disk at the middle joint, and a brass disk at the lower tip. The pendulum is released from rest and swings freely under gravity. The camera is fixed and does not move. Dice. A clean 2D physics animation of a single six-sided die being rolled on a flat surface inside a wooden box with low walls. The die starts tumbling from the left side and comes to rest showing a single upward face clearly. The animation is slow, smooth, and physically accurate. The die is white with large black circular pips, viewed from directly above. Cards. A single playing card is drawn from the top of the face-down deck and flipped face-up onto the table. The card is shown clearly face-up in the final frame. Lottery. A physics animation of a transparent lottery ball tumbler containing 20 numbered white balls (1–20). The machine spins and one ball is ejected into a clear tube at the top. The animation ends with a single ball clearly visible in the tube, its number legible. Smooth, realistic motion. Roulette. A clean physics animation of a European roulette wheel spinning rapidly. A single white ball is released onto the spinning wheel and eventually settles into one of the numbered pockets. The final frame clearly shows the ball at rest with the pocket number and colour clearly visible. The animation is smooth, realistic, and viewed from directly above.

H

VLM Extraction Prompts

For each scene the visual input is passed to Gemini 3.1 Pro alongside the scene-specific prompt below: the final frame for the terminal-state scenes (both Galton boards, dice, cards, lottery, roulette) and the full video for the direction-of-motion scenes (ball fork, double pendulum, walking). The prompt asks for either a discrete outcome label or null when the outcome is not unambiguously visible. Each query is repeated three times and aggregated by majority vote (Section 3.3). Physical Galton board. Where is the metallic ball? If there are multiple metallic balls in the image, or if the ball appears to straddle two bins or has not clearly settled in a single bin, reply with ‘null’. Otherwise reply with only the bin number it has landed in as a single integer. 24

Animated Galton board. Where is the green ball? If there are multiple green balls in the image, or if the ball appears to straddle two bins or has not clearly settled in a single bin, reply with ‘null’. Otherwise reply with only the bin number it has landed in as a single integer. Ball fork. You are shown a video of a ball rolling into a Y-shaped fork. Through which channel does the ball ultimately exit: left or right? Only reply ‘null’ if the ball never clearly exits through either channel. Otherwise reply with only ‘left’ or ‘right’. Walking. You are shown a video of a person approaching a T-junction. Has the person clearly chosen to walk left or right? If the person has not yet reached the junction or has not committed to a direction, reply with ‘null’. Otherwise reply with only ‘left’ or ‘right’. Double pendulum. You are shown a video of a double pendulum. Does the pendulum maintain the correct rigid structure of a normal double pendulum (two rods, three balls) throughout the video? If the structure is deformed, broken, or grows extra arms or balls at any point, reply with ‘null’. Otherwise, where is the lowest pendulum weight at the end of the video — to the left or right of the central pivot? Reply with only ‘left’, ‘right’, or ‘null’. Dice. Is there one clearly dominant top face on the die? A slight tilt is acceptable as long as one face is clearly uppermost. If the die is balanced on an edge with no single face dominant, reply with ‘null’. Otherwise reply with only the number on the top face as a single integer between 1 and 6. Cards. You are shown the final frame of a card-draw video. Look at the playing card and identify its suit. Reply ‘null’ if any of the following apply: the card is face-down, partially obscured, or no card is visible; the symbols on the card are deformed, blurry, or distorted to the point where the suit is ambiguous; the card shows different suit symbols (e.g. a heart and a club on the same card) — a real playing card has only one suit, so inconsistent symbols indicate a rendering artefact; the suit symbols are otherwise unrecognisable as one of the four standard suits. Otherwise reply with only ‘hearts’, ‘diamonds’, ‘clubs’, or ‘spades’. Lottery. Is exactly one lottery ball visible inside the clear display tube? If no ball has been drawn into the tube, or if multiple balls are visible in the tube, reply with ‘null’. Otherwise reply with only the number shown on that ball as a single integer between 1 and 20. Roulette. This is the final frame of a roulette video. Step 1 — Find the white ball: Can you see a distinct white roulette ball resting inside the wheel? If the ball is not visible (the frame is blurry, the ball has disappeared, or you only see the wheel pattern with no ball), reply ‘null’. If you see multiple balls, reply ‘null’. Step 2 — Check location: Reply ‘null’ if the ball is: outside the wheel’s outer pocket ring (bounced off); sitting on the central hub or spinner; or resting on the inner sloped wooden surface between the hub and the pocket ring (i.e. it has rolled inward and is not inside any numbered pocket). Step 3 — Identify colour: Look at the exact pocket the ball’s centre sits in. Pocket colours are red, black, or green. Important: these wheels are often filmed from above. From an overhead angle the ball may appear to sit on or near the outer rim when it is actually resting at the top edge of a numbered pocket — if a numbered pocket is directly below the ball, the ball is in that pocket. Report that pocket’s colour. If the ball is equally split between two pockets with no dominant side, reply ‘null’. Otherwise report the colour of whichever pocket contains most of the ball. Reply with only ‘red’, ‘black’, ‘green’, or ‘null’.

I

Generation-Setting Ablations

Two generation settings vary across models (Table 1): output resolution and video duration. We ablate each in turn to confirm neither drives the calibration results. 25

Table 9: SeeDance-2.0 at its benchmarked 480p vs 720p (same frames, prompts, 32 seeds; only resolution changes). ρ = nvalid /32; p = MC χ2 ; TV vs the scene reference. Strongly-miscalibrated scenes (p < 0.001) are unchanged by resolution. †: ≤ 3 valid generations, so no χ2 test is reported (main-text dagger rule). ρ

p

TV

Scene

480p

720p

480p

720p

480p

720p

Physical board Animated board Ball fork Walking Pendulum Dice Cards Lottery Roulette

0.59 0.88 1.00 1.00 0.06 1.00 0.91 0.75 0.84

0.47 0.81 0.97 1.00 0.09 1.00 0.91 0.88 0.72

0.088 <0.001 0.051 0.051 † <0.001 <0.001 0.049 0.004

<0.001 <0.001 0.719 0.111 † <0.001 <0.001 <0.001 <0.001

0.275 0.657 0.188 0.188 0.000† 0.417 0.500 0.400 0.121

0.327 0.467 0.048 0.156 0.500† 0.438 0.509 0.643 0.190

Table 10: Uniform 5-second duration ablation: aggregate mnTV (lower is better) at each model’s benchmarked duration vs. a uniform 5 s. WAN-2.7, SeeDance-2.0 and HappyHorse-1.0 are re-generated and re-scored at 5 s; Runway Gen-4.5 and Cosmos3-Super are already ∼5 s (reused); Veo 3.1 cannot run at 5 s (only {4, 6, 8} s) and is shown at its benchmarked 4 s. Model WAN-2.7 SeeDance-2.0 HappyHorse-1.0 Veo 3.1 Runway Gen-4.5 Cosmos3-Super

I.1

Benchmark dur.

mnTV (benchmark)

mnTV (5 s)

3s 4s 3s 4s 5s ∼5 s

0.58 0.39 0.54 0.65 0.48 0.63

0.56 0.40 0.59 — 0.48 0.63

Resolution ablation (SeeDance-2.0, 480p vs. 720p)

SeeDance-2.0 is the only model we run at 480p rather than 720p (Table 1). To test whether its miscalibration is an artefact of the lower resolution, we regenerate all nine scenes at 720p—identical conditioning frames, prompts, and 32 seeds, changing only the resolution—then re-extract and re-score them. Table 9 compares the two. The strongly-miscalibrated scenes (animated board, dice, cards) are unchanged (p < 0.001 at both resolutions); the borderline cells mostly worsen at 720p (lottery, roulette and the physical board become more significant), with only the ball fork and walking improving. SeeDance’s calibration is therefore no better at the higher resolution, so its miscalibration is not explained by the 480p setting. I.2

Duration ablation

Per-model video durations differ (Table 1; 3–5 s). To check that this does not drive the results, we re-generate the models that were not already at ∼5 s and are capable of this—WAN-2.7, SeeDance-2.0, HappyHorse1.0—at a uniform 5 s (identical conditioning frames, prompts, and 32 seeds; only the duration changes) and re-score them. Runway Gen-4.5 (5 s) and Cosmos3-Super (∼5.04 s) are already ∼5 s; Veo 3.1 cannot run at 5 s—its API accepts only {4, 6, 8} s—so it stays at its benchmarked 4 s. Table 10 compares the aggregate mnTV. The shifts are small (≤ 0.05) and the ordering is unchanged—SeeDance-2.0 remains the lowest-error model—so the mixed durations do not explain the calibration deficits.

J

Classifier-Free Guidance Sweep on Cosmos3-Super Dice

The baseline Cosmos3-Super dice evaluation reported in Table 2 utilises a default Classifier-Free Guidance (CFG) scale of 6.0, yielding an empirical Total Variation Distance (TV) of 0.28 (p = 0.011). To investigate 26

whether this miscalibration represents a stable architectural property or an artefact of the default guidance scale, we conduct a parametric sweep over CFG ∈ {1.0, 1.5, 3.0, 4.5, 6.0, 7.5, 9.0}, regenerating the full N = 32 allocation at each guidance value with only the guidance scale varied. All sweep points share a fixed 61-frame (2.5 s) generation horizon. We provide the calibration metrics against guidance scale in Table 11 and visualise the corresponding empirical distributions in Figure 14. Table 11: Cosmos3-Super dice analysis at seven guidance scale values Total Variation (TV) and p values characterise calibration on the valid sample against the uniform {1, . . . , 6} reference; “modal face” is the face on which the empirical distribution peaks. Note that the two lowest-TV settings (CFG 1.0, 1.5) also have the lowest scorability ρ, so their TV is computed on fewer valid generations. Guidance Scale

ρ↑

TV ↓

p value

Modal face

1.0 1.5 3.0 4.5

0.66 0.78 0.97 1.00 0.97 1.00 1.00

0.17 0.17 0.34 0.26 0.31 0.45 0.51

0.699 0.394 0.003 0.042 0.004 <0.001 <0.001

1 5 5 5 5 1 5

6.0 (default) 7.5 9.0

Cosmos3-Super dice: empirical distribution by CFG CFG = 1.0 (nvalid = 21)

Proportion

Reference (uniform 1/6)

CFG = 3.0 (nvalid = 31)

0.4 0.2 0.0

1

2

3

4

Face

5

6

1

2

CFG = 4.5 (nvalid = 32) Proportion

CFG = 1.5 (nvalid = 25)

3

4

Face

5

6

1

2

CFG = 6.0 (nvalid = 31)

3

4

Face

5

6

1

2

CFG = 7.5 (nvalid = 32)

3

4

Face

5

6

5

6

CFG = 9.0 (nvalid = 32)

0.4 0.2 0.0

1

2

3

4

Face

5

6

1

2

3

4

Face

5

6

1

2

3

4

Face

5

6

1

2

3

4

Face

Figure 14: Empirical dice distribution at each Classifier Free Guidance (CFG) value (leftmost panel: uniform reference). At the lowest guidance settings (CFG 1.0–1.5) the distribution is close to uniform; as CFG increases, mass concentrates onto faces 1 and 5 and face 6 is essentially absent for CFG ≥ 3.0. Higher guidance sharpens the collapse onto the modal faces. Note that nvalid (shown per panel) falls sharply at low CFG, so the near-uniform low-CFG panels rest on fewer valid generations. We observe that calibration is strongly dependent on the Classifier-Free Guidance (CFG) scale, revealing a strict trade-off against sample scorability. For CFG ≥ 3.0, scorability saturates (ρ ≥ 0.97) while the Total Variation Distance (TV) varies non-monotonically—attaining a shallow minimum of 0.26 at CFG = 4.5 before rising to 0.51 at CFG = 9.0. Yet every configuration in this regime remains significantly miscalibrated (p ≤ 0.042). Attenuating guidance further to CFG ∈ [1.0, 1.5] suppresses the TV to 0.17, yielding the distribution being statistically indistinguishable from uniform (p = 0.70 and p = 0.39; non-rejected). However, this calibration improvement incurs a strict penalty in validity: scorability deflates to ρ = 0.78 at CFG = 1.5 and ρ = 0.66 at CFG = 1.0, as the model produces substantially more unscorable sequences. 27

This behaviour aligns with the foundational mechanics of CFG (Ho & Salimans, 2021): higher guidance sharpens the conditional distribution toward the prominent modes of p(x | c), reducing effective coverage of the distributional support (Astolfi et al., 2024), while lower guidance recovers diversity at the expense of prompt-following and, hence, scorability. The baseline Cosmos3-Super dice miscalibration is therefore partially an artefact of its default setting (CFG = 6.0). The calibration gap can be closed entirely by reducing guidance, but at the cost of operational scorability. Notably, the concentration on faces 1 and 5 as well as the near-total absence of face 6 persists for all CFG ≥ 3.0 and dissolves only within the lowest, low-scorability settings. We could not replicate this parametric sweep on WAN-2.7, SeeDance-2.0, HappyHorse-1.0, Veo 3.1, or Runway Gen-4.5, as their public Replicate APIs do not expose an adjustable guidance scale parameter.

K

Explicit Target-Face Ablation on Dice

The evaluations on the dice scene investigate whether models produce a uniform marginal distribution under outcome-agnostic instructions. A natural corollary is whether a model can be systematically steered to a designated outcome by embedding the target face directly into the prompt text. For each face N ∈ {1, . . . , 6}, we append the suffix conditional sentence, “The die comes to rest showing the N -pip face upward.”, to the baseline conditioning prompt. We generate five random seeds per (model, N ) pair and extract the resulting physical state via the VLM pipeline. As in the guidance sweep, the Cosmos3-Super generations here use the generator’s default 61-frame (2.5 s) horizon rather than the 121-frame horizon of the main table; compliance is a within-ablation steerability comparison, so this does not affect the conclusions. We define compliance as the proportion of valid generations where the outcome matches the requested face. Under an unsteerable random baseline, expected compliance is 1/6 ≈ 16.7%. We report the empirical compliance rates per model in Table 12, and the corresponding cross-conditional distributions are visualised via the requested-versus-generated outcome confusion matrices in Figure 15.

Explicit-target-face dice ablation: 5 seeds per (model, requested face)

3

1

1 1

3

3 1

4

4 5

5 6

1

1 1

3 2

3

4

5

Rolled face

1 6

5

2

1

3

1

5

3

2

3

4

3

2

5

4

1

6

3

2

5

6

1

2

3

4

Rolled face

null

4 2

3 5

6 1

2

3

4

5

Rolled face

6

1

5

2

2

3

3

4

1

4

3

2

5

5

6

3

null

1

Runway 4.5 (29% compliance)

1

2

1

5

null

Requested face

Requested face

4

1 4

4

Veo 3.1 (43% compliance) 1

3

HappyHorse (17% compliance)

Requested face

2

SeeDance-2.0 (63% compliance) 1

1

2

2

1

1 1

1

2

1

1

2

2

4

3

2

5

2

1

6

3

1

4

5

3

1

2

3

Rolled face

1

1

1 1

6

null

2 2

3

4

5

Rolled face

6

null

Cosmos3-Super (33% compliance)

Requested face

5 Requested face

Requested face

WAN-2.7 (46% compliance) 1

1

4

2

1

1 1

1

2

2

3

1

3

5

1

3

6

1

1

3

3

4

5

3

1

4

1

2

Rolled face

1 6

null

Figure 15: Per-model confusion matrices for the explicit target-face dice ablation, 5 seeds per (model, requested face). Rows index the requested face, columns index the rolled face (plus a final null column for invalid video generations). Diagonal cells (correctly-followed instruction) are outlined. Numbers in parentheses are overall compliance across all requested faces. 28

Table 12: Overall compliance on the explicit target-face dice ablation. Chance compliance under no instructionfollowing is 1/6 ≈ 16.7%. Model

Compliance

SeeDance-2.0 WAN-2.7 Veo 3.1 Cosmos3-Super Runway Gen-4.5 HappyHorse

63.3% 46.4% 43.3% 33.3% 28.6% 16.7% (chance)

The conditioning image anchors which faces can be requested. The reference frame for the dice scene displays only two visible facets: the 1-pip and 5-pip faces, while the 2, 3, 4, and 6-pip facets remain occluded. The compliance pattern in Figure 15 track this geometric bias directly. The visible 1 and 5 facets are the most susceptible to steering, specifically, 60–100% compliance across the majority of models. In contrast, the entirely occluded opposite 6 face proves the most resilient to prompt conditioning with 0% compliance for five of the six models, rising to 40% exclusively for Veo 3.1. For the remaining hidden intermediate facets (2, 3, and 4) we observe intermediate compliance levels. This structural alignment mirrors the baseline outcome distributions reported in Table 2, confirming that both the unsteered marginal distributions and the conditional prompt compliance are strongly biased toward the explicit visual features present in the initial conditioning frame. Calibration is not the same as capability. High prompt-steering compliance does not imply baseline physical calibration. For example, SeeDance-2.0 achieves the highest aggregate steering accuracy (63.3%), maintaining a 60–100% compliance rate across five of the six requested faces. This demonstrates that the architecture is fully capable of generating these hidden facets when explicitly conditioned to do so. However, in the outcome-agnostic evaluation (Table 2), this same model concentrates 56% of its unsteered mass on face 1 and generates face 2 only 3% of the time, despite successfully generating face 2 in 60% of the trials when specifically instructed. The baseline miscalibration therefore cannot be attributed to an underlying architectural capability deficit; the underrepresented categories remain fully renderable when steered. Instead, the model lacks a calibrated default: absent explicit outcome instructions, the conditional distribution p(x | c) remains heavily skewed toward the visually anchored modes of the conditioning frame. This behaviour, however, does not generalise universally across all architectures. Across its 30 evaluation sequences, HappyHorse-1.0 fails to generate faces 2, 3, 4, or 6 entirely; Veo 3.1 never produces face 2; and WAN-2.7 completely fails to generate face 6. In these regimes, the effects of a biased calibration default are observationally indistinguishable from structural generation failures. Consequently, the severe marginal deviations reported in the baseline benchmark may indeed reflect a genuine capability gap rather than a purely uncalibrated default distribution.

29

Related documents

• CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?#625378
• CaliBench: Are the Stochastic Dynamics of Video World Models Physically Calibrated?#623105
• Geometric Inconsistency Localization in Multi-View Image Sets#1105805
• Input Resolution Matters: Real-Time Object Detection Latency#1070369
• Measuring multi-calibration#147106