arXiv:2606.05966v1 [cs.DB] 4 Jun 2026
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs Tianyi Tang
Zhuoyi Lin†
Zeyu Feng
CFAR, IHPC, Agency for Science, Technology and Research (A*STAR) Singapore [email protected]
I2 R, Agency for Science, Technology
CFAR, IHPC, Agency for Science, Technology and Research (A*STAR) Singapore [email protected]
and Research (A*STAR) Singapore [email protected]
Tianyi Ma
Yew-Soon Ong
Ivor Tsang
Nanyang Technological University Singapore CFAR, IHPC, Agency for Science, Technology and Research (A*STAR) Singapore [email protected]
Nanyang Technological University Singapore CFAR, IHPC, Agency for Science, Technology and Research (A*STAR) Singapore [email protected]
Nanyang Technological University Singapore CFAR, IHPC, Agency for Science, Technology and Research (A*STAR) Singapore [email protected]
Haiyan Yin† CFAR, IHPC, Agency for Science, Technology and Research (A*STAR) Singapore [email protected]
Abstract
1
Introduction
Understanding and reasoning about the physical world is the foundation of intelligent behavior, yet state-of-the-art vision-language models (VLMs) still fail at causal physical reasoning, often producing plausible but incorrect answers. To address this gap, we introduce CausalPhys, a benchmark of over 3,000 carefully curated video- and image-based questions spanning four domains: Perception, Anticipation, Intervention, and Goal Orientation. Each question is paired with an expert-annotated causal graph capturing object–attribute–event dependencies, enabling interpretable and fine-grained evaluation of causal understanding. Building on this, we formulate a causal-graph-grounded metric that quantitatively measures how well a model’s chain-of-thought reasoning aligns with the correct causal relations, moving beyond answer-only accuracy and enabling systematic diagnosis of VLMs’ causal reasoning failures. Using this metric, we conduct a comprehensive analysis of leading VLMs, revealing systematic gaps in capturing causal dependencies and underscoring the need for causality-aware learning. To address these limitations, we further propose Causal Rationaleinformed Fine-Tuning (CRFT), which explicitly aligns VLM reasoning with causal structures. Extensive experiments demonstrate that CRFT substantially enhances both reasoning accuracy and interpretability across multiple model backbones. By unifying dataset curation, causal evaluation, and causality-informed learning, CausalPhys establishes a strong foundation for advancing modern VLMs toward causally grounded physical reasoning. Our code and dataset are available at https://github.com/haorentang/CausalPhys.
Understanding and reasoning about the physical world lies at the heart of intelligence, enabling agents to act robustly and adaptively in real-world environments [22, 48]. Yet today’s vision-language models (VLMs) remain far from human intuition, often struggling with even basic physical interactions. Robust physical reasoning demands more than visual pattern recognition: agents must infer intrinsic object properties [10, 57], track spatial and temporal relations among entities [52, 56], interpret evolving physical scenes, and anticipate how interactions unfold to guide planning and avoid costly errors [6, 16]. Humans, by contrast, perform such reasoning effortlessly, guided by an intuitive grasp of physical causality that emerges early in cognitive development [7, 11, 40]. How to equip VLMs with this level of causally grounded understanding remains a central open challenge. Bridging this gap is essential for advancing embodied AI systems that are both reliable and trustworthy. Recent VLMs excel at multimodal tasks such as visual question answering [3, 53], object recognition [8, 45], and image captioning [34, 35]. Yet extending these successes to dynamic physical reasoning in realistic environments remains an open challenge [6, 11, 16, 50]. Relying solely on perception-driven capabilities has proven insufficient for building generalist embodied agents [9, 17, 32, 38], often leading to brittle behaviors such as mishandling fragile objects or misjudging grasp affordances. As a concrete example, Fig. 2 (Intervention) illustrates that inferring the orientation of a door relative to the camera viewpoint from limited observations is far from trivial. Such reasoning demands sensitivity to latent spatial structures, occluded relationships, and viewpoint transformations that are invisible in isolated images. Ultimately, these cases † Corresponding authors.
Tang et al.
4 Perception (951) Subset
# Anticipation (900)
#Question Subset
#Question
Optical Inference
252
Collision Forecasting
300
Containability
201
Deformation
200
Scene Reconstruction
200
Fluid Flow
200
Mechanics Reasoning
298
Intention Speculation
200
Ô Intervention (573) Subset
◎ Goal Orientation (638)
#Question Subset
#Question
Spatial Manipulation
151
Object Relocation
190
Action Substitution
99
Tool Assignment
100
Temporal Shifting
149
Behavior Selection
229
Viewpoint Transformation
174
Trajectory Optimization
119
Figure 1: Hierarchical Taxonomy of CausalPhys spanning four categories. Each of the four major categories corresponds to a causal range [43] (𝑃 {𝑌 | 𝑋 }, 𝑃 {𝑌 | do(𝑋 )}, 𝑃 {Goal | do(𝑋 )}, 𝑃 {𝑌 | do(𝑋 ′ )}). The outer segments enumerate sixteen subcategories that instantiate these primitives.
Table 1: Statistics of CausalPhys. The benchmark comprises 3,062 video- and image-based questions spanning 4 causal domains and 16 fine-grained subsets, capturing a broad spectrum of physical reasoning abilities.
hinge on anticipating how the world changes under interventions or viewpoint shifts, an ability naturally framed through causally informed reasoning. This inferential capacity allows agents to bypass the pitfalls of spurious visual correlations, anchoring their understanding in the structural causal dependencies that dictate physical transitions. Bridging high-dimensional multimodal perception with explicit causal grounding enables a transition from mere observation to intervention-aware reasoning, ultimately fostering a more consistent, generalizable, and mathematically rigorous representation of physical reality. However, integrating causal reasoning into VLMs remains a fundamental challenge, and we identify three critical gaps. (1) Current models primarily learn statistical associations from observational data rather than underlying causal mechanisms, limiting their ability to reason in dynamic, real-world environments. (2) Prior benchmarks rarely include explicit causal annotations, motivating our construction of a dataset with expert-annotated causal graphs for rigorous evaluation of causal dependencies. (3) Existing efforts largely emphasize evaluation rather than training, leaving open how to effectively instill causal reasoning within multimodal systems. These challenges call for benchmarks and training paradigms that explicitly foster causally informed reasoning, moving VLMs beyond surface correlations toward genuine physical understanding. To tackle these challenges, we introduce CausalPhys, a comprehensive benchmark of over 3,000 expertly curated video- and image-based questions spanning four domains: Perception, Anticipation, Intervention, and Goal Orientation, across 16 subcategories (Fig. 1 and Table 1). A distinguishing feature of CausalPhys is that each question is paired with an expert-annotated causal graph capturing physical interactions and dependencies, enabling mechanism-level and interpretable evaluation of VLM reasoning. By coupling visual reasoning with Causal Directed Acyclic Graphs
(DAGs), our framework establishes a rigorous foundation for evaluating not just what a model predicts, but why it fails—bridging the gap between surface-level pattern matching and systematic physical world understanding. Utilizing CausalPhys, we identify a systemic decoupling between visual recognition and causal dependency modeling in state-of-the-art VLMs. To address this, we propose Causal Rationale-informed Fine-Tuning (CRFT), a framework that enforces alignment between a model’s latent reasoning and causal graphs. This approach enhances zero-shot generalization and interpretability, bridging the gap between superficial pattern matching and robust physical world understanding. This work is the first to establish a unified framework connecting benchmarking, evaluation, and model improvement for causal physical reasoning in VLMs. We aim to deliver actionable insights and close the gap between VLMs and physical world understanding, advancing embodied AI by learning from human reasoning capabilities. Overall, this paper makes three key contributions: (1) We introduce CausalPhys, the first benchmark coupling physical reasoning tasks with explicit, expert-annotated causal graphs, enabling mechanism-level and interpretable evaluation beyond surface-level accuracy. (2) We develop a causal-graph-grounded metric that evaluates whether a model’s reasoning aligns with correct causal dependencies, providing fine-grained diagnostic insights beyond answer-only metrics. (3) We propose CRFT, a causally guided VLM fine-tuning strategy that leverages causal graphs to enhance VLMs’ accuracy and interpretability in physical environments.
2
Related Works
Physical Benchmarks. Early efforts in physical reasoning benchmarks laid the groundwork for evaluating agents’ understanding
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Intervention
Perception
✨ Question: What is the orientation of the ✨ Question: Where is the blue painting classroom door relative to the camera’s viewpoint in the second image? A: Front B: Right
C: Left D: Back
located respective to camera? A: Front Left B: Back Left
C: Front Right D: Back Right
Anticipation
✨ Question: Will the two cars collide ? A: Yes
B: No
Goal -Orientation
✨ Question: In which direction
should the black player hit the ball? A: Left
B: Right
Figure 2: Overview of the CausalPhys benchmark. CausalPhys categorizes causally-informed physical understanding across four fundamental domains: (i) Intervention, (ii) Perception, (iii) Anticipation, and (iv) Goal-Orientation. Beyond ground-truth question–answer pairs, each question is accompanied by a carefully annotated causal directed acyclic graph (DAG) that captures the underlying object–attribute–event dependencies governing the physical dynamics of the scene. of basic interactions under simplified conditions [6, 50, 62]. Classical benchmarks such as CLEVRER and CoPhy [10, 57] target elementary visual primitives of spheres, cubes, and rigid-body collisions, thereby focusing on fundamental perceptual and causal concepts. Subsequent multimodal datasets [5, 24, 25, 28, 39, 59] extended this direction toward commonsense reasoning grounded in linguistic or textual knowledge. While valuable for probing conceptual understanding, these settings often abstract away the perceptual and dynamical complexity inherent to real-world physics. In parallel, spatial VQA benchmarks [36, 47, 52, 56] explored geometric relationships and spatial reasoning in 3D environments, marking an important step toward holistic physical scene understanding. Recent large-scale benchmarks such as PhysBench [11] and MVPBench [16] advance this line by systematically evaluating models’ ability to perceive, anticipate, and describe physical events across diverse settings. Both focus on physics-centric understanding through answer-based evaluation, emphasizing prediction accuracy rather than the underlying reasoning process. CausalPhys complements these efforts by introducing expert-annotated causal graphs that capture object–attribute–event dependencies, accompanied by causal-graph-grounded evaluation metrics for mechanism-level assessment. We further propose a causally-informed fine-tuning strategy that aligns VLM reasoning with causal structure, enabling more consistent and interpretable physical understanding. Causal Reasoning Datasets. While causal reasoning has been extensively studied for LLMs [29, 30, 37, 46], equivalent efforts in the VLMs remain comparatively nascent. Early multimodal studies encode causal knowledge at a focused level of granularity. For example, CELLO [9] represents nodes as perceptible objects and models local relations such as “object 1 supports object 2,” while other works instantiate structural causal models primarily for interpretability rather than data-driven reasoning [18]. Recent VLM benchmarks,
exemplified by CausalVLBench [32] and Causal3D [38], introduce fixed-structure causal graphs within synthetic or template-based scenes, where both entities and relations are predefined in the prompts. These controlled settings are valuable for isolating local causal dependencies but insufficient for probing holistic, multiobject interactions in real-world physical environments. In contrast, CausalPhys generalizes this paradigm through a flexible-form causal annotation schema grounded in expert-annotated DAGs with rich human insights that capture object–attribute–event dependencies across diverse static and dynamic scenes. It is further accompanied by causal evaluation criteria with graph-grounded reasoning metrics and a causally informed fine-tuning strategy, enabling more consistent and interpretable causal reasoning in VLMs.
3
The CausalPhys Benchmark
We introduce CausalPhys, a large-scale, rigorously curated benchmark for evaluating VLMs on causally-informed physical reasoning. We outline the benchmark design and formalism for causally informed physical reasoning in Sec. 3.1, detail the expert-driven annotation workflow in Sec. 3.2, and introduce a causal-graph-grounded evaluation framework in Sec. 3.3 for mechanism-level reasoning. Comprehensive results in Sec. 3.4 reveal systematic reasoning gaps in current VLMs and highlight clear directions toward causally grounded model improvement.
3.1
Benchmark Overview
CausalPhys comprises over 3,000 expert-curated image- and videobased questions (Fig. 1) drawn from 11 public datasets, covering diverse physical scenarios ranging from collisions to fluid dynamics (see Appendix A.1). Each benchmark instance includes a visual scene, a question, a ground-truth answer, and an expert-annotated
Tang et al.
Table 2: Comparison of CausalPhys with existing physical reasoning benchmarks. While prior datasets are limited by synthetic environments, restricted diversity, or missing causal structure, CausalPhys uniquely integrates real-world data, diverse scenes, and fine-grained causal annotations. Data Type Dataset
Data Source
Causal Structure
Causal Node
Size Real-World Data
Scene Diversity
Annotation
Flexibility
Object
Attribute
Event
✓ ✓ ✓
✓
✓
✗
✗ ✗ ✗ ✗ ✗
✓ ✓ ✗ ✗ ✗
✗ ✗ ✗ ✗ ✗ ✗
✓
✓
✓
Image
Video
✓ ✓ ✓ ✓
✗ ✗ ✗
✓ ✗ ✗
✗ ✗ ✗
✓ ✓ ✗
✓ ✓ ✓
✓ ✓ ✓
✗ ✗ ✗
✗ ✗ ✗ ✗ ✗
✓
✓
✓
✓
✓
CELLO [9] Causal3D [38] CausalVLBench [32] PhysBench [11] Causal VQA [17] MVP Bench [16]
14,000+ 10,000+ 700+ 1,000+
✓
CausalPhys (Ours)
3,000+
✓
✗
causal graph capturing object–attribute–event dependencies. An illustrative example is shown in Fig. 2. Taxonomy. We organize CausalPhys around the levels of causal understanding defined by Pearl’s causal ladder [43]: association, intervention, and counterfactuals. This hierarchy guides four task categories: Perception, Anticipation, Intervention, and GoalOrientation (Fig. 1). Perception corresponds to the association rung, testing whether models recognize objects and physical attributes. Anticipation and Intervention probe the interventional level, assessing a model’s ability to predict or reason under explicit manipulations. Goal-Orientation aligns with counterfactual reasoning, inferring actions that would achieve a desired outcome. This causal grounding offers a principled framework for assessing how far VLMs can ascend the causal hierarchy of physical reasoning. Each of the four categories is further divided into four subcategories targeting specific physical mechanisms, geometry, dynamics, interaction, and transformation, providing a fine-grained and systematic evaluation of VLMs across diverse aspects of the physical world. Causal Graph Representation (typed DAG). Each question in CausalPhys is paired with a structured causal graph that exposes the underlying physical mechanism, allowing models to be evaluated on how they reason, not just whether they are correct. Formally, each graph is a directed acyclic graph G = (V, E), where nodes V represent causal variables and edges E denote directed dependencies. To capture the heterogeneous components of physical reasoning, every node 𝑣 ∈ V is defined as a typed tuple (𝑣𝜏 , 𝑣𝑛 , 𝑣𝑑 ): 𝑣𝜏 specifies the semantic type, 𝑣𝑛 gives the name of entity, and the optional 𝑣𝑑 stores descriptive details. Nodes are typed into three semantic classes: Objects (O), representing physical entities (e.g., door, ball); Attributes (A), representing their states (e.g., position, distance, velocity); and Events (E), representing transformations or interactions (e.g., rotating, colliding, pouring). This typed DAG formalism provides a unified representation of the causal dynamics that link what exists, how it changes, and why, forming the backbone of mechanism-level reasoning in CausalPhys. Causal Dependencies. Directed edges in the causal graph express how physical quantities and events influence one another. These edges cover a broad range of dependencies, including attributes
describing objects, events acting on objects, events modifying attributes, and cascaded event-event interactions where one transformation triggers another. Formally, for any nodes 𝑢, 𝑣 ∈ V, (𝑢, 𝑣) ∈ E
⇐⇒
𝑢 is a direct cause of 𝑣 .
This design yields a fine-grained and typed dependency structure that comprehensively represents the causal mechanisms underlying each physical scenario, enabling evaluation not only of what a model predicts but also how it arrives at that prediction.
3.2
Data Collection Workflow
We constructed CausalPhys through a rigorously designed and fully auditable workflow to ensure causal correctness, annotation reliability, and reproducibility (see Appendix A.4 for full documentation). The benchmark’s visual foundations are derived from established datasets (Appendix A.2), upon which ten STEM-background experts, including researchers in physics and robotics, synthesized all questions and causal graphs. Each annotator completed a rigorous calibration phase (Appendix A.3) focusing on our formal causal-graph schema and physical reasoning taxonomy, ensuring high inter-annotator consistency and precise grounding of visual events into causal structures. We continuously monitored interannotator agreement across graph annotations, with all conflicts adjudicated by senior annotators. Comprehensive statistics on annotator profiles and agreement metrics are provided in Appendix A.1. Our workflow proceeds in five tightly controlled stages: (a) Data Acquisition: instances sourced from 11 public datasets with complete provenance tracking; (b) Question Formulation: annotators design physically grounded questions paired with verified answers; (c) Data Processing: media are standardized and aligned with their corresponding annotations; (d) Causal Graph Construction: each instance is encoded from a Mermaid draft into a typed JSON causal graph with validators; (e) Quality Assurance: all items undergo double annotation and adjudication, removing samples with insufficient visual evidence, inconsistent annotations, or textual biases. To ensure transparency and community extensibility, we release the entire pipeline end-to-end, including dataset selection scripts, annotation guidelines, Mermaid DAG templates, and JSONschema validators (Appendix A). This level of documentation makes CausalPhys one of the few multimodal reasoning benchmarks that is highly transparent, auditable, and reproducible by design.
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
CoT Generated by VLMs
Input Query VLM Reasoning
Question: Where is the TV located respective to the camera in the second image? A: Front B: Right
EF-Q1
DC-Q1
RA-Q1
EF-Q2
DC-Q2
RA-Q2
EF-Q3
DC-Q3
EF Score: 1
DC Score: 0.33
RA Score: 0.5
Accuracy Score: 0
Causal Graph
Causal-Aware Questions Entity Faithfulness (EF)
In the first image, the camera faces the TV, so the TV is in f r ont. Th e seco n d i m a g e shows the camera pointing in the opposite direction, as evidenced by the change in background…… Result: Behind
C: Left D: Back
LLM-Based Judging
(E.g., EF-Q1: Does the “camera” presents in the rationale?)
Question Construction
Description Correctness (DC) (E.g., DC-Q1: Does the “Location of the TV in first image” is mentioned to be “Front” in the rationale?)
Relation Awareness (RA) (E.g., RA-Q1: Does the causal relation between “Location of the TV in first image” and “Location of the TV in second image” stated?)
Figure 3: Evaluation pipeline for causal-graph-grounded metrics. Given a visual input and a corresponding query, the tested VLM first produces a rationale-styled CoT and a final answer. A judge LLM then evaluates the reasoning outputs using three causal-aware metrics: Entity Faithfulness (EF), Description Correctness (DC), and Relation Awareness (RA), and computes the final Accuracy (ACC).
3.3
Evaluation Metrics
Our proposed evaluation metrics move beyond traditional ‘answeronly’ evaluation toward a mechanism-level diagnostic paradigm. To this end, we formulate a novel causal-graph-grounded evaluation framework (Fig. 3). Given a visual input 𝑋 and a query 𝑄, the evaluated model produces a rationale 𝑅 and a final answer 𝑌 to be jointly evaluated: (𝑅, 𝑌 ) = VLM(𝑋, 𝑄).
(1)
LLM-based Judging. The rationale evaluation compares the model’s generated explanation 𝑅, expressed in natural language, against the ground-truth causal graph. We employ a dedicated judge LLM (e.g., GPT-4o [26]) to make binary (True/False) decisions for each evaluation item: M (𝑧, 𝑅) ∈ {0, 1}, where M (𝑧, 𝑅) = 1 if the judge confirms that the target element 𝑧 (an entity, relation, or description) is correctly reflected in 𝑅. The judge operates under strictly fact-based criteria derived from the ground-truth causal graph, verifying the presence and correctness of relevant entities, attributes, and causal dependencies. This design ensures the evaluation remains both semantically aware and objectively grounded, scaling reliably across natural-language rationales. Reliability of the LLM-as-Judge. We validate the judge against a trained human annotator and a second frontier model, and confirm its robustness to paraphrasing; the full agreement study (Cohen’s 𝜅 for LLM–Human, Cross-LLM, and paraphrase stability) is reported in Appendix B.1. Metric Definitions. Beyond answer accuracy, CausalPhys measures whether a model’s reasoning is causally aligned with the underlying physical mechanisms. Given the ground-truth answer 𝑌 ∗ and causal graph G = (V, E), we define four complementary metrics that jointly capture correctness, causal grounding, and descriptive coherence: (1) Accuracy (ACC): measures whether the predicted answer matches the ground truth. ACC = 1{ 𝑌 = 𝑌 ∗ }.
(2)
(2) Entity Faithfulness (EF): assesses whether the rationale explicitly mentions the entities in the causal graph. 1 ∑︁ M (𝑣𝑛 , 𝑅). (3) EF = |V | 𝑣 ∈ V (3) Relation Awareness (RA): tests whether the rationale captures the directed causal dependencies specified in the graph. 1 ∑︁ RA = M ((𝑢𝑛 , 𝑣𝑛 ), 𝑅). (4) |E | (𝑢,𝑣) ∈ E
(4) Description Correctness (DC): evaluates whether the rationale describes each attribute or event consistently with its ground-truth annotation 𝑣𝑑 . ∑︁ 1 DC = M (𝑣𝑑 , 𝑅). (5) |𝐴 ∪ 𝐸| 𝑣 ∈ (𝐴∪𝐸 )
These metrics provide a multi-level diagnostic of causal reasoning, jointly capturing outcome accuracy, structural grounding, and descriptive fidelity.
3.4
Benchmark Results of VLMs on CausalPhys
We evaluate a broad suite of VLMs on CausalPhys, revealing three key patterns that characterize their strengths and limitations in causal physical reasoning. (I) Understanding physical relations remains a fundamental bottleneck for VLMs. While modern Vision–Language Models (VLMs) demonstrate proficiency in purely perceptual tasks, their performance precipitously declines when reasoning necessitates an understanding of physical relations (Fig. 4, Table 3). Relation-intensive subsets, specifically viewpoint transformation and optical inference, pose the greatest challenge; most models perform near-chance (below 40% or ≈ 0.3), exposing a systemic inability to integrate spatial geometry with causal dependencies. Apparent successes in tasks like trajectory prediction are often artifacts of spurious pattern recognition (e.g., object presence) rather than genuine causal synthesis. Consequently, the robust encoding of spatial and dynamical relations remains a critical open challenge.
Tang et al.
Open-Source Models
Closed-Source Models
Model Inte
rnV
L3[ 6
0]
Qw en3 -V
L [5 4]
Qw en2 .5-V L [4 4]
Qw en2 -VL
[55
]
Mis
Phi
-4-M
Lla m
a [2
0]
ulti mo
dal
[1]
tral -Sm all-
GP 3.2
[49
T-4 o
]
GP [42 ]
T-4 o
Gem ini-
Cla
2.5-
-mi
ni[4 2
Fla
]
ude
sh [
13]
-So nne
t-4
Size # Anticipation
78B
32B
3B
7B
11B
5.6B
24B
-
-
-
-
Accuracy (ACC) ↑ Entity Faithfulness (EF) ↑ Relation Awareness (RA) ↑ Description Correctness (DC) ↑ 4 Perception
0.5800 0.6211 0.2338 0.4243
0.5189 0.5910 0.2088 0.3428
0.2944 0.2700 0.0797 0.1217
0.5222 0.5100 0.1710 0.2586
0.3333 0.5290 0.1736 0.2789
0.5533 0.5570 0.1719 0.3012
0.4100 0.4926 0.1808 0.2559
0.6011 0.5935 0.2346 0.3979
0.5911 0.5706 0.2021 0.3303
0.5822 0.5820 0.2061 0.3714
0.5322 0.5798 0.2238 0.3481
Accuracy (ACC) ↑ Entity Faithfulness (EF) ↑ Relation Awareness (RA) ↑ Description Correctness (DC) ↑ Ô Intervention
0.6257 0.7873 0.3976 0.4664
0.5689 0.7822 0.3884 0.4049
0.4490 0.7112 0.2692 0.3483
0.5205 0.6562 0.2325 0.3133
0.3985 0.6965 0.2490 0.3457
0.5573 0.7141 0.2822 0.3501
0.4826 0.7221 0.3680 0.3479
0.5889 0.7738 0.3407 0.4621
0.5983 0.7687 0.3027 0.4014
0.5920 0.7411 0.3092 0.4574
0.5868 0.7349 0.3488 0.4218
Accuracy (ACC) ↑ Entity Faithfulness (EF) ↑ Relation Awareness (RA) ↑ Description Correctness (DC) ↑ ◎ Goal-Orientation
0.5707 0.6547 0.2858 0.5985
0.4799 0.5941 0.2666 0.5516
0.3246 0.3954 0.1493 0.2991
0.4764 0.5563 0.2003 0.5661
0.3211 0.5044 0.1762 0.3781
0.4852 0.5501 0.1925 0.4659
0.4852 0.6240 0.2483 0.5562
0.5707 0.6592 0.2762 0.5742
0.5131 0.6295 0.2496 0.5873
0.5567 0.6451 0.2464 0.5827
0.5672 0.6941 0.3076 0.5278
Accuracy (ACC) ↑ Entity Faithfulness (EF) ↑ Relation Awareness (RA) ↑ Description Correctness (DC) ↑ ¡ Average
0.5799 0.7064 0.2855 0.4339
0.5172 0.6978 0.2856 0.3564
0.3103 0.5372 0.1819 0.1341
0.4906 0.6207 0.1931 0.2475
0.3009 0.5762 0.2078 0.2264
0.4483 0.6440 0.2036 0.3153
0.4796 0.6858 0.2410 0.3817
0.5878 0.6784 0.2376 0.3447
0.5157 0.6764 0.2238 0.3250
0.5439 0.6614 0.1966 0.3385
0.4702 0.6470 0.2238 0.3439
Accuracy (ACC) ↑ Entity Faithfulness (EF) ↑ Relation Awareness (RA) ↑ Description Correctness (DC) ↑
0.5924 0.6968 0.3052 0.4720
0.5268 0.6795 0.2914 0.4040
0.3514 0.4862 0.1729 0.2278
0.5065 0.5871 0.2002 0.3308
0.3445 0.5862 0.2046 0.3073
0.5199 0.6226 0.2166 0.3501
0.4611 0.6287 0.2641 0.3669
0.5888 0.6795 0.2760 0.4397
0.5630 0.6652 0.2468 0.3994
0.5725 0.6597 0.2437 0.4308
0.5428 0.6634 0.2783 0.4037
[2]
Table 3: Benchmark evaluation results on CausalPhys. We report performance of state-of-the-art open- and closed-source VLMs across four domains (Anticipation, Perception, Intervention, and Goal Orientation). Metrics include Accuracy (ACC), Entity Faithfulness (EF), Relation Awareness (RA), and Description Correctness (DC). Results reveal that while models achieve moderate accuracy and entity-level consistency, they struggle with relation-level reasoning (RA), indicating persistent gaps in capturing causal dependencies. These systematic weaknesses underscore the need for causally-informed approaches such as our proposed CRFT.
(II) Performance parity between open-source and proprietary systems. A salient finding from CausalPhys is that open-source models perform on par with proprietary systems across nearly all categories (e.g., InternVL3 vs. GPT-4o), diverging from established trends on general multimodal benchmarks. This parity suggests that proprietary scale and private datasets are insufficient to bridge the gap in physical reasoning. Instead, model capacity emerges as the primary determinant of success: smaller variants (e.g., Qwen2-VL 3B) consistently underperform relative to their larger counterparts (7B, 32B). This pattern underscores that while scaling is essential, current paradigms have yet to translate increased parameter counts into superior causal generalization. (III) The persistent gap between entity recognition and relational reasoning. Across all categories, we identify a consistent Entity Faithfulness (EF)-Relation Awareness (RA) gap: models reliably identify constituent objects and attributes (high EF, ≈ 0.7) but fail to synthesize them into coherent causal structures (low RA, ≈ 0.2-0.3). This disparity highlights a structural decoupling in current VLMs: while the vision encoder successfully extracts individual semantic tokens, the multimodal fusion layers fail to compose these entities into a valid physical world model. This failure to bridge the gap from “what” to “how” represents a critical bottleneck for deploying VLMs in dynamic, real-world environments.
Statistical reliability. A bootstrap sensitivity analysis (10,000 resamples) confirms that these patterns are not artifacts of sample composition: overall accuracy and the model ranking remain stable across resamples within a narrow 95% CI, indicating that CausalPhys yields statistically reliable comparisons. Full numbers and per-model statistics are reported in Appendix B.3.
4
From Answers to Reasons: Causal Rationale Fine-Tuning
The empirical evidence from our benchmark (Sec. 3.4) converges on a key conclusion: optimal VLM performance is achieved when models generate not only the conclusion, but also the causal structure. In light of this observation, we propose Causal Rationale Fine-Tuning (CRFT), which acts as a structural regularizer for the model’s reasoning manifold. By aligning latent reasoning chains with grounded causal structures, CRFT effectively mitigates the ‘Guesser’ behavior prevalent in standard SFT, fostering zero-shot generalization and ensuring that accuracy is a byproduct of robust causal grounding rather than shallow heuristic shortcuts. Gold Rationale Construction. Given an instance (𝑋, 𝑄, 𝑌 ∗, G) as defined in Section 3.1, we generate gold causal rationales 𝑅𝑔𝑜𝑙𝑑 utilizing a teacher LLM (e.g., GPT-4o [42]). Each rationale is required to (i) explicitly reference nodes and edges in G, (ii) trace
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Figure 4: Radar plots comparing 11 tested VLMs over 16 CausalPhys subcategories. Evaluation spans four causal-graphgrounded metrics: Accuracy (ACC), Relation Awareness (RA), Description Correctness (DC), and Entity Faithfulness (EF). intermediate causal implications, and (iii) conclude with 𝑌 ∗ . This ensures that rationales are faithful to the causal graph, providing structured supervision beyond free-form text. Training Objective. For training, we concatenate the gold-rationale and the ground-truth answer into a single sequence (𝑅𝑔𝑜𝑙𝑑 , 𝑌 ∗ ) and fine-tune the target VLM 𝜋𝜃 to maximize its likelihood under a weighted supervision: h ∑︁ LCRFT (𝜃 ) =−E 𝜆𝑟 log 𝜋𝜃 (𝑠𝑡 |𝑋, 𝑄, 𝑠 <𝑡 ) 𝑡 ∈idx(𝑅𝑔𝑜𝑙𝑑 )
+𝜆𝑦
∑︁
i
log 𝜋𝜃 (𝑠𝑡 |𝑋, 𝑄, 𝑠 <𝑡 ) ,
Table 4: Comparison of Qwen2-VL-7B variants on CausalPhys. CoT-SFT is a chain-of-thought SFT baseline with rationale length matched to CRFT, isolating the effect of causal structure from that of supervision richness. Methods
ACC ↑
EF ↑
RA ↑
DC ↑
Qwen2-VL-7B Qwen2-VL-7B SFT (answer-only) Qwen2-VL-7B CoT-SFT
0.5349 0.6762 0.6570
0.5978 0.3247 0.4280
0.2130 0.0911 0.2080
0.2905 0.2645 0.2160
Qwen2-VL-7B CRFT
0.7066
0.5969
0.2554
0.3493
(6)
𝑡 ∈idx(𝑌 ∗ )
where 𝜆𝑟 and 𝜆𝑦 balance rationale and answer supervision. By anchoring fine-tuning to causal rationales, CRFT drives VLMs to internalize causal mechanisms instead of merely memorizing surface correlations. The model is guided not just to predict the correct answer, but to trace why the answer follows, aligning its reasoning with the ground-truth causal graph. This shift transforms evaluation into learning: it produces predictions that are more accurate, reasoning that is more interpretable, and models that are more reliable for physical decision-making. This specific alignment is critically dependent on the CausalPhys, where every instance
comes with explicit causal structure and gold rationale, making CRFT both principled and practically feasible. We implement CRFT using a strategic 90/10 split of the CausalPhys corpus, allocating the majority for causal alignment while reserving a distinct subset for rigorous out-of-distribution evaluation. This configuration is empirically validated to be highly efficient, demonstrating that even a moderate scale of expert-guided rationales can effectively regularize the VLMs’ reasoning manifold and catalyze the emergence of robust causal grounding. Experiment results. For the experiments, we split the dataset into training and testing sets, applied uniformly across all subcategories.
Tang et al.
Causal Rationale Fine-Tuning (CRFT)
Common VLM Reasoning Input Query VLM Reasoning without Causal Inference
✨✨Question: How many hanging lamps are there in the room? A: 1 B: 3
There are three ceiling lamps in the image. They are hanging from a single fixture near the top right corner of the large circular mirror.
Input Query
Golden Rationale
✨Question: How many
The presence of a mirror introduces light reflection, which creates the illusion of additional lamps. As a result, the perceived number of lamps becomes four instead of the actual two......
hanging lamps are there in the room? Answer: 2
Teacher LLM
Causal Graph
Ground Truth
C: 2 D: 4
Ground truth: C
Figure 5: Workflow of Causal Rationale Fine-Tuning (CRFT). Left: Baseline VLM reasoning fails when causal dependencies are not modeled. Right: CRFT uses causal graphs and teacher-generated gold rationales to guide VLMs toward physically grounded and causally coherent reasoning, jointly supervising both the rationale and the final answer.
As shown in Table 4, the SFT (answer-only fine-tuning) results reveal a critical limitation: although SFT achieves a satisfactory Accuracy (ACC) score, its performance on Entity Faithfulness (EF), Description Correctness (DC), and especially Relation Awareness (RA) drops dramatically. This suggests that answer-only supervision encourages the model to optimize for surface-level prediction accuracy, but at the cost of its ability to capture and reflect the underlying causal reasoning process. In other words, SFT fine-tuning tends to make the model behave like a “guesser,” prioritizing concluding final answers based on shallow experience rather than over structured, interpretable reasoning chains. To verify that the gains stem from causal structure rather than longer supervision, we compare against a CoT-SFT baseline with rationale length matched to CRFT but free-form rather than graphgrounded. CoT-SFT trails CRFT on every causal metric (EF 0.4280 vs. 0.5969, RA 0.2080 vs. 0.2554), and its accuracy (0.6570) even dips below answer-only SFT (0.6762). This indicates that unstructured rationales inject noise without coherent causal grounding, and that the structured, graph-anchored signal in CRFT, not rationale length alone, is the key driver of the improvement. In contrast, the proposed CRFT explicitly integrates causal relations into the learning strategy. The results demonstrate that CRFT not only preserves competitive accuracy but also substantially improves EF, DC, and RA scores compared to the original model, the answer-only SFT variant, and the CoT-SFT baseline. This indicates that CRFT effectively encourages the model to ground its answers in a more faithful and structured causal rationale, aligning outputs more closely with human-like reasoning.
Cross-benchmark generalization. To test whether CRFT learns transferable causal reasoning rather than overfitting to CausalPhys, we evaluate the CRFT-trained model on PhysBench [11], an independent physical-reasoning benchmark, without any further tuning. As shown in Table 5, CRFT improves Qwen2-VL-7B by +4.0% overall (58.3→62.3), with consistent gains on Dynamics (+3.8) and Relationships (+8.1) and no degradation elsewhere. The largest improvement on relational reasoning mirrors the EF– RA gap identified in our benchmark, indicating that the causal
Table 5: Cross-benchmark transfer on PhysBench. The CRFTtrained model is evaluated on PhysBench with no further tuning, improving overall accuracy by +4.0%. Model Qwen2-VL-7B (Vanilla) + CRFT
Dynamics
Relationships
Property
Scene
Overall
50.6 54.4
73.0 81.1
61.7 61.7
54.1 56.8
58.3 62.3
grounding instilled by CRFT transfers across benchmarks rather than memorizing CausalPhys-specific patterns.
5
Conclusion
In this paper, we introduced CausalPhys, a benchmark that evaluates physical reasoning through expert-annotated causal graphs providing explicit mechanism-level supervision. It enables systematic and interpretable evaluation of VLM reasoning across perception, anticipation, intervention, and goal-oriented physical tasks. Empirical results show that even state-of-the-art VLMs degrade sharply when causal consistency is required, revealing a persistent gap between visual correlation and causally coherent physical reasoning. To bridge this gap, we propose Causal Rationale FineTuning (CRFT), a fine-tuning paradigm that explicitly aligns VLM reasoning with causal structures. Looking forward, CausalPhys establishes a foundation for studying causal generalization across stochastic dynamics, multi-agent interactions, and embodied environments. As immediate next steps, we are extending this line of work toward AI assurance and data security and privacy, aiming for causally coherent, human-aligned AI systems that are trustworthy in real-world deployment.
6
Ethical Considerations
We have carefully considered the ethical implications of constructing and releasing CausalPhys, and evaluation pipeline to ensure that the benchmark adheres to widely accepted standards of research integrity and social responsibility. • Informed Consent and Human Subjects. All activities comply with the ACM Publications Policy on Research Involving Human Participants. We obtained explicit informed
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
consent from all annotators, who were briefed on research goals and their right to withdraw at any stage. The protocol followed institutional ethical guidelines. • Annotator Welfare. Annotators were compensated at fairmarket rates. We implemented a pre-screening pipeline to ensure no exposure to harmful or offensive content, prioritizing participant psychological safety. • Data Privacy and Licensing. Source visual data (Appendix A.2) strictly adhere to original licensing terms, excluding any content with redistribution restrictions. We performed rigorous manual audits to remove personal identifiers, ensuring no biometric or sensitive metadata is released. • Responsible Usage. The dataset is restricted to non-commercial research. We conducted bias audits to ensure content neutrality and provide documentation to prevent misuse in sensitive automated decision-making contexts.
7
Acknowledgments
This research is supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG-NMLP-2024-003), the National Research Foundation, Singapore and Infocomm Media Development Authority under its Trust Tech Funding Initiative, and the National Research Foundation, Singapore under its Smart Nation & Digital Government Translational R&D Grant (TRANS) (Award No. TRANS2023-TGC03). Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of the National Research Foundation, Singapore, the Agency for Science, Technology and Research, or the Infocomm Media Development Authority. We would also like to thank Hongtian Cai, Jianghan Zhu, Chenfei Liu, Jiageng Wang, Jing Wang, Huiying Zheng, Jiali Yu, and Xuanlin Zhang for their dedicated efforts in data collection, data annotation and figure preparation, which were essential to the completion of this work.
References [1] Marah Abdin, Jyoti Aneja, Harkirat Behl, et al. 2024. Phi-4 Technical Report. arXiv:2412.08905 [cs.CL] https://arxiv.org/abs/2412.08905 [2] Anthropic. 2025. Introducing Claude 4. https://www.anthropic.com/news/claude4. Accessed: 2025-09-25. [3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision. 2425–2433. [4] Samuel Ayman. [n. d.]. Cup Dataset [Kaggle]. Accessed: 2025-11-21. [5] Alisson Azzolini, Junjie Bai, Hannah Brandon, Jiaxin Cao, Prithvijit Chattopadhyay, Huayu Chen, Jinju Chu, Yin Cui, Jenna Diamond, Yifan Ding, et al. 2025. Cosmos-reason1: From physical common sense to embodied reasoning. arXiv preprint arXiv:2503.15558 (2025). [6] Daniel M Bear, Elias Wang, Damian Mrowca, Felix J Binder, Hsiao-Yu Fish Tung, RT Pramod, Cameron Holdaway, Sirui Tao, Kevin Smith, Fan-Yun Sun, et al. 2021. Physion: Evaluating physical prediction from vision in humans and machines. arXiv preprint arXiv:2106.08261 (2021). [7] Susan Carey. 2000. The origin of concepts. Journal of Cognition and Development 1, 1 (2000), 37–41. [8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In European conference on computer vision. Springer, 213–229. [9] Meiqi Chen, Bo Peng, Yan Zhang, and Chaochao Lu. 2024. Cello: Causal evaluation of large vision-language models. arXiv preprint arXiv:2406.19131 (2024). [10] Zhenfang Chen, Kexin Yi, Yunzhu Li, Mingyu Ding, Antonio Torralba, Joshua B Tenenbaum, and Chuang Gan. 2022. Comphy: Compositional physical reasoning of objects and events from videos. arXiv preprint arXiv:2205.01089 (2022).
[11] Wei Chow, Jiageng Mao, Boyi Li, Daniel Seita, Vitor Guizilini, and Yue Wang. 2025. Physbench: Benchmarking and enhancing vision-language models for physical world understanding. arXiv preprint arXiv:2501.16411 (2025). [12] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2024. Scaling Instruction-Finetuned Language Models. J. Mach. Learn. Res. 25 (2024), 70:1–70:53. https://jmlr.org/papers/v25/23-0870.html [13] Gheorghe Comanici, Eric Bieber, Mike Schaekermann, et al. 2025. Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities. arXiv:2507.06261 [cs.CL] https: //arxiv.org/abs/2507.06261 [14] Yutao Cui, Chenkai Zeng, Xiaoyu Zhao, Yichun Yang, Gangshan Wu, and Limin Wang. 2023. SportsMOT: A Large Multi-Object Tracking Dataset in Multiple Sports Scenes. In IEEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023. IEEE, 9887–9897. doi:10.1109/ICCV51070. 2023.00910 [15] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. 2021. The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 43, 11 (2021), 4125–4141. doi:10.1109/TPAMI.2020.2991965 [16] Zhuobai Dong, Junchao Yi, Ziyuan Zheng, Haochen Han, Xiangxi Zheng, Alex Jinpeng Wang, Fangming Liu, and Linjie Li. 2025. Seeing is Not Reasoning: MVPBench for Graph-based Evaluation of Multi-path Visual Physical CoT. arXiv preprint arXiv:2505.24182 (2025). [17] Aaron Foss, Chloe Evans, Sasha Mitts, Koustuv Sinha, Ammar Rizvi, and Justine T Kao. 2025. CausalVQA: A Physically Grounded Causal Reasoning Benchmark for Video Models. arXiv preprint arXiv:2506.09943 (2025). [18] Jiarun Fu, Lizhong Ding, Hao Li, Pengqi Li, Qiuning Wei, and Xu Chen. 2025. Unveiling and causalizing cot: A causal pespective. arXiv preprint arXiv:2502.18239 (2025). [19] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. 2017. The "Something Something" Video Database for Learning and Evaluating Visual Common Sense. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017. IEEE Computer Society, 5843–5851. doi:10.1109/ICCV.2017.622 [20] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. 2024. The Llama 3 Herd of Models. arXiv:2407.21783 [cs.AI] https://arxiv.org/abs/2407.21783 [21] Kristen Grauman, Andrew Westbury, Eugene Byrne, Vincent Cartillier, and al Zachary Chavis et. 2025. Ego4D: Around the World in 3,600 Hours of Egocentric Video. IEEE Trans. Pattern Anal. Mach. Intell. 47, 11 (2025), 9468–9509. doi:10. 1109/TPAMI.2024.3381075 [22] Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. 2021. Embodied intelligence via learning and evolution. Nature communications 12, 1 (2021), 5721. [23] Victor-Louis De Gusseme, Thomas Lips, Remko Proesmans, Julius Hietala, Giwan Lee, Jiyoung Choi, Jeongil Choi, Geon Kim, and al Phayuth Yonrith et. 2025. A Dataset and Benchmark for Robotic Cloth Unfolding Grasp Selection: The ICRA 2024 Cloth Competition. arXiv:2508.16749 [cs.RO] https://arxiv.org/abs/2508. 16749 [24] Yunzhuo Hao, Jiawei Gu, Huichen Will Wang, Linjie Li, Zhengyuan Yang, Lijuan Wang, and Yu Cheng. 2025. Can mllms reason in multimodality? emma: An enhanced multimodal reasoning benchmark. arXiv preprint arXiv:2501.05444 (2025). [25] Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008 (2024). [26] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024). [27] Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, Bo Zhang, Chaoyou Fu, Peng Gao, and Hongsheng Li. 2025. MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency. CoRR abs/2502.09621 (2025). arXiv:2502.09621 doi:10.48550/ARXIV.2502.09621 [28] Zhihuan Jiang, Zhen Yang, Jinhao Chen, Zhengxiao Du, Weihan Wang, Bin Xu, and Jie Tang. 2024. Visscience: An extensive benchmark for evaluating k12 educational multi-modal scientific reasoning. arXiv preprint arXiv:2409.13730 (2024).
Tang et al.
[29] Zhijing Jin, Yuen Chen, Felix Leeb, Luigi Gresele, Ojasv Kamal, Zhiheng Lyu, Kevin Blin, Fernando Gonzalez Adauto, Max Kleiman-Weiner, Mrinmaya Sachan, et al. 2023. Cladder: Assessing causal reasoning in language models. Advances in Neural Information Processing Systems 36 (2023), 31038–31065. [30] Thomas Jiralerspong, Xiaoyin Chen, Yash More, Vedant Shah, and Yoshua Bengio. 2024. Efficient causal graph discovery using large language models. arXiv preprint arXiv:2402.01207 (2024). [31] Kantine. [n. d.]. DOMOTIC PouringCoffee Expert Dataset [Hugging Face]. Accessed: 2025-11-20. [32] Aneesh Komanduri, Karuna Bhaila, and Xintao Wu. 2025. CausalVLBench: Benchmarking Visual Causal Reasoning in Large Vision-Language Models. arXiv preprint arXiv:2506.11034 (2025). [33] Ville Kuosmanen. [n. d.]. AGILEX Clean Pour Water Dataset [Hugging Face]. Accessed: 2025-11-20. [34] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning. PMLR, 19730–19742. [35] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International conference on machine learning. PMLR, 12888–12900. [36] Jianing Li, Xi Nan, Ming Lu, Li Du, and Shanghang Zhang. 2024. Proximity qa: Unleashing the power of multi-modal large language models for spatial proximity analysis. arXiv preprint arXiv:2401.17862 (2024). [37] Zhiyuan Li, Heng Wang, Dongnan Liu, Chaoyi Zhang, Ao Ma, Jieting Long, and Weidong Cai. 2025. Multimodal Causal Reasoning Benchmark: Challenging Multimodal Large Language Models to Discern Causal Links Across Modalities. In Findings of the Association for Computational Linguistics: ACL 2025. Association for Computational Linguistics, Vienna, Austria, 5509–5533. doi:10.18653/v1/ 2025.findings-acl.288 [38] Disheng Liu, Yiran Qiao, Wuche Liu, Yiren Lu, Yunlai Zhou, Tuo Liang, Yu Yin, and Jing Ma. 2025. Causal3d: A comprehensive benchmark for causal learning from visual data. arXiv preprint arXiv:2503.04852 (2025). [39] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35 (2022), 2507–2521. [40] Michael McCloskey, Allyson Washburn, and Linda Felch. 1983. Intuitive physics: the straight-down belief and its origin. Journal of Experimental Psychology: Learning, Memory, and Cognition 9, 4 (1983), 636. [41] Daniel C. Moura, Shizhan Zhu, and Orly Zvitia. 2025. Nexar Dashcam Collision Prediction Dataset and Challenge. In IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2025, Nashville, TN, USA, June 11-15, 2025. Computer Vision Foundation / IEEE, 2583–2591. https://openaccess.thecvf.com/content/CVPR2025W/WAD/html/Moura_ Nexar_Dashcam_Collision_Prediction_Dataset_and_Challenge_CVPRW_ 2025_paper.html [42] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, and et al. 2024. GPT-4 Technical Report. arXiv:2303.08774 [cs.CL] https://arxiv.org/abs/2303.08774 [43] Judea Pearl. 2009. Causality. Cambridge university press. [44] Qwen, :, An Yang, Baosong Yang, Beichen Zhang, et al. 2025. Qwen2.5 Technical Report. arXiv:2412.15115 [cs.CL] https://arxiv.org/abs/2412.15115 [45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763. [46] Goutham Rajendran, Simon Buchholz, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. 2024. Learning interpretable concepts: Unifying causal representation learning and foundation models. arXiv preprint arXiv:2402.09236 (2024). [47] Fatemeh Shiri, Xiao-Yu Guo, Mona Golestan Far, Xin Yu, Gholamreza Haffari, and Yuan-Fang Li. 2024. An empirical analysis on spatial reasoning capabilities of large multimodal models. arXiv preprint arXiv:2411.06048 (2024). [48] Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Martín-Martín, Fei Xia, Kent Elliott Vainio, Zheng Lian, Cem Gokmen, Shyamal Buch, Karen Liu, et al. 2022. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In Conference on robot learning. PMLR, 477–490. [49] Mistral AI Team. 2025. Mistral Small 3: Apache 2.0, 81% MMLU, 150 tokens/s. https://mistral.ai/news/mistral-small-3. Accessed: 2025-09-25. [50] Hsiao-Yu Tung, Mingyu Ding, Zhenfang Chen, Daniel Bear, Chuang Gan, Josh Tenenbaum, Dan Yamins, Judith Fan, and Kevin Smith. 2023. Physion++: Evaluating physical scene understanding that requires online inference of different physical properties. Advances in Neural Information Processing Systems 36 (2023), 67048–67068. [51] Casper Van Engelenburg, Fatemeh Mostafavi, Emanuel Kuhn, Yuntae Jeon, Michael Franzen, Matthias Standfest, Jan van Gemert, and Seyran Khademi.
2024. MSD: A Benchmark Dataset for Floor Plan Generation of Building Complexes. In European Conference on Computer Vision. Springer, 60–75. [52] Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. 2024. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 19757– 19767. [53] Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017. Visual question answering: A survey of methods and datasets. Computer Vision and Image Understanding 163 (2017), 21–40. [54] An Yang, Anfeng Li, Baosong Yang, et al. 2025. Qwen3 Technical Report. arXiv:2505.09388 [cs.CL] https://arxiv.org/abs/2505.09388 [55] An Yang, Baosong Yang, Binyuan Hui, et al. 2024. Qwen2 Technical Report. arXiv:2407.10671 [cs.CL] https://arxiv.org/abs/2407.10671 [56] Jihan Yang, Shusheng Yang, Anjali W Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. 2025. Thinking in space: How multimodal large language models see, remember, and recall spaces. In Proceedings of the Computer Vision and Pattern Recognition Conference. 10632–10643. [57] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. 2019. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442 (2019). [58] Baiqiao Yin, Qineng Wang, Pingyue Zhang, Jianshu Zhang, Kangrui Wang, Zihan Wang, Jieyu Zhang, Keshigeyan Chandrasegaran, Han Liu, Ranjay Krishna, Saining Xie, Manling Li, Jiajun Wu, and Li Fei-Fei. 2025. Spatial Mental Modeling from Limited Views. CoRR abs/2506.21458 (2025). arXiv:2506.21458 doi:10.48550/ ARXIV.2506.21458 [59] Xinyu Zhang, Yuxuan Dong, Yanrui Wu, Jiaxing Huang, Chengyou Jia, Basura Fernando, Mike Zheng Shou, Lingling Zhang, and Jun Liu. 2025. Physreason: A comprehensive benchmark towards physics-based reasoning. arXiv preprint arXiv:2502.12054 (2025). [60] Jinguo Zhu, Weiyun Wang, Zhe Chen, et al. 2025. InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models. arXiv:2504.10479 [cs.CV] https://arxiv.org/abs/2504.10479 [61] Mingye Zhu, Yi Liu, Zheren Fu, Quan Wang, and Yongdong Zhang. 2025. InToken Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback. arXiv:2511.09865 [cs.CL] https://arxiv.org/abs/2511.09865 [62] Mingwei Zhu, Leigang Sha, Yu Shu, Kangjia Zhao, Tiancheng Zhao, and Jianwei Yin. 2023. Benchmarking sequential visual input reasoning and prediction in multimodal large language models. arXiv preprint arXiv:2310.13473 (2023).
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Supplementary Material A Data Collection and Annotation Guidelines Our dataset comprises over 3,000 instances covering a wide range of real-world scenarios, including household activities, sports, traffic, and cooking, making the annotation process both diverse and nontrivial. To ensure annotation quality and domain robustness, we recruited ten well-trained STEM graduate students as annotators, with each annotator responsible for roughly 300 instances spanning two assigned subcategories. The annotation pipeline is structured into three stages: (1) Data Collection: sampling and temporally clipping raw videos from multiple publicly available datasets. (2) Question Construction: designing causal-grounded questions aligned with the four categories: Perception, Anticipation, Intervention, and Goal-Orientation. (3) Data Annotation: creating causal graphs, writing rationales, and producing final labels formatted in JSON.
A.1
Data Collection
We curate data from a broad range of publicly available video and image datasets, including EPIC-Kitchens [15], SportsMOT [14], Something-Something [19], Ego4D [21], CausalVQA [17], MSD [51], Pouring Water [31, 33], Nexar Collision Prediction [41], MindCube [58], Cups [4], and Robotic Clothes [23]. These sources were deliberately chosen because they contain rich temporal interactions, object contacts, force-transfer patterns, and human–object manipulation dynamics, which are elements that naturally encode causal cues essential for constructing our four reasoning categories. Based on assigned subcategories, annotators select source materials that adequately support causal-grounded question creation. All instances are then manually reviewed and annotated following our unified guidelines. Table 6 summarizes the mapping between visual sources and subcategories. Data formats. Our dataset includes three modalities: images, image sequences, and videos. Images may originate from image-centric datasets, individual frames extracted from videos, or frames sampled from image sequences. Image sequences are either sourced directly from sequence-oriented datasets or constructed by sampling up to 8 evenly spaced frames from a video. Video instances are curated as short clips capped at 5 seconds, a choice that balances annotation effort while providing sufficient temporal context for identifying causal dependencies such as state transitions, collisions, occlusions, or tool-use dynamics. Image/Video resolution. Most raw footage appears in high-resolution formats (e.g., 1920 × 1080). To ensure consistent visual appearance and simplify downstream processing, we uniformly downsample all images and videos to 442 × 442. This normalization reduces cross-dataset variability, preserves essential spatial cues for causal interpretation, and maintains compatibility with GPU memory constraints during model training. Scene and causal pattern diversity. The assembled data collectively covers a wide spectrum of causal patterns, including force
propagation, action–reaction events, viewpoint changes, occlusion–disocclusion cycles, object deformation, everyday manipulation tasks, and safety-critical scenarios (e.g., vehicle collisions). Such diversity ensures that the dataset captures both short-term physical dynamics and higher-level causal structures, providing a comprehensive basis for constructing causal graphs and evaluating process-level reasoning in VLMs.
A.2
Question Creation
All CausalPhys questions, answer options, and causal graphs were entirely authored by our annotators and do not reuse, rephrase, or re-annotate any existing benchmark questions. The only reused component is the raw visual frames, which serve solely as the visual input. We categorize causal-grounded questions into four major types: Perception, Anticipation, Intervention, and Goal-Orientation, each targeting a different level of the causal reasoning hierarchy. Perception. Perception questions assess factual understanding that requires reasoning beyond surface-level recognition. Typical templates include “Can . . . be . . . ?”, “What is . . . ?”, or “Where is . . . ?”. Examples include containability (“Can the cups be nested?”), which requires structural reasoning about object geometry, and spatial queries such as “Where is the chair relative to the camera?” that may require analyzing reflections or occlusions. Anticipation. Anticipation questions require predicting future outcomes based on current observations. Each question includes an explicit evidence cue from which the prediction must be derived. Common templates include “Will . . . ?”, “What will . . . ?”, or “Where will . . . ?”. Examples include fluid flow (“Where will the liquid flow?”) and collision prediction (“Will the car collide?”). Intervention. Intervention questions introduce a hypothetical action or modification, corresponding to a causal do-operator, and evaluate the effect under this manipulated scenario. They are typically phrased as “If . . . , what will . . . ?”. Examples include viewpoint transformation (“If the viewpoint changes, where will the door appear?”) and action substitution (“If the defense player could jump one meter higher, would he block the ball?”). Goal-Orientation. Goal-Orientation questions specify a target objective and ask for the optimal action required to achieve it. The reasoning must incorporate constraints, affordances, and physical feasibility in the scene. Templates include “To achieve . . . , what should . . . ?” or “If we want to . . . , what should . . . ?”. Examples include tool assignment (“If we want to disassemble the toy car, which tool should we use?”) and trajectory selection (“If we want to score a goal, which direction should we shoot?”). Specific Instructions. Annotators follow a standardized set of instructions to ensure consistency and clarity: • Each question must be grounded in an image, a sequence of images, or a video clip. • All questions must be written in clear, grammatically correct English. • Questions must be unambiguous and answerable exclusively via the provided multiple-choice options.
Tang et al.
Table 6: Mapping between publicly available visual sources and the subcategories of CausalPhys. Only the raw visual inputs (video frames or images) originate from these datasets while the questions are self-designed. Checkmark (✓) indicates coverage.
Subcategory
Type Perception Scene Reconstruction Mechanics Reasoning Containability Optical Inference Intervention Viewpoint Transformation Spatial Manipulation Temporal Shifting Action Substitution Goal-orientation Object Relocation Trajectory Optimization Tool Assignment Behavior Selection Anticipation Collision Forecasting Deformation Fluid Flow Intention Speculation
[21 ]
C VQ ausal A[ 17]
Kit Epic c he n[1 5]
Video
Video
Video
Eg o4D
✓
MS D
[51
]
Wa Pour ter ing [31 , 33
S MO ports T[1 4]
Ne x Pre Colli ar dic sion tio n[4 1]
Seq. Images
Video
Video
],
Image
✓
58]
R Clo obot the ic s[2 3]
S Som ome eth thing ing [19 ]
Seq. Images
Video
Video
Cu
ps[
4]
Image
✓ ✓
✓
✓ ✓ ✓ ✓ ✓
✓ ✓
✓ ✓
✓ ✓
✓ ✓ ✓
✓ ✓ ✓ ✓
✓ ✓ ✓ ✓
✓ ✓ ✓ ✓ ✓
✓
✓
✓
• Annotators must submit: (i) the question, (ii) the answer options, (iii) the correct answer, and (iv) the causal graph. • All fields must strictly follow the required formatting specifications to maintain structural consistency. Review Process. To ensure correctness, fairness, and robustness, all annotations undergo a three-stage review pipeline: (1) Self-check: initial verification by the annotator. (2) Independent peer review: cross-review by another annotator to detect ambiguity or logical inconsistencies. (3) Textual/visual bias check: each question is validated to ensure it cannot be answered using only the text or only the visual input. This multi-layered process ensures high-quality, causally grounded questions and promotes consistency across the benchmark.
A.3
Cu be[
Mi nd
• "graph": The annotated ground-truth causal graph, consisting of nodes (objects, attributes, or events) and edges (relations between nodes). Causal Graph Annotation. Although the final annotations are stored in JSON, directly constructing causal graphs in raw JSON is cumbersome and error-prone. To streamline the annotation process, we employ Mermaid as an intermediate graph-editing interface. Mermaid provides real-time visualization, intuitive graph manipulation, and significantly reduces structural annotation errors. For example: If the causal entity corresponds to the object vase, the node is annotated as O:Vase. If the entity corresponds to the attribute: the shape of the vase is a narrower opening and wider body, the node is annotated as A: shape: narrower opening and wider body. The annotated causal graph is saved in .mmd format as shown in Figure 7 and then automatically converted into the standardized JSON format.
Annotation Structure
Annotation example. As illustrated in Figure 6, each annotation instance is stored in JSON format and contains the following keys: • "id": The identifier of the current data instance. • "question": The question created for this instance. • "ground_truth_answer": The ground-truth answer, denoted by one of A|B|C|D. • "path": The path(s) to the visual data. This field supports three types of input: a single image path, a single video path, or a list of image paths representing an image sequence. • "category": One of the four main categories. • "sub_category": One of the sixteen subcategories.
A.4
Annotation Platform
To facilitate large-scale data labeling, we designed a dedicated annotation platform that automatically iterates through the video and image dataset, presenting each instance sequentially for annotation. This system significantly reduces manual effort by streamlining the workflow and ensuring that annotators can focus entirely on content creation rather than file handling. For each data instance, the platform provides an interactive graphical user interface (GUI), as illustrated in Figure 8. On the left-hand side of the interface, the video (or image) corresponding to the current instance is displayed, allowing annotators to carefully
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Reliability Check
EF
RA
DC
LLM–Human Agreement (𝜅) 0.86 0.83 0.76 Cross-LLM Consistency (𝜅) 0.89 0.88 0.78 Paraphrase Stability (𝜅) 0.92 0.86 0.96
Overall 0.82 0.84 0.90
Table 7: Reliability evaluation of the LLM-as-Judge on 320 samples (20 per subcategory). LLM–Human reports Cohen’s 𝜅 between GPT-5.4 and a human annotator; Cross-LLM reports 𝜅 between GPT-5.4 and Claude Sonnet 4.6; Paraphrase Stability reports consistency under semantically equivalent rewrites.
observe the scene. On the right-hand side, several editable blocks guide the annotation process. Specifically, annotators can: • Formulate a Question. Each instance requires a question relevant to the visual content. The question is editable in a designated text block, enabling annotators to phrase it in a multiple-choice format that reflects the underlying causal or goal-oriented reasoning of the scenario. • Define Answer Options. Alongside the question, annotators specify multiple-choice options (e.g., actions that could be taken in the video). These options allow for structured evaluation of models on causal reasoning and decisionmaking. • Select the Ground-Truth Answer. From the defined options, annotators must identify the correct choice, which is recorded as the ground-truth label. This ensures that the dataset captures unambiguous supervisory signals for training and evaluation. • Construct the Causal Graph. To go beyond questionanswer annotation, our platform integrates a causal graph editor using Mermaid syntax. Annotators can input nodes (representing objects, attributes, or events) and edges (capturing causal dependencies). The tool automatically renders a visual preview of the graph, allowing annotators to validate the structure before saving. This ensures that every instance is accompanied not only by a question-answer pair but also by a structured causal representation. GUI provides utility functions such as previewing the rendered causal graph, clearing inputs, saving the current annotation, or moving to the previous/next video in the dataset. Together, these functions make the annotation process more efficient, standardized, and less error-prone. Overall, this annotation platform supports a unified pipeline for curating complex datasets that combine natural video/image observations with structured causal annotations, thus laying the foundation for high-quality benchmarks in causal reasoning tasks.
B Additional Experiments and Analyses B.1 LLM-as-Judge Reliability Validation Prior work shows that LLM-based evaluators provide semantically informed, objectively grounded judgments [27]. We validate our setup on 320 instances. While the main paper uses GPT-4o as the judge, here we re-judge with two stronger, more recent SOTA
Teacher
ACC ↑
EF ↑
RA ↑
DC ↑
CRFT (GPT-4o) CRFT (Claude Sonnet 4)
0.7066 0.7133
0.5969 0.6133
0.2554 0.2667
0.3493 0.3333
Table 8: Teacher-swap analysis. Re-generating all gold rationales with an independent teacher (Claude Sonnet 4) yields comparable CRFT performance, confirming that CRFT learns teacher-invariant causal structure. Rationale-level similarity between teachers: ROUGE-L = 0.38, BERTScore = 0.91.
models (GPT-5.4 and Claude Sonnet 4.6) and assess three complementary axes of reliability (Table 7). Both agreement settings exceed 0.80 (almost-perfect), and LLM–Human agreement is comparable to Cross-LLM, confirming that the judge is human-comparable, reproducible, and not sensitive to the choice of judge model. Human Cross-Validation. Comparing GPT-5.4 against a trained human annotator (blinded to the model outputs) on the EF/RA/DC questions yields a strong overall agreement of Cohen’s 𝜅 = 0.82 (EF 0.86, RA 0.83, DC 0.76), indicating human-comparable reliability when evaluating causal-graph–grounded rationales. Cross-LLM Consistency. Re-judging the same instances with Claude Sonnet 4.6 under identical prompts gives 𝜅 = 0.84 overall (EF 0.89, RA 0.88, DC 0.78). Because this matches the LLM–Human level, the evaluation is stable across frontier judges and not an artifact of the specific model. Paraphrase Stability. Rewriting each question into three semantically equivalent paraphrases and counting a verdict as consistent only when all paraphrases agree, the judge achieves a 0.90 consistency rate, showing strong invariance to surface wording.
B.2
Teacher-Invariant Causal Structure
To verify that CRFT learns the underlying causal structure rather than imitating the style of a specific teacher LLM, we re-generate all training rationales with an independent teacher (Claude Sonnet 4) under the identical graph-anchored prompt and re-train Qwen2VL-7B with the same configuration. At the rationale level the two teachers differ substantially in surface form (ROUGE-L = 0.38) yet convey near-identical causal content (BERTScore = 0.91), indicating that the supervision is graph-anchored rather than style-anchored. Consequently, the two resulting models perform comparably across all metrics (Table 8; ACC 0.707 vs. 0.713, RA 0.255 vs. 0.267), confirming that CRFT captures teacher-invariant causal structure rather than teacher-specific stylistic priors.
B.3
Bootstrap Sensitivity Analysis
To confirm the results are not artifacts of sample composition, we resample the evaluation set with replacement 10,000 times and recompute the metrics, reporting 95% CIs as the 2.5/97.5 percentiles. Over the full benchmark (N = 3,062), overall accuracy is 58.9% (bootstrap std 0.88%, 95% CI [57.2%, 60.6%]). The narrow interval and the stable model ranking across resamples indicate that CausalPhys yields statistically reliable comparisons; per-model statistics are released with the artifacts.
Tang et al.
Figure 6: An illustrative example of our JSON-based causal annotation format. Each instance includes a question, visual input, ground-truth answer, and a structured causal graph. This format ensures that each component is easy to access, verify, and integrate into downstream evaluation pipelines.
Figure 7: Mermaid graph annotation of a causal graph. It supports real-time visualization and easy editing, enabling annotators to interactively refine node–edge structures.This format is also highly interpretable and can be seamlessly converted to our JSON causal annotation schema.
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Figure 8: Designed GUI-based annotation platform. It integrates visual input display, question design, causal graph editing, and rendered graph preview, providing an easily accessible and well-visualized interface that greatly reduces annotation effort.
Tang et al.
C
Prompts
This section describes the prompt configurations used across different LLMs and VLMs under various evaluation settings. We present the prompts for: (i) VQA with rationale, (ii) VQA answer-only, (iii) gold rationale generation, and (iv) rationale correctness evaluation.
C.1
VQA Prompt with Rationale
The VLM reasoning prompt instructs the model to produce a stepby-step rationale prior to answering the question. The agent is asked to carefully analyze the input image or image sequence, articulate its reasoning process, and then provide the final answer. This prompt is used for both VLM evaluation and for training and evaluating CRFT, as it explicitly elicits interpretable intermediate reasoning. VQA Prompt with Rationale You are a precise Vision–Language QA assistant. ## Goals - Read the user’s question and (if provided) a SEQUENCE of images in the given order - Provide a one-sentence rationale and your answer ## Sequence Handling - If multiple images are provided, treat them as an ordered sequence (e.g., frames of a video) - Consider temporal consistency and cross-frame cues when reasoning ## Conservative Reasoning - Rely only on information available in the images and the question - Be explicit and concise; avoid speculation ## Hard Format Constraints (must obey exactly) Output MUST include: 1. Generate a clear, step-by-step rationale (max 8 sentences) wrapped in <rationale>...</rationale> 2. Your answer must be in EXACTLY ONE CAPITAL LETTER: A, B, C, or D wrapped in <result>...</result>
C.2
Gold Rationale Generation
Based on the ground-truth causal graph, the teacher LLM is required to generate a gold rationale that reflects the reasoning implied by the nodes and edges. The agent is provided with the question, causal graph,visual input and ground-truth answer. The rationale must be written in natural language as a reasoning process leading to the final answer, rather than as a description of the causal graph itself. This gold rationale will be used for further CRFT supervision.
relationships ## Task Generate a clear, step-by-step rationale that answers the question in natural language. ## Requirements 1. Write an objective, answer-focused rationale in natural language 2. Treat the supporting information as reference only (do not describe it) 3. Write ONE coherent paragraph (max 8 sentences) that flows naturally 4. Include relevant elements from the reference only when needed for reasoning (do not enumerate them) 5. Follow the correct logical order: causes must appear before their effects 6. If an element has a description, state it clearly and exactly as provided 7. Use natural, everyday language (avoid terms like "entity", "relation", "graph", "structure") 8. Ensure proper grammar and spelling 9. Make the explanation easy to understand and self-contained 10. Present the reasoning as a logical analysis of the situation ## Output format - Single paragraph only - No bullet points, lists, or special formatting - Plain English text - Complete explanation that follows the logical reasoning sequence ## Important The supporting information (entities, descriptions, relations) is for reference only. Do NOT describe or list it. Use it implicitly to justify the answer. Focus on explaining why the answer is correct in plain language.
C.3
Rationale Correctness Judgement Prompt
Based on the defined rationale evaluation metrics, three categories of questions will be constructed. The evaluator LLM will be given a rationale generated by the VLM along with a sequence of true/false questions. Its task is to evaluate the rationale by answering each question and output the results as a list in YAML format. This prompt is used in rationale evaluation. LLM as a judge causal relationship prompt You are a meticulous evaluator. Read the problem and the model’s rationale, then answer a list of True/False questions strictly based on that rationale. Do not use outside knowledge or the image. If the rationale is ambiguous or does not state the fact, answer False.
Gold Rationale Generation Prompt You are a reasoning assistant that analyzes visual scenarios and provides step-by-step reasoning. ## Input format You will receive: - A question about the visual scenario - A image or a sequence of images - A ground truth answer (A, B, C, or D) - Supporting information about objects, their properties, and
Answer using ONLY the specified YAML schema. Do not add extra commentary. INPUT - problem: The multiple-choice question with options - rationale: The model’s rationale paragraph(s) - questions: A list of True/False questions. Each item has: - id: opaque identifier (string)
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
- text: the T/F question JUDGING PRINCIPLES - True only if the rationale explicitly supports the statement with clear mention or an unambiguous entailment. - False if absent, unclear, contradicted, or only weakly implied. - Allow synonyms/coreference (e.g., “kicker” for “Fighter”), but do not infer beyond text. - For causal relation questions, require a clear causal/influence expression (e.g., X causes/leads to/affects Y; Y depends on X). Mere co-occurrence is insufficient. OUTPUT FORMAT (YAML) answers: - id: <string> answer: true|false EXAMPLE problem: "Which direction should he kick to hit the target? A. Left B. Right" rationale: "The pad is to the left of the kicker; therefore he should kick left to hit it." questions: - id: "0" text: "Is object ’pad’ mentioned in the rationale?" - id: "1" text: "Is the causal relation between ’Kick direction’ and ’Pad Location’ correctly expressed?" answers: - id: "0" answer: true - id: "1" answer: true
D Experiment Details D.1 Metric Questions Construction In Section 3.2 of the main paper, we introduced three LLM-as-Judge evaluation metrics, Entity Faithfulness (EF), Relation Awareness (RA), and Description Correctness (DC). To reduce the degrees of freedom in free-form model outputs and ensure stable judgement, we convert each metric into a set of canonical True/False verification questions. For any entity represented as 𝑣 = (⟨type⟩, ⟨name⟩, ⟨description⟩), and for any entity pair (𝑣 1, 𝑣 2 ), the corresponding questions are instantiated as follows: • EF: “Does the ⟨type1 ⟩ ⟨name1 ⟩ appear in the rationale?” • DC: “Is the ⟨type1 ⟩ ⟨name1 ⟩ described as ‘⟨description1 ⟩’ in the rationale?” • RA: “Is a direct causal relation between ⟨type1 ⟩ ⟨name1 ⟩ and ⟨type2 ⟩ ⟨name2 ⟩ explicitly stated in the rationale?”
Figure 9: Detailed evaluation of CRFT. CRFT shows promising improvements across multiple reasoning categories, with especially strong gains in Anticipation, suggesting that causal-rationale fine-tuning is particularly effective for forward physical reasoning.
D.2
CRFT Details
Graph-Anchored Rationale Construction. Each instance is paired with a human-annotated causal graph 𝐺 = (𝑉 , 𝐸) containing object, attribute, and event nodes. We prompt GPT-4o to convert this graph into a textual rationale that faithfully reflects its structure. GPT-4o does not generate the graph itself; it only verbalizes the humanprovided graph, making the supervision graph-anchored rather than model-anchored. We further verify that the resulting rationales neither introduce information absent from the graph nor directly recite the graph. Training Objective. As described in Section 4 of the main paper, CRFT minimizes a mixed objective consisting of (i) the rationalelevel cross-entropy loss Lrat and (ii) the answer-level loss Lans . Following prior work showing that combining intermediate rationale supervision with final-answer supervision yields a denser and more stable learning signal [12], we optimize both components: L = 𝜆rat Lrat + 𝜆ans Lans . This mixed supervision is particularly useful in our setting, where MCQ answers are often only one token and answer-only optimization can be unstable. A small sweep over 𝜆rat ∈ 0.1, 0.2, 0.5 and 𝜆ans ∈ 0.5, 1, 2 showed that 𝜆rat = 0.2 and 𝜆ans = 1 yield the most stable optimization, consistent with prior findings on balanced rationale–answer weighting [61]. Training Configuration. We use a 9:1 split, resulting in 2,765 training instances and 297 test instances. Qwen-VL-7B is fine-tuned with LoRA on a single NVIDIA L40S 40GB GPU for 6 epochs, taking approximately 8 hours. We use batch size 8, learning rate 5 × 10−5 , LoRA rank 16, and dropout 0.05. Detailed Evaluation of CRFT.. Figure 9 evaluates CRFT across all sixteen CausalPhys subcategories using ACC, EF, RA, and DC. CRFT shows consistent improvements across diverse physical reasoning tasks, especially in anticipation-oriented scenarios.
Tang et al.
E
Case Study
Based on the case studies below, we observe a consistent pattern: when a model’s rationale is well aligned with the ground-truth causal graph, its final answer is usually correct. In contrast, when this alignment breaks down, the model’s prediction becomes unreliable and often appears to rely on guesswork rather than genuine reasoning. The following cases provide detailed examples from three representative models on selected instances.
This highlights the distinctive value of CausalPhys: its finegrained causal annotations expose reasoning failures that conventional VQA benchmarks cannot detect, enabling more diagnostic evaluation and guiding the development of VLMs with genuinely causal reasoning.
Fluid Flow
Collision Prediction Question: Based on the current frames, will the car collide? Options: A. Yes B. No Ground-truth: A
Causal Graph:
GPT-4o Response: A. Based on the sequence of frames, a collision appears highly likely. The silver car on the left is aggressively merging into the same lane as the dashcam car, with very limited space between them. As the sequence progresses, the silver car's rear moves directly into the dashcam car’s path, and by the final frame, there is almost no visible gap between the two vehicles. Given the relative positions and trajectories, unless one of the drivers reacts immediately—such as braking or swerving—a side-swipe collision at the front-left of the dashcam car and the rear-right of the silver car seems imminent.
Gemini 2.5 Flash Response: A. Based on the sequential dashcam frames, a collision is extremely likely and appears to be in the process of occurring. The white car on the left is seen rapidly moving closer to the vehicle recording the footage across all three images. The gap between the two vehicles closes dramatically from the top frame to the middle frame, and by the bottom frame, the proximity is so close that the white car has either made contact with the recording vehicle or an impact is unavoidable due to the high speed, heavy rain, and lack of space for an evasive maneuver.
Claude Sonnet 4.5 Response: B. Based on these three sequential dashcam frames, no imminent collision appears likely at this moment. The vehicle is traveling in rainy conditions with reduced visibility, and there's a white/light-colored vehicle visible ahead in the same lane. The following distance appears adequate, and there's no evidence of sudden braking or rapid closing distance between the frames. However, the wet road conditions are a significant safety concern that requires extra caution. The driver should continue maintaining a safe following distance and reduce speed appropriately for the weather conditions. While the current situation appears stable, wet pavement reduces traction and increases stopping distances, so continued vigilance is essential.
Question: Where will the liquid from the container expected to be poured? Options: A. Into the other container B. No outflow C. Onto the table D. Onto the hand
Ground-truth : C
Causal Graph:
GPT-4o Response: A.The robotic arm is holding a tilted container positioned directly above a transparent cup on the table. The orientation of the container indicates a pouring motion is occurring or about to occur. The second frame confirms that the container is tipped further, aligned with the mouth of the cup. There is no hand present in the scene, and the table is not directly underneath the container's spout. The most logical destination for the liquid, given the alignment and position, is into the cup. Therefore, the liquid is expected to be poured into the other container.
Gemini 2.5 Flash Response: A. The images show a robotic arm holding a container (a pitcher or measuring cup) that is tilted downward. Directly beneath the spout of the container is an empty glass or cup. The arm's movement is clearly positioned to direct the liquid from the upper container into the lower glass. Therefore, the intended and expected action is the pouring of the liquid from the container into the other container (the glass).
Claude Sonnet 4.5 Response: A. The images show a red robotic arm holding a container in a tilted pouring position over another container on the table. The positioning and angle of the container being held suggests a controlled pouring motion. The receiving container is directly beneath the tilted container, positioned to catch any liquid. This setup indicates a deliberate transfer operation between containers. The robotic arm's precise positioning makes it clear the liquid is intended to flow into the container below, not onto the table or hand.
Figure 11: Sample cases of Fluid Flow Figure 10: Sample cases of Collision Prediction
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Intention Speculation
Deformation
Question: What will the person's next move be? Options: A. Pick up cafetiere plunger B. Pick up cutlery C. Close cupboard D. Put plate into cupboard
Question: What will the deformation of the garment be if the robot arm continue its action? Options: A. Compress Vertically B. Strech Horizontally and Vertically C. Compress Horizontally and Vertically D. No deformation expected
Ground-truth : C
Ground-truth : B
Causal Graph:
Causal Graph:
GPT-4o Response: D. The person is holding a plate near an open cupboard that already contains stacked plates. Their hand and the plate are aligned with the shelf space inside the cupboard. The scene shows no movement toward the cafetiere plunger or cutlery. The cupboard is open, indicating the person is in the process of placing something inside rather than closing it immediately. Therefore, the most likely next move is to put the plate into the cupboard.
GPT-4o Response: B. The robot arm is grasping two corners of a soft fabric garment and appears to be pulling them apart. In the second frame, the garment is visibly stretched outward in both horizontal and vertical directions. The fabric is under tension from both grippers, which are moving away from each other at an angle. This setup indicates a biaxial stretching action. There is no compression or folding motion observed. Therefore, the garment will stretch horizontally and vertically as the robot continues its action.
Gemini 2.5 Flash Response: A. The images show a person placing a white plate onto the top of a stovetop burner or an empty pot on the stovetop. In the third frame, the hand is still near or on the plate, but the hand's fingers are also pointing toward a shiny object to the right of the plate. This shiny object, with a visible handle, is the plunger part of a cafetière (French press) which is typically placed on the grounds after the hot water is added. Given the kitchen setting and the likely task of making coffee, the person will probably pick up the cafetière plunger next to continue the preparation.
Gemini 2.5 Flash Response: C. The images show a robotic arm holding a blue garment or cloth on a table. The first frame shows the arm grasping the cloth, and the second frame shows the arm pulling the cloth toward itself. This action, pulling the cloth from its anchored position on the table, will cause the material to be gathered and bunched up as it is pulled. Therefore, the resulting deformation will be a compression in both the horizontal and vertical directions as the material is pulled into a smaller, thicker volume.
Claude Sonnet 4.5 Response: A. The sequential frames show a person's hand moving in a kitchen environment near what appears to be a cafetiere (French press) on the counter. The hand positioning and movement trajectory across the three frames suggest the person is reaching toward the cafetiere area. The natural progression of making coffee with a French press involves picking up the plunger after the coffee has steeped. There are no visible plates, cutlery being handled, or cupboard doors open in the frames. The context and hand motion strongly indicate the person is about to grasp the cafetiere plunger as the next logical action.
Claude Sonnet 4.5 Response: B. The images show a robotic arm manipulating a blue garment on an orange surface in front of shelving units. Between the two frames, the robot arm appears to be in different positions, suggesting it's actively moving and pulling the fabric. The garment shows signs of being spread out or stretched across the surface. Given the robot's grasping action and the typical behavior of fabric manipulation tasks, continued pulling would cause the material to stretch in multiple directions. The fabric would extend both horizontally as it's pulled across the table surface and vertically as tension is applied, resulting in stretching in both dimensions rather than compression.
Figure 12: Sample cases of Intention Speculation
Figure 13: Sample cases of Deformation
Tang et al.
Tool Assignment Question: Q: Which tool should be use to disassemble the toy? Options: A. Box-End Wrench B. Open-End Wrench C. Hex Screwdriver D.Phillips screwdriver Ground-truth : B Causal Graph:
Object Relocation Question: To take out the pot, how many levels should the hand move? Options: A. One layer upward B. Two layers upward C. One layer downward D. No movement required
Ground-truth : A
GPT-4o Response: D. The toy has screws visible on its wheels and joints. The screws have a crossshaped head, which is characteristic of Phillips screws. Neither box-end nor open-end wrenches fit this type of screw, and a hex screwdriver would be for hexagonal heads. Therefore, the correct tool to disassemble this toy is a Phillips screwdriver.
Gemini 2.5 Flash Response: D. The image displays a yellow toy construction vehicle that is assembled using screws. A close look at the heads of the visible screws on the wheels, the cab, and the scoop shows a cross-shaped indentation. This cross-shaped pattern is the defining feature of a Phillips head screw. Therefore, the appropriate tool required to disassemble this toy is a Phillips screwdriver.
Claude Sonnet 4.5 Response: D. The image shows a yellow toy construction vehicle (appears to be a front loader or excavator) being handled by someone. Toy vehicles like this are typically assembled with Phillips head screws, which are the most common fastener type used in toy manufacturing. The visible construction and design of the toy suggest standard toy assembly methods that rely on crosshead screws. Phillips screwdrivers are the standard tool for disassembling plastic toys, as they provide good grip and are safe for consumer products. Hex screwdrivers and wrenches are less common in toy construction and would typically be visible if present.RetryClaude can make mistakes. Please double check responses.
Figure 14: Sample cases of Tool Assignment
Causal Graph:
GPT-4o Response: B. The hand is currently reaching toward a lower shelf that contains vegetables and other items. The pot is clearly placed on the shelf directly above the hand’s current position. To grasp and take out the pot, the hand needs to move upward by one shelf level. There is no need to reach two layers up or down, and the current hand position is not yet aligned with the pot. Hence, one upward movement is sufficient.
Gemini 2.5 Flash Response: A. The image shows a hand reaching into the refrigerator, currently on the second shelf from the top (the first main shelf below the top drawer). The pot is clearly visible resting on the shelf directly above the hand's current position. Therefore, to reach and take out the pot, the hand needs to move one layer upward.
Claude Sonnet 4.5 Response: C. The image shows a hand reaching into a refrigerator with a fisheye lens view. The hand is currently positioned at an upper shelf level in the refrigerator. A pot or container appears to be visible on a lower shelf below where the hand currently is. To reach and take out the pot, the hand would need to move down one shelf level from its current position. The refrigerator shelves are clearly delineated, showing distinct levels, and the pot is one layer below the hand's current reach.
Figure 15: Sample cases of Object Relocation
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Trajectory Optimization Question: Which direction should the player shoot the ball to goal? Options: A. Left B. Right C. Straight ahead
Ground-truth : B
Action Selection Question: To make the lighter fall onto the table, what should be done? Options: A. Lift the plate C. Drag the plate horizontally
B. Flip the plate D. Drop the plate
Causal Graph: Ground-truth : B
Causal Graph:
GPT-4o Response: A.The goalkeeper is already leaning slightly to their right (the player’s left), but hasn't committed to a dive yet. However, the right side of the goal (from the player's view) is crowded with visual distractions and possibly more expected by the keeper. The left side (from the kicker’s view) appears more open and has more net space available. Given the positioning and body language, shooting left increases the chance of scoring. It's a strategic placement to avoid the goalkeeper’s anticipated reach.
GPT-4o Response: B. The lighter is currently resting on top of the plate. Simply lifting or dragging the plate horizontally would keep the lighter on the plate due to gravity and friction. Dropping the plate would likely keep the lighter on it until impact. Flipping the plate would invert its surface, causing the lighter to lose support and fall onto the table. Therefore, flipping the plate is the direct action to make the lighter fall.
Gemini 2.5 Flash Response: A. The image captures a soccer player preparing to take a penalty kick against a goalkeeper. The goalkeeper has already started to lean and move toward the viewer's right side of the goal (the player's left). To maximize the chance of scoring, the player should aim for the opposite side of the goal, which is the viewer's left side (the player's right), where the goalkeeper is creating a large, unguarded space. Shooting Left will capitalize on the keeper's committed movement to the other side.
Gemini 2.5 Flash Response: B. The lighter is resting inside a small brown plate, which is being held by a hand. To make the lighter fall onto the table from this position, the plate must be turned over, or flipped, allowing gravity to pull the lighter out. Lifting the plate would keep the lighter inside, dragging it horizontally would move both items together, and dropping the plate would likely cause the plate and lighter to fall together, not the lighter alone onto the table. Flipping the plate is the action that will release the lighter onto the surface below it.
Claude Sonnet 4.5 Response: A. The image shows a penalty kick scenario with a player preparing to shoot and a goalkeeper in the goal. The goalkeeper appears to be positioned slightly toward the right side of the goal from the shooter's perspective. In penalty situations, shooters typically aim away from where the goalkeeper is positioned or leaning to maximize scoring chances. Since the goalkeeper shows positioning bias toward the right, shooting to the left would exploit the open space and provide the best scoring opportunity. The left side of the goal appears more vulnerable based on the goalkeeper's stance and position.
Claude Sonnet 4.5 Response: B. The image shows a hand holding a brown plate with a lighter resting in the center of it above a textured surface (appears to be a table or ground). The lighter is currently stable on the plate's surface due to gravity. To make the lighter fall onto the table below, the plate would need to be flipped or turned over, which would cause the lighter to slide or drop off due to the change in orientation. Simply lifting the plate would keep the lighter on it, dragging horizontally wouldn't dislodge it, and dropping the plate would likely keep the lighter on the plate as they fall together. Flipping creates the necessary change in angle to release the lighter.
Figure 16: Sample cases of Trajectory Optimization
Figure 17: Sample cases of Action Selection
Tang et al.
Viewpoint Transformation
Spatial Manipulation
Question: Where is the TV located respective to the camera in the second image? Options: A. Front B. Back C. Left D. Right
Question: Can the player in yellow could still receive the ball if it were passed 50 cm more to the right from the current trajectory? Option: A. Yes B. No
Ground-truth : C
Ground-truth : A
Causal Graph:
Causal Graph:
GPT-4o Response: B. In the second image, the camera is facing toward the front of the shoe, which is placed on a table. In the previous image (now shown second), the same table setup is present, but from the opposite side, showing the entryway and hall. Since the first image is taken from the direction opposite the TV (which was visible in the earlier question), it means the TV is behind the camera in this second image. Thus, from the perspective of the current camera view, the TV is located at the back.
GPT-4o Response: A. In the last frame, the player in yellow is already extending his arm to the right and is positioned ahead of the ball's trajectory. The pass is close to reaching him, and his body momentum is directed forward with room to adjust slightly. A shift of 50 cm to the right is within a reachable margin, especially at this sprinting pace. No defenders are tightly obstructing that side, and he appears ready to receive it. Therefore, he could still receive the ball with that adjustment.
Gemini 2.5 Flash Response: B. The second image shows the camera facing the opposite direction from the first image. The camera is positioned on the bed, pointing out of the room toward a doorway leading to what appears to be a dining area. The TV, which was visible in the background of the first image, is now located behind the camera's current viewing direction.
Gemini 2.5 Flash Response: A. The images show a basketball game where a player in yellow is running on the court near the three-point line. This player is being closely guarded by a player in blue, but the yellow player is ahead of the defender and moving towards the basket. A 50 cm shift to the right from the current passing trajectory would still place the ball in an area the running player could likely reach or adjust to, as they are already in motion and have a slight lead on their defender. They would simply need to take a step or two to the right to receive the pass.
Claude Sonnet 4.5 Response: B. In the second image, the camera is positioned looking at shoes on what appears to be a bed or table surface in the foreground. The TV is visible mounted on the wall in the distant background of the room, behind the main subject (the shoes). The spatial relationship shows the TV is located behind where the camera is focused, placing it at the back relative to the camera's position. The room layout with furniture and the TV placement on the far wall confirms this orientation.Retry
Figure 18: Sample cases of Viewpoint Transformation
Claude Sonnet 4.5 Response: B. The three sequential frames show a basketball game with players in yellow and another team on the court. Analyzing the ball's current trajectory and the position of the yellow player, a shift of 50 cm (approximately 20 inches) to the right would significantly alter the pass direction. The yellow player appears to be positioned to receive the ball at its current trajectory, but a 50 cm rightward deviation would likely take the ball outside their immediate reach or interception zone. Given typical player spacing and movement capabilities in basketball, such a deviation would require the player to make a substantial adjustment that may not be feasible given their current position and momentum. The distance and angle suggest the ball would pass beyond their effective receiving range.
Figure 19: Sample cases of Spatial Manipulation
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Temporal Shifting Question: Assuming Photo N was taken right now, which of the following scenes is not possible after 5 hours?? Option: A:Photo A B: Photo B C: Photo C
Ground-truth : A
Action Substitution Question: If the bottle were pushed suddenly forward, in which direction would it most likely fall? Options: A. Towards the slipper. B. Towards the wall.
C. Towards the plant.
Causal Graph: Ground-truth : C
N
A
B
C
GPT-4o Response: B. Photo N shows Marina Bay Sands during early evening with the sky still bright. After 5 hours, it would be well into the night. Photo A and C, which show the scene in daylight, would no longer be possible. However, Photo B shows the area at twilight or early night, with the lights on but not completely dark. Since Photo N already shows evening light, 5 hours later would be too late for the lighting conditions in Photo B. Therefore, Photo B is the one not possible after 5 hours.
Gemini 2.5 Flash Response: A. The images all depict the Marina Bay Sands hotel and the ArtScience Museum in Singapore. Photo N is taken at dusk or twilight, as indicated by the deep blue sky, artificial lights beginning to come on, and reflections on the water. If Photo N was taken right now, the scene five hours later would be well into the night. Photo B shows the scene at night, with all the building lights fully illuminated, which is a possible state after five hours. Photo A and Photo C show the scene in full daylight with bright skies. For the environment to transition from dusk to full daylight, a complete 12-hour cycle including a full night and a sunrise must occur, which is far more than five hours. Therefore, the scenes shown in Photo A or Photo C are not possible five hours after the dusk scene in Photo N.
Claude Sonnet 4.5 Response: C. Photo N shows Marina Bay Sands in Singapore during the blue hour (early evening/dusk) with blue lighting on the buildings. After 5 hours from this time, it would be approximately late night or early morning hours. Photo A shows the scene at sunset/golden hour, Photo B shows it at night with purple/blue illumination, and Photo C shows it in broad daylight. Since Photo N is taken at dusk and we're looking 5 hours ahead, we'd be in the middle of the night or very early morning. Photo C, which shows full daylight conditions, would not be possible 5 hours after dusk, as it would still be nighttime.
Figure 20: Sample cases of Temporal Shifting
Causal Graph:
GPT-4o Response: A.The person is holding a bottle near the edge of a table, facing an open area. The slippers are directly in front of the edge, while the wall and plant are off to the sides. Gravity would cause the bottle to fall straight down if pushed gently. Given the visible orientation, it would land toward the direction where the slippers are placed. Therefore, the bottle would most likely fall towards the slippers.
Gemini 2.5 Flash Response: C. The question requires inferring the bottle's location relative to a table's edge, based on its surroundings. Although the bottle is being held, the question hypothesizes its fall from a table. Looking at the surroundings: the plant is visible on the left side and behind the bottle (suggesting it is on the side of the assumed table edge where a fall is possible). The slippers are on the ground on the right side near the wall. The wall is on the right side. A gentle push usually means the object falls straight down from the edge. Assuming the person is holding the bottle near the edge of a table extending into the foreground, if the table edge is along the line separating the bottle from the plant/slippers, the bottle is likely to fall toward the largest open space. Given the visual context where the bottle is being held between the plant and the wall/slippers, and the common orientation for such a video, the bottle would likely fall towards the plant (C), assuming the table edge is closest to the plant side or that is the direction of the open drop.
Claude Sonnet 4.5 Response: A. The image shows a bottle positioned on what appears to be a ledge or raised surface near a corner where two walls meet. There is a slipper visible on the lower ground level below and in front of the bottle's position. When an object is pushed gently off an edge, it falls primarily due to gravity in a downward direction, with some forward momentum from the push. The bottle's position on the elevated surface places it directly above the area where the slipper is located. The wall is behind the bottle, making a fall toward it unlikely, and there's no clear indication of a plant in the immediate falling trajectory.
Figure 21: Sample cases of Action Substitution
Tang et al.
Optical Inference
Mechanics Reasoning
Question: Where are the computer monitors located related to the camera?
Question: How the Hand moves relative to the Cup?
Option: A: Front left C: Front right
Option: A: Left C: Right
Ground-truth : B
B: Back left D: Back right Causal Graph:
GPT-4o Response: D.The computer monitors are visible inside the mirror, meaning they are actually positioned behind the camera. In the reflection, they appear on the left, but because it’s a mirror, their real location is on the opposite side. Since the mirror reverses left and right, the true position relative to the camera is at the back right. Therefore, the computer monitors are located at the back right of the camera.
Gemini 2.5 Flash Response: B. The image is a photo of a mirror resting on the floor, showing the reflection of the room behind the person taking the photo. The camera is being held by the person whose legs are visible in the image. The computer monitors are clearly visible in the reflection on the polished desk surface. Since the monitors are seen in the mirror's reflection, they are located behind the camera's position. Observing the reflection, the monitors and desk are situated to the viewer's right side of the mirror, which translates to the camera's left side in the actual room. Therefore, the computer monitors are located back left relative to the camera.
Claude Sonnet 4.5 Response: D. The image shows a mirror reflection of a room setup. In the mirror, we can see computer monitors on a desk. Since we're viewing through a mirror, the spatial relationships are reversed from the camera's actual perspective. The monitors appear in the mirror's reflection on what would be the right side of the actual room. To determine their true location relative to the camera, we need to account for the mirror reversal - what appears on the right in the mirror is actually behind and to the right of the camera's position. The monitors are located back right relative to where the camera is positioned.
Figure 22: Sample cases of Optical Inference
Ground-truth : C
B: Up D: Down Causal Graph:
GPT-4o Response: C.Across the three frames, the hand starts to the left of the cup and gradually moves toward the right. The fingers stay extended and the motion continues horizontally relative to the cup. The position of the cup remains fixed on the table, confirming that only the hand is in motion. There is no vertical movement (up or down) observed. Therefore, the hand is moving to the right relative to the cup.
Gemini 2.5 Flash Response: D. The image sequence shows a hand moving closer to a cup that is positioned below it. In the top frame, the hand is positioned above the cup. In the middle frame, the hand has moved closer to the cup. By the bottom frame, the hand is almost touching the rim or contents of the cup. Since the hand is visibly lowering towards the object beneath it over the course of the frames, the hand is moving down relative to the stationary cup.
Claude Sonnet 4.5 Response: D. The three sequential images show a hand approaching a cup from above. In the first frame, the hand is positioned high above the cup. In the second frame, the hand has moved closer to the cup. In the third frame, the hand appears to be even closer or making contact with the cup. The progression clearly shows a downward trajectory as the hand moves from a higher position toward the cup below. This is consistent with a reaching or grasping motion where someone is moving their hand down to pick up or interact with the cup.
Figure 23: Sample cases of Mechanics Reasoning
Causal Scaffolding for Physical Reasoning: A Benchmark for Causally-Informed Physical World Understanding in VLMs
Scene Reconstruction
Containability
Question: Which direction is the ping-pong ball flying to? Option: A: To the red player
B: To the blue player
Ground-truth : B
Causal Graph:
Question: Which glass on the table can be nested? Option: A. B. C. D. Ground-truth : A
Causal Graph:
D B C A
GPT-4o Response: B.The red player is in a forehand motion with their paddle extended, indicating they have just hit the ball. The ball is clearly visible in midair near the blue player’s side of the table. The blue player is also in a ready stance, anticipating the return. This suggests the ball is moving toward the blue player. Therefore, the ball is flying to the blue player.
Gemini 2.5 Flash Response: B. The image captures a moment in a table tennis match where the red player is positioned to strike the ball with an overhead motion (likely a smash or loop drive), while the blue player is positioned across the net, ready to receive. The ping-pong ball is visible just to the right of the red player's head, having just been struck or about to be struck by the red player. Given the red player's hitting motion and the ball's location, the ball is currently flying away from the red player and towards the blue player.
Claude Sonnet 4.5 Response: B. The image shows a table tennis match with a player in red on one side and a player in blue on the other. Based on the body positioning and racket angles of both players, the red player appears to have just made contact with or completed a shot. The blue player's stance suggests they are preparing to receive or react to an incoming ball. In table tennis, after one player hits the ball, it travels across the net to the opponent. Since the red player appears to be in a follow -through position after hitting, the ball would be traveling toward the blue player's side of the table.
Figure 24: Sample cases of Scene Reconstruction
GPT-4o Response: B. Glass B has straight sides and a simple cylindrical shape, which allows it to be nested with similar glasses. In contrast, glasses A, C, and D have wider bowls or curved shapes that prevent easy stacking. Nested glasses typically require uniform diameters from top to bottom. The flatbottomed design of Glass B supports stacking without tipping. This makes it the most practical option on the table for nesting.
Gemini 2.5 Flash Response: D. The glass labeled D is a wine or beer tasting glass with a distinct stem and a narrower bowl opening. The glass labeled C is a similar type of glass, and the glasses labeled D and C appear to be of the same or a very similar style and size. Given that they are the same type of stemmed glassware, it's possible that glass D could be inverted and placed inside glass C (or vice versa), or that multiples of this specific style of stemmed glass are designed to be stacked (nested) on top of each other by their base/foot. Glasses A and B are different styles (A is a tumbler, B is a decanter/pitcher) that cannot be reliably nested with the others.
Claude Sonnet 4.5 Response: C. Looking at the glasses on the table, glass C appears to be a tall, cylindrical tumbler or highball glass with straight sides. This type of glass design with uniform diameter from top to bottom is ideal for nesting, as multiple identical glasses can stack inside each other. Glass B appears to be a wine glass with a wider bowl that tapers, making it less suitable for nesting. Glass D looks like it might be a stemmed glass as well. The straight-sided cylindrical design of glass C makes it the single best candidate for nesting with another identical glass.
Figure 25: Sample cases of Containability