arXiv CSopen access

Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

artificial intelligence, reasoning, knowledge representation

Driver-WM: A Driver-Centric Traffic-Conditioned Latent World Model for In-Cabin Dynamics Rollout Haozhuang Chi1 , Daosheng Qiu2 , Hao Su3 , Haochen Liu1 , Zirui Li1 , Haoruo Zhang1 , and Chen Lv1B Nanyang Technological University, Singapore 2 Hubei University, Wuhan, China 3 Osaka University, Osaka, Japan Corresponding email: [email protected]

arXiv:2605.05092v1 [cs.RO] 6 May 2026

Abstract. Safe L2/L3 driving automation requires anticipating humanin-the-loop reactions during shared-control transitions. While most driving world models forecast the external environment, in-cabin intelligence remains strictly recognition-oriented and lacks multi-step rollout capabilities for driver dynamics. We introduce Driver-WM, a driver-centric latent world model that rolls out in-cabin dynamics causally conditioned on out-cabin traffic context. This formulation unifies physical kinematics forecasting with auxiliary behavioral and emotional semantic recognition. Operating in a compact latent space constructed from frozen visionlanguage features, Driver-WM adopts a dual-stream architecture to separately encode external traffic and internal driver states. These streams are directionally coupled via a gated causal injection mechanism, which uses a learned vector gate to modulate external contextual perturbations while strictly enforcing temporal causality. Evaluations on a multi-task assistive driving benchmark demonstrate that Driver-WM yields robust long-horizon geometric forecasting for reactive high-motion maneuvers and improves semantic alignment for both driver and traffic states. Finally, the explicit external-to-internal conditioning allows for controlled test-time interventions to systematically analyze mechanism responses. Project page: Driver-WM Keywords: World model · Driver monitor system · Vision-language model · Human-in-the-loop driving automation · Autonomous driving

Introduction

Autonomous driving has evolved from isolated driver assistance functions toward human-centered intelligent systems [4, 5]. Currently, the majority of real-world driving automation systems are deployed at the SAE L2/L3 level [25], where driving responsibility continuously shifts between the autonomous policy and

H. Chi et al. (a)

(c)

X frames

⋯

emotion distraction

In-Cabin Video

Neural Networks

X frames

in-cabin

Driver States

⋯

In-Cabin Video face pose body gesture

(b) X frames

X frames

⋯

Out-Cabin Video

Y frames

⋯

out-cabin

object states trajectory

World Model

Future Environment

Out-Cabin Video

World Model

emotion distraction

⋯

Future Driver Dynamics

Fig. 1: The comparison of three paradigms: (a) Regular driver monitoring systems (DMS) for driver-state recognition. (b) Standard world models for future environment forecasting. (c) Driver-WM (ours) that performs multi-step rollout of internal driver dynamics explicitly conditioned on synchronized external traffic observations.

human supervision in a shared-control (mixed-autonomy) setting [31]. Although recent advances in end-to-end systems [11, 16] and vision-language-action (VLA) paradigms [7,41] have notably improved the understanding and reasoning capabilities of perception and planning [11, 13, 17], the ultimate safety of current system domains still requires the human in the loop. In practice, many safety-critical failures are associated with inadequate takeover readiness rather than incorrect scene understanding [30], and the risk increases when the system operates beyond its functional domain under evolving traffic interactions [28]. Motivated by the need to reason about long-horizon safety and interaction, world models have recently emerged as a principled framework for autonomous driving [14]. By predicting how the external environment evolves conditioned on current observations and actions, they have been widely adopted for forward simulation, maneuver reasoning, and policy training, ranging from unified fullstack driving systems to generative traffic simulators and driving foundation models [8, 10, 29]. To support efficient long-horizon prediction and scalable closedloop rollout, latent world models have emerged that operate dynamic prediction in compressed latent space [16, 38]. However, existing world models are primarily environment-oriented (Fig. 1b): they model how roads, surrounding agents, and vehicles evolve, while the driver is usually treated as a post-hoc source of risk. They follow an environment-to-action paradigm and do not model how external driving events drive the driver’s internal evolution, leaving posture, gaze, motor readiness, and reaction behavior outside the rollout loop. In parallel, in-cabin modules, including DMS [6,22,34] (Fig. 1a) and emerging LLM-based cockpit assistants [26], remain recognition- and interaction-oriented rather than predictive. DMS primarily focuses on recognizing instantaneous states such as distraction or fatigue from short temporal windows [12, 21], while LLM-based systems emphasize dialogue, intent understanding, and high-level reasoning [18,26]. Although effective for detection and interaction, both paradigms lack the capability to predict how the driver’s physical and cognitive state will evolve over time in response to changing traffic conditions, treating the driver as an observed entity rather than a dynamical system. As a result, in-cabin intelligence remains limited to recognition-based alerts and cannot support anticipatory

Driver-WM

safety reasoning or systematic analysis of how traffic context modulates driver responses over a prediction horizon. This is critical for L2/L3 driving automation, where safe design requires anticipating not only environment evolution but also driver responses seconds ahead and under hypothetical interventions. A world model that ignores the driver’s internal dynamics is therefore incomplete for shared-control operation. To address these issues, we propose Driver-WM, a driver-centric latent world model that rolls out future driver dynamics conditioned on synchronized incabin and out-cabin observations. In this context, dynamics denotes the temporal evolution of the driver’s internal state in a compact latent space. This underlying evolution supports both physically grounded kinematics (decoded as future skeleton trajectories) and semantic factors (e.g., driver behavior and emotion, predicted as auxiliary regularizers). Instead of generating pixels, Driver-WM forecasts structured 2D skeleton keypoints of the driver’s pose and motion [9, 40], providing an efficient target that is stable for long-horizon prediction and naturally aligned with in-cabin supervision. Driver-WM operates in a compact latent space, where frozen vision–language model (VLM) [3, 15] features serve as a perceptual encoder to compress high-dimensional visual inputs, and a lightweight worldmodel core learns latent dynamics with a past/future rollout protocol initialized by observed latents. It adopts a dual-stream latent architecture to separately represent traffic context and driver state, and couples them through a gated interaction module that controls how external driving events drive internal dynamics. This formulation explicitly models how the environment shapes human reactions, enabling both realistic driver response forecasting and controlled testtime interventions for mechanism analysis. In summary, our contributions are threefold: – Driver-centric latent world model for dynamics rollout: multi-step forecasting of in-cabin driver dynamics, unifying kinematic trajectories and auxiliary semantic factors, causally conditioned on observed traffic context. – Directionally coupled dual-stream dynamics with gated injection: explicit external-to-internal coupling that supports controllable conditioning and controlled intervention analysis. – Foundation-derived state interface with unified decoding: frozen VLM features serve as a compact perceptual interface, supporting geometric rollout with auxiliary semantic regularization.

Related Work

2.1

World Models in Autonomous Driving

Most driving world models formulate future prediction as conditional video generation in pixel space. Examples include GAIA-1 [10], DriveDreamer [29, 37], Vista [8] for controllable rollouts, and DriveDreamer4D [36] for 4D scene generation. While pixel-space rollouts offer high visual fidelity, they often entangle semantics, geometry, and physics with low-level rendering. Recent efforts such

H. Chi et al.

as MAD [24] attempt to explicitly decouple motion and appearance to improve generation efficiency. However, these approaches remain strictly focused on the external environment; enforcing rigorous structural constraints remains challenging in a pixel-generation formulation. Latent-state world models address this limitation by predicting and planning in a compact state space. In autonomous driving, LAW [16] regularizes end-to-end driving via temporal consistency, while World4Drive [38] constructs an intention-aware latent model for complex interactions. Yet, whether employing motion-decoupled video generators or latent-state planners, existing methods predominantly model the evolution of the external scene, leaving the human driver unmodeled in the predictive state. Driver-WM diverges from this scene-centric paradigm by introducing a driver-centric latent dynamics model causally conditioned on out-cabin traffic context. Instead of raw video synthesis, we utilize skeleton trajectory rollout as a physically grounded target, decoupling human kinematics from visual appearance and enabling explicit analysis of how external driving events shape predicted driver behavior.

2.2

Driver and Cabin State Modeling

In-cabin perception is commonly studied as discrete state recognition, aiming to infer discrete driver states or activities, such as distraction or drowsiness [32, 35], from visual observations [6]. Drive&Act [22] provides a benchmark for fine-grained driver activity recognition with multi-modal streams, and AIDE [34] further links in-cabin observations with synchronized out-cabin traffic context. Recent unified multi-task frameworks have reported strong semantic recognition results on this benchmark; for example, MMTL-UniAD [21]. Driver-related prediction has also been explored in maneuver/intention anticipation, e.g., Brain4Cars [12]. These works primarily focus on predicting discrete categorical labels. As such, they do not explicitly model continuous driver motion responses to external driving events (e.g., ego maneuvers such as braking or turning), which are important for predictive safety assessment. Human motion forecasting predicts future skeletal trajectories from historical motion. SiMLPe [9] shows that compact architectures can capture strong kinematic priors, while MotionBERT [40] demonstrates the benefit of transferable skeleton representations via large-scale pretraining. Recent studies further incorporate context and interactions: TRiPOD [1] studies pose forecasting in the wild, and Waymo-3DSkelMo [39] models pedestrian skeletal reactions under traffic interactions. In contrast, driver motion inside the cabin is still often treated as isolated estimation without explicit conditioning on the evolving traffic context. To bridge the gap between classification-based driver monitoring and motion forecasting without explicit conditioning on out-cabin traffic context, Driver-WM explicitly parameterizes the directional coupling from out-of-cabin driving events to driver motion, formulating traffic-conditioned driver responses as latent dynamics and decoding them into continuous skeleton rollouts.

Driver-WM

2.3

Foundation Models for Embodied Perception

Vision–language foundation models (VLMs) provide strong open-vocabulary representations and are widely used as perception backbones; instruction-tuned families such as Qwen-VL [2, 3, 27] further improve semantic grounding. Many works adopt a frozen-backbone paradigm [15], keeping large encoders fixed while learning lightweight interfaces for downstream adaptation. In embodied AI and autonomous driving, VLMs are frequently used for symbolic reasoning and planning with discrete outputs, including general embodied systems such as PaLME [7] and RT-2 [41], as well as driving-oriented VLMs like DriveLM [26]. Rather than utilizing VLMs to produce discrete symbolic outputs, Driver-WM treats frozen Qwen3-VL features as a continuous state interface for latent dynamics. This formulation enables efficient multi-step physical rollouts that remain semantically consistent with the visual scene, bypassing the need for pixel-level generation.

Method

3.1

Problem Formulation

We formulate driver-centric cabin world modeling as externally conditioned temporal forecasting. We use in-/out-cabin to denote temporally synchronized raw observations, and internal/external to denote their derived latent states. Given these streams, we learn a world model to roll out the driver’s internal dynamics conditioned on the observed external context, and decode the resulting states into physically plausible skeleton trajectories. In contrast to conventional driver monitoring that focuses on per-frame recognition, our primary objective is multi-step rollout; semantic predictions are used only as auxiliary regularizers. Throughout this paper, dynamics refers to the latent-state evolution governing the driver’s future responses, whereas kinematics refers specifically to the decoded geometric skeleton trajectories. Semantic predictions provide clip-level regularization and are not used as temporal conditioning inputs. in T out out T Let oin 1:T = {ot }t=1 and o1:T = {ot }t=1 denote synchronized in-cabin and out-cabin observations over horizon T . Driver-WM treats oout as an external context that conditions the driver’s internal evolution. Driver-WM adopts a past/future protocol with T = Tobs + Tpred with causal rollouts. Both streams are initialized with ground-truth latents for t ≤ Tobs . For t > Tobs , Driver-WM performs closed-loop rollouts of external and internal latents, while enforcing that the update at step t+1 only accesses histories up to step t. Crucially, the external rollout serves exclusively to sustain continuous environmental conditioning for the driver, rather than to reconstruct future traffic scenes explicitly. Dual-stream latent states are formulated in Driver-WM. Specifically, we represent the driver state at time t as an internal latent zint ∈ RD and a set t ext ext D V of multi-view external latents Zt = {zt,v ∈ R }v=1 , where V is the number of out-cabin views (e.g., front/left/right) and D=2048 is the hidden dimension of the frozen Qwen3-VL vision encoder. Accordingly, the external history is

H. Chi et al.

t-X

t+1 t+Y

⋯ ⋯

𝒇𝒇𝒆𝒆𝒆𝒆𝒆𝒆 𝜽𝜽

� 𝒆𝒆𝒆𝒆𝒆𝒆 Out-Cabin Features 𝒁𝒁 ≤𝒕𝒕

𝒈𝒈𝒕𝒕

⋯

Predicted In-Cabin Features in-cabin

out-cabin

𝒛𝒛� 𝒆𝒆𝒆𝒆𝒆𝒆 𝒕𝒕+𝟏𝟏

Concat

t+1 t+Y

𝓛𝓛𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔 Physical 𝓛𝓛𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑 Priors 𝓛𝓛𝒍𝒍𝒍𝒍𝒍𝒍

Latent Consistency Cross

𝓛𝓛𝒂𝒂𝒂𝒂𝒂𝒂 Entropy

Loss Constraints ⋯

Predicted Out-Cabin Features

Semantic Heads

t-1 t-Y

𝑪𝑪𝐭𝐭𝐭𝐭𝜽𝜽 𝒎𝒎𝒕𝒕

t+Y

Task Heads

𝒊𝒊𝒊𝒊𝒊𝒊 � ≤𝒕𝒕 In-Cabin Features 𝒁𝒁

𝒛𝒛� 𝒊𝒊𝒊𝒊𝒊𝒊 𝒕𝒕+𝟏𝟏

Y frames

Renderer (Optional)

⋯

𝒛𝒛� 𝒊𝒊𝒊𝒊𝒊𝒊 𝒕𝒕+𝟏𝟏 𝒇𝒇𝒊𝒊𝒊𝒊𝒊𝒊 𝜽𝜽

MLP & Sigmoid

Out-Cabin Video

t+1 t+Y

View Mean Pooling & Projection

X+Y frames (right-view) ⋯ ⋯

Qwen3-VL

X+Y frames (left-view) ⋯ ⋯

t-1 t-Y

⋯

t+1 t+2 t+3

Driver Kinematics Head

In-Cabin Video

t-X

Concat

Directed Gated Causal Injection

Projection

X frames ⋯

X+Y frames (front-view) ⋯ ⋯

Causal Cabin World Model

Frozen

Prompt

⋯

Future In-Cabin Video Y frames

𝑫𝑫𝒔𝒔𝒔𝒔𝒔𝒔𝒔𝒔 𝑫𝑫𝒅𝒅𝒅𝒅𝒅𝒅 𝑫𝑫𝒅𝒅𝒅𝒅𝒅𝒅 𝑫𝑫𝒕𝒕𝒕𝒕𝒕𝒕

𝑫𝑫𝒗𝒗𝒗𝒗𝒓𝒓

⋯

face pose body gesture

Skeleton Trajectory

⋯

emotion distraction

Classification Labels Future Driver Dynamics

Fig. 2: Overall Architecture of Driver-WM. From synchronized in/out-cabin videos, a frozen Qwen3-VL extracts dual-stream latent features. Pooled external history ˆ ext perturbs the internal transition via a directed Gated Causal Injection with a vector Z̄ ≤t gate gt , yielding an updated internal latent ẑint t+1 . Internal latents are autoregressively rolled out to forecast future states, decoded into skeleton trajectories and auxiliary semantic predictions (e.g., driver behavior) for regularization.

PV ext ext Zext = V1 v=1 zext t,v ≤t = {zτ,v }τ ≤t, v . We further define a pooled external vector z̄t ext and its history Z̄ext ≤t = {z̄τ }τ ≤t , which is used by the dynamics core. During the rollout stage, the external context of Driver-WM is advanced by ext ˆext z̄ˆext t+1 = fθ (z̄t ), and the internal state is updated by an externally conditioned transition: \label {eq:rollout_core} \hat {\mathbf {z}}^{\text {int}}_{t+1} = \mathcal {F}_{\theta }\!\left (\hat {\mathbf {Z}}^{\text {int}}_{\le t},\; \hat {\bar {\mathbf {Z}}}^{\text {ext}}_{\le t}\right ). (1) int ext ˆext We initialize ẑint 1:Tobs = z1:Tobs and z̄1:Tobs = z̄1:Tobs , and compute rollout losses only on the future window. Driver-WM formulates multi-task decoding with auxiliary regularizers. A geometric head decodes rolled-out internal latents into skeleton keypoints:

\label {eq:skel_decode} \hat {\mathbf {s}}_{t}=D_{\text {skel}}\!\left (\hat {\mathbf {z}}^{\text {int}}_{t}\right ), \quad \hat {\mathbf {s}}_{t}\in \mathbb {R}^{K\times 2}\;(\text {or }\mathbb {R}^{K\times 3}),

(2)

where K is the number of keypoints (e.g., K=136 for HALPE-style skeleton). We also attach lightweight auxiliary heads on zint and zext to regularize the latent space with semantic factors. 3.2

VLM-based Perception and Latent Interface

We model temporal dynamics in a compact semantic space rather than raw pixels. We adopt Qwen3-VL [3] as a frozen perceptual backbone and use its representation as the latent interface, following the frozen-encoder adaptation paradigm (e.g., BLIP-2 [15]). Frozen features. Let Evlm (·) denote the visual perception stack of Qwen3VL. Given synchronized observations at time t, we extract ftin = Evlm (oin t ) and

Driver-WM

out ft,v = Evlm (oout t,v ), where v ∈ {1, . . . , V } indexes the out-cabin views and D=2048. We pre-extract and cache {ftin , ftout }Tt=1 and exclude the VLM from the training graph to reduce computation and stabilize optimization, so that learning focuses on the world-model core for driver dynamics. View-conditioned interface. To disambiguate camera sources in a shared semantic space, we inject a learnable view embedding \bar {\mathbf {f}}^{\text {in}}_{t}=\mathbf {f}^{\text {in}}_{t}+\mathbf {e}_{\text {view}}(\text {in}),\qquad \bar {\mathbf {f}}^{\text {out}}_{t,v}=\mathbf {f}^{\text {out}}_{t,v}+\mathbf {e}_{\text {view}}(v),\qquad \mathbf {e}_{\text {view}}(\cdot )\in \mathbb {R}^{D}, (3)

where v is the view identifier (e.g., {front, left, right, in-cabin}). We obtain f̄tin out and f̄t,v accordingly. Latent State Interface. By default, we use an identity latent interface: \mathbf {z}^{\text {int}}_{t}=\bar {\mathbf {f}}^{\text {in}}_{t},\qquad \mathbf {z}^{\text {ext}}_{t,v}=\bar {\mathbf {f}}^{\text {out}}_{t,v}.

(4) PV 1

=V We apply mean pooling across views to obtain z̄ext t 3.3

ext v=1 zt,v .

Causal Driver World Model

We formulate the world model as a lightweight gated module, in contrast to heavy ext T block-based architectures. Given dual-stream latents {(zint t , z̄t )}t=1 (Sec. 3.1– 3.2), we learn a directed world model that rolls out the driver’s internal dynamics conditioned on external context. We enforce two inductive biases: (i) temporal causality (the update at t+1 only depends on context up to t) and (ii) directed coupling (external → internal via an explicit gate rather than symmetric fusion). In our main configuration, we apply a causal temporal self-attention pre-encoding to each stream before rollout; the bidirectional variant is used only as a non-causal upper bound. ˆ ext Causal context summary. Let Ẑint ≤t and Z̄≤t denote the internal/external histories available up to step t. We compute an external-to-internal context summary mt ∈ RD via cross-attention: \label {eq:ctx_attn} \mathbf {m}_{t} = \mathrm {Ctx}_{\theta }\!\left (\hat {\mathbf {Z}}^{\text {int}}_{\le t},\, \hat {\bar {\mathbf {Z}}}^{\text {ext}}_{\le t}\right ).

(5)

Temporal causality is enforced by truncating the histories passed to Ctxθ at each rollout step (i.e., only providing representations ≤ t). Internal transition. We predict a candidate next internal latent via a lightweight transition predictor fθint : \label {eq:int_transition} \tilde {\mathbf {z}}^{\text {int}}_{t+1} = f^{\text {int}}_{\theta }\!\left (\hat {\mathbf {z}}^{\text {int}}_{t}\right ). (6) In practice, we parameterize a diagonal Gaussian transition during training and use the mean prediction for deterministic inference; KL regularization is enabled only in the probabilistic ablation (Supplementary). Gated causal coupling. External driving events do not affect the driver equally; we introduce a vector-valued gate gt ∈ (0, 1)D to regulate contextual perturbation: \label {eq:gated_fusion} \mathbf {g}_t=\sigma \!\left (\mathrm {MLP}_{g}\!\left (\hat {\bar {\mathbf {z}}}^{\text {ext}}_{t}\right )\right ), \qquad \hat {\mathbf {z}}^{\text {int}}_{t+1} = (\mathbf {1}-\mathbf {g}_t)\odot \tilde {\mathbf {z}}^{\text {int}}_{t+1} \;+\; \mathbf {g}_t\odot \mathbf {m}_{t}.

(7)

For controlled interventions analysis, we bypass the learned gate with a scalar int override λCA , ẑint t+1 = (1 − λCA )z̃t+1 + λCA mt .

3.4

H. Chi et al.

Unified Decoding and Optimization Objectives

Driver-WM rolls out internal latents ẑint 1:T and uses them as a shared substrate for (i) skeleton rollout (primary) and (ii) auxiliary semantic predictions (regularizers). We adopt a lightweight one core, multiple heads design to keep the temporal core compact. Geometric decoding. We decode the rolled-out internal latents into skeleton keypoints via a Spatial-Temporal Graph Convolutional Network (ST-GCN): \label {eq:skel_head} \hat {\mathbf {s}}_{t}=D_{\text {skel}}\!\left (\hat {\mathbf {z}}^{\text {int}}_{t}\right ).

(8)

For 2D rollout, ŝt ∈ [0, 1]K×2 stores normalized joint coordinates (sigmoid output), where K is the number of joints (e.g., K=136 for HALPE). Dskel uses human skeletal adjacency as the graph topology. We supervise {ŝt } with regression loss Lskel over the rollout horizon; model selection is driven by multi-step skeleton accuracy (e.g., MPJPE). Physical priors for plausible rollouts. Beyond pointwise regression, we regularize long-horizon kinematics with Lphys = λbone Lbone + λsmooth Lsmooth + λseat Lseat , where Lseat is an optional soft cabin/ROI feasibility penalty (Supplementary). We impose a structural consistency term by matching the predicted and ground-truth edge lengths on the kinematic tree: \label {eq:bone_loss} \mathcal {L}_{\text {bone}} = \sum _{t}\sum _{(i,j)\in \mathcal {E}} \left | \left \|\hat {\mathbf {s}}_{t,i}-\hat {\mathbf {s}}_{t,j}\right \|_2 - \left \|\mathbf {s}_{t,i}-\mathbf {s}_{t,j}\right \|_2 \right |,

(9)

where E denotes kinematic tree edges. To suppress jitter and stabilize rollout, we impose temporal smoothness using first/second-order differences \label {eq:smooth_loss} \mathcal {L}_{\text {smooth}} = \sum _{t}\left \|\Delta \hat {\mathbf {s}}_{t}-\Delta \mathbf {s}_{t}\right \|_2^2 + \sum _{t}\left \|\Delta ^2 \hat {\mathbf {s}}_{t}\right \|_2^2,

(10)

with ∆ŝt = ŝt+1 − ŝt and ∆2 ŝt = ŝt+2 − 2ŝt+1 + ŝt . When auxiliary 3D keypoints are available (or obtained by lifting), we also evaluate a 3D counterpart of Lbone in the Supplementary Material. All physical terms can be toggled for controlled ablations. Auxiliary semantic heads. We apply last-step pooling for clip-level recognition:

\label {eq:int_aux} \hat {\mathbf {y}}^{\text {dbr}}&=D_{\text {dbr}}(\hat {\mathbf {z}}^{\text {int}}_{T}),\qquad \hat {\mathbf {y}}^{\text {der}}=D_{\text {der}}(\hat {\mathbf {z}}^{\text {int}}_{T}),\\ \label {eq:ext_aux} \hat {\mathbf {y}}^{\text {tcr}}&=D_{\text {tcr}}(\bar {\mathbf {z}}^{\text {ext}}_{T}),\qquad \hat {\mathbf {y}}^{\text {vcr}}=D_{\text {vcr}}(\bar {\mathbf {z}}^{\text {ext}}_{T}), (12) trained with cross-entropy losses and used exclusively as auxiliary regularizers. External semantics are decoded from the final external context z̄ext T , while internal semantics use the rolled-out state ẑint . T Latent rollout consistency. Besides supervising decoded skeletons, we regularize the internal latent rollout to match the target latent evolution. We support

Driver-WM

two modes: \label {eq:latent_direct} \mathcal {L}_{\text {lat}}^{\text {direct}} &= \sum _{t=1}^{T-1} \left \| \hat {\mathbf {z}}^{\text {int}}_{t+1}-\mathbf {z}^{\text {int}}_{t+1} \right \|_2^2,\\ \label {eq:latent_vel} \mathcal {L}_{\text {lat}}^{\text {vel}} &= \sum _{t=1}^{T-1} \left \| (\hat {\mathbf {z}}^{\text {int}}_{t+1}-\hat {\mathbf {z}}^{\text {int}}_{t}) - (\mathbf {z}^{\text {int}}_{t+1}-\mathbf {z}^{\text {int}}_{t}) \right \|_2^2,

(13b) and compute the loss on the rollout (future) window unless stated otherwise. The final objective aggregates latent consistency, skeleton supervision, physical priors, auxiliary semantics: \label {eq:total_loss} \mathcal {L} = \lambda _{\text {lat}}\mathcal {L}_{\text {lat}} + \lambda _{\text {skel}}\mathcal {L}_{\text {skel}} + \lambda _{\text {aux}}\mathcal {L}_{\text {aux}} + \lambda _{\text {phys}}\mathcal {L}_{\text {phys}},

Experiments

4.1

Experimental Setup

(14)

Dataset, protocol, and targets. We evaluate Driver-WM on the AIDE assistive driving benchmark [34] to validate its capacity for both kinematic rollout and auxiliary semantic recognition. Each video is segmented into 3-second clips with 10 uniformly sampled frames (∼3.3 Hz). We follow a 5→5 time-causal protocol: given Tobs =5 observed steps, we roll out Tpred =5 future steps in latent space, decoding the internal states into HALPE-136 2D skeletons (K=136). Rollout losses (latent and skeleton) are computed exclusively on the future window (t > Tobs ). We attach four auxiliary classification heads as semantic regularizers: 7-class Driver Behavior (DBR) and 5-class Emotion (DER) for the in-cabin stream, alongside 3-class Traffic Context (TCR) and 5-class Vehicle Condition (VCR) for the outcabin stream. Ground-truth semantic labels are used exclusively to supervise these auxiliary heads. They are strictly excluded from serving as conditioning inputs to the temporal transition or gating pathways (i.e., label embeddings are disabled) to ensure a purely vision-driven rollout. Implementation details. We pre-extract per-frame features (D=2048) using a frozen Qwen3-VL-2B [3] with a fixed prompt to avoid label leakage. Models are trained for 80 epochs using AdamW (learning rate 5×10−5 , batch size 32) on A5000 GPUs. At inference, temporal causality is enforced by truncating the internal/external histories passed to the context interaction and gating modules at each step (strictly ≤ t). Unless stated otherwise, models are trained from scratch; warm-start variants are detailed in the Supplementary. We additionally verify strict zero-lookahead numerically: replacing the unobservable future external suffix with noise or zeros changes the predicted skeletal tensors by at most 5 × 10−7 (max abs diff; see Supplementary). Metrics and baselines. We report horizon-averaged MPJPE (px), diagonalnormalized MPJPE (d-nMPJPE, %), and [email protected]; [email protected] and perhead Accuracy are deferred to the Supplementary.√ We compute d-nMPJPE as 100 × MPJPE/d with the image diagonal d = 19202 + 10802 , and use a

H. Chi et al.

Table 1: Main results on AIDE under the 5→5 rollout. We report horizonaveraged geometric errors (MPJPE in px, d-nMPJPE in %) and semantic Macro-F1. Model

All

PCK

MPJPE↓ d-nMPJPE↓ MPJPE↓ d-nMPJPE↓ @0.05↑ Motion-only & offline baselines Zero-Velocity† [23] 52.89 ST-GCN [33] 110.98 SiMLPe [9] 106.38 MotionBERT [40] 73.51 Offline Enc-Dec [23] 75.87

F1↑

– – – – –

139.19 158.10 156.29 141.53 144.05

6.32 7.18 7.09 6.42 6.54

85.95 60.97 63.45 78.01 77.17

8.27 18.96 37.26 56.70 53.82

14.94 22.66 41.69 52.71 50.68

Controlled contextual baselines (identical VLM interface) Static pooling (no rollout)‡ 68.50 3.11 Single-stream 90.87 4.13 Late fusion 82.59 3.75 Cross-Attn only 80.14 3.64

134.56 147.44 143.31 142.41

6.11 6.69 6.51 6.47

72.55 59.63 65.07 66.43

54.04 24.56 45.87 62.52

60.07 24.35 49.10 68.75

Ours Driver-WM (main) Driver-WM (full pretrained) Non-causal (bidir)

138.03 138.03 136.50

6.27 6.27 6.20

71.66 70.99 71.39

68.07 72.61 90.15 68.34 73.22 73.97 90.54 62.09 73.35 74.46 88.65 68.82

71.47 72.68 72.15

2.40 5.04 4.83 3.34 3.45

DBR DER TCR VCR

3.24 3.30 3.27

26.39 14.00 26.39 14.00 87.85 65.97 87.94 69.09

† Copy-last-frame baseline favored by ℓ metrics under inertial continuity. 2 ‡ Static pooling baseline (no rollout) can yield low horizon-averaged ℓ errors via mean-regression. 2 − Externally grounded semantics (TCR/VCR) are inapplicable for motion-only baselines. HM The High-Motion subset: the Top 10% clips ranked by a ground-truth motion score computed on the future window as the mean per-joint frame-to-frame displacement in pixel coordinates (contains 60/609 test clips; see Supplementary).

threshold of 0.05d for [email protected]. Semantic performance is measured by Macro-F1. We compare against motion-only forecasters (Zero-Velocity [23], ST-GCN [33], SiMLPe [9], MotionBERT [40]) and an offline encoder-decoder reference [23] representing the non-causal motion bound. To rigorously isolate architectural gains from visual backbone disparities, we additionally construct controlled contextual baselines (e.g., static pooling, single-stream, late fusion) under the exact same frozen Qwen3-VL interface. All learnable methods follow the identical 5→5 rollout protocol. 4.2

Main Results

Table 1 reports the comparison under the fixed 5→5 causal rollout. To focus on paradigm-level differences, we defer dedicated ablations and controlled variants to Sec. 4.3. Motion-only baselines and an inertia effect under ℓ2 -style metrics. An apparent outlier in Table 1 is the Zero-Velocity baseline, which attains the lowest overall d-nMPJPE. This is consistent with naturalistic driving clips dominated by mild motion and inertial continuity: copying the last observed pose exploits this inertia to reduce an ℓ2 -style geometric metric globally. However, autonomous safety hinges on reactive, high-dynamic moments. As illustrated in Fig. 3(b), when focusing on the safety-critical High-Motion tail, the Zero-Velocity baseline’s error grows sharply as the horizon extends. While simpler baselines benefit from strong spatial priors at initial steps, Driver-WM successfully captures reactive dynamics, significantly mitigating the rapid error accumulation observed in motion-only models and achieving competitive long-horizon fidelity (e.g., h=5 MPJPE 155.82 vs. S0 178.62) on the High-Motion tail. While stronger motion-only forecasters

Driver-WM

(e.g., MotionBERT) improve kinematic fidelity, they do not explicitly condition on out-cabin context and degrade on the High-Motion tail at longer horizons. Finally, the offline variant serves strictly as a non-causal reference. Contextual paradigms and a static mean-regression trap. Introducing external visual context changes the trade-off substantially under the same frozen Qwen3-VL interface. Notably, the static pooling baseline (no rollout) can appear strong on horizon-averaged ℓ2 metrics, including on the HM subset, since a time-invariant, time-averaged hypothesis may reduce the average penalty when reactive maneuvers exhibit temporal ambiguity under sparse sampling. However, this baseline does not produce an autoregressive trajectory and therefore cannot support horizon-wise degradation analysis or test-time mechanism probes. In addition, its externally grounded Macro-F1 remains low (Table 1), indicating limited balanced recognition of traffic context. In contrast, Driver-WM explicitly models multi-step dynamics conditioned on traffic observations and yields robust long-horizon rollouts on the safety-critical High-Motion tail (Fig. 3(b)). Driver-WM. Driver-WM delivers robust long-horizon performance on the safetycritical High-Motion tail (Fig. 3(b)), while maintaining competitive horizonaveraged geometric errors (Table 1). With gated, time-causal injection of external context, it mitigates the long-horizon degradation observed in motion-only forecasters and improves semantic consistency across heads under the same interface. The full pretrained variant yields additional but modest gains on internal semantics with comparable rollout errors. The non-causal bidirectional variant serves as an offline reference and does not yield consistent gains over the causal design in this setting, supporting causal temporal modeling as a deployment-consistent inductive bias. Robustness. Across three random initialization seeds, the main Driver-WM configuration demonstrates high stability, yielding a consistent All d-nMPJPE of 3.25±0.01% alongside a robust TCR F1 score of 89.26±1.33. In contrast, suboptimal fusion designs exhibit notably larger variance; for instance, the M2 variant fluctuates with an All MPJPE of 79.93±2.35 px. The complete multi-seed table and a reference comparison to recent classification-only methods are provided in the Supplementary Material. Note that these approaches do not perform kinematic rollout and are therefore not directly comparable to our setting. 4.3

Ablation Studies on Driver-WM Architecture

To isolate the contributions of key design choices in Driver-WM, we evaluate controlled variants around the main model under the identical 5→5 causal rollout. Table 2 summarizes representative ablations. External context is necessary for externally grounded semantics. Severing the out-cabin stream leads to a substantial degradation on the externally grounded head (TCR), while also reducing DER. Although re-training without external context yields competitive averaged MPJPE due to dominant low-motion segments, it lacks access to environmental triggers. This gap is revealed by test-time interventions on the jointly trained model: removing external context

H. Chi et al.

Table 2: Ablations on Driver-WM. Protocol and metrics follow Table 1. Geometric metrics are inapplicable for the no pose head variant. Variant

All

PCK

MPJPE↓ d-nMPJPE↓ MPJPE↓ d-nMPJPE↓ @0.05↑

DBR DER TCR VCR F1↑

F1↑

Driver-WM (main)

71.47

3.24

138.03

6.27

71.66

68.07 72.61 90.15 68.34

External Context w/o ext context

71.74

3.26

136.94

6.21

71.13

71.33 63.56 26.39 14.00

Dynamics Core & Training Choices RSSM/GRU dyn. 78.42 3.56 +Skeleton-Pre 70.92 3.22

136.46 136.04

6.19 6.18

67.48 72.09

67.15 72.29 88.26 63.74 69.55 75.65 88.33 70.24

Regularization & Interface Diagnostics KL bottleneck 70.56 3.20 no-prompt feat. 76.54 3.47 no pose head – –

136.50 142.08 –

6.20 6.45 –

71.71 68.28 –

69.89 71.54 88.84 64.68 68.88 69.92 76.87 58.59 67.22 70.92 88.83 69.44

increases rollout error substantially (e.g., +15.2 px at h=5; Table 3), indicating that Driver-WM strictly relies on traffic cues for reactive forecasting. Dynamics core and training choices. Replacing the transition kernel with an RSSM/GRU-style recurrent dynamics degrades geometric fidelity and lowers semantic consistency, suggesting that self-attention is better suited for longhorizon conditional rollouts under high-dimensional context features. Skeleton pre-training yields only a limited effect relative to our main model and does not consistently improve semantics, indicating that the main gains of Driver-WM are not driven by warm-starting the skeleton predictor. Regularization and interface diagnostics. Introducing a KL bottleneck results in marginal changes across metrics, implying that the dominant performance driver in this setting is the causal injection structure rather than probabilistic regularization. Removing prompted external features consistently degrades both geometric and semantic metrics, highlighting the importance of the external feature interface for aligning driver dynamics with traffic context. Finally, removing the pose forecasting head retains competitive semantic performance on TCR but removes kinematic rollouts altogether; this diagnostic underscores the distinction between discriminative semantics and physically grounded forecasting, motivating kinematic prediction as the core foundation in Driver-WM. 4.4

Interventions and Qualitative Analysis

We perform post-hoc test-time interventions on the trained Driver-WM (main) to assess (i) whether external context is effectively used and (ii) whether the injection pathway is necessary and controllable. Specifically, we intervene on the external stream (Ext) via cross-video content swapping, large temporal shifting, view dropping, or complete removal; or we intervene on the injection pathway by either clamping the learned gate g to a constant or bypassing it with a scalar override λCA ; do(g=0/1) is equivalent to do(λCA =0/1) under clamping or bypass, while λCA > 1 serves as an over-injection stress test. We use interventions as mechanism probes, without making causal effect estimation claims.

Driver-WM

Table 3: Controlled interventions on Driver-WM (main, test set). We report absolute deviations (∆) in pixel space between the intervened rollouts and the factual (unintervened) rollouts. Larger deviations indicate higher sensitivity to the corresponding intervention. h=5 denotes the last predicted step. Intervention do(·)

∆All

Target

∆HM

∆h=5

∆Head

∆Hands

Factual (Driver-WM, main) None

0.000

do(Ext = Swap_clip) do(Ext = ∅) do(Ext = Shift_large) do(Ext = DropView)

Cross-video content swap Remove out-cabin features Large temporal offset Drop one external view

5.363 12.953 0.793 0.419

4.794 8.691 0.892 0.100

8.042 15.208 1.009 0.478

5.462 7.385 0.661 0.392

3.502 12.937 0.018 0.265

do(λCA =0) / do(g=0) do(λCA =1) / do(g=1) do(λCA =2)

Disable pathway Force injection Over-injection

89.641 62.232 116.751 26.733 11.480 42.170 43.015 23.959 52.142

36.783 15.861 33.909

95.666 26.727 43.695

180

MPJPE (px) ↓

160 140

120

Zero-Velocity MotionBERT Static pooling Driver-WM (Ours)

100 80 60

Factual

𝒅𝒐(𝐄𝐱𝐭 = 𝐒𝐰𝐚𝐩)

𝒅𝒐(𝝀𝑪𝑨 = 𝟎)

(a)

h=1

h=4

h=5

Prediction Horizon (steps)

h=2

h=3

(b)

Fig. 3: Mechanism and dynamics. (a) Controlled interventions: On the same clip, swapping the out-cabin context or disabling injection (λCA =0) alters reactive hand motion; frames are aligned to the maximal injection step. (b) High-Motion tail: Horizon-wise MPJPE shows the zero-velocity baseline degrades with horizon, while Driver-WM substantially reduces long-horizon error compared to motion-only baselines.

Pathway necessity and injection strength. Table 3 shows that disabling the injection pathway (λCA =0 or g=0) yields a substantial increase in geometric error across All/HM and at the final horizon, with particularly strong degradation on hand-related joints. Forcing unmodulated injection (λCA ∈{1, 2}) also degrades rollout accuracy, indicating that external signals should be injected with a learned, time-varying intensity rather than applied unconditionally. Effects of external context. Both swapping external context (Swap_clip) and removing it entirely (Ext=∅) induce consistent deviations, with a larger degradation when the external stream is removed. The effect is more pronounced at longer horizons and on reactive joints (hands). In contrast, large temporal shifts and dropping a single view produce comparatively minor deviations in this setting, suggesting empirical robustness to coarse temporal misalignment and partial viewpoint loss. This is consistent with early mean-pooling of multi-view external latents before entering the dynamics core. Visualizing causal responses. Figures 3 and 4 provide qualitative counterparts of Table 3. In particular, Fig. 3 visualizes the degradation under do(λCA =0). Fig. 4 overlays the predicted factual and intervened skeleton trajectories onto the

H. Chi et al.

In-Cabin do(Ext = ∅) Driver-WM

Left

Out-Cabin Right

Front

𝑓𝑓5

Observation Boundary

𝑓𝑓6

𝑓𝑓7

𝑓𝑓8

𝑓𝑓9

𝑓𝑓10

Fig. 4: Qualitative results of driver dynamics rollout and causal interventions. Setting: 5→5 rollout (Tobs =5 frames observed, Tpred =5 frames predicted). We compare the factual rollout of Driver-WM against the intervened rollout without external traffic context (do(Ext=∅)). To avoid any video-generation artifacts, we directly overlay the 5-step predicted skeletons (f6 ∼f10 ) onto the corresponding ground-truth in-cabin frames in pixel space. The intervention yields a clearly different hand/upper-body kinematic trajectory in the future window.

corresponding ground-truth in-cabin frames (future window f6 ∼f10 ) to highlight the kinematic divergence induced by intervention.

Conclusion

We present Driver-WM, a dual-stream autoregressive world model that forecasts in-cabin driver dynamics explicitly conditioned on out-cabin traffic context. The architecture unifies the prediction of geometric skeleton trajectories and auxiliary semantic factors via a time-causal, gated cross-attention pathway, which dynamically injects external environmental cues into the internal latent rollout. Evaluations on the AIDE benchmark expose a critical inertia effect in standard ℓ2 metrics, where motion-only baselines exploit static extrapolation to attain deceptively low errors but degrade substantially on reactive maneuvers. Driver-WM circumvents this mean-regression trap, achieving robust kinematic fidelity alongside strong semantic alignment with the driving scene. Controlled test-time interventions corroborate the structural design: disabling the causal injection causes severe rollout degradation, while unmodulated injection leads to destabilization. This confirms the necessity of learned, time-varying contextual gating. Overall, the evidence establishes that reliable driver forecasting requires

Driver-WM

modeling human behavior as a proactive rollout causally grounded in the evolving traffic environment.

Acknowledgements This work is supported by A*STAR under the RIE2025 Industry Alignment Fund – Industry Collaboration Projects (IAF-ICP) Funding Initiative (Award: I2501E0045), as well as cash and in-kind contribution from the industry partner(s).

References 1. Adeli, V., Ehsanpour, M., Reid, I., Niebles, J.C., Savarese, S., Adeli, E., Rezatofighi, H.: TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild . In: 2021 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 13370– 13380. IEEE Computer Society, Los Alamitos, CA, USA (Oct 2021). https://doi. org/10.1109/ICCV48922.2021.01314, https://doi.ieeecomputersociety.org/ 10.1109/ICCV48922.2021.01314 2. Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., Zhou, J.: Qwen-VL: A versatile vision-language model for understanding, localization, text reading, and beyond (2024), https://openreview.net/forum?id=qrGjFJVl3m 3. Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., Ge, W., Guo, Z., Huang, Q., Huang, J., Huang, F., Hui, B., Jiang, S., Li, Z., Li, M., Li, M., Li, K., Lin, Z., Lin, J., Liu, X., Liu, J., Liu, C., Liu, Y., Liu, D., Liu, S., Lu, D., Luo, R., Lv, C., Men, R., Meng, L., Ren, X., Ren, X., Song, S., Sun, Y., Tang, J., Tu, J., Wan, J., Wang, P., Wang, P., Wang, Q., Wang, Y., Xie, T., Xu, Y., Xu, H., Xu, J., Yang, Z., Yang, M., Yang, J., Yang, A., Yu, B., Zhang, F., Zhang, H., Zhang, X., Zheng, B., Zhong, H., Zhou, J., Zhou, F., Zhou, J., Zhu, Y., Zhu, K.: Qwen3-vl technical report (2025) 4. Chen, L., Wu, P., Chitta, K., Jaeger, B., Geiger, A., Li, H.: End-to-end autonomous driving: Challenges and frontiers. IEEE Trans. Pattern Anal. Mach. Intell. 46(12), 10164–10183 (Dec 2024). https://doi.org/10.1109/TPAMI.2024.3435937, https: //doi.org/10.1109/TPAMI.2024.3435937 5. Chen, L., Li, Y., Huang, C., Li, B., Xing, Y., Tian, D., Li, L., Hu, Z., Na, X., Li, Z., Teng, S., Lv, C., Wang, J., Cao, D., Zheng, N., Wang, F.Y.: Milestones in autonomous driving and intelligent vehicles: Survey of surveys. IEEE Transactions on Intelligent Vehicles 8(2), 1046–1056 (2023). https://doi.org/10.1109/TIV. 2022.3223131 6. Chi, H., Yang, H., Yang, L., Lv, C.: Vlm-dm: Visual language models for multitask domain adaptation in driver monitoring. In: 2025 IEEE Intelligent Vehicles Symposium (IV). pp. 1280–1285 (2025). https://doi.org/10.1109/IV64158.2025. 11097620 7. Driess, D., Xia, F., Sajjadi, M.S.M., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., Huang, W., Chebotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., Florence, P.: Palm-e: an embodied multimodal language model. In: Proceedings of the 40th International Conference on Machine Learning. ICML’23, JMLR.org (2023)

H. Chi et al.

8. Gao, S., Yang, J., Chen, L., Chitta, K., Qiu, Y., Geiger, A., Zhang, J., Li, H.: Vista: A generalizable driving world model with high fidelity and versatile controllability. In: Globerson, A., Mackey, L., Belgrave, D., Fan, A., Paquet, U., Tomczak, J., Zhang, C. (eds.) Advances in Neural Information Processing Systems. vol. 37, pp. 91560–91596. Curran Associates, Inc. (2024). https://doi.org/ 10.52202/079017-2906, https://proceedings.neurips.cc/paper_files/paper/ 2024/file/a6a066fb44f2fe0d36cf740c873b8890-Paper-Conference.pdf 9. Guo, W., Du, Y., Shen, X., Lepetit, V., Alameda-Pineda, X., Moreno-Noguer, F.: Back to mlp: A simple baseline for human motion prediction. In: 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). pp. 4798–4808 (2023). https://doi.org/10.1109/WACV56688.2023.00479 10. Hu, A., Russell, L., Yeo, H., Murez, Z., Fedoseev, G., Kendall, A., Shotton, J., Corrado, G.: GAIA-1: A Generative World Model for Autonomous Driving. arXiv e-prints arXiv:2309.17080 (Sep 2023). https://doi.org/10.48550/arXiv.2309. 17080 11. Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., Wang, W., Lu, L., Jia, X., Liu, Q., Dai, J., Qiao, Y., Li, H.: Planning-oriented autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 17853–17862 (June 2023) 12. Jain, A., Koppula, H.S., Soh, S., Raghavan, B., Singh, A., Saxena, A.: Brain4cars: Car that knows before you do via sensory-fusion deep learning architecture (2016), https://arxiv.org/abs/1601.00740 13. Jiang, B., Chen, S., Xu, Q., Liao, B., Chen, J., Zhou, H., Zhang, Q., Liu, W., Huang, C., Wang, X.: Vad: Vectorized scene representation for efficient autonomous driving. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 8306–8316 (2023). https://doi.org/10.1109/ICCV51070.2023.00766 14. Kong, L., Yang, W., Mei, J., Liu, Y., Liang, A., Zhu, D., Lu, D., Yin, W., Hu, X., Jia, M., Deng, J., Zhang, K., Wu, Y., Yan, T., Gao, S., Wang, S., Li, L., Pan, L., Liu, Y., Zhu, J., Tsang Ooi, W., Hoi, S.C.H., Liu, Z.: 3D and 4D World Modeling: A Survey. arXiv e-prints arXiv:2509.07996 (Sep 2025). https://doi.org/10.48550/ arXiv.2509.07996 15. Li, J., Li, D., Savarese, S., Hoi, S.: Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In: Proceedings of the 40th International Conference on Machine Learning. ICML’23, JMLR.org (2023) 16. Li, Y., Fan, L., He, J., Wang, Y., Chen, Y., Zhang, Z., Tan, T.: Enhancing End-toEnd Autonomous Driving with Latent World Model. arXiv e-prints arXiv:2406.08481 (Jun 2024). https://doi.org/10.48550/arXiv.2406.08481 17. Liu, H., Huang, Z., Huang, W., Yang, H., Mo, X., Lv, C.: Hybrid-prediction integrated planning for autonomous driving. IEEE Transactions on Pattern Analysis and Machine Intelligence 47(4), 2597–2614 (2025). https://doi.org/10.1109/ TPAMI.2025.3526936 18. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. In: Proceedings of the 37th International Conference on Neural Information Processing Systems. NIPS ’23, Curran Associates Inc., Red Hook, NY, USA (2023) 19. Liu, W., Guo, Q., Wang, Z., Wang, W., Yang, L., Qiao, Y., Wang, L., Li, Z., Lv, C., Zhang, S., Xi, J., Liu, H.: Uv-m3tl: A unified and versatile multimodal multi-task learning framework for assistive driving perception (2026), https://arxiv.org/ abs/2602.01594 20. Liu, W., Qiao, Y., Wang, Z., Guo, Q., Chen, Z., Zhou, M., Li, X., Wang, L., Li, Z., Liu, H., Wang, W.: Tem3-learning: Time-efficient multimodal multi-task

Driver-WM

learning for advanced assistive driving. In: Laugier, C., Renzaglia, A., Atanasov, N., Birchfield, S., Cielniak, G., {De Mattos}, L., Fiorini, L., Giguere, P., Hashimoto, K., Ibanez-Guzman, J., Kamegawa, T., Lee, J., Loianno, G., Luck, K., Maruyama, H., Martinet, P., Moradi, H., Nunes, U., Pettre, J., Pretto, A., Ranzani, T., Ronnau, A., Rossi, S., Rouse, E., Ruggiero, F., Simonin, O., Wang, D., Yang, M., Yoshida, E., Zhao, H. (eds.) IROS 2025 - 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems, Conference Proceedings. pp. 6156–6163. IEEE International Conference on Intelligent Robots and Systems, Institute of Electrical and Electronics Engineers Inc., United States (2025). https://doi.org/10.1109/ IROS60139.2025.11246637, publisher Copyright: © 2025 IEEE.; 2025 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2025 ; Conference date: 19-10-2025 Through 25-10-2025 21. Liu, W., Wang, W., Qiao, Y., Guo, Q., Zhu, J., Li, P., Chen, Z., Yang, H., Li, Z., Wang, L., Tan, T., Liu, H.: MMTL-UniAD: A Unified Framework for Multimodal and Multi-Task Learning in Assistive Driving Perception . In: 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 6864–6874. IEEE Computer Society, Los Alamitos, CA, USA (Jun 2025). https://doi.org/10. 1109/CVPR52734.2025.00644, https://doi.ieeecomputersociety.org/10.1109/ CVPR52734.2025.00644 22. Martin, M., Roitberg, A., Haurilet, M., Horne, M., Reiß, S., Voit, M., Stiefelhagen, R.: Drive&act: A multi-modal dataset for fine-grained driver behavior recognition in autonomous vehicles. In: 2019 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 2801–2810 (2019). https://doi.org/10.1109/ICCV.2019. 00289 23. Martinez, J., Black, M.J., Romero, J.: On Human Motion Prediction Using Recurrent Neural Networks . In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 4674–4683. IEEE Computer Society, Los Alamitos, CA, USA (Jul 2017). https://doi.org/10.1109/CVPR.2017.497, https: //doi.ieeecomputersociety.org/10.1109/CVPR.2017.497 24. Rahimi, A., Gerard, V., Zablocki, E., Cord, M., Alahi, A.: Mad: Motion appearance decoupling for efficient driving world models (2026), https://arxiv.org/abs/2601. 09452 25. SAE On-Road Automated Vehicle Standards Committee: Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems. SAE Standard J3016 (2014) 26. Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beißwenger, J., Luo, P., Geiger, A., Li, H.: Drivelm: Driving with graph visual question answering. In: Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part LII. p. 256–274. Springer-Verlag, Berlin, Heidelberg (2024). https://doi.org/10.1007/978-3-031-72943-0_15, https://doi.org/10.1007/978-3-031-72943-0_15 27. Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., Lin, J.: Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution (2024), https://arxiv.org/abs/2409.12191 28. Wang, W., Wang, L., Zhang, C., Liu, C., Sun, L.: Social interactions for autonomous driving: A review and perspectives. Found. Trends Robot 10(3–4), 198–376 (Nov 2022). https://doi.org/10.1561/2300000078, https://doi.org/ 10.1561/2300000078

H. Chi et al.

29. Wang, X., Zhu, Z., Huang, G., Chen, X., Zhu, J., Lu, J.: Drivedreamer: Towards realworld-drive world models for autonomous driving. In: Computer Vision – ECCV 2024: 18th European Conference, Milan, Italy, September 29–October 4, 2024, Proceedings, Part XLVIII. p. 55–72. Springer-Verlag, Berlin, Heidelberg (2024). https://doi.org/10.1007/978-3-031-73195-2_4, https://doi.org/10.1007/ 978-3-031-73195-2_4 30. Weaver, B.W., DeLucia, P.R.: A systematic review and meta-analysis of takeover performance during conditionally automated driving. Human Factors 64(7), 1227– 1260 (2022) 31. Xing, Y., Lv, C., Cao, D., Hang, P.: Toward human-vehicle collaboration: Review and perspectives on human-centered collaborative automated driving. Transportation Research Part C: Emerging Technologies 128, 103199 (2021). https://doi.org/ https://doi.org/10.1016/j.trc.2021.103199 32. Xing, Y., Lv, C., Wang, H., Cao, D., Velenis, E., Wang, F.Y.: Driver activity recognition for intelligent vehicles: A deep learning approach. IEEE Transactions on Vehicular Technology 68(6), 5379–5390 (2019). https://doi.org/10.1109/TVT. 2019.2908425 33. Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence. AAAI’18/IAAI’18/EAAI’18, AAAI Press (2018) 34. Yang, D., Huang, S., Xu, Z., Li, Z., Wang, S., Li, M., Wang, Y., Liu, Y., Yang, K., Chen, Z., Wang, Y., Liu, J., Zhang, P., Zhai, P., Zhang, L.: Aide: A vision-driven multi-view, multi-modal, multi-tasking dataset for assistive driving perception. In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 20402–20413 (2023). https://doi.org/10.1109/ICCV51070.2023.01871 35. Yang, H., Liu, H., Hu, Z., Nguyen, A.T., Guerra, T.M., Lv, C.: Quantitative identification of driver distraction: A weakly supervised contrastive learning approach. IEEE Transactions on Intelligent Transportation Systems 25(2), 2034–2045 (2024). https://doi.org/10.1109/TITS.2023.3316203 36. Zhao, G., Ni, C., Wang, X., Zhu, Z., Zhang, X., Wang, Y., Huang, G., Chen, X., Wang, B., Zhang, Y., Mei, W., Wang, X.: Drivedreamer4d: World models are effective data machines for 4d driving scene representation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 12015–12026 (June 2025) 37. Zhao, G., Wang, X., Zhu, Z., Chen, X., Huang, G., Bao, X., Wang, X.: Drivedreamer2: Llm-enhanced world models for diverse driving video generation (2024), https: //arxiv.org/abs/2403.06845 38. Zheng, Y., Yang, P., Xing, Z., Zhang, Q., Zheng, Y., Gao, Y., Li, P., Zhang, T., Xia, Z., Jia, P., Lang, X., Zhao, D.: World4drive: End-to-end autonomous driving via intention-aware physical latent world model. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 28632–28642 (October 2025) 39. Zhu, G., Fan, S., Dai, H., Ho, E.S.L.: Waymo-3dskelmo: A multi-agent 3d skeletal motion dataset for pedestrian interaction modeling in autonomous driving. In: Proceedings of the 33rd ACM International Conference on Multimedia. p. 13184–13190. MM ’25, Association for Computing Machinery, New York, NY, USA (2025). https://doi.org/10.1145/3746027.3758273, https://doi.org/10. 1145/3746027.3758273

Driver-WM

40. Zhu, W., Ma, X., Liu, Z., Liu, L., Wu, W., Wang, Y.: MotionBERT: A Unified Perspective on Learning Human Motion Representations . In: 2023 IEEE/CVF International Conference on Computer Vision (ICCV). pp. 15039–15053. IEEE Computer Society, Los Alamitos, CA, USA (Oct 2023). https://doi.org/10. 1109/ICCV51070.2023.01385, https://doi.ieeecomputersociety.org/10.1109/ ICCV51070.2023.01385 41. Zitkovich, B., Yu, T., Xu, S., Xu, P., Xiao, T., Xia, F., Wu, J., Wohlhart, P., Welker, S., Wahid, A., Vuong, Q., Vanhoucke, V., Tran, H., Soricut, R., Singh, A., Singh, J., Sermanet, P., Sanketi, P.R., Salazar, G., Ryoo, M.S., Reymann, K., Rao, K., Pertsch, K., Mordatch, I., Michalewski, H., Lu, Y., Levine, S., Lee, L., Lee, T.W.E., Leal, I., Kuang, Y., Kalashnikov, D., Julian, R., Joshi, N.J., Irpan, A., Ichter, B., Hsu, J., Herzog, A., Hausman, K., Gopalakrishnan, K., Fu, C., Florence, P., Finn, C., Dubey, K.A., Driess, D., Ding, T., Choromanski, K.M., Chen, X., Chebotar, Y., Carbajal, J., Brown, N., Brohan, A., Arenas, M.G., Han, K.: Rt-2: Vision-languageaction models transfer web knowledge to robotic control. In: Tan, J., Toussaint, M., Darvish, K. (eds.) Proceedings of The 7th Conference on Robot Learning. Proceedings of Machine Learning Research, vol. 229, pp. 2165–2183. PMLR (06–09 Nov 2023), https://proceedings.mlr.press/v229/zitkovich23a.html

H. Chi et al.

Appendix A

Discussions

Towards a better understanding of this work, we supplement intuitive questions that may arise. Note that the following list does not indicate the manuscript was submitted to a previous venue or not. Q1: Why does Driver-WM simulate the driver instead of detecting attention or monitoring takeover readiness? A driver monitor system can tell us whether the driver is currently attentive, but it cannot tell us how the driver will respond to our next maneuver. In L2/L3 driving, the vehicle’s actions directly influence driver behavior—smooth deceleration, sudden braking, or lane changes all change how quickly and safely a driver re-engages. Instead, Driver-WM explicitly simulates this interaction. Predicting structured driver embodiment conditioned on external views offers the Driver-WM a causal handle on human behavior, enabling planning that anticipates and shapes driver response instead of reacting to its failure. Q2: Why can’t we formulate Driver-WM predicting takeover time or a readiness score? Predicting driver monitoring signals provides only summarized supervision. Two drivers with the same predicted takeover time can behave very differently—one may smoothly regain control, while another may hesitate or overcorrect. Faced with the hidden behavior multi-modalities underlying future monitoring signals, our Driver-WM predicts the structure of the response: posture, hand movement, attention semantics, and their timing. This allows the planner to evaluate different maneuvers and choose the one that leads to the safest outcome, instead of simply disengaging when a threshold is crossed. Q3: What capabilities does Driver-WM unlock that are currently unavailable in L2/L3? L2 and L3 systems rely on conservative rules: warn early, disengage quickly, or brake hard when uncertainty rises. This reduces comfort and usable autonomy. With the Driver-WM, the system can anticipate how the driver will re-engage and plan accordingly—slowing earlier, choosing safer lanes, or delaying a takeover request until the driver is more ready. This leads to smoother handovers, fewer false alarms, and a single unified control logic that works across both L2 and L3, rather than requiring two separate systems. As autonomy capabilities expand, the same model scales naturally by adjusting risk tolerance and prediction horizons. This reduces engineering duplication, simplifies validation, and positions the system for higher autonomy without redesigning the human–machine interface.

Additional Implementation Details

This appendix provides additional definitions and implementation details omitted from the main paper due to space constraints.

Driver-WM

B.1

AIDE Dataset and High-Motion Subsets

While the fundamental task protocol and temporal sampling strategy (5 → 5 causal rollout at ∼3.3 Hz) are detailed in the main text, we provide additional specific dataset configurations necessary for strict reproducibility. Data Partition and Keypoint Topology. We strictly adhere to the official AIDE evaluation split without performing subject-held-out re-splitting. This yields an exact distribution of 1,884 clips for training, 405 for validation, and 609 for testing. For kinematic modeling, we utilize the official HALPE-136 2D skeleton annotations. This topology is densely structured with K=136 joints, specifically comprising 26 body joints, 68 facial landmarks, and 42 hand joints (21 per hand). Explicitly modeling the 42 hand joints is critical for analyzing driver reactions to out-cabin traffic events. The specific semantic class definitions used for the auxiliary regularizers are listed in Table 4. High-Motion (HM) Subset Definition. To rigorously evaluate the model’s capability in safety-critical, reactive scenarios, we define a High-Motion (HM) tail on the test set. This subset is established by ranking clips according to a ground-truth motion score, computed strictly on the future window {t = Tobs + 1, . . . , T }. Let K×2 sgt denote the ground-truth keypoints in original pixel coordinates. The t ∈R motion score is defined as the mean per-joint frame-to-frame displacement: \label {eq:hm_motion_score} \mathrm {motion\_score} = \frac {1}{(T_{\mathrm {pred}}-1)K} \sum _{t=T_{\mathrm {obs}}+1}^{T-1} \sum _{k=1}^{K} \left \| \mathbf {s}^{gt}_{t+1,k}-\mathbf {s}^{gt}_{t,k} \right \|_2.

(15)

We rank all 609 test clips by Eq. (15) and select the top 10%. This yields a fixed subset of NHM =60 clips, corresponding to a motion score threshold of ≥ 72.05 pixels. The HM subset is universally shared across all horizons and compared baselines to ensure a completely fair evaluation of reactive dynamics. Table 4: AIDE task label sets (class order follows the official AIDE benchmark visualization). Task Classes DBR DER TCR VCR

Normal Driving; Looking Around; Making Phone; Body Movement; Talking; Smoking; Dozing Off Peace; Anxiety; Weariness; Happiness; Anger Smooth Traffic; Waiting; Traffic Jam Forward Moving; Parking; Turning; Backward Moving; Changing Lane

B.2

Prompt-Guided Feature Extraction with Frozen VLM

To construct the dual-stream external and internal context for Driver-WM, we pre-extract frame-level features using a frozen vision-language model (QwenVL series, 2B). The extraction is performed offline, operating strictly without

H. Chi et al.

access to any ground-truth dataset annotations or multi-task labels to prevent information leakage. Fixed Text Prompt. To explicitly steer the frozen encoder toward interactionaware semantics rather than generic object recognition, we pair every input frame with a fixed natural-language prompt. The exact prompt string used across all splits and camera views is: “Extract multimodal features that capture: (A) external traffic cues (vehicles, signals, road agents), (B) how these cues act as triggers for driver motion responses, and (C) the short-term evolution (1-2 seconds) of driver behavior in response to the changing road context.” This single, static instruction acts as a textual prior, forcing the model to encode causal and dynamic elements within the scene into its hidden states. Encoding and Pooling Strategy. For each synchronized timestamp, the RGB frame and the fixed prompt are processed through the VLM. We extract the hidden states from the final layer and apply mean pooling over the output sequence dimension. This yields a single D = 2048 dimensional feature vector per frame. For each 3-second clip, the extracted frame-level features are stacked into a T × D tensor (where T =10) and cached offline for each respective view. out Usage in Driver-WM. The cached feature tensors (ftin and ft,v ) are treated as fixed observations strictly for the historical window (t ≤ Tobs ). During the training and evaluation of Driver-WM, gradients are not propagated into the VLM. For the future prediction window (t > Tobs ), the external context is advanced autoregressively in the latent space. The internal rollout transition ˆ ext ), at time step t accesses only the causally valid external history up to t (Z̄ ≤t rigorously preserving zero-lookahead temporal causality.

B.3

Pretraining Strategies and Warm-start Variants

To disentangle spatial geometric decoding from temporal causal dynamics, we decouple the pretraining of the Driver-WM components into two isolated streams, corresponding to the warm-start variants evaluated in our ablation studies. Skeleton Decoder Warm-start (S-Pre). Rather than using teacher-forced visual latents, we warm-start the spatial decoding head (Dskel ) via a Coordinate AutoEncoder formulation. Specifically, the ground-truth 2D skeleton sequences sgt are projected into a proxy latent space zskel using a temporary lightweight encoder, and subsequently reconstructed by the ST-GCN decoder. This phase is optimized solely using the geometric regression loss Lskel (without causal injection or auxiliary semantics). As analyzed in the main text, the resulting downstream gains on the AIDE benchmark are modest. This diagnostic confirms that the dominant error in long-horizon High-Motion rollouts stems from contextconditioned internal transitions rather than spatial decoding capacity; once the temporal transition core is robustly trained, warm-starting the spatial decoder provides limited additional benefits.

Driver-WM

Table 5: Numerical invariance tests for strict time-causality (no leakage). We report max abs diff on predicted skeleton tensors under two perturbations applied to the unobservable future external suffix. Test (max abs diff ) Prefix-equivalence (suffix-zero) Suffix randomization (strict future)

2.980232×10−7 2.980232×10−7

Internal Dynamics Pretraining on Auxiliary Data. We optionally warm-start the internal transition core fθint on Drive&Act using latent multi-step prediction without external context. Given an internal latent window Zint 1:L , we take the last int K state zint and autoregressively predict K future steps {ẑ L L+k }k=1 . The objective is configured as an L2 loss, cosine loss, or a mixed direction–magnitude loss, and feature normalization is controlled by the pretraining configuration. The resulting weights are used to initialize fθint before fine-tuning on AIDE with the full Driver-WM model.

B.4

Strict Zero-Lookahead Verification

To rule out any potential future information path during evaluation, we conduct numerical invariance tests on the predicted skeletal tensors. Specifically, at each rollout horizon h ∈ {1, . . . , 5}, we perturb the future portion beyond the causal cutoff either by (i) zeroing the suffix (prefix-equivalence proxy), or (ii) replacing it with randomized noise (suffix randomization). Table 5 reports the maximum absolute difference (max abs diff) between the perturbed and the original predictions. All differences remain below 3 × 10−7 across horizons, numerically confirming strict time-causality. We also rerun the same forward pass under fixed seed and evaluation mode, obtaining a self-consistency max abs diff of 3.576279×10−7 .

B.5

Rationale for the ST-GCN Decoder.

Recent motion forecasters (e.g., MotionBERT, SiMLPe) demonstrate strong performance as coordinate-to-coordinate sequence models. However, we deliberately adopt a lightweight ST-GCN as our skeleton decoder. This choice aligns with the latent world model paradigm: the complex temporal dynamics and context-conditioned reactions are fully resolved within the latent transition core. Consequently, the decoding head’s primary role is spatial rather than temporal— projecting the rolled-out semantic latent state back to physically plausible joint coordinates. ST-GCN provides the explicit skeletal topology (kinematic graph) necessary for this spatial decoding. Employing a heavy spatio-temporal forecaster as the head would introduce redundant temporal modeling and confound the evaluation, blurring whether performance gains stem from our Gated Causal Injection or simply a high-capacity pose module.

H. Chi et al.

B.6

Causal Context Operator and Coupling Variants

Implementation of Ctxθ . We implement Ctxθ as a lightweight Perceiver-style cross-attention module. Temporal causality is enforced by construction: at rollout ext step t, the module only receives truncated prefixes Ẑint ≤t and Z̄≤t , without any explicit attention masking. Concretely, we first mean-pool the internal history and map it through an MLP to produce a small set of latent queries. These queries attend to keys and values obtained from the external prefix (with low-rank bottleneck projections). The attention output is projected to a global context vector and broadcast-added to the internal sequence, yielding a context-modulated history. We take the last context-modulated state as the summary mt ∈ RD used in \label {eq:gated_fusion} \mathbf {g}_t=\sigma \!\left (\mathrm {MLP}_{g}\!\left (\hat {\bar {\mathbf {z}}}^{\text {ext}}_{t}\right )\right ), \qquad \hat {\mathbf {z}}^{\text {int}}_{t+1} = (\mathbf {1}-\mathbf {g}_t)\odot \tilde {\mathbf {z}}^{\text {int}}_{t+1} \;+\; \mathbf {g}_t\odot \mathbf {m}_{t}.

(16)

Gate input variants. The default gate uses only the current pooled external latent: \mathbf {g}_t=\sigma \!\left (\mathrm {MLP}_{g}\!\left (\bar {\mathbf {z}}^{\text {ext}}_{t}\right )\right )\in (0,1)^D. (17) When explicit external semantic attributes are available, we concatenate their embedding: \mathbf {g}_t=\sigma \!\left (\mathrm {MLP}_{g}\!\left ([\bar {\mathbf {z}}^{\text {ext}}_{t},\,\mathbf {e}_{c,t}]\right )\right ). (18) Definition of the Maximal Injection Step. For the qualitative visualization of causal responses in the main text, the “maximal injection step” is defined as the time step t that yields the maximum mean activation PDacross all dimensions of 1 the vector gate gt . Specifically, tmax = arg maxt ( D d=1 gt,d ). This provides a scalar proxy to temporally align frames where the external context exerts the strongest overall influence on the internal dynamics. B.7

Multi-view Out-cabin Fusion with View Embeddings

AIDE provides three out-cabin cameras (front/left/right) and one in-cabin camera. Let ftout,v = Evlm (oout,v ) ∈ RD denote the Qwen3-VL feature from out-cabin t view v ∈ {front, left, right}. We first inject a learnable view embedding, then fuse the out-cabin views into a single external latent. Our default fusion aligns with the main text by first extracting the view-conditioned representations and subsequently performing mean pooling: \bar {\mathbf {f}}^{\text {out},v}_t = \mathbf {f}^{\text {out},v}_t + \mathbf {e}_{\text {view}}(v), \qquad \mathbf {z}^{\text {ext}}_{t,v} = \bar {\mathbf {f}}^{\text {out},v}_t, \qquad \bar {\mathbf {z}}^{\text {ext}}_t = \frac {1}{3}\sum _{v}\mathbf {z}^{\text {ext}}_{t,v}.

(19)

For the in-cabin stream (single view), we use: \mathbf {z}^{\text {int}}_t = \mathbf {f}^{\text {in}}_t + \mathbf {e}_{\text {view}}(v_{\text {in}}).

(20)

(Alternative fusion schemes, e.g., learned attention pooling over views, can be plugged in, but are not required for the main results.)

Driver-WM

B.8

Skeleton Representation and Teacher-based Distillation

Skeleton topology and kinematic tree. We represent the driver using a HALPE-136 style skeleton with K=136 keypoints. Let E denote the set of kinematic edges (bone pairs) used for bone-length regularization (Eq. (24)). In our implementation, E follows the standard HALPE whole-body tree (body+hands), and we keep the joint ordering consistent across all splits. Coordinate normalization and missing-joint masking. We compute skeleton losses in normalized image coordinates. Given a keypoint (x, y) in pixel coordinates for a frame of size (W, H), we normalize it as: \tilde {x}=x/W,\qquad \tilde {y}=y/H,

(21)

and compute MPJPE/PCK on the normalized coordinates. To handle occlusions and low-confidence detections from the teacher, we use a per-joint confidence mask mt ∈ {0, 1}K : \mathcal {L}_{\text {skel}}= \sum _{t}\frac {1}{\sum _{i}m_{t,i}+\epsilon }\sum _{i=1}^{K} m_{t,i}\left \|\hat {\mathbf {s}}_{t,i}-\mathbf {s}_{t,i}\right \|_2^2,

(22)

where mt,i = 1 if the teacher confidence exceeds a threshold and 0 otherwise. B.9

Physical and Kinematic Priors

To ensure that the autoregressive rollouts remain anatomically and geometrically plausible over long horizons, we supplement the primary regression loss with a composite physical prior: \label {eq:total_phys_loss} \mathcal {L}_{\text {phys}} = \lambda _{\text {bone}}\mathcal {L}_{\text {bone}} + \lambda _{\text {smooth}}\mathcal {L}_{\text {smooth}} + \lambda _{\text {seat}}\mathcal {L}_{\text {seat}}.

(23)

Kinematic Tree Consistency. For variants utilizing 3D keypoint lifting or supervision, we enforce constant bone lengths across the kinematic tree E (e.g., the HALPE-136 whole-body topology). The bone-length consistency loss is defined as: \label {eq:bone_loss_supp} \mathcal {L}_{\text {bone}} = \sum _{t=1}^{T} \sum _{(i,j) \in \mathcal {E}} \left | \| \hat {\mathbf {s}}_{t,i} - \hat {\mathbf {s}}_{t,j} \|_2 - \| \mathbf {s}_{t,i} - \mathbf {s}_{t,j} \|_2 \right |,

(24)

where si,j denote the ground-truth joint coordinates. Temporal Smoothness. To mitigate high-frequency jitter in self-conditioned rollouts, we penalize first- and second-order temporal differences: \label {eq:smooth_loss_supp} \mathcal {L}_{\text {smooth}} = \sum _{t} \| \Delta \hat {\mathbf {s}}_{t} - \Delta \mathbf {s}_{t} \|_2^2 + \sum _{t} \| \Delta ^2 \hat {\mathbf {s}}_{t} \|_2^2, where ∆ŝt = ŝt+1 − ŝt and ∆2 ŝt = ŝt+2 − 2ŝt+1 + ŝt .

(25)

H. Chi et al.

Cabin ROI Feasibility. We define a soft Region-of-Interest (ROI) penalty to discourage key joints J (e.g., head, hips) from drifting outside anatomically plausible cabin boundaries. Given ROI = [xmin , xmax ] × [ymin , ymax ], the loss is formulated as: \begin {split} \mathcal {L}_{\text {seat}} = \sum _{t} \sum _{j \in \mathcal {J}} \Big ( & \text {ReLU}(x_{t,j} - x_{\max }) + \text {ReLU}(x_{\min } - x_{t,j}) \\ & + \text {ReLU}(y_{t,j} - y_{\max }) + \text {ReLU}(y_{\min } - y_{t,j}) \Big ). \end {split} (26)

B.10

Stochastic Transition Core and KL Bottleneck

In our probabilistic ablation variants, the transition operator Fθ is factorized into a stochastic internal predictor fθ followed by conditioned contextual injection. Reparameterized Transition. The predictor fθ outputs the parameters of a diagonal Gaussian distribution representing the candidate next state: \label {eq:int_transition_supp} (\boldsymbol {\mu }_{t+1}, \log \boldsymbol {\sigma }_{t+1}) = f_{\theta }(\hat {\mathbf {z}}^{\text {int}}_{t}), \quad \tilde {\mathbf {z}}^{\text {int}}_{t+1} = \boldsymbol {\mu }_{t+1} + \boldsymbol {\epsilon } \odot \boldsymbol {\sigma }_{t+1}, \quad \boldsymbol {\epsilon } \sim \mathcal {N}(\mathbf {0}, \mathbf {I}).

(27)

For the deterministic Driver-WM used in our main results, we set σ t+1 = 0. Supervised KL Regularization. When stochastic modeling is enabled, we regularize int int int the predicted density qθ (z̃int t+1 | ẑt ) with a conditional prior p(zt+1 ) = N (zt+1 , I) centered at the ground-truth latent. This supervised probabilistic bottleneck enforces latent consistency via: \label {eq:kl_cond} \mathcal {L}_{\text {KL}} = \sum _{t=1}^{T-1} \text {KL}\left ( \mathcal {N}(\boldsymbol {\mu }_{t+1}, \text {diag}(\boldsymbol {\sigma }^2_{t+1})) \| \mathcal {N}(\mathbf {z}^{\text {int}}_{t+1}, \mathbf {I}) \right ).

(28)

The KL term admits the following closed-form solution: \mathcal {L}_{\text {KL}} = \frac {1}{2} \sum _{t=1}^{T-1} \sum _{i=1}^{D} \left ( \sigma _{t+1,i}^2 + (\mu _{t+1,i} - z^{\text {int}}_{t+1,i})^2 - 1 - \log \sigma _{t+1,i}^2 \right ).

(29)

Unified Optimization Objective. The total training objective integrates the geometric rollout loss, latent consistency, and the aforementioned priors: \label {eq:total_loss_supp} \mathcal {L} = \lambda _{\text {lat}}\mathcal {L}_{\text {lat}} + \lambda _{\text {skel}}\mathcal {L}_{\text {skel}} + \lambda _{\text {aux}}\mathcal {L}_{\text {aux}} + \beta \mathcal {L}_{\text {KL}} + \lambda _{\text {phys}}\mathcal {L}_{\text {phys}}.

(30)

In practice, model selection is driven by h=1 . . . 5 MPJPE, while LKL and Lphys serve as controllable regularizers to stabilize long-term kinematic rollouts. B.11

Additional Training Details (Fill-in Table)

For completeness and reproducibility, we summarize key hyperparameters in Table 6.

Driver-WM

Hyperparameter

Setting Architecture Details

VLM feature dim D 2048 (Qwen3-VL 2B, frozen) Latent interface: identity (view embedding only) TransitionMLP Causal Temporal core 1 layer, 4 heads, Df f =4096 Gated Cross-Attn 1 layer, 4 heads Data Processing & Setup Sampling strategy Tobs / Tpred Target pose format

10 frames uniformly sampled over 3.0s 1.5s (5 frames) / 1.5s (5 frames) HALPE-136 (136 keypoints) Optimization Setup

Optimizer Learning rate schedule Weight decay Batch size / Max epochs Loss weights

AdamW (β1 = 0.9, β2 = 0.999) Cosine annealing, base LR 5 × 10−5 1 × 10−5 32 / 100 (Best epoch typically ∼78) λskel =1.0, λsem =1.0 (equal weighting)

Table 6: Implementation details of the proposed Driver-WM.

H. Chi et al.

Additional Results

Table 7: Comprehensive performance summary of all evaluated models (seed 42). We report horizon-averaged geometric errors (MPJPE in px and diagonalnormalized %), PCK, and semantic metrics (Accuracy and Macro-F1). This supplementary table provides exhaustive metrics across all classification and forecasting tasks to manifest the multi-task capacity. For the no pose head variant, kinematic forecasting metrics are not applicable (marked as “–”) as the skeletal decoder is removed. For motion-only & offline baselines, externally grounded semantics (TCR/VCR) are inapplicable and marked as “–”. Rank Model 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 —

Zero-Velocity Static pooling (no rollout) KL bottleneck +Skeleton-Pre Driver-WM (main) w/o ext context Non-causal (bidir) Driver-WM (full pretrained) MotionBERT Offline Enc-Dec no-prompt feat. RSSM/GRU dyn. Cross-Attn only Late fusion Single-stream SiMLPe ST-GCN no pose head

All MPJPE HM MPJPE PCK PCK (px) (%) (px) (%) @0.05 @0.10

DBR (Acc/F1)

DER (Acc/F1)

TCR (Acc/F1)

VCR (Acc/F1)

52.89 68.50 70.56 70.92 71.47 71.74 72.15 72.68 73.51 75.87 76.54 78.42 80.14 82.59 90.87 106.38 110.98 –

40.72 / 8.27 69.62 / 54.04 75.37 / 69.89 74.06 / 69.55 73.23 / 68.07 73.40 / 71.33 73.40 / 73.35 75.37 / 73.22 67.98 / 56.70 67.32 / 53.82 73.89 / 68.88 70.77 / 67.15 67.49 / 62.52 67.98 / 45.87 54.52 / 24.56 59.44 / 37.26 51.23 / 18.96 72.74 / 67.22

59.61 / 14.94 73.56 / 60.07 77.83 / 71.54 81.61 / 75.65 79.47 / 72.61 74.22 / 63.56 81.44 / 74.46 80.62 / 73.97 66.34 / 52.71 64.70 / 50.68 79.31 / 69.92 79.80 / 72.29 75.53 / 68.75 68.80 / 49.10 62.07 / 24.35 66.34 / 41.69 61.08 / 22.66 77.50 / 70.92

–/– 65.52 / 26.39 92.45 / 88.84 92.28 / 88.33 93.10 / 90.15 65.52 / 26.39 92.12 / 88.65 93.43 / 90.54 –/– –/– 85.39 / 76.87 92.12 / 88.26 92.12 / 87.94 92.78 / 87.85 65.52 / 26.39 –/– –/– 92.94 / 88.83

–/– 53.86 / 14.00 81.61 / 64.68 82.92 / 70.24 82.59 / 68.34 53.86 / 14.00 81.94 / 68.82 81.94 / 62.09 –/– –/– 77.18 / 58.59 81.44 / 63.74 82.92 / 69.09 82.92 / 65.97 53.86 / 14.00 –/– –/– 83.74 / 69.44

2.40 3.11 3.20 3.22 3.24 3.26 3.27 3.30 3.34 3.45 3.47 3.56 3.64 3.75 4.13 4.83 5.04 –

139.19 134.56 136.50 136.04 138.03 136.94 136.50 138.03 141.53 144.05 142.08 136.46 142.41 143.31 147.44 156.29 158.10 –

6.32 6.11 6.20 6.18 6.27 6.21 6.20 6.27 6.42 6.54 6.45 6.19 6.47 6.51 6.69 7.09 7.18 –

85.95% 72.55% 71.71% 72.09% 71.66% 71.13% 71.39% 70.99% 78.01% 77.17% 68.28% 67.48% 66.43% 65.07% 59.63% 63.45% 60.97% –

93.61% 89.10% 88.33% 88.30% 88.16% 88.20% 88.79% 88.25% 92.94% 93.07% 86.45% 86.69% 85.75% 85.28% 82.05% 86.29% 84.88% –

Driver-WM

Table 8: Comprehensive multi-seed robustness analysis (3 seeds: 42, 123, 456). We report the mean and standard deviation (µ ± σ) for all geometric and semantic metrics under the 5→5 rollout protocol. For motion-only baselines, externally grounded semantics (TCR/VCR) are physically inapplicable and marked as “–”. †: copy-last-frame baseline. Model

All MPJPE (px)↓ (%)↓

HM MPJPE (px)↓ (%)↓

DBR Acc↑

DER F1↑

Acc↑

TCR F1↑

Motion-only & offline baselines Zero-Velocity† 52.89±0.0 2.40±0.0 139.19±0.0 6.32±0.0 40.72±0.0 8.27±0.0 59.61±0.0 14.94±0.0 MotionBERT 73.74±0.5 3.35±0.0 142.61±1.1 6.47±0.1 66.77±1.0 54.96±1.6 66.28±2.2 52.37±1.1 Offline Enc-Dec 76.91±1.3 3.49±0.1 145.14±1.5 6.59±0.1 66.34±0.9 50.61±2.9 64.20±2.1 50.56±0.4

Acc↑

F1↑

Acc↑

VCR F1↑

PCK @0.05↑

PCK @0.10↑

– – –

85.95±0.0 93.61±0.0 77.77±0.4 92.89±0.4 76.78±0.3 92.69±0.3

Controlled contextual baselines (identical VLM interface) Single-stream 92.26±1.8 4.19±0.1 146.98±0.5 6.67±0.0 45.32±8.0 13.70±9.4 60.43±1.4 18.08±5.4 65.52±0.0 26.39±0.0 53.86±0.0 14.00±0.0 59.19±1.1 81.57±0.5 Late fusion 79.93±2.4 3.63±0.1 140.89±2.1 6.39±0.1 68.42±1.2 53.02±7.5 72.25±3.9 54.23±8.0 92.45±0.3 88.23±0.9 82.92±0.2 67.99±2.3 65.98±1.0 86.04±0.7 Cross-Attn only 79.27±0.9 3.60±0.0 141.23±1.0 6.41±0.1 66.61±2.1 61.31±1.6 77.45±1.9 69.27±1.9 92.12±0.7 88.18±0.7 83.36±0.4 68.72±2.0 66.75±0.4 85.85±0.1 Driver-WM variants Driver-WM (main) 71.66±0.3 3.25±0.0 138.34±1.2 6.28±0.1 73.34±1.3 69.09±1.8 80.35±0.8 73.08±1.0 92.94±0.8 89.26±1.3 82.81±0.4 66.54±1.6 71.40±0.3 88.21±0.3 KL bottleneck 71.53±0.9 3.25±0.0 137.85±1.4 6.26±0.1 73.89±1.9 67.43±2.4 78.38±1.4 71.08±1.2 92.67±0.2 89.13±0.5 82.27±1.4 66.40±3.5 71.17±0.4 88.31±0.1 +Skeleton-Pre 72.32±1.3 3.28±0.1 138.35±2.0 6.28±0.1 72.58±1.3 68.04±2.9 80.51±1.4 73.98±1.7 92.77±0.9 89.01±1.1 81.88±2.4 69.57±0.7 70.93±1.1 87.94±0.3 RSSM/GRU dyn. 76.48±1.7 3.47±0.1 137.21±1.1 6.23±0.1 71.53±0.7 67.70±1.4 79.86±0.9 72.29±1.5 92.34±0.2 87.74±0.5 81.50±0.9 63.91±0.8 68.46±0.9 86.97±0.3 w/o ext context 71.74±0.0 3.26±0.0 136.94±0.0 6.21±0.0 73.40±0.0 71.33±0.0 74.22±0.0 63.56±0.0 65.52±0.0 26.39±0.0 53.86±0.0 14.00±0.0 71.13±0.0 88.20±0.0

Table 9: Reference semantic comparison on AIDE (test set). We report taskwise Accuracy (Acc, %) for the four AIDE semantic tasks. Note: classification-only SOTAs are purely discriminative and cannot perform multi-step kinematic rollouts; therefore, kinematic metrics (MPJPE/PCK) are inapplicable. AIDE names the 4th task as VCR (Vehicle Condition Recognition), while several later works denote the same task as VBR (Vehicle Behavior Recognition) for the same label set. Model

Paradigm

Classification-only SOTAs (no kinematic rollout) UV-M3TL [19] Classif. TEM3 -Learning [20] Classif. MMTL-UniAD [21] Classif.

Rollout DBR Acc DER Acc TCR Acc VCR/VBR Acc ×† ×† ×†

73.82 69.31 73.61

77.39 75.00 76.67

96.57 96.29 93.91

Driver-WM (ours): kinematic rollout + auxiliary semantics Driver-WM (seed 42) Rollout+Aux ✓ 73.23 79.47 93.10 Driver-WM (3 seeds, mean±std) Rollout+Aux ✓ 73.34±1.31 80.35±0.83 92.94±0.75

87.07 86.11 85.00 82.59 82.81±0.38

† Classification-only methods are optimized for clip-level recognition and do not generate future kinematics; MPJPE/PCK and test-time rollout interventions are not applicable.

Additional Post-hoc Visualizations

do(λ = 0)

do(Ext = ∅)

Driver-WM

H. Chi et al.

𝑓𝑓6

𝑓𝑓7

𝑓𝑓8

𝑓𝑓9

𝑓𝑓10

Fig. 5: Additional post-hoc visualizations with the optional frozen renderer. Five uniformly sampled future frames are shown for one representative test clip under the 5→5 rollout protocol. Rows correspond to ground truth (GT), the factual rollout of Driver-WM, and two interventions: do(Ext=∅) and do(λCA =0). The renderer is used only for qualitative interpretation and is not part of training or evaluation. Compared with the factual rollout, the two interventions induce visibly different head and upperbody motion trends, consistent with the skeleton-level intervention analysis reported in the main paper.

For qualitative interpretation only, we additionally visualize selected rollouts using the optional frozen renderer illustrated in the main paper. This renderer is an off-the-shelf post-hoc visualization utility and is not part of the reported training, evaluation, or model selection pipeline. Accordingly, all quantitative results in the main paper and this supplement are computed strictly from skeleton rollout metrics and auxiliary semantic predictions, rather than from rendered pixel-space appearance. Figure 5 shows one representative clip under the same 5→5 rollout protocol used throughout the paper. Columns correspond to five uniformly sampled future frames across the prediction window, and rows compare the ground truth (GT), the factual rollout of Driver-WM, and two intervened rollouts: do(Ext=∅) and do(λCA =0). These rendered frames are provided solely to improve visual interpretability of the predicted driver motion under different contextual conditions. The renderer reveals clear differences in the temporal trend of driver responses under different conditions. In this example, the factual rollout remains comparatively stable and closer to the ground-truth future sequence. In contrast, the two interventions induce visibly different head and upper-body responses from the

Driver-WM

second sampled frame onward: removing external context leads to a more evident upward head motion, while disabling the contextual pathway induces a stronger head-turning tendency. These rendered frames serve as a qualitative counterpart to the skeleton-level intervention analysis reported in the main paper.

Notation and Definitions

Table 10 summarizes key symbols and conventions. Bold lowercase letters denote vectors and bold uppercase letters denote matrices. Subscripts indicate time indices. Hat ˆ· denotes predictions (rollouts). Table 10: Summary of key notations and definitions. Symbol

Description Time and sequences

t ∈ {1, . . . , T } Tobs , Tpred , T x1:T = {xt }Tt=1 ˆ·

Time index. Observed steps, predicted steps, and total horizon (T = Tobs + Tpred ). A generic sequence over time. Prediction/rollout operator, e.g., x̂t . Observations

out oin t , ot

In-cabin and out-cabin observations at time t (out-cabin can be multi-view). Perception backbone and features

Evlm (·) ftin ∈ RD ftout ∈ RD eview (v) ∈ RD f̄t Aϕ (·)

Frozen VLM visual encoder. In-cabin feature: ftin = Evlm (oin t ). Out-cabin feature: ftout = Evlm (oout t ) (per-view if applicable). Learnable view embedding for view identifier v. View-conditioned feature: f̄t = ft + eview (v). Optional residual adapter; default interface is identity. Dual-stream latent states

Internal (driver/cabin) latent state at time t. Multi-view external latents at time t; v indexes views and V is the total number. PV ext z̄ext Mean-pooled external latent: z̄ext = V1 t t v=1 zt,v . int in ext out (Default interface) zt = f̄t and zt,v = f̄t,v . ẑint Rolled-out (predicted) internal latent at time t. t ext int ext ext Ẑint Stacked histories up to time t: Ẑint , Z̄ ≤t ≤t ≤t = {ẑτ }τ ≤t , Z̄≤t = {z̄τ }τ ≤t . zint ∈ RD t V Zext = {zext t t,v }v=1

H. Chi et al.

Table 10 (continued from previous page) Symbol

Description Time-causal interaction and gated coupling

Ctxθ (·) mt ∈ RD fθ (·) z̃int t+1 gt ∈ (0, 1)D ẑint t+1

Time-causal interaction operator (instantiated as causal crossattention). ext External-to-internal context summary: mt = Ctxθ (Ẑint ≤t , Z̄≤t ). Internal transition predictor producing (µt+1 , log σ t+1 ) from current internal latent. Candidate next internal latent via reparameterization: z̃int t+1 = µt+1 + ϵ ⊙ σ t+1 (deterministic case uses µt+1 ). Vector-valued gate: gt = σ(MLPg (z̄ext t )). int Gated causal update: ẑint = (1 − g t ) ⊙ z̃t+1 + gt ⊙ mt . t+1 Outputs and heads

K Dskel (·) ŝt Ddbr , Dder Dtcr , Dvcr

Number of skeleton keypoints (e.g., K=136 for HALPE-136). Skeleton decoding head. K×2 Predicted 2D skeleton: ŝt = Dskel (ẑint . t ) ∈ R Auxiliary semantic heads attached to internal latents. Auxiliary semantic heads attached to external latents. Optional probabilistic bottleneck

int qθ (z̃int t+1 | ẑt ) LKL

Diagonal Gaussian transition N (µt+1 , diag(σ 2t+1 )). Supervised KL regularizer (ablation only). Main model is deterministic. Training objectives

Lskel Lphys Llat Laux L

Skeleton regression loss on {ŝt }. Physical priors (bone-length, temporal smoothness, ROI feasibility). Latent rollout consistency loss. Auxiliary semantic supervision losses. Total objective: L = λlat Llat + λskel Lskel + λaux Laux + λphys Lphys .

Record · ID 158550 · SHA-256 6701d1ebff2d43d6
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.