arXiv CSopen access

Causality-Aware End-to-End Autonomous Driving via Ego-Centric Joint Scene Modeling

Causality-Aware End-to-End Autonomous Driving via Ego-Centric Joint Scene Modeling

arXiv:2605.13646v1 [cs.RO] 13 May 2026

Seokha Moon1 , Minseung Lee1 , Joon Seo1 , Jinkyu Kim1,2 , Jungbeom Lee1 1 Korea University, 2 Kakao Mobility

Abstract End-to-end autonomous driving, which bypasses traditional modular pipelines by directly predicting future trajectories from sensor inputs, has recently achieved substantial progress. However, existing methods often overlook the causal interdependencies in ego-vehicle planning, ignoring the reciprocal relations between the ego vehicle and surrounding agents. This causal oversight leads to inconsistent and unreliable trajectory predictions, especially in interaction-critical scenarios where ego decisions and neighboring agent behaviors must be reasoned about jointly. To address this limitation, we propose CaAD, a Causality-aware end-toend Autonomous Driving framework that captures these dependencies within a shared latent scene representation. First, we propose a ego-centric joint-causal modeling module that builds on the marginal prediction branch, and learns causal dependencies between the ego vehicle and interaction-relevant agents. Second, we employ a causality-aware policy alignment stage implemented with joint-mode embeddings to align the stochastic ego policy with planning-oriented closed-loop feedback computed from surrounding traffic and map context. On the Bench2Drive and NAVSIM benchmarks, CaAD demonstrates strong closed-loop planning performance, achieving a Driving Score of 87.53 and Success Rate of 71.81 on Bench2Drive, and a PDMS of 91.1 on NAVSIM.

1

Introduction

End-to-end (E2E) autonomous driving has recently emerged as a promising alternative to traditional modular pipelines [27, 31, 35, 37, 54, 56, 58, 63, 64], directly predicting future trajectories or control actions from sensor inputs [4, 6, 7, 13, 17, 46]. By learning a unified mapping from perception to planning, E2E models reduce the reliance on hand-crafted intermediate representations and mitigate error accumulation across separately optimized stages. With the availability of large-scale driving datasets and closed-loop benchmarks [1, 2, 9, 15], most recent E2E planners are trained primarily with imitation learning (IL), where the model is optimized to reproduce expert driving behavior [5, 6, 7, 13, 17, 47]. This paradigm has led to substantial progress, demonstrating the effectiveness of data-driven planners in learning complex driving policies from expert behaviors. Despite this progress, existing E2E methods still struggle to represent mutually consistent futures of the ego vehicle and surrounding agents in highly interactive traffic, largely because they model all agents marginally (Step 1 of Fig. 1(a)). Reliable driving requires more than predicting an individually plausible ego trajectory; the predicted ego motion must remain coherent with the future behavior of nearby agents. In real-world driving, a merge may succeed only if a neighboring vehicle yields, while an overtake may be safe only when surrounding vehicles maintain compatible motions. Similarly, crossing an intersection may depend on the behavior of conflicting traffic. Such interaction dependencies have been extensively studied in the modular motion forecasting task, where independent marginal futures are known to be insufficient for faithfully representing multi-agent scenes [36, 37, 38, 44, 50]. In contrast, prior E2E driving methods still predict the ego plan and surrounding-agent motion as loosely coupled outputs, relying mainly on shared latent features to Preprint.

Encoder

Multi-view
 Images

Marginal

Embeddings

Agent 3

Agent 1

Agent 2

Ego

Reinforcement

Learning

Step 1. Marginal Prediction Learning

Scene-Agnostic

Ego Policy

Marginal Ego Policy

Marginal Predictions

(a) Prior Methods

Step 2. Causality-Limited Policy Optimization

Marginal Prediction

Encoder Multi-view
 Images

Marginal Agent

Embeddings

Joint-mode

Embeddings

Agent

Selection

Reinforcement

Learning Joint Ego Policy Select Ignore

Scene-Aware

Ego-Policy Reference Policy

Ego-Centric Joint Prediction

Step 1. Ego-Centric Joint-Causal Modeling

Ego Policy

Step 2. Causality-Aware Policy Alignment (b) CaAD (Ours)

Figure 1: Comparison between prior E2E methods and CaAD. (a) Prior methods first perform marginal prediction learning and then optionally adopt RL-based policy optimization on marginal representations, resulting in causality-limited policy refinement. (b) We first performs ego-centric joint-causal modeling and then applies causality-aware policy alignment on joint-mode embeddings, where each sampled ego rollout is evaluated. capture interactions implicitly [5, 13, 17, 46, 47]. Consequently, trajectories that are reasonable in isolation may form a globally inconsistent traffic scene, resulting in unreliable closed-loop behavior during critical interactions like merging or yielding. To address this limitation, we propose CaAD, a Causality-aware end-to-end Autonomous Driving framework that captures these dependencies by organizing shared features into ego-conditioned joint scene representations rather than using shared latent context alone. As shown by Step 1 in Fig. 1(b), CaAD performs ego-centric joint-causal modeling through interaction between marginal agent embeddings and joint-mode embeddings, where marginal embeddings retain actor-specific motion evidence and joint-mode embeddings encode ego-centric scene hypotheses for modeling interactions. This ego-centric formulation is well suited for planning-oriented scene modeling, where uncertainty is primarily driven by the subset of agents critical to the ego-vehicle’s safety and feasibility. The resulting interaction yields joint-mode representations that facilitate coupled prediction of the ego-plan and relevant agent behaviors. This enables the planner to reason over interactive futures while maintaining the computational efficiency of planning-oriented E2E architectures. Supervised imitation alone does not directly optimize closed-loop driving quality, since pointwise trajectory losses may underweight key objectives such as safety, map compliance, comfort, and longhorizon progress [14, 25]. Accordingly, although some methods [25, 39, 43] further refine the ego policy with reward or preference signals, such refinement remains causality-limited, as it is still built on marginal or ego-focused planning representations. In contrast, as shown in Step 2 of Fig. 1(b), our method performs causality-aware policy alignment on ego-centric joint-mode embeddings, allowing reinforcement learning to refine an ego policy that has already been shaped by joint-causal modeling. This keeps surrounding-agent prediction supervised while shifting the ego policy toward safer and more coherent interaction outcomes, leading to state-of-the-art performance on Bench2Drive [15] and NAVSIM [9]. Our contributions can be summarized as follows: • Joint-causal modeling. To the best of our knowledge, CaAD is the first end-to-end driving framework to explicitly incorporate ego-centric joint prediction. By facilitating interactions between marginal agent embeddings and ego-centric joint-mode embeddings, CaAD constructs shared representations which enable the model to learn causality-aware scene dynamics. • Causality-aware policy alignment. We employ a causality-aware policy alignment stage built on ego-centric, joint-mode embeddings to align a stochastic ego policy with planning2

oriented, closed-loop feedback. This feedback is computed against the surrounding traffic and map context, while keeping the surrounding-agent forecasting supervised. • State-of-the-art closed-loop performance. CaAD achieves state-of-the-art performance on both Bench2Drive and NAVSIM. On Bench2Drive, it achieves a Driving Score of 87.53 and a Success Rate of 71.81. On NAVSIM, it attains the best PDMS of 91.1.

2

Related Work

2.1

End-to-End Planning and Reinforcement Learning for Autonomous Driving

End-to-end autonomous driving learns planning-oriented representations directly from sensor observations, offering an alternative to modular perception–prediction–planning pipelines [4, 6, 7, 13, 17, 46, 47, 52]. To better capture multi-modality and uncertainty, generative planning methods have further emerged, ranging from anchor-based methods [5, 45, 46, 47] to diffusion- or flow-based trajectory generation [29, 53, 61]. Recent studies have also explored world-model-based planning, motivated by its ability to model future scene evolution [22, 23, 24, 55, 59, 60, 62]. In parallel, reinforcement learning and preference-based optimization have been used to refine imitation-trained driving policies toward objectives beyond behavior cloning [3, 18, 20, 25, 33, 39, 51, 57]. However, existing approaches still primarily optimize the ego planner rather than explicitly modeling reciprocal ego–agent future dynamics. In contrast, our proposed CaAD introduces ego-centric joint prediction and a scene-level reinforcement learning stage, enabling coupled prediction of the ego and ego-relevant surrounding agents in a shared latent space. 2.2

Joint and Interaction-Aware Motion Forecasting

In interactive driving, independently predicted marginal futures can be mutually inconsistent, motivating joint models for scene-consistent agent futures [10, 34, 36, 37, 38, 41, 44, 50]. Recent methods improve interaction-aware forecasting with scene-centric [37], factorized [44], latent-variable [10], graph-based joint future prediction [34], transformer query-based prediction and refinement [41], token-based language modeling [38], self-supervised pre-training [50], and retrocausal/instructable formulations [49]. Complementary safety-aware planners further estimate future collision risks and drivable-region compliance to support interpretable end-to-end driving [19]. However, these works mainly address ego-side prediction or safety-aware ego planning, rather than explicitly optimizing a globally coordinated multi-agent plan. CaAD augments marginal forecasting with a selective joint module that captures causal dependencies between the ego vehicle and interaction-relevant actors. A more detailed discussion of earlier end-to-end planners, RL-based policy refinement, and joint forecasting models is deferred to Appendix A.

3

Method

CaAD addresses the loose coupling between ego planning and surrounding-agent forecasting in interactive end-to-end driving, where individually plausible futures can be interaction-inconsistent. As shown in Fig. 2, CaAD augments a query-based E2E planner with ego-centric joint-causal modeling through marginal–joint embedding interaction. Marginal embeddings preserve actor-specific evidence, while ego-centric joint-mode embeddings organize predictions into shared scene modes (Sec. 3.2), yielding scene representations that align the ego trajectory with interaction-relevant agent futures. Building on these ego-centric joint mode representations, a causality-aware policy alignment stage further refines the stochastic ego policy using planning-oriented feedback evaluated under the predicted scene context (Sec. 3.3). 3.1

Preliminaries

Our method builds on a query-based end-to-end driving framework [47] that unifies detection, mapping, motion prediction, and planning within a shared task-query architecture. Its unified decoder iteratively refines task queries through temporal attention, agent-agent attention, and interaction attention that couples planning and ego queries with detection and map queries. The resulting marginal agent embeddings encode actor-specific motion cues. We denote the current marginal embedding of 3

(a) Overall Architecture

CaAD

Update

Initial Queries

𝑧

Joint Prediction Marginal Prediction

Reward

GRPO Sampling

Unified Decoder

Encoder

Agent-Mode Attention

Training-Only

Multi-view Images

(c) Agent-Mode Attention

(b) Causality-aware Policy Alignment

×𝑁!"#$!"%

Joint Scene Optimization

(e) Joint Prediction

(d) Marginal Prediction

𝑄

𝑆! 𝑆#

Head

% H

Head

𝑉

Scene Feature

⋮

Split

Joint-mode Embedding

𝐾

Transformer Decoder

Marginal Embedding

𝐸𝑔𝑜 Planned Trajectories

Anchor Embedding

𝑆" Interaction-relevant Agent Selection

/

/

/

Ego / Plan / Agent / Mode Queries

Ego-centric Mode Assignment

Selected Trajectory

/

Selected / Ignored

Figure 2: Overview of the proposed CaAD. (a) CaAD builds on a query-based architecture and augments the marginal prediction branch with a training-only ego-centric joint prediction branch. (b) A causality-aware policy alignment stage further aligns the stochastic ego policy with planning-oriented rewards computed from surrounding traffic and map context. (c) Agent-Mode Attention enables marginal agent embeddings and joint-mode embeddings to interact to learn scene representations. (d) The marginal branch preserves stable trajectory prediction, (e) while the joint branch supervises interaction-relevant agents under ego-centric scene modes. agent i by xai ∈ RC , and the current ego planning embedding by xe ∈ RC , omitting refinement-layer indices for readability. Throughout this section, N denotes the number of surrounding agents and C the feature dimension. The backbone also predicts ego trajectories at multiple granularities: we e use the temporally sampled trajectory τtp for ego-centric mode assignment and the spatial trajectory e τsp as a geometric cue for selecting interaction-relevant agents. However, because these marginal embeddings are optimized primarily for actor-wise prediction, they do not explicitly represent shared scene modes conditioned on ego motion. This limitation motivates the joint-causal modeling module introduced next. 3.2

Ego-Centric Joint-Causal Modeling

CaAD augments the marginal motion prediction branch with a ego-centric joint-causal modeling to improve scene-level consistency between ego planning and surrounding-agent forecasting. The e module introduces M ego-centric scene modes, where each mode m predicts an ego trajectory τ̂m a N and surrounding-agent trajectories {τ̂i,m }i=1 under the same scene hypothesis. To avoid unnecessary coupling, we apply joint supervision only to agents that are relevant to the ego maneuver, while leaving the remaining agents to the marginal prediction branch. Marginal–joint embedding interaction. Building on the decoded ego and agent embeddings defined in Sec. 3.1, we introduce joint-mode embeddings to instantiate ego-centric scene modes. For surrounding agent i, let zai,m ∈ RC denote the agent joint-mode embedding associated with the m-th scene mode, and let zem ∈ RC denote the ego joint-mode embedding for the same m-th mode. We then form ego and agent mode stacks by concatenating each decoded embedding with its M joint-mode embeddings, namely Ue = [xe ; ze1 ; . . . ; zeM ] and Uai = [xai ; zai,1 ; . . . ; zai,M ], where Ue , Uai ∈ R(1+M )×C . As illustrated in Fig. 2(c), the Agent-Mode Attention module views each stack as a compact mode-wise token sequence, consisting of the decoded embedding followed by M joint-mode embeddings, and applies attention [48] along the mode dimension: Ūe = AttnAM (Ue , Ue , Ue ) ,

Ūai = AttnAM (Uai , Uai , Uai ) . 4

(1)

(a) All-Actor Winner-Takes-All

Agg. L2

Lowest

Error

Others

2

1

Agg. L2

2

(b) Ego-Centric Winner-Takes-All

2

Lowest

Error

Others

1

Minimum Scene-Aggregate L2

Ego

Mode 3

Mode 2

Mode 1

Ego

Mode 3

Mode 2

Mode 1

Ego

3

(i) Select relevant agents Relevant

Agent

Irrelevant

Agent

(ii) Select mode by ego L2

GT Trajectory

All Actor Prediction

Ego-Centric Prediction

Marginal Prediction

Figure 3: Comparison of joint mode selection strategies. (a) Conventional methods select the joint mode based on the aggregated L2 error over all predicted ego and agent trajectories. (b) Ego-centric winner-takes-all selection chooses the joint mode using the ego trajectory L2 error and supervises interaction-relevant agent responses under the selected ego mode. In implementation, we collect the ego and all surrounding-agent stacks into a batch of per-entity mode sequences of shape (1 + M ) × C, and apply Agent-Mode Attention independently to each sequence. Since the marginal embeddings are produced by the unified decoder, they already contain ego, agent, and map context. Agent-Mode Attention then integrates this entity-level context into each corresponding mode embedding, allowing the joint-mode embeddings to attend scene information specific to the ego or agent entities. The first output token updates the decoded ego or agent embedding, while the remaining tokens become refined joint-mode embeddings. After the final refinement layer, a joint refinement head decodes the m-th ego mode embedding and the corresponding agent mode embeddings into an egoe a a e centric scene hypothesis Sm = (τ̂m , τ̂1,m , . . . , τ̂N,m ), where τ̂m denotes the predicted ego trajectory a for mode m, and τ̂i,m denotes the predicted trajectory of agent i under the same mode. Thus, the shared mode index m defines an ego-centric future scene hypothesis in which the ego trajectory and surrounding-agent futures are organized under a common scene mode. Interaction-relevant agent selection. A key design choice is that we do not force all agents to follow ego-centric joint supervision. Several agents are far from the ego vehicle and are better handled by standard marginal forecasting. We therefore select an interaction set Aint using the ground-truth e ego spatial trajectory τsp and the marginal agent predictions: 
a e Aint = i ∃k ∈ Ki , s.t. collision τ̃i,k , τsp = true , (2) a where τ̃i,k ∈ Ki is the k-th marginal prediction of agent i, and Ki denotes the set of marginal multi-modal planning candidates considered for collision checking. We use the spatial trajectory because a temporally sampled ego trajectory does not always expose the intended path clearly, with path geometry often entangled with speed or timing. By contrast, the spatial trajectory directly reflects where the ego intends to move, providing a stable and interpretable cue for interaction relevance. This is also consistent with the sparsity of real-world interactions: only a subset of surrounding agents should meaningfully respond to changes in ego intent, and restricting supervision to this subset avoids over-conditioning irrelevant actors.

Ego-centric mode assignment. A conventional all-actor winner-takes-all rule selects the aggregate scene mode, e.g., the one with the lowest summed L2 error for all agent trajectories [36, 37, 50]. In end-to-end driving, however, we organize the joint representation around the ego trajectory to better align it with the ego planning objective. As shown in Fig. 3, CaAD accordingly selects the joint mode based on the ego trajectory and applies the selected mode only to interaction-relevant agents. Formally, we define the selected joint scene mode as: e e m⋆ = arg min d(τ̂m , τtp ), m

(3)

where d(·, ·) is a masked trajectory distance. Agents in Aint are then supervised at the same egoselected mode m⋆ , while non-selected agents keep marginal supervision with their own best marginal 5

a a mode ki⋆ = arg mink d(τ̃i,k , τi,gt ). This design focuses scene-mode learning on reciprocal dependencies around ego motion and avoids assigning every actor in the scene to the same joint mode. The joint mode classification loss is formulated as: X X Ljoint_cls = Lecls (m⋆ ) + Lacls,i (m⋆ ) + Lacls,i (ki⋆ ), (4) i∈Aint

i∈A / int

where Lcls denotes the focal loss [30] for classification. 3.3

Causality-Aware Policy Alignment

The key novelty of this stage is that policy refinement operates on the ego-centric joint-mode representations learned in Sec. 3.2, rather than on isolated marginal or ego-only planning embeddings. Each sampled ego rollout is therefore evaluated together with the corresponding surrounding-agent futures under the same joint mode, grounding the update in a coupled future hypothesis. To improve closed-loop driving quality beyond supervised imitation, we implement this causality-aware policy alignment with a post-training GRPO objective [11, 40], which is shown in Fig. 2(b). Concretely, the ego trajectory head is parameterized as a Gaussian policy. For each joint mode m, we sample G ego rollouts and score the g-th rollout with a planning-oriented reward rm,g , using a PDMS-style [9] penalty-plus-quality decomposition adapted for training, as detailed in Appendix B.1. This aligns the ego policy using complete scene outcomes and encourages safer, more coherent future interactions. Gaussian trajectory policy. Classic E2E methods typically formulate future trajectory prediction as a deterministic regression task. To incorporate reinforcement learning, however, the model must parameterize a trajectory distribution instead. We therefore interpret the original regression output as the predictive mean µ and additionally predict the standard deviation σ. The supervised joint regression term is then written as a Gaussian negative log-likelihood: X X Ljoint_reg = Lereg (m⋆ ) + Lareg,i (m⋆ ) + Lareg,i (ki⋆ ), i∈Aint

1 Lreg (m̃) = 2

µm̃ − τgt 2 σm̃

2

i∈A / int

!

(5)

+ 2 log σm̃ + log 2π ,

where τgt denotes the corresponding ground-truth trajectory target. This converts the original trajectory regression head into a stochastic policy head during training; during inference, we use only the predicted mean trajectory µ. Policy alignment. We first sample ego-only rollouts for each mode, represented as τ̄m ∈ RG×Tfut ×2 , where Tfut denotes the prediction horizon. We then normalize the rewards rm,g within each mode group and define the truncated advantage following [66]:  rm,g − mean(rm ) −1, if collision occurs, Am,g = , Ãm,g = (6) max(0, Am,g ), otherwise. std(rm ) We then optimize the clipped GRPO objective over sampled ego rollouts:       πθ πθ LGRPO = −Em,g min Ãm,g , clip , 1 − ϵ, 1 + ϵ Ãm,g , πθold πθold

(7)

where πθ and πθold denote the current and old policies, respectively, and ϵ controls the update step size for training stability. Compared with plain reward maximization, the group-relative formulation is useful here because it asks the model to rank multiple candidate ego rollouts under the same scene context and to move probability mass toward the better ones. This makes the update closely aligned with the goal of interactive planning, where the critical question is often not whether a trajectory is feasible in isolation, but whether it is preferable to other plausible choices under the same scene. Importantly, we apply reinforcement learning only to the ego policy, while keeping surrounding-agent prediction supervised. This avoids noisy per-agent rewards and stabilizes the joint scene predictor. The total objective functions are deferred to Appendix B.2. 6

Table 1: Comparison of methods on Bench2Drive [15] under open-loop and closed-loop metrics. Avg. L2 is averaged over the 2-second prediction horizon at 2 Hz intervals. ∗ denotes methods that incorporate RL-based policy refinement during training, and † denotes the ground-truth expert. Open-loop Metric

Method

Closed-loop Metric

Avg. L2 ↓

Driving Score ↑

Success Rate (%) ↑

Think2Drive [22]

–

91.85

85.41

–

–

UniAD [13] VAD [17] SparseDrive [46] GenAD [61] DriveTransformer [16] Hydra-NeXt [26] MomAD [42] HiP-AD [47] VADv2 [5] SafeDrive [19]

0.73 0.91 0.87 – 0.62 0.92 0.82 0.69 – –

45.81 42.35 44.54 44.81 63.46 73.86 47.91 86.77 76.15 66.77

16.36 15.00 16.71 15.90 35.01 50.00 18.11 69.09 50.46 42.40

129.21 157.94 170.21 – 100.64 197.76 174.91 203.12 178.24 –

43.58 46.01 48.63 – 20.78 20.68 51.20 19.36 37.81 –

DIVER∗ [43] ReCogDrive∗ [25]

1.11 –

68.90 71.36

36.75 45.45

72.34 138.18

22.34 17.45

CaAD (Ours)∗

0.68

87.53

71.81

172.53

34.34

†

Efficiency ↑

Comfortness ↑

Table 2: Comparison of different methods under ability metrics. ∗ denotes methods that incorporate RL-based policy refinement during training Ability (%) ↑

Method Merging

Overtaking

Emergency Brake

Give Way

Traffic Sign

Mean

UniAD [13] VAD [17] DriveTransformer [16] Hydra-NeXt [26] HiP-AD [47]

14.10 8.11 17.57 40.00 50.00

17.78 24.44 35.00 64.44 84.44

21.67 18.64 48.36 61.67 83.33

10.00 20.00 40.00 50.00 40.00

14.21 19.15 52.10 50.00 72.10

15.55 18.07 38.60 53.22 65.98

DIVER∗ [43] ReCogDrive∗ [25]

35.08 29.73

25.09 20.00

41.09 69.09

50.00 20.00

59.21 71.34

42.09 42.03

CaAD (Ours)∗

60.00

86.67

83.33

50.00

71.34

70.27

4

Experiments

4.1

Experimental Setup

We evaluate CaAD on two complementary benchmarks: Bench2Drive [15], which emphasizes closedloop performance in interaction-intensive scenarios, and NAVSIM [9], which evaluates planning quality under its official protocol. On Bench2Drive [15], we report the official closed-loop metrics together with fine-grained driving-ability scores for challenging maneuvers such as merging, overtaking, and emergency braking. For ablation studies, we additionally use Bench2Drive-mini subset (details in Appendix C.2). On NAVSIM [9], we report PDMS along with its component metrics, which provide a more interpretable breakdown of planning behavior in terms of safety, progress, and ride quality. Our model uses a sparse query-based end-to-end planner backbone [47] and follows the standard training and evaluation setup unless otherwise specified. Additional details on dataset splits, metric definitions, preprocessing, training schedules, optimizer settings, hardware configuration, and ablation settings are provided in Appendix C. 4.2

Comparison with SOTA Methods

Evaluation on Bench2Drive. As shown in Tab. 1, we compare CaAD with state-of-the-art methods on the closed-loop Bench2Drive benchmark [15]. CaAD achieves state-of-the-art performance, attaining a Driving Score of 87.53 and a Success Rate of 71.81. Notably, its Driving Score is close to that of Think2Drive [22], the expert policy used to collect the training data. This result suggests that the proposed joint-causal scene modeling and causality-aware policy alignment improve scene-level decision making, thereby improving overall planning performance. Tab. 2 further shows that the advantage is particularly pronounced for interaction-critical abilities, such as Merging and Overtaking, where it is essential to reason about how surrounding agents respond to the ego maneuver. The methods 7

Table 3: Compact ablation on Bench2Drive-mini. Checkmarks indicate temporal/spatial ego cues for relevant-agent selection, joint prediction on selected agents with ego-centric mode assignment, and policy alignment on marginal or joint scene representations. † denotes reproduced under the same 6-epoch ablation setting. Model

Agent Selection Temp. Cue

Spatial Cue

Joint Prediction Selected Agents

Policy Align

Ego-centric

Marginal

Joint

HiP-AD† (Base) [47] Model A Model B Model C Model D Model E (Ours)

✓ ✓ ✓ ✓ ✓

✓ ✓ ✓ ✓

✓ ✓ ✓

✓

Closed-loop

Ability

DS ↑

SR(%) ↑

83.12

62.96

Mean ↑ 63.83

87.43 (4.31↑) 86.90 (3.78↑) 84.76 (1.64↑) 88.61 (5.49↑) 91.05 (7.93↑)

64.81 (1.85↑) 68.52 (5.56↑) 64.81 (1.85↑) 75.93 (12.97↑) 79.63 (16.67↑)

66.38 (3.45↑) 66.92 (3.09↑) 56.29 (7.54↓) 72.66 (8.83↑) 76.51 (12.68↑)

marked with ∗ provide a particularly informative comparison: DIVER [43] and ReCogDrive [25] incorporate RL-based refinement and outperform several purely imitation-learned methods, yet their improvements remain uneven across abilities. This suggests that policy refinement alone is insufficient when the underlying scene representation remains marginal. Overall, this pattern supports the effectiveness of the proposed joint-causal scene modeling, which provides a causality-aware scene representation that makes subsequent policy alignment substantially more reliable. Evaluation on NAVSIM. As shown in Table 4, Table 4: Closed-loop planning performance on CaAD also performs strongly on NAVSIM [9], NAVSIM [9]. The upper block lists non-RL methachieving the best PDMS of 91.1 and the ods, and the lower block lists RL-based methods. strongest collision-related results. The gains PDMS Components PDMS↑ are not confined to the aggregate score: com- Method NC↑ DAC↑ EP↑ TTC↑ pared with prior RL-used methods [25, 39, 43], 97.9 92.4 79.3 93.0 84.0 CaAD consistently improves safety-critical com- PARA-Drive [52] [23] 96.4 95.4 81.7 88.7 84.6 ponents, where reasoning about ego-agent in- LAW GoalFlow [53] 98.3 93.8 79.8 94.3 85.7 teractions is particularly important. This result DiffusionDrive [29] 98.2 96.2 82.2 94.7 88.1 96.0 80.4 93.1 86.6 Hydra-MDP++ [21] 97.6 suggests that policy alignment is more effective WOTE [24] 98.4 96.6 81.7 94.5 88.0 when applied to causality-aware scene embedRL-based methods dings than to marginal representations. In CaAD, DIVER [43] 98.5 96.5 82.6 94.9 88.3 policy alignment refines an ego policy that has 98.5 98.1 84.3 94.8 90.0 DriveDPO [39] already been shaped by joint-causal scene mod- ReCogDrive [25] 97.9 97.3 87.3 94.9 90.8 eling, using rewards computed from the groundCaAD (Ours) 99.4 98.3 84.9 95.9 91.1 truth futures of the jointly considered agents together with map context. As a result, the aligned policy produces safer rollouts with larger collision margins while maintaining strong rule compliance and competitive route progress. 4.3

Ablation Studies

Effectiveness of joint-mode design and causality-aware policy alignment. Tab. 3 shows that joint-causal scene modeling is already highly effective before policy alignment. Compared to the reproduced baseline, Model D, which incorporates spatial agent selection, joint prediction of the selected agents, and ego-centric assignment, improves Driving Score, Success Rate, and Ability mean by an average of 5.49, 12.97, and 8.83 points, respectively. This suggests that interactive planning benefits from organizing futures of relevant agents under the ego-selected scene mode, rather than relying solely on marginal predictions. Building on this stronger representation, causality-aware policy alignment (Model E) yields the best overall performance, further improving over Model D by 2.44 Driving Score, 3.70 Success Rate, and 3.85 Ability mean points, and over the reproduced baseline by 7.93, 16.67, and 12.68 points, respectively. Importantly, these gains do not arise from policy alignment alone. Although Model A performs policy alignment directly on marginal embeddings, it improves over the reproduced baseline by only 4.31 Driving Score, 1.85 Success Rate, and 3.45 Ability mean points. In contrast, Model D already surpasses Model A by 1.18, 11.12, and 6.28 points without policy alignment, and Model E further exceeds Model A by 3.62, 14.82, and 10.13 points. Together, these comparisons show that joint-causal scene modeling provides a stronger planning representation than marginal policy alignment alone, serving as a prerequisite for effective policy alignment in interactive driving. 8

T1

→

T2

→

T3

Ours

HiP-AD

Ours

HiP-AD

⏱

Figure 4: Qualitative examples of interaction-aware planning. We visualize consecutive frames in two challenging interactive scenarios, where the sky-blue trajectory denotes the spatial trajectory and the red trajectory denotes the temporal trajectory.

Impact of ego trajectory representation for agent selection. Tab. 3 compares temporal and spatial ego cues for selecting interaction-relevant agents. Under the same joint-prediction and ego-centric assignment setting, replacing the temporal cue in Model B with the spatial cue in Model D improves Driving Score, Success Rate, and Ability mean by 1.71, 7.41, and 5.74 points, respectively. This suggests that the spatial cue provides a clearer signal of the ego maneuver for relevant-agent selection, whereas the temporal cue can be entangled with speed or timing and lead to weaker supervision. Importance of Ego-Centric Assignment. Tab. 3 also isolates role of ego-centric assignment is even more direct. When spatial selection and joint prediction are kept fixed, removing ego-centric assignment from Model D to obtain Model C lowers Driving Score, Success Rate, and Ability mean by 3.85, 11.12, and 16.37 points, respectively. This substantial degradation shows that identifying relevant agents is not sufficient; their futures must also be organized under the ego-selected scene mode. These results validate ego-centric assignment as a key component for effective interactionsparse supervision. 4.4

Qualitative Results

Fig. 4 shows qualitative results of CaAD in challenging interactive scenarios. (a) CaAD reasons about surrounding vehicles and proceeds through the gap between agents, avoiding the deadlock that may occur when agent responses are not modeled jointly with the ego plan. (b) CaAD first waits for the interacting agent to pass before proceeding, whereas a planner without joint interaction modeling may enter prematurely, stop in front of the agent, and fail to make progress. These results show that ego-conditioned joint prediction enables CaAD to produce more coherent and socially compliant closed-loop behaviors in dense multi-agent traffic. Additional qualitative results on NAVSIM [9], together with further qualitative examples for the ablation study, are provided in the Appendix D.2.

5

Conclusion

We propose CaAD, a Causality-aware end-to-end Autonomous Driving framework that combines joint-causal scene modeling with causality-aware policy alignment for planning-oriented scene modeling. By learning ego-conditioned joint scene representations, CaAD achieves state-of-the-art performance on Bench2Drive and NAVSIM while improving interaction-consistent planning. We hope CaAD encourages a new direction for end-to-end driving, where future planning is grounded not only in ego trajectory prediction but also in causality-aware modeling of interactive scene dynamics. Handling long-tail social behaviors and validating transfer to more diverse real-world settings remain important directions for future work. 9

References [1] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. [2] Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810, 2021. [3] Raphael Chekroun, Marin Toromanoff, Sascha Hornauer, and Fabien Moutarde. Gri: General reinforced imitation and its application to vision-based autonomous driving. Robotics, 12(5):127, 2023. [4] Li Chen, Penghao Wu, Kashyap Chitta, Bernhard Jaeger, Andreas Geiger, and Hongyang Li. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10164–10183, 2024. [5] Shaoyu Chen, Bo Jiang, Hao Gao, Bencheng Liao, Qing Xu, Qian Zhang, Chang Huang, Wenyu Liu, and Xinggang Wang. Vadv2: End-to-end vectorized autonomous driving via probabilistic planning. arXiv preprint arXiv:2402.13243, 2024. [6] Kashyap Chitta, Aditya Prakash, and Andreas Geiger. Neat: Neural attention fields for endto-end autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15793–15803, 2021. [7] Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. IEEE transactions on pattern analysis and machine intelligence, 45(11):12878–12895, 2022. [8] Daniel Dauner, Marcel Hallgarten, Andreas Geiger, and Kashyap Chitta. Parting with misconceptions about learning-based vehicle motion planning. In Conference on Robot Learning, pages 1268–1281. PMLR, 2023. [9] Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, et al. Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking. Advances in Neural Information Processing Systems, 37:28706–28719, 2024. [10] Roger Girgis, Florian Golemo, Felipe Codevilla, Martin Weiss, Justin A. D’Souza, Samira Ebrahimi Kahou, Felix Heide, and Christopher Pal. Latent variable sequential set transformers for joint multi-agent motion prediction. In International Conference on Learning Representations, 2022. [11] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. [13] Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17853–17862, 2023. [14] Jyh-Jing Hwang, Runsheng Xu, Hubert Lin, Wei-Chih Hung, Jingwei Ji, Kristy Choi, Di Huang, Tong He, Paul Covington, Benjamin Sapp, et al. Emma: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262, 2024. [15] Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving. Advances in Neural Information Processing Systems, 37:819–844, 2024. 10

[16] Xiaosong Jia, Junqi You, Zhiyuan Zhang, and Junchi Yan. Drivetransformer: Unified transformer for scalable end-to-end autonomous driving. arXiv preprint arXiv:2503.07656, 2025. [17] Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8340–8350, 2023. [18] Bo Jiang, Shaoyu Chen, Qian Zhang, Wenyu Liu, and Xinggang Wang. Alphadrive: Unleashing the power of vlms in autonomous driving via reinforcement learning and reasoning. arXiv preprint arXiv:2503.07608, 2025. [19] Jungho Kim, Jiyong Oh, Seunghoon Yu, Hongjae Shin, Donghyuk Kwak, and Jun Won Choi. Safedrive: Fine-grained safety reasoning for end-to-end driving in a sparse world. arXiv preprint arXiv:2602.18887, 2026. [20] Derun Li, Changye Li, Yue Wang, Jianwei Ren, Xin Wen, Pengxiang Li, Leimeng Xu, Kun Zhan, Peng Jia, Xianpeng Lang, et al. Learning personalized driving styles via reinforcement learning from human feedback. arXiv preprint arXiv:2503.10434, 2025. [21] Kailin Li, Zhenxin Li, Shiyi Lan, Yuan Xie, Zhizhong Zhang, Jiayi Liu, Zuxuan Wu, Zhiding Yu, and Jose M Alvarez. Hydra-mdp++: Advancing end-to-end driving via expert-guided hydra-distillation. arXiv preprint arXiv:2503.12820, 2025. [22] Qifeng Li, Xiaosong Jia, Shaobo Wang, and Junchi Yan. Think2drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in carla-v2). In European Conference on Computer Vision, pages 142–158. Springer, 2024. [23] Yingyan Li, Lue Fan, Jiawei He, Yuqi Wang, Yuntao Chen, Zhaoxiang Zhang, and Tieniu Tan. Enhancing end-to-end autonomous driving with latent world model. arXiv preprint arXiv:2406.08481, 2024. [24] Yingyan Li, Yuqi Wang, Yang Liu, Jiawei He, Lue Fan, and Zhaoxiang Zhang. End-to-end driving with online trajectory evaluation via bev world model. arXiv preprint arXiv:2504.01941, 2025. [25] Yongkang Li, Kaixin Xiong, Xiangyu Guo, Fang Li, Sixu Yan, Gangwei Xu, Lijun Zhou, Long Chen, Haiyang Sun, Bing Wang, et al. Recogdrive: A reinforced cognitive framework for end-to-end autonomous driving. arXiv preprint arXiv:2506.08052, 2025. [26] Zhenxin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Zuxuan Wu, and Jose M Alvarez. Hydra-next: Robust closed-loop driving with open-loop training. arXiv preprint arXiv:2503.12030, 2025. [27] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: learning bird’s-eye-view representation from lidar-camera via spatiotemporal transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. [28] Xiaodan Liang, Tairui Wang, Luona Yang, and Eric Xing. Cirl: Controllable imitative reinforcement learning for vision-based self-driving. In Proceedings of the European conference on computer vision (ECCV), pages 584–599, 2018. [29] Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, et al. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 12037–12047, 2025. [30] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollar. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. [31] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela L Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA), pages 2774–2781. IEEE, 2023. 11

[32] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. [33] Yiren Lu, Justin Fu, George Tucker, Xinlei Pan, Eli Bronstein, Rebecca Roelofs, Benjamin Sapp, Brandyn White, Aleksandra Faust, Shimon Whiteson, et al. Imitation is not enough: Robustifying imitation with reinforcement learning for challenging driving scenarios. In 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7553–7560. IEEE, 2023. [34] Wenjie Luo, Cheolho Park, Andre Cornman, Benjamin Sapp, and Dragomir Anguelov. JFP: Joint future prediction with interactive multi-agent modeling for autonomous driving. In Proceedings of The 7th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1457–1467. PMLR, 2023. [35] Seokha Moon, Hyun Woo, Hongbeen Park, Haeji Jung, Reza Mahjourian, Hyung-gun Chi, Hyerin Lim, Sangpil Kim, and Jinkyu Kim. Visiontrap: Vision-augmented trajectory prediction guided by textual descriptions. In European Conference on Computer Vision, pages 361–379. Springer, 2024. [36] Nigamaa Nayakanti, Rami Al-Rfou, Aurick Zhou, Kratarth Goel, Khaled S Refaat, and Benjamin Sapp. Wayformer: Motion forecasting via simple & efficient attention networks. arXiv preprint arXiv:2207.05844, 2022. [37] Jiquan Ngiam, Benjamin Caine, Vijay Vasudevan, Zhengdong Zhang, Hao-Tien Lewis Chiang, Jeffrey Ling, Rebecca Roelofs, Alex Bewley, Chenxi Liu, Ashish Venugopal, et al. Scene transformer: A unified architecture for predicting multiple agent trajectories. arXiv preprint arXiv:2106.08417, 2021. [38] Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S. Refaat, Rami Al-Rfou, and Benjamin Sapp. MotionLM: Multi-agent motion forecasting as language modeling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. [39] Shuyao Shang, Yuntao Chen, Yuqi Wang, Yingyan Li, and Zhaoxiang Zhang. Drivedpo: Policy learning via safety dpo for end-to-end autonomous driving. arXiv preprint arXiv:2509.17940, 2025. [40] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [41] Shaoshuai Shi, Li Jiang, Dengxin Dai, and Bernt Schiele. Motion transformer with global intention localization and local movement refinement. In Advances in Neural Information Processing Systems, 2022. [42] Ziying Song, Caiyan Jia, Lin Liu, Hongyu Pan, Yongchang Zhang, Junming Wang, Xingyu Zhang, Shaoqing Xu, Lei Yang, and Yadan Luo. Don’t shake the wheel: Momentum-aware planning in end-to-end autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 22432–22441, 2025. [43] Ziying Song, Lin Liu, Hongyu Pan, Bencheng Liao, Mingzhe Guo, Lei Yang, Yongchang Zhang, Shaoqing Xu, Caiyan Jia, and Yadan Luo. Diver: Reinforced diffusion breaks imitation bottlenecks in end-to-end autonomous driving. arXiv preprint arXiv:2507.04049, 2025. [44] Qiao Sun, Xin Huang, Brian C Williams, Hang Zhao, et al. M2i: From factored marginal to joint trajectory prediction for highly interactive environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16116–16126, 2022. [45] Wenchao Sun, Xuewu Lin, Keyu Chen, Zixiang Pei, Xiang Li, Yining Shi, and Sifa Zheng. Sparsedrivev2: Scoring is all you need for end-to-end autonomous driving. arXiv preprint arXiv:2603.29163, 2026. 12

[46] Wenchao Sun, Xuewu Lin, Yining Shi, Chuang Zhang, Haoran Wu, and Sifa Zheng. Sparsedrive: End-to-end autonomous driving via sparse scene representation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 8795–8801. IEEE, 2025. [47] Yingqi Tang, Zhuoran Xu, Zhaotie Meng, and Erkang Cheng. Hip-ad: Hierarchical and multigranularity planning with deformable attention for autonomous driving in a single decoder. arXiv preprint arXiv:2503.08612, 2025. [48] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017. [49] Royden Wagner, Omer Sahin Tas, Felix Hauser, Marlon Steiner, Dominik Strutz, Abhishek Vivekanandan, Carlos Fernandez, and Christoph Stiller. Retromotion: Retrocausal motion forecasting models are instructable. arXiv preprint arXiv:2505.20414, 2025. [50] Royden Wagner, Omer Sahin Tas, Marvin Klemp, and Carlos Fernandez. Jointmotion: joint self-supervision for joint motion prediction. arXiv preprint arXiv:2403.05489, 2024. [51] Yan Wang, Wenjie Luo, Junjie Bai, Yulong Cao, Tong Che, Ke Chen, Yuxiao Chen, Jenna Diamond, Yifan Ding, Wenhao Ding, et al. Alpamayo-r1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail. arXiv preprint arXiv:2511.00088, 2025. [52] Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15449–15458, 2024. [53] Zebin Xing, Xingyu Zhang, Yang Hu, Bo Jiang, Tong He, Qian Zhang, Xiaoxiao Long, and Wei Yin. Goalflow: Goal-driven flow matching for multimodal trajectories generation in endto-end autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 1602–1611, 2025. [54] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10), 2018. [55] Zhenjie Yang, Xiaosong Jia, Qifeng Li, Xue Yang, Maoqing Yao, and Junchi Yan. Raw2drive: Reinforcement learning with aligned world models for end-to-end autonomous driving (in carla v2). arXiv preprint arXiv:2505.16394, 2025. [56] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11784–11793, 2021. [57] Zhenlong Yuan, Chengxuan Qian, Jing Tang, Rui Chen, Zijian Song, Lei Sun, Xiangxiang Chu, Yujun Cai, Dapeng Zhang, and Shuo Li. Autodrive-r2 : Incentivizing reasoning and selfreflection capacity for vla model in autonomous driving. arXiv preprint arXiv:2509.01944, 2025. [58] Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, and Raquel Urtasun. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8660–8669, 2019. [59] Bozhou Zhang, Nan Song, Jingyu Li, Xiatian Zhu, Jiankang Deng, and Li Zhang. Future-aware end-to-end driving: Bidirectional modeling of trajectory planning and scene evolution. arXiv preprint arXiv:2510.11092, 2025. [60] Jinqing Zhang, Zehua Fu, Zelin Xu, Wenying Dai, Qingjie Liu, and Yunhong Wang. Resworld: Temporal residual world model for end-to-end autonomous driving. arXiv preprint arXiv:2602.10884, 2026. [61] Wenzhao Zheng, Ruiqi Song, Xianda Guo, Chenming Zhang, and Long Chen. Genad: Generative end-to-end autonomous driving. In European Conference on Computer Vision, pages 87–104. Springer, 2024. 13

[62] Yupeng Zheng, Pengxuan Yang, Zebin Xing, Qichao Zhang, Yuhang Zheng, Yinfeng Gao, Pengfei Li, Teng Zhang, Zhongpu Xia, Peng Jia, et al. World4drive: End-to-end autonomous driving via intention-aware physical latent world model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 28632–28642, 2025. [63] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4490–4499, 2018. [64] Zikang Zhou, Jianping Wang, Yung-Hui Li, and Yu-Kai Huang. Query-centric trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 17863–17873, 2023. [65] Julian Zimmerlin, Jens Beißwenger, Bernhard Jaeger, Andreas Geiger, and Kashyap Chitta. Hidden biases of end-to-end driving datasets. arXiv preprint arXiv:2412.09602, 2024. [66] Jialv Zou, Shaoyu Chen, Bencheng Liao, Zhiyu Zheng, Yuehao Song, Lefei Zhang, Qian Zhang, Wenyu Liu, and Xinggang Wang. Diffusiondrivev2: Reinforcement learning-constrained truncated diffusion modeling in end-to-end autonomous driving, 2025.

14

Appendix

A

Additional Related Work

A.1

End-to-End Planning for Autonomous Driving

End-to-end autonomous driving learns planning-oriented representations directly from sensor observations, offering an alternative to modular pipelines [4, 6, 7, 13, 17, 46, 47, 61, 65]. Early methods such as TransFuser [7] and NEAT [6] predict planning-relevant waypoints from fused sensor features or BEV-based attention-field representations. More recent approaches incorporate stronger intermediate structures. UniAD [13] unifies perception, prediction, and planning in a planning-oriented framework, while VAD [17] and subsequent works [29, 46, 47, 61] improve vectorized scene modeling, efficiency, multimodal ego planning, or generative future modeling. To better capture multi-modality and uncertainty, generative planning methods have further emerged, ranging from anchor-based methods [5, 45, 46, 47] to diffusion- or flow-based trajectory generation [29, 53, 61]. Recent studies have also explored world-model-based planning, motivated by its ability to model future scene evolution [22, 23, 24, 55, 59, 60, 62]. Nevertheless, the final objective in much of this literature remains tightly centered on ego planning. These approaches lack a fully explicit joint model of ego-agent future dynamics. Consequently, reciprocal dependencies in highly interactive driving scenarios may be captured only partially or implicitly. A.2

Reinforcement Learning for End-to-End Planning

Although end-to-end driving is dominated by imitation learning (IL), reinforcement learning (RL) provides supervisory signals for objectives beyond behavior cloning, such as safety and humanpreference alignment [3, 18, 20, 25, 28, 33, 39, 51, 57]. Early methods such as CIRL [28] and GRI [3] combine expert demonstrations with RL for vision-based autonomous driving. More recent work performs post-training refinement of supervised planners or policies, including BC-SAC [33], TrajHF [20], and DriveDPO [39], using RL or preference-based objectives to improve safety, reasoning, or preference alignment. Recent works have further improved reasoning and planning capabilities beyond imitation-centric training [18, 25, 51, 57] for vision-language-action (VLA) models. However, these approaches still optimize primarily the ego policy or planner. RL serves mainly as an auxiliary signal beyond IL, rather than as an explicit objective for jointly cooperative multi-agent trajectory generation. Instead, our proposed CaAD employs a joint-causal policy alignment stage to fine-tune the driving policy with rewards that promote globally coherent, safe, and socially compliant behavior. A.3

Joint Prediction for Multi-Agent Interaction

In interactive driving, independently predicted marginal futures can be mutually inconsistent, motivating joint models for scene-consistent futures. Scene Transformer [37] introduces a scene-centric transformer for joint, marginal, and conditional prediction, while M2I [44] improves tractability by factorizing interactions into influencer–reactor pairs. Other interaction-aware approaches include JFP [34], which learns consistent multi-agent trajectories; AutoBots [10], which models joint trajectory distributions with latent-variable set transformers; MTR [41], which combines intention localization with refinement; MotionLM [38], which autoregressively models discrete motion tokens; and JointMotion [50], which pretrains joint predictors via self-supervision. More recently, RetroMotion [49] decomposes forecasting into marginal and pairwise joint predictions, enabling retrocausal refinement and instruction-based modification. Complementary to these forecasting methods, SafeDrive [19] adds fine-grained safety reasoning to end-to-end driving by estimating agentspecific collision risks and drivable-region compliance over future timesteps. Overall, these works mainly address interaction-aware prediction and safety-aware ego planning, rather than explicitly optimizing a globally coordinated multi-agent plan.

B

Additional Method Details

B.1

Reward Function

For RL stage on Bench2Drive [15], each sampled ego rollout is evaluated together with the jointly predicted futures of surrounding agents under the same scene mode. We follow the penalty-plusquality decomposition of PDMS used in NAVSIM [9], but adapt several subscores to make the reward more suitable for scene-level GRPO [40]. In particular, the original discrete PDMS subscores can assign identical rewards to many rollouts sampled within the same scene group. Such ties reduce the informativeness of the relative ranking signal used by GRPO. We therefore preserve the safety-oriented structure of PDMS while making selected subscores continuous.  r=

 Y

P

w∈{EP,T T C,C} weightw × scorew

scorep  × {z

}

safety/rule penalties

.

P

w∈{EP,T T C,C} weightw

p∈{N C,DAC,DD}

|

!

|

{z

driving-quality term

(8)

}

Here, NC denotes no at-fault collision, DAC drivable-area compliance, DD driving-direction compliance, EP ego progress, TTC time-to-collision margin, and C comfort. We use weights 5.0, 5.0, and 2.0 for EP, TTC, and C, respectively. Continuous subscore modifications. We modify the following subscores. The no-collision subscore remains discrete because at-fault collisions are safety-critical. These changes produce denser rewards and reduce ties among rollouts sampled in the same scene group, making the reward better matched to GRPO’s relative comparison objective. Driving-direction compliance. Let dopp denote the maximum accumulated opposite-direction progress. Instead of using the original three-level score, we linearly interpolate between the compliance threshold 2.0 m and violation threshold 6.0 m:   dopp ≤ 2.0, 1.0, opp −2.0 scoreDD = 1 − d6.0−2.0 , 2.0 < dopp < 6.0,  0.0, dopp ≥ 6.0. Drivable-area compliance. Instead of assigning zero reward whenever any off-road timestep occurs, we use the fraction of timesteps that remain within the drivable area: T

scoreDAC = 1 −

1X 1{ego is outside the drivable area at t}. T t=1

Time-to-collision. Instead of a binary TTC score, we score the rollout by the first TTC-infraction timestep: scoreT T C =

min(tTTC , T ) , T

where tTTC is the first TTC-infraction timestep and T is the rollout horizon. If no infraction occurs, the score is 1. Comfort. Instead of a hard conjunction over comfort thresholds, each comfort metric receives a continuous violation score:   ∆k , scoreC = min sk , sk = exp − k αk where ∆k is the violation magnitude for comfort metric k, and αk is a scale factor based on the corresponding threshold. 16

B.2

Training Objective

For completeness, we collect the full optimization objective here. Let Le2e denote the original supervised objective, including detection, mapping, planning, and marginal motion prediction [47]. The final objective is L = Le2e + λjoint (Ljoint_reg + Ljoint_cls ) + λrl LGRPO ,

(9)

where λjoint and λrl are weighting coefficients that control the contributions of the joint prediction loss and the reinforcement learning objective, respectively. This design preserves reliable sceneconditioned prediction under supervision, while allowing RL to refine only the preference over ego actions. As a result, supervised ego-conditioned joint prediction captures how the scene is organized around ego intent, whereas scene-aware GRPO improves closed-loop action selection under the predicted scene.

C

Additional Experimental Details

C.1

Dataset and Metrics

Bench2Drive and Metrics. Our primary evaluation uses Bench2Drive [15], a challenging closedloop benchmark designed to stress interactive driving. It provides 950 clips for training and 50 clips for open-loop validation, while closed-loop performance is measured on 220 predefined routes. We report the official metrics, including Driving Score (DS), Success Rate (SR), Efficiency, and Comfortness, and further break performance down into interaction-centric driving abilities such as Merging, Overtaking, Emergency Braking, Give Way, and Traffic Sign compliance. Ablation studies are conducted on the subsampled Bench2Drive benchmark [15], which is representative of the full dataset. NAVSIM and Metrics. We additionally evaluate on NAVSIM [9], a planning-oriented benchmark based on data-driven non-reactive simulation. Under the official NAVSIM v1 protocol, each predicted ego trajectory is rolled out for 4 seconds, while background actors follow their logged future trajectories. This setup measures planning quality in a standardized setting that isolates the effect of the ego policy. The primary NAVSIM metric is the Predictive Driver Model Score (PDMS), which summarizes planning performance by combining safety-critical penalties with a weighted quality term [9]. We also report its component metrics: no at-fault collisions (NC), drivable area compliance (DAC), ego progress (EP), and time-to-collision within bound (TTC). NC and DAC capture safety and rule compliance, whereas EP, and TTC reflect route progress, collision margin, and ride smoothness. Reporting both PDMS and its components makes the evaluation easier to interpret, since improvements may arise from different aspects of planning behavior. Since the NAVSIM planning metrics are also used to construct the reward in our scene-aware RL stage. C.2

Bench2Drive-mini for Ablation Studies

For closed-loop ablation experiments, we use the Bench2Drive-mini split [47]. Evaluating every ablation on all 220 routes is computationally expensive, so we utilized the 54-route subset, corresponding to around 25% of the full benchmark. This subset includes one representative route from each of the 43 scenarios, together with 11 additional randomly selected routes. Therefore, it preserves full scenario coverage while serving as a computationally efficient proxy for the full benchmark. We use exactly the same Route IDs listed in Tab. A1. Table A1: Bench2Drive-mini [15] Route IDs used for closed-loop ablations. Split

Route IDs

43 scenario routes

1711, 1773, 1790, 1825, 1852, 1956, 2050, 2082, 2084, 2115, 2127, 2144, 2164, 2201, 2204, 2273, 2373, 2390, 2416, 2509, 2534, 2664, 2709, 2790, 3086, 3248, 3364, 3436, 3464, 3540, 3561, 3936, 14194, 14842, 17563, 17752, 23658, 23695, 23771, 23901, 26458, 28087, 28099

11 random routes

1792, 2086, 2129, 2283, 2539, 2668, 26406, 26956, 27494, 27532, 28154

17

Table A2: Ablation on Bench2Drive [15] full routes set. † denotes reproduced results. Closed-loop

Model DS ↑

SR (%) ↑

Efficiency ↑

Comfortness ↑

Mean ↑

86.38

66.82

200.11

16.89

63.33

86.90 (0.52↑) 87.53 (1.15↑)

65.45 (1.37↓) 71.81 (4.99↑)

203.39 (3.28↑) 172.53 (27.58↓)

14.73 (2.16↓) 34.34 (17.45↑)

65.36 (2.03↑) 70.27 (6.94↑)

A. HiP-AD† (Base) [47] B. All-agent Policy Alignment C. Ego-centric Policy Alignment (Ours)

C.3

Ability

Implementation Details

Implementation for Bench2Drive. Our implementation uses a sparse query-based planner with multi-granularity ego queries, following the released baseline configuration [47] unless otherwise specified. Input images are resized to 640 × 352, and target points and high-level commands are embedded into the planner. Ego status is predicted during both training and inference. In addition, e we sample 2m trajectories for τsp . We use the predicted temporal and spatial trajectories only for longitudinal and lateral control, respectively, while ego-centric joint prediction and SRL are used solely as training-time supervision. Training is conducted in three stages. We first train the detection module for 12 epochs, followed by the planning module with joint prediction for another 12 epochs. We then further optimize the model with reinforcement learning for 6 epochs. All experiments are conducted on 8 NVIDIA A6000 GPUs with a total batch size of 48. The initial learning rate is set to 2 × 10−4 for the first two stages and 2 × 10−5 for the reinforcement learning stage. We use the AdamW optimizer [32] with a weight decay of 0.01. For the ablation study, we train all models under the same setting, with 6 epochs for stage two and 3 epochs for stage three. Implementation for NAVSIM. For NAVSIM, we follow the implementation of DiffusionDrive [29]. We use ResNet-34 [12] as the image encoder and additionally provide rasterized BEV LiDAR as input. Three front-view camera images are downsampled and concatenated into an input resolution of 1024 × 256. The model is trained from scratch on the navtrain split for 80 epochs using the AdamW optimizer [32] with a learning rate of 6 × 10−4 , and is then further optimized for 20 epochs with an additional PDMS-based causality-aware policy alignment stage.

D

Additional Experiments

D.1

Additional Quantitative Results

Ego-centric vs. All-agent Policy Alignment. We further ablate the scope of reinforcement learning by comparing ego-centric joint policy alignment with all-agent policy alignment, where rollout sampling and policy optimization are applied not only to the ego vehicle but also to each surrounding agent. Tab. A2 shows that all-agent policy alignment slightly improves Driving Score and Ability mean, but reduces Success Rate. In contrast, ego-centric causality-aware policy alignment achieves the best overall performance. This indicates that applying alignment to all agents is not necessarily beneficial, as per-agent sampling can introduce noisy credit assignment and destabilize supervised surrounding-agent forecasting. Ego-centric alignment instead preserves a reliable joint scene predictor and focuses policy optimization on the ego decisions that directly determine closed-loop performance. Evaluation on the nuScenes Open-loop Table A3: Open-loop evaluation on nuScenes [1] val set. Setting. We additionally evaluate CaAD † indicates reproduced results. in the open-loop setting on nuScenes [1]. L2 (m) ↓ Collision (%) ↓ Method Compared with the reproduced base1s 2s 3s Avg. 1s 2s 3s Avg. line [47], CaAD achieves a lower L2 error [17] 0.41 0.70 1.05 0.72 0.03 0.19 0.43 0.21 but a slightly higher collision rate. This im- VAD-Base GenAD [61] 0.28 0.49 0.78 0.52 0.08 0.14 0.34 0.19 plies that CaAD optimizes a closed-loop SparseDrive [46] 0.29 0.58 0.96 0.61 0.01 0.05 0.18 0.08 scene-level interaction, suggesting a per- DriveTransformer [16] 0.16 0.30 0.55 0.33 0.01 0.06 0.15 0.07 [42] 0.31 0.57 0.91 0.60 0.01 0.05 0.22 0.09 formance mismatch can occur between MomAD HiP-AD [47] 0.28 0.53 0.87 0.56 0.01 0.05 0.15 0.07 closed-loop and open-loop evaluation [8]. HiP-AD† [47] 0.31 0.63 1.05 0.67 0.00 0.03 0.14 0.06 Despite this, CaAD shows stronger closed- CaAD (Ours) 0.28 0.56 0.94 0.59 0.02 0.02 0.19 0.08 loop performance, suggesting improved interaction-aware planning beyond open-loop trajectory matching. 18

T2

T1

T3

(+) JCSM

Baseline

Stuck

(+) CPA

Collision

Figure A1: Qualitative examples of interaction-aware planning under module ablations. JCSM and CPA denote joint-causal scene modeling and causality-aware policy alignment, respectively. We visualize consecutive frames in challenging interactive scenarios, where the sky-blue trajectory denotes the spatial trajectory and the red trajectory denotes the temporal trajectory.

Front Left

Collision object trajectory

Front

Collision object T time bbox

DiffusionDrive Trajectory

Front Right

DiffusionDrive T time bbox

CaAD Trajectory

BEV

CaAD T time bbox

GT Trajectory

Ego Origin bbox

Figure A2: Qualitative NAVSIM [9] collision cases. Each row shows three camera views and BEV. Green, red, and purple trajectories denote CaAD, DiffusionDrive [29], and ground truth, respectively. In BEV, all boxes except the purple ego box are drawn at the same future time step T : orange denotes the collision-related object, red denotes the DiffusionDrive-predicted ego pose, and green denotes the CaAD-predicted ego pose. The purple box denotes the current ego pose at t = 0.

19

D.2

Additional Qualitative Results

Qualitative Comparison of Components. Fig. A1 compares qualitative behaviors under different module configurations. The baseline [47] tends to exploit locally available free space without sufficiently reasoning about surrounding-agent interactions, which can cause the ego vehicle to stop and become stuck in front of another vehicle. Ego-centric joint-causal modeling improves interaction reasoning by organizing relevant agent futures around ego motion, but may still suffer from failures such as colliding with a traffic cone. With Causal-aware policy alignment, the ego policy is further aligned using planning-oriented feedback, producing safer and more coherent interaction-aware behavior. Qualitative Results for NAVSIM. Fig. A2 shows collision cases on NAVSIM [9] between DiffusionDrive [29] and our method, CaAD. While DiffusionDrive [29] often predicts ego trajectories that overlap with collision-related objects at future time steps, CaAD generates trajectories that avoid these objects while remaining close to the ground-truth driving behavior. The BEV visualization further shows that this difference comes from the predicted ego trajectory region at the same future time step, highlighting the ability of CaAD to produce safer and more interaction-aware plans in object-conflict scenarios.

E

Discussions

Limitations. Despite the strong overall closed-loop results, CaAD still struggles with rare social behaviors. In particular, we did not observe successful cases in which the ego vehicle reliably clears space for an approaching emergency vehicle, a failure mode that is also common in prior end-toend driving frameworks [26, 43, 47]. More broadly, the gains are strongest in interaction-critical scenarios, while performance can still degrade when key agents are missed or when the scene contains rare conventions that are underrepresented in the training data. Our evaluation is also limited to camera-based simulation benchmarks. Social Impact. CaAD can contribute to safer and more socially compliant autonomous driving by improving how end-to-end planners reason about interactions between the ego vehicle and surrounding agents. This is particularly beneficial in interaction-critical scenarios such as merging, yielding, overtaking, and intersection crossing, where reliable driving requires coordinated behavior among multiple traffic participants. More broadly, CaAD provides a positive step toward end-to-end autonomous driving systems that better capture real-world traffic dynamics and improves reliability for autonomous driving.

20