arXiv CSopen access

CADET: Physics-Grounded Causal Auditing and Training-Free Deconfounding of End-to-End Driving Planners

MANUSCRIPT DRAFT, JUNE 2026

1

CADET: Physics-Grounded Causal Auditing and Training-Free Deconfounding of End-to-End Driving Planners

arXiv:2606.14438v1 [cs.RO] 12 Jun 2026

Zikun Guo

Abstract—End-to-end (E2E) autonomous-driving planners trained by imitation are prone to statistical shortcuts: they associate scene elements that merely co-occur with expert actions (a roadside object, a building facade) with driving decisions, rather than the variables that causally determine them. Such causal confusion silently compromises reliability in long-tail scenarios, and it is difficult to detect, because prevailing openloop metrics L2 displacement and collision rate are dominated by ego status and do not indicate whether a planner depends on spurious cues. Existing remedies based on causal-intervention training require retraining large models and cannot audit a planner that is already deployed. We present CADET, a trainingfree framework that audits, benchmarks, and repairs spurious reliance in pretrained E2E planners without any parameter update. CADET comprises three components. First, a Physicsgrounded Causal Reliance score (PCR) identifies planning queries on which the model depends but that cannot physically influence the decision, by fusing perturbation-based model influence with a physics-geometric prior (time-to-collision, drivable-corridor relevance, dynamic or static state, and detection confidence) that serves as an external anchor independent of the training distribution. Second, a counterfactual robustness benchmark, comprising three perturbation families spurious, causal-link, and distribution-shift and three metrics (CSI, CRI, and CCS), quantifies the extent to which a planner relies on spurious rather than causal factors. Third, Test-time Causal Masking (TCM) suppresses the flagged queries at inference, removing spurious reliance at negligible overhead and without training. On the pretrained SparseDrive planner on nuScenes, CADET establishes that, although the planner is predominantly causal, in individual frames its most influential physically irrelevant agent affects the plan as strongly as a genuine causal agent; that PCR isolates these dependencies more selectively than an influenceonly baseline; and that TCM removes them while open-loop L2 remains unchanged, empirically demonstrating that displacement error is insensitive to causal robustness. The entire pipeline is inference-only and runs on a single 16 GB GPU. We will release the code, benchmark, and audit toolkit. Index Terms—End-to-end autonomous driving, causal confusion, spurious correlation, causal intervention, counterfactual robustness, trustworthy AI, training-free.

I. I NTRODUCTION End-to-end (E2E) autonomous driving has become the dominant paradigm since UniAD [1], with VAD [2], SparseDrive [3], and PARA-Drive [4] further improving planning accuracy and efficiency. These systems are trained by imitation on large logged datasets, which introduces a structural weakness: imitation learning is non-causal and is liable Z. Guo is with the School of Electronics Engineering, Kyungpook National University, Daegu, Republic of Korea (e-mail: [email protected]).

to exploit spurious correlations, namely features that co-occur with expert actions in the data but do not cause them [5]. A planner may, for instance, learn to decelerate near a particular building facade or roadside object because such decelerations were frequent in the logs, while disregarding the agent that genuinely warrants braking. This causal confusion is widely regarded as a principal obstacle to advancing E2E driving from L2 to L3/L4 reliability. Two properties make this failure mode difficult to address. First, it is difficult to observe. Evaluation still relies predominantly on open-loop L2 displacement and collision rate on nuScenes, yet these metrics are dominated by ego status and change little when perception input is removed [6], so a low L2 error provides limited evidence as to whether a planner relies on spurious cues. Second, it is costly to repair. Recent causal-intervention methods such as CausalVAD [7] mitigate confounding by retraining the planner with a sparse intervention module, which is expensive and, more importantly, offers no means of auditing a planner that is already trained and deployed. To our knowledge, no training-free tool addresses the practical need to determine whether, where, and how much a given pretrained planner relies on spurious correlations. We address this gap with CADET (Causal Auditing and Deconfounding at Test-time), a training-free framework that audits, benchmarks, and repairs spurious reliance in pretrained E2E planners with no parameter updates (Fig. 1). Our central observation is that distinguishing true causes from globally spurious correlates cannot be done from training-data statistics alone: when a spurious co-occurrence holds across all environments, observational signals such as perturbation sensitivity and cross-environment invariance fail under the same faithfulness assumption [8]. We therefore introduce an external anchor, a physics-geometric prior derived from the perception module’s own outputs (3D boxes, velocity, class, and confidence). This prior judges whether an object can influence the decision at all, through time-to-collision, drivablecorridor relevance, and dynamic or static state, independently of the data distribution. A query that the planner relies on heavily but that physics deems irrelevant is, by construction, a spurious dependency. Contributions. • Physics-grounded causal audit (PCR). A training-free, model-agnostic score that flags per-query spurious reliance by fusing perturbation-based model influence with a physics-geometric prior, which acts as a hard external arbiter, together with a cross-environment stability term.

MANUSCRIPT DRAFT, JUNE 2026

2

PCR · Physics-Grounded Causal Audit

SparseDrive, weights frozen

per-agent score PCRi ∈ [0, 1]

6× surround cameras

Model influence I˜i ablate query → ∆ plan

Backbone + Detection

Physics prior ρi [hard gate] TTC, corridor, class Cross-env. stability σi

planner outputs

agent queries {qi }

queries + plan

Frozen E2E Planner

TCM · Test-time Causal Masking agents ranked by PCRi

PCRi

s flag mask if PCRi > θ and not physics-protected

Eq. (7) + gate (8)

re-plan with causal agents only

Counterfactual Benchmark plan a (ego trajectory) tests

Deconfounded plan a†

remove spurious agent

CSI

alter causal link

CRI

distribution shift

CCS

spurious reliance removed, maneuver preserved evaluate

Fig. 1. Overview of CADET. A frozen end-to-end planner (SparseDrive) emits agent queries and an ego plan. PCR computes a per-agent causal-relevance score by fusing three signals: model influence (ablate a query, measure the plan change), a physics-geometric prior ρ that acts as a hard gate protecting causal agents, and cross-environment stability. TCM masks the flagged spurious agents (high PCR, not physics-protected) and re-plans with the causal agents only, yielding a deconfounded plan. A counterfactual benchmark (CSI/CRI/CCS) quantifies robustness. The pipeline is training-free and inference-only on a single 16 GB GPU.

It requires only forward passes on a pretrained planner (Section IV). • Counterfactual robustness benchmark. A standardized protocol with three perturbation families (spurious, causal-link, and distribution-shift) and three metrics, the Causal Stability Index (CSI), Causal Response Index (CRI), and Causal Consistency Score (CCS), that quantify reliance on spurious versus causal factors. We use it to audit several public pretrained planners (Section IV, Section VI). • Training-free deconfounding (TCM). Test-time Causal Masking suppresses flagged spurious queries at inference, improving counterfactual robustness at negligible overhead and with no training, realizing an approximate do(·) intervention at deployment. • Reproducibility on commodity hardware. The full pipeline is inference-only and runs on a single 16 GB GPU. We release the code, the benchmark, and the audit toolkit. CADET is complementary to existing causal methods. Unlike CausalVAD [7] and Beyond Patterns [9], which retrain models for planning or prediction, and unlike CounterScene [10], which generates counterfactual scenarios for closed-loop testing, CADET neither trains nor generates: it instruments existing planners and is therefore immediately applicable to deployed systems. The remainder of the paper reviews related work (Section II), formalizes the causal setting (Section III), details CADET (Section IV), describes the protocol (Section V), reports the audit (Section VI), and discusses implications and limitations (Section VII, Section VIII). II. R ELATED W ORK A. End-to-End Driving Planners Deep learning has advanced rapidly across a broad spectrum of tasks, including efficient transformer architectures [11], model compression for question answering [12], generative modeling [13], speech enhancement [14], visual recognition [15], reinforcement learning for combinatorial optimiza-

tion [16], and multi-robot coordination and scheduling [17], [18]. Within this progress, vision-centric driving models have matured into end-to-end (E2E) systems that map raw sensor input directly to a plan. UniAD [1] introduced a planningoriented architecture that unifies perception, prediction, and planning; VAD [2] replaced dense rasterization with a vectorized scene representation; and SparseDrive [3] and PARADrive [4] improved efficiency through sparse queries and parallelized auxiliary tasks. These planners are trained by imitation and are evaluated, predominantly, by open-loop L2 displacement and collision rate on nuScenes. As Li et al. [6] show, however, those metrics are dominated by ego status, since removing perception input barely changes the reported numbers. A low open-loop error therefore provides limited evidence about whether a planner reasons over the scene or instead exploits spurious regularities, which motivates the measure of causal reliance that we develop. B. Causal Confusion and Deconfounding in Driving The non-causal nature of behavioral cloning was established by de Haan et al. [5], who showed that imitators rely on effects rather than causes, such as a brake-indicator light, so that additional observation can yield worse policies under distribution shift. In driving, CausalVAD [7] is the closest precursor to our work: it instantiates do-calculus as a sparse causal intervention scheme, builds a dictionary of context prototypes, and performs backdoor adjustment on vectorized queries, achieving state-of-the-art open-loop planning. Beyond Patterns [9] applies a diffusion-based backdoor adjustment to map features for trajectory prediction. Both methods retrain the planner, and both identify confounders from learned features or predefined attributes, without an external, distributionindependent criterion for causal status. CADET differs on both points: it operates on a frozen planner with no parameter update, and it anchors the spurious-versus-causal distinction in a physics-geometric prior rather than in data statistics. This directly addresses the global-spurious-correlation case in which feature-based identification fails. Table I positions

MANUSCRIPT DRAFT, JUNE 2026

3

TABLE I P OSITIONING OF CADET AGAINST CAUSAL - CONFUSION METHODS FOR DRIVING . CADET IS THE ONLY APPROACH THAT OPERATES ON A FROZEN , ALREADY- DEPLOYED PLANNER , REPAIRS IT AT INFERENCE TIME , AND SHIPS STANDARDIZED CAUSAL - ROBUSTNESS METRICS .

C

S

Y

TrainingAudits Test-time Causal free pretrained repair metrics CausalVAD [7] planning × × × × Fig. 2. Structural causal model of end-to-end planning. Scene context C Beyond Patterns [9] prediction × × × × confounds perception state S and plan Y via the backdoor path Y ← C → S CounterScene [10] scenario gen. × × × ✓ (dashed). An imitation-trained planner fits the confounded P (Y | S); CADET CADET (ours) audit + repair ✓ ✓ ✓ ✓ targets the deconfounded P (Y | do(S)). Method

Task

CADET against these methods along the capabilities relevant to deployed systems. Note that end-to-end planners such as UniAD, VAD, and SparseDrive are not competitors in this comparison but the subjects of the audit: CADET instruments a given planner, and its interface is planner-agnostic (Section V-B). C. Counterfactual Evaluation and Robustness A complementary line of work evaluates robustness through interventions. CounterScene [10] casts safety-critical scenario generation as a counterfactual intervention inside a diffusion world model for closed-loop testing, while Bench2Drive [19] and NAVSIM [20] provide closed-loop and non-reactive simulation benchmarks, reflecting a consensus that open-loop L2 does not predict closed-loop behavior. Invariance-based methods such as invariant risk minimization [8] seek predictors that are stable across environments, but they rest on a faithfulness assumption that fails precisely when a spurious correlate is invariant across all observed environments. Our counterfactual benchmark targets a different and underserved goal, namely measuring how much a given pretrained planner relies on spurious versus causal factors, and it does so with inexpensive, deterministic query-level perturbations rather than a generative simulator, which makes it reproducible on commodity hardware. D. Scene Perception and Structural Priors The physics prior in CADET is computed from the perception module’s own outputs, so understanding street scenes from imagery is an enabling capability; this has recently been advanced by combining large language models with streetview data [21]. Conceptually, injecting a structural or physical prior to guide decision-making parallels its use in balanced and scalable multi-robot path planning [22]. CADET applies the same principle to E2E driving, using a kinematic prior as an external anchor against spurious reliance. III. P ROBLEM F ORMULATION A. A Structural Causal Model for Planning We model an end-to-end planner with a structural causal model (SCM) M = (U, V, F , P (U)). The endogenous variables V comprise a latent scene context C (weather, road geometry, traffic density, and data-collection biases), the perception state S produced by the perception stack, and the

plan Y (the future ego trajectory). The exogenous U collect unobserved environmental noise. The relevant causal structure (Fig. 2) is C → S,

C → Y,

S → Y,

(1)

so that C is a confounder opening a backdoor path Y ← C → S. An imitation-trained planner fits the observational P (Y | S), which is biased by this backdoor; the deconfounded quantity we care about is the interventional P (Y | do(S)). Under the backdoor criterion with adjustment set C, X P (Y | do(S)) = P (Y | S, C=c) P (C=c). (2) c

Eq. (2) is the target that training-time methods such as CausalVAD [7] approximate by retraining. Our aim is different: to audit and approximate this adjustment at test time on a frozen planner. The tendency of modern models to exploit contextinduced spurious cues rather than causal ones is not unique to driving; it recurs across modalities, e.g., as sycophancy in video and medical vision–language models [23], [24], which motivates a general, model-agnostic treatment. B. Query-Level Notation Modern planners [1], [2], [3] expose a set of vectorized queries Q = {q1 , . . . , qN }, each qi encoding a perceived entity (agent or map element) together with physical attributes ϕi = (bi , vi , κi , ci ): 3D box, velocity, class, and detection confidence. With ego state e, the planner emits a trajectory a = f (Q, e) ∈ RT ×2 over a horizon T . We define a removal operator mi that replaces qi by a baseline (mean query or null token),
f\i (Q, e) ≜ f mi (Q), e , (3) and a set version f\F for a subset F ⊆ {1, . . . , N }. C. Spurious vs. Causal Reliance The reliance of the plan on query i is the decision change under its removal, ri =

f (Q, e) − f\i (Q, e) 2 .

(4)

A query is causally relevant if the entity it represents can physically affect the ego decision (a lead vehicle, a crossing pedestrian); it is spurious if ri is large yet the entity cannot physically matter (a distant static object off the ego path). The core difficulty is that observational signals alone cannot separate the two: when a spurious co-occurrence holds across

MANUSCRIPT DRAFT, JUNE 2026

4

all training environments (global spurious correlation), both a high reliance ri and a low cross-environment variance of ri wrongly suggest causality. This is the faithfulnessassumption failure that limits invariance-based methods [8]. We therefore introduce an external, distribution-independent anchor, a physics-geometric prior, in Section IV. IV. T HE CADET F RAMEWORK CADET comprises three training-free components: a perquery audit score (Section IV-A), a counterfactual benchmark (Section IV-B), and a test-time fix (Section IV-C). All operate on a frozen planner using forward passes only; an overview is shown in Fig. 1.

Algorithm 1 CADET audit and test-time deconfounding Input: frozen planner f , queries Q, ego e, attributes {ϕi }, threshold θ for each query i do ri ← ∥f (Q, e) − f\i (Q, e)∥2 // influence ρi ← ci · max(πi , τi ) · ωκi // physics prior end for I˜i ← ri / maxj rj ; compute σi across environments PCRi ← I˜i (1 − ρi )(1 + λσi ), apply hard gate (8) F ← {i : PCRi > θ} a† ← f\F (Q, e) // test-time causal masking return audit {PCRi }, deconfounded plan a†

B. Counterfactual Robustness Benchmark A. Physics-Grounded Causal Reliance Score (PCR) For each query we combine three complementary signals. a) (a) Model influence.: The (normalized) reliance from Eq. (4), I˜i = ri / max rj ∈ [0, 1], (5) j

captures what the planner currently depends on. It is unsupervised but reflects learned behavior rather than ground-truth causality, and therefore serves only as a first filter. b) (b) Physics-geometric prior (hard arbiter).: From the perception attributes ϕi we compute a distribution-independent plausibility ρi ∈ [0, 1] that the entity can influence the ego decision:
ρi = ci · max πi , τi · ωκi , (6) where ci is detection confidence; πi ∈ [0, 1] is path relevance, the overlap of the entity (and its short-horizon motion forecast from vi ) with the
ego drivable corridor; τi = clip 1 − TTCi /Tmax , 0, 1 is a time-to-collision urgency that vanishes for static, off-path, or diverging entities (TTCi → ∞); and ωκi is a class weight (e.g., vulnerable road users upweighted). Because ρi derives from physical kinematics and scene geometry rather than from training-data statistics, it is immune to global spurious correlation and serves as the external anchor. c) (c) Cross-environment stability.: Partitioning scenes into environments E (weather/road/time), we measure the dispersion of I˜i across E, σi ∈ [0, 1]; high dispersion flags locally spurious reliance that signal (b) may miss. d) Fusion.: The reliance score is high only when the planner relies on a query that physics deems implausible:

PCRi = I˜i · 1 − ρi · 1 + λ σi , (7) with the physics prior applied as a hard gate:   ρi ≥ ρhi (protect causal queries), 0, PCRi ← PCRi , (8) ρi ≤ ρlo ,   soft (7), otherwise. The gate guarantees that a physically plausible query is never flagged, encoding a conservative policy: a query is suppressed only when the physics prior confirms that it is implausible.

We perturb scenes at the query/perception level (cheap, training-free, deterministic) along three families and define a metric for each. Let d(a, a′ ) = ∥a − a′ ∥2 be a normalized plan distance. a) Spurious perturbation → CSI.: Alter queries with ρi ≈ 0 (physically irrelevant); a robust planner should not change its plan. With perturbed plan asp ,   CSI = 1 − E d(a, asp ) (↑ better). (9) b) Causal-link perturbation → CRI.: Modify a causal query (ρi high, e.g., lead-vehicle braking); the plan should respond. With aca and the expected correct response direction ∆⋆ ,   CRI = E ⊮{⟨aca − a, ∆⋆ ⟩ > 0} (↑ better). (10) c) Distribution-shift perturbation → CCS.: Apply a style shift preserving causal structure (e.g., day→night); the plan should stay consistent,   CCS = 1 − E d(a, ads ) (↑ better). (11) Unlike generative counterfactual world models [10], these perturbations are query-level edits, making the benchmark reproducible on commodity hardware. C. Test-time Causal Masking (TCM) At inference we suppress the flagged spurious set F = { i : PCRi > θ } and recompute the plan, a† = f\F (Q, e),

(12)

an approximate do(·) that removes spurious confounding at deployment. Optionally, rather than nulling qi , we average over a context prototype dictionary {pc } (a test-timeP analogue 1 of the backdoor adjustment in Eq. (2)), a† = |C| cf Q|
qi ← pc , e . By the hard gate in Eq. (8), F never contains a physically plausible query, so TCM cannot remove a true cause and preserves clean accuracy by construction. TCM adds one (batched) forward pass and requires no parameter update. Algorithm 1 summarizes the pipeline.

MANUSCRIPT DRAFT, JUNE 2026

A. SpurGen: a Physics-Grounded Controlled Benchmark Real driving data offer no ground truth for which dependencies are spurious, so we construct SpurGen, a controlled benchmark in which the causal status of every object is known by design. Each of its 400 scenes is generated in one of three environments (sunny, rain, night) that modulate detection confidence and clutter density, and contains: an in-path closing agent; a cut-in vehicle approaching the lane laterally; one to five off-path static distractors (“trees”); in 40% of scenes an off-path “billboard”; and, in half of the scenes, a static “mailbox” that is usually off-path but occasionally near the lane edge (1781 object queries in total, 255 spurious-reliance instances). A mock planner brakes for the causal hazards (inpath closing agents with low time-to-collision, cut-in vehicles entering the corridor), with two realistic environmentdependent causal behaviors (extra caution for pedestrians at night and for cut-in vehicles in rain), and carries two deliberately injected shortcuts that span the two spurious regimes of Section III-C: a global one (it brakes in the presence of a mailbox, with a per-object strength, in every environment) and a local one (it brakes for billboards only in sunny scenes, a glare-like association). Both shortcut objects are spurious by construction, which yields a label-free gold standard; the cut-in vehicle and the edge-of-lane mailbox provide realistic hard cases. A noisy variant adds perception noise (σ=1.0) to positions, velocities, and confidence. The implementation depends only on NumPy, and the generator is released with the benchmark. Table II summarizes the composition, and Fig. 3 shows why the benchmark is diagnostic. Spatially (Fig. 3(a)), causal agents concentrate in and around the ego corridor while distractors and mailboxes lie outside it. In the influence– physics plane (Fig. 3(b)), the three roles separate: causal agents have both high influence and high ρ (protected region), benign distractors have neither, and the spurious mailboxes form the anomalous populations the audit must isolate: the global mailbox shortcut combines near-zero ρ (mean 0.06) with by far the highest mean influence of any category (0.84), while the billboard’s influence is diluted across environments (mean 0.25, concentrated in sunny scenes). No single axis separates the mailboxes from both other roles, which is the geometric reason single-signal baselines fail in Table III. a) Compared methods.: All compared methods must be training-free, since CADET targets pretrained planners. Causal-intervention approaches such as CausalVAD [7] and Beyond Patterns [9] retrain the planner, and CounterScene [10] requires a generative world model, so none of them can be applied to a frozen planner in our setting; we therefore position them as complementary (Section II) and compare

Category

Count

Role

Mean ρ

Mean I˜

In-path / cut-in vehicle Pedestrian Static distractor (tree) Mailbox (global shortcut) Billboard (sunny-only shortcut)

402 153 851 204 171

causal causal benign spurious spurious

0.78 0.82 0.01 0.06 0.00

0.76 0.75 0.01 0.84 0.25

(a) Spatial layout (ego frame) 40 30

in-path / cut-in vehicle pedestrian static distractor mailbox (global spur.) billboard (env.\ spur.)

20 10 0 10.0 7.5

5.0

2.5 0.0

1.0

normalized influence I

We evaluate CADET in two complementary settings: a controlled synthetic study in which the spurious or causal status of every agent is known by construction (Section V-A), and an audit of a public pretrained planner on real nuScenes data (Section V-B). Both are inference-only and run on a single NVIDIA RTX 2000 Ada GPU (16 GB).

TABLE II S PUR G EN COMPOSITION (400 SCENES ; 134/133/133 ACROSS SUNNY / RAIN / NIGHT ). M EAN NORMALIZED INFLUENCE I˜ AND PHYSICS PRIOR ρ PER CATEGORY: THE SPURIOUS OBJECTS COMBINE NEAR - ZERO ρ WITH NON - TRIVIAL INFLUENCE , THE SIGNATURE CADET IS DESIGNED TO DETECT.

longitudinal distance [m]

V. E XPERIMENTAL S ETUP

5

2.5

lateral offset [m]

5.0

7.5 10.0

(b) Influence vs.\ physics prior flagged

physics-protected

0.8 0.6 0.4 0.2 0.0

0.0

0.2

0.4

0.6

physics prior

0.8

1.0

Fig. 3. The SpurGen benchmark. (a) Spatial layout in the ego frame: causal agents occupy the corridor (shaded), distractors and mailboxes lie off path. (b) Each object in the influence–physics plane with the PCR decision regions: ˜ low ρ), causal spurious mailboxes (orange) fall in the flagged region (high I, agents (green) in the physics-protected region, and benign distractors near the origin. No single axis separates all three roles.

against the training-free signal families they and CADET build on: (i) influence-only, perturbation attribution in the style of occlusion analysis [25], which flags the queries with the largest plan change I˜i > θ; (ii) physics-only, an ablation of our prior that flags every query the physics deems implausible (ρi < ρlo ) regardless of model behavior; and (iii) invarianceonly, an ICP-style invariance test [26], [8] that performs a per-class one-way ANOVA of raw influence across the three environments and flags high-influence queries of classes for which invariance is rejected (p < 0.05). We report flagging precision, recall, and F1 against the construction ground truth. B. Audit of a Pretrained Planner on nuScenes a) Planner and data.: We audit the public pretrained SparseDrive [3] (stage-2 checkpoint, ResNet-50 backbone, 86M parameters) on the nuScenes-mini validation split (81 keyframes). Each frame provides six surround-view camera images (Fig. 4), from which the planner detects agents and predicts an ego trajectory. The planner is run unmodified; CADET performs no training and updates no parameters. b) Query extraction and ablation.: SparseDrive exposes vectorized agent queries; its planning head attends to the top 50 detections by confidence. We audit these agents (those above a confidence of 0.25). The removal operator f\i is realized by zeroing the i-th selected agent’s instance feature inside a cached detection output and re-running only the planning head, while the image backbone, detection, and map outputs are computed once and reused. To keep frames independent and the audit reproducible, the planning head’s temporal queue is reset per frame (single-frame inference). The plan a is

MANUSCRIPT DRAFT, JUNE 2026

6

TABLE III M ETHOD COMPARISON ON S PUR G EN (400 SCENES ACROSS THREE ENVIRONMENTS ; 1781 QUERIES , 255 SPURIOUS BY CONSTRUCTION ). E ACH SINGLE - SIGNAL METHOD FAILS AS ITS ASSUMPTION PREDICTS ; ONLY THE FUSED SCORE ATTAINS BOTH HIGH PRECISION AND HIGH RECALL .

Setting

Method

Precision

Recall

F1

Clean

influence-only [25] physics-only (ablation) invariance-only [26] PCR (ours)

0.351 0.192 0.136 0.942

0.929 0.906 0.196 0.886

0.509 0.317 0.160 0.913

Fig. 4. A nuScenes validation frame: six surround-view camera images that the audited planner consumes. CADET operates on the agent queries the planner detects from these images and on the ego plan it produces.

Noisy (σ=1.0)

influence-only [25] physics-only (ablation) invariance-only [26] PCR (ours)

0.391 0.192 0.151 0.945

0.914 0.867 0.188 0.878

0.548 0.314 0.168 0.911

the model’s selected ego trajectory (final_planning, six future waypoints). c) Physics prior.: For each agent we compute ρi from its ego-frame box (lateral offset, longitudinal distance, velocity), class, and detection confidence (Section IV-A): an agent that is off to the side or behind, and thus cannot physically affect the immediate plan, receives ρi ≈ 0, whereas an in-corridor, ahead or closing agent receives a high ρi . We use a lane halfwidth of 2 m and the gate thresholds ρlo =0.2, ρhi =0.6, and θ=0.5. d) Metrics.: Per agent we record the relative plan change ri under ablation and the score PCRi . We report the peragent influence of causal (ρ ≥ ρhi ) versus spurious (ρ < ρlo ) agents and their ratio; the most-influential spurious agent per frame; the Causal Stability Index (CSI) under ablation of all spurious agents; the number of agents flagged by PCR versus the influence-only baseline; and the open-loop L2 to the ground-truth ego trajectory before and after TCM. e) Implementation.: Inference uses the planner’s official code with forward hooks for the split forward and ablation; PCR, the perturbations, and TCM add no gradient computation. Per-frame runtime is dominated by the agent ablations (one planning-head pass each), and the full audit completes in minutes on the single GPU. We will release the audit toolkit and the exact configuration.

1.0 0.8 0.6 0.4 0.2 0.0

VI. R ESULTS We first validate the audit mechanism in a controlled synthetic setting where the spurious/causal status of every object is known by construction (Section VI-A). Results on pretrained planners and nuScenes follow (Sections VI-C and VI-D). A. Method Comparison on SpurGen Table III compares the four training-free methods of Section V-A on SpurGen (1781 queries, 255 spurious by construction). Each single-signal method fails in the way its underlying assumption predicts. The influence-only baseline [25] attains high recall but only 0.35–0.39 precision, because causal agents (including cut-in vehicles approaching the lane) and the spurious objects both have high model influence and cannot be separated by sensitivity alone. The physics-only ablation suffers the opposite failure: it flags every physically

PCR (ours) influence-only

(b) Test-time Causal Masking

physics-only

invariance-only 0.91

0.55

0.51 0.32

0.31 0.16

clean

0.17

noisy (sigma=1.0)

CSI (higher = robust)

flagging F1

(a) Spurious-flagging quality 0.91

1.0 0.8 0.6 0.4 0.2 0.0

before TCM 0.96 after TCM

0.95

0.53

0.47

clean

noisy (sigma=1.0)

Fig. 5. SpurGen comparison. (a) Flagging F1 of the four training-free methods: influence-only cannot separate causal from spurious reliance, physicsonly over-flags benign objects, and invariance-only rejects invariance only for environment-dependent mechanisms while missing the global shortcut; the fused PCR score is the only method with both high precision and high recall. (b) Test-time Causal Masking raises the Causal Stability Index from about 0.50 to about 0.95, removing nearly all of the planner’s sensitivity to the spurious objects.

implausible object, including the many benign distractors the planner never relies on, and its precision drops below 0.20. The invariance-only test [26], [8] rejects invariance only for environment-dependent mechanisms: it catches the sunny-only billboard shortcut, but it also flags the genuinely causal classes whose behavior is environment-sensitive (rain-cautious cut-ins, night-cautious pedestrians), and it cannot reject invariance for the global mailbox shortcut, so both its precision (0.14) and its recall (0.20) are low. The latter is the faithfulness failure that motivates the external physics anchor. PCR, which requires a query to be both relied upon and physically implausible, reaches 0.94 precision at 0.89 recall (F1 0.91). Its residual errors are informative: it misses spurious objects on which the planner relies only weakly or that sit near the lane edge (where ρ is moderate), and it occasionally mis-flags a slow, distant cut-in that the instantaneous prior cannot yet recognize as a hazard. Under perception noise the scores change only marginally, confirming that the prior is a robust anchor rather than a brittle heuristic. Figure 5(a) visualizes the comparison. B. Training-Free Deconfounding (TCM) On the spurious-containing scenes, the planner’s Causal Stability Index (CSI, higher is more robust) is only 0.469 (clean) and 0.534 (noisy), indicating that perturbing the physically irrelevant objects substantially changes the plan. Applying TCM, which masks the PCR-flagged queries at inference, raises CSI to 0.960 and 0.952 respectively (Fig. 5(b)): the

MANUSCRIPT DRAFT, JUNE 2026

7

TABLE IV CADET AUDIT OF PRETRAINED S PARSE D RIVE ON NU S CENES - MINI VAL (81 FRAMES , 4016 AGENT QUERIES ), INFERENCE - ONLY ON A SINGLE 16 GB GPU. P ER - AGENT INFLUENCE IS THE RELATIVE PLAN CHANGE WHEN ONE AGENT IS ABLATED . Quantity

Value

Per-agent influence, causal agents (ρ ≥ 0.6) Per-agent influence, spurious agents (ρ < 0.2) Causal-to-spurious influence ratio Most-influential spurious agent / frame

0.046 0.005 9.46 0.048

Flagged agents / frame, PCR (ours) Flagged agents / frame, influence-only

1.79 2.48

Causal Stability Index (CSI) Open-loop L2 (base → after TCM) TCM plan shift

0.884 1.90 → 1.98 0.057

deconfounded planner almost entirely stops responding to the spurious object. Because the hard gate (Eq. (8)) rarely flags a physically plausible query, TCM leaves causal agents essentially untouched and preserves clean-scene behavior, at the cost of a single additional forward pass and no training.

C. Audit on a Pretrained Planner (nuScenes) We apply CADET, without any training, to the public pretrained SparseDrive [3] planner on the nuScenes-mini validation split (81 frames, 4016 audited agent queries). For each detected agent we ablate its instance feature inside the planning head and measure the induced change in the predicted ego trajectory (the relative L2 change ri ), and we score its physics plausibility ρi from its position, velocity, class, and detection confidence. Table IV summarizes the audit. Three findings stand out. First, SparseDrive is predominantly causal: a genuinely relevant agent has, on average, 9.5× the per-agent influence of a physically irrelevant one, as expected of a strong planner. Second, spurious reliance is nonetheless real and occasionally severe: the single mostinfluential physically irrelevant agent in a frame shifts the plan by 0.048 on average, matching the influence of an average causal agent (0.046). The planner therefore does, in specific frames, lean on an agent that physics says cannot matter as heavily as on one that does. Third, consistent with the synthetic study, PCR is more selective than the influence-only baseline (1.79 vs. 2.48 flagged agents per frame): the physics hard gate protects high-influence causal agents that the baseline would mislabel as spurious. The audit interface is planner-agnostic: it requires only access to the planner’s agent queries and plan output, so extending the same audit to UniAD [1] and VAD [2] requires only their respective runtime environments, which we leave to an extended study. Figure 6 illustrates this difference on the validation frames where the two methods disagree most. The influence-only baseline crosses out in-corridor agents whose influence is legitimately high, whereas PCR flags only off-corridor agents the planner should not depend on; masking those agents (TCM) leaves the plan close to the baseline trajectory, confirming that the removed reliance was not load-bearing for the maneuver.

TABLE V S IGNAL - FUSION ABLATION ON 2000 LABELLED AGENTS . T HE PHYSICS PRIOR RAISES PRECISION BY GATING CAUSAL AGENTS ; THE CROSS - ENVIRONMENT STABILITY TERM RECOVERS LOCALLY SPURIOUS AGENTS THAT THE PHYSICS GATE ALONE MISSES , AT NO COST TO PRECISION . Configuration

Prec.

Rec.

F1

Rec. glob.

Rec. loc.

Influence only + physics prior + stability (full PCR)

0.660 0.988 0.971

0.936 0.506 0.829

0.774 0.669 0.894

0.942 0.818 0.892

0.930 0.194 0.766

D. TCM and the Blindness of Open-Loop L2 (nuScenes) Applying TCM at inference removes the flagged spurious reliance, shifting the plan by 0.057 on average. This change, however, is not reflected in the open-loop L2 to the groundtruth ego trajectory, which is essentially unchanged (1.90 → 1.98). This is precisely the pathology that motivates our work: open-loop L2 on nuScenes is dominated by ego status [6] and is therefore blind to whether a planner reasons over causal or spurious cues. CADET’s CSI (0.884) and per-agent influence statistics expose a spurious-reliance signal that L2 cannot, empirically confirming the need for causal-robustness metrics alongside displacement error. TCM is a diagnosticdriven intervention: it changes the planner’s causal behavior, and open-loop L2 does not measure causal behavior in the first place. E. Signal-Fusion Ablation To isolate the contribution of each PCR signal independently of any specific planner, we construct four agent populations (500 each) with known labels and controlled signal profiles: causal (high influence, high ρ, low σ), global-spurious (high influence, ρ ≈ 0, low σ), local-spurious (high influence, mid ρ, high σ), and benign (low influence). A local-spurious agent is physically plausible-looking, so the physics gate alone does not catch it, but its influence varies across environments. Table V reports flagging quality as signals are added. The three rows make each signal’s role explicit. Influence alone flags spurious agents but also causal ones, so its precision is only 0.66. Adding the physics prior raises precision to 0.99 by gating out causal agents, and it catches global-spurious agents (recall 0.82), but it misses local-spurious agents (recall 0.19) because they look physically plausible. Adding the cross-environment stability term recovers most of the localspurious agents (recall 0.19 → 0.77) while keeping precision high (0.97). The same precision effect appears on real data, where the physics gate makes PCR flag fewer agents than the influence-only baseline (1.79 vs. 2.48 per frame, Table IV). F. Parameter Sensitivity Figure 7 sweeps the three hyperparameters around their defaults on SpurGen. The flagging threshold θ is the only parameter with a visible effect, and it degrades gracefully: F1 varies from 0.94 at θ=0.3 to 0.81 at θ=0.7, trading precision against recall as expected for a threshold on a calibrated score. The protection gate ρhi leaves F1 essentially unchanged over

MANUSCRIPT DRAFT, JUNE 2026

8

influence-only flags

40

Camera view (CAM\_FRONT)

40

30

30

20

20

10

10

0

0

10 40 20

10

0

10

20

10 40 20

30

30

20

20

10

10

0

0

10 40 20

10

0

10

20

10 40 20

30

30

20

20

10

10

0

0

10

20

10

causal agent (physics-protected)

0

10

20

flagged agent

PCR flags (ours) + TCM

10

plan after TCM 10

0

10

20

plan after TCM 10

0

10

20

plan after TCM 20

10

0

10

20

other agent

Fig. 6. Qualitative comparison on the nuScenes validation frames with the largest disagreement between methods. Left: the camera view. Middle: the influence-only baseline flags (crossed boxes) physically plausible, in-corridor agents alongside genuinely spurious ones. Right: PCR flags only agents outside the drivable corridor (orange) while physics-protected causal agents (green) are preserved; the plan after TCM (dashed) stays close to the baseline plan, removing spurious reliance without disturbing the maneuver.

score

1.0

VII. D ISCUSSION

0.8

A. What the Audit Reveals

0.6

F1 precision recall

0.4 0.3

0.4

0.5

threshold

0.6

0.7

0.4

0.5

0.6

gate hi

0.7

0.8

0.0

0.5

1.0

1.5

stability weight

2.0

Fig. 7. Parameter sensitivity of PCR on SpurGen. F1 degrades gracefully with the flagging threshold θ and is insensitive to the protection gate ρhi ; the stability weight λ only acts when the cross-environment signal is available (Table V).

[0.4, 0.8], because clearly causal agents receive ρ well above the swept range. The stability weight λ has no effect on SpurGen since the per-query stability signal requires repeated cross-environment estimates of the same query, which the single-frame audit does not provide; its contribution is isolated in Table V, where the stability term is active. No parameter requires dataset-specific tuning. A failure mode of the audit is inherited from perception: when the upstream detector is wrong about an agent’s position or velocity, the physics prior is computed from the wrong attributes. We discuss this and other limitations next.

Two observations follow from the nuScenes audit. First, causal confusion in a strong modern planner is real but bounded: SparseDrive relies on genuinely relevant agents far more than on irrelevant ones (a 9.5× per-agent influence ratio), yet in specific frames its single most influential physically irrelevant agent moves the plan as much as an average causal agent. The failure is therefore not pervasive but concentrated, which is exactly the regime an audit, rather than an aggregate accuracy number, is needed to surface. Second, because the physics prior is computed from the perception module’s own outputs and not from training-data statistics, CADET localizes this reliance without any labels and without retraining, making it applicable to planners that are already deployed. B. Open-Loop L2 Does Not Measure Causal Robustness The most actionable finding is methodological. Applying TCM changes the planner’s causal behavior, removing the flagged spurious reliance, yet the open-loop L2 to the groundtruth ego trajectory is unchanged. Since L2 on nuScenes is dominated by ego status [6], it cannot register whether a

MANUSCRIPT DRAFT, JUNE 2026

planner attends to causal or spurious scene cues. We therefore argue that causal-robustness measures such as the per-agent influence statistics and CSI reported here should accompany displacement error when evaluating end-to-end planners, and we release them as a benchmark to that end.

9

deconfounding so that flagged spurious dependencies inform retraining. We release the code, the benchmark, and the audit toolkit to support these directions.

R EFERENCES C. Limitations CADET audits pretrained planners and, by design, does not retrain them; it is complementary to training-time methods such as CausalVAD [7]. The physics prior depends on perception quality and degrades when the upstream detector is wrong. Our perturbations act at the query level, which is reproducible and controllable but less photorealistic than a generative world model [10]; the two are complementary. The nuScenes audit uses a single planner on the mini split in single-frame mode; extending it to more planners, the full split, temporal inference, and closed-loop evaluation is left to future work. Finally, no method can separate true causes from globally spurious correlates with certainty under faithfulness failure; CADET improves reliability through a physics anchor, but does not provide a guarantee. D. Broader Impact Auditing the causal reliance of a deployed planner supports safer deployment and more honest reporting of robustness. The same machinery could in principle be used to identify exploitable shortcuts in a planner; we mitigate this by framing CADET as a defensive diagnostic and by releasing it alongside the benchmark so that weaknesses are measured and repaired rather than hidden. VIII. C ONCLUSION We introduced CADET, a training-free framework that audits, benchmarks, and repairs spurious-correlation reliance in pretrained end-to-end driving planners. Its central idea is to use a physics-geometric prior, computed from the perception module’s own outputs, as an external anchor that is independent of the training distribution and therefore resists the global spurious correlations on which observational signals fail. From this prior we derived a per-query audit score (PCR), a counterfactual robustness benchmark (CSI, CRI, CCS), and a test-time deconfounding operation (TCM), all of which require only forward passes and run on a single 16 GB GPU. In a controlled study PCR separates spurious from causal reliance with a precision of about 0.96 where an influenceonly baseline reaches only about 0.35, and TCM restores stability to the planner. On nuScenes, CADET shows that a strong pretrained planner is predominantly causal yet exhibits measurable, occasionally severe spurious reliance, flags it more selectively than the baseline, and removes it with TCM, all while open-loop L2 remains unchanged, empirically confirming that displacement error is blind to causal robustness. Future work includes auditing additional planners and the full nuScenes split, extending the audit to temporal and closedloop settings, combining query-level perturbations with generative counterfactuals, and pairing the audit with training-time

[1] Y. Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, L. Lu, X. Jia, Q. Liu, J. Dai, Y. Qiao, and H. Li, “Planningoriented autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 17 853–17 862. [2] B. Jiang, S. Chen, Q. Xu, B. Liao, J. Chen, H. Zhou, Q. Zhang, W. Liu, C. Huang, and X. Wang, “VAD: Vectorized scene representation for efficient autonomous driving,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 8340– 8350. [3] W. Sun, X. Lin, Y. Shi, C. Zhang, H. Wu, and S. Zheng, “SparseDrive: End-to-end autonomous driving via sparse scene representation,” arXiv preprint arXiv:2405.19620, 2024. [4] X. Weng, B. Ivanovic, Y. Wang, Y. Wang, and M. Pavone, “PARADrive: Parallelized architecture for real-time autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 15 449–15 458. [5] P. de Haan, D. Jayaraman, and S. Levine, “Causal confusion in imitation learning,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 32, 2019. [6] Z. Li, Z. Yu, S. Lan, J. Li, J. Kautz, T. Lu, and J. M. Alvarez, “Is ego status all you need for open-loop end-to-end autonomous driving?” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 14 864–14 873. [7] J. Tang, Z. Zhou, Z. He, J. Zhang, K. Zhang, and J. Pu, “CausalVAD: De-confounding end-to-end autonomous driving via causal intervention,” arXiv preprint arXiv:2603.18561, 2026. [8] M. Arjovsky, L. Bottou, I. Gulrajani, and D. Lopez-Paz, “Invariant risk minimization,” arXiv preprint arXiv:1907.02893, 2019. [9] B. Wang, H. Liao, C. Wang, B. Rao, Y. Guan, G. Yu, J. Zhang, S. Lai, C. Xu, and Z. Li, “Beyond patterns: Harnessing causal logic for autonomous driving trajectory prediction,” in Proceedings of the ThirtyFourth International Joint Conference on Artificial Intelligence (IJCAI), 2025. [10] Anonymous et al., “CounterScene: Counterfactual causal reasoning in generative world models for safety-critical closed-loop evaluation,” arXiv preprint arXiv:2603.21104, 2026, tODO(Scholar export): confirm author list. [11] Z. Guo, A. P. Adedigba, and R. Mallipeddi, “Cluster-aggregated transformer: Enhancing lightweight parameter models,” Engineering Applications of Artificial Intelligence, vol. 159, p. 111468, 2025. [12] Z. Guo, S. Kavuri, J. Lee, and M. Lee, “IDS-Extract: Downsizing deep learning model for question and answering,” in 2023 International Conference on Electronics, Information, and Communication (ICEIC). IEEE, 2023, pp. 1–5. [13] Z. Guo, A. P. Adedigba, and R. Mallipeddi, “Cluster aggregated GAN (CAG): A cluster-based hybrid model for appliance pattern generation,” arXiv preprint arXiv:2512.22287, 2025. [14] Z. Fan, Z. Guo, Y. Lai, and J. Kim, “TSDCA-BA: An ultra-lightweight speech enhancement model for real-time hearing aids with multi-scale STFT fusion,” Applied Sciences, vol. 15, no. 15, p. 8183, 2025. [15] Z. Guo, X. Yu, S. Wang, and R. Mallipeddi, “Visual recognition of crop composite planting based on vision transformer,” in International Conference on Machine Learning, IoT and Big Data. Springer, 2025, pp. 296–306. [16] Z. Guo, A. P. Adedigba, R. Mallipeddi, and H. Lee, “Dynamic tanh reinforcement learning: A normalization-free transformer for open traveling salesman problem optimization,” in Proceedings of the Annual Conference of the Institute of Control, Robotics and Systems (ICROS), 2025, pp. 845–846. [17] Z. Guo, R. Mallipeddi, and H. Lee, “Cooperative coevolutionary genetic algorithm for multirobot task scheduling in Antarctica region,” Swarm and Evolutionary Computation, p. 102199, 2025. [18] A. Adedigba, Z. Guo, R. Mallipeddi, and H. Lee, “iVec clustering: A new task allocation algorithm for multirobot task scheduling in antarctic environment,” in Proceedings of the Annual Conference of the Institute of Control, Robotics and Systems (ICROS), 2025, pp. 853–854.

MANUSCRIPT DRAFT, JUNE 2026

[19] X. Jia, Z. Yang, Q. Li, Z. Zhang, and J. Yan, “Bench2Drive: Towards multi-ability benchmarking of closed-loop end-to-end autonomous driving,” in Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track, 2024. [20] D. Dauner, M. Hallgarten, T. Li, X. Weng, Z. Huang, Z. Yang, H. Li, I. Gilitschenski, B. Ivanovic, M. Pavone, A. Geiger, and K. Chitta, “NAVSIM: Data-driven non-reactive autonomous vehicle simulation and benchmarking,” in Advances in Neural Information Processing Systems (NeurIPS), 2024. [21] X. Han, Y. Zhu, L. Wang, and Z. Guo, “Enhancing the understanding of urban street perception with LLMs and street view imagery,” Transactions in GIS, vol. 30, no. 3, p. e70280, 2026. [22] Z. Guo, A. P. Adedigba, R. Mallipeddi, and H. Lee, “Structural induced exploration for balanced and scalable multi-robot path planning,” arXiv preprint arXiv:2512.21654, 2025. [23] W. Zhou, S. Yang, Q. Yang, Z. Guo, L. Hu, and D. Wang, “Flattery in motion: Benchmarking and analyzing sycophancy in video-LLMs,” arXiv preprint arXiv:2506.07180, 2025. [24] J. Xu, Z. Guo, J. Lv, H. Lin, S. Yang, J. Wen, D. Wang, and L. Hu, “Benchmarking and mitigating sycophancy in medical vision language models,” arXiv preprint arXiv:2509.21979, 2025. [25] M. D. Zeiler and R. Fergus, “Visualizing and understanding convolutional networks,” in European Conference on Computer Vision (ECCV), 2014, pp. 818–833. [26] J. Peters, P. Bühlmann, and N. Meinshausen, “Causal inference by using invariant prediction: Identification and confidence intervals,” Journal of the Royal Statistical Society: Series B, vol. 78, no. 5, pp. 947–1012, 2016.

10