UniDrive: A Unified Vision-Language and Grounding Framework for Interpretable Risk Understanding in Autonomous Driving Xiaowei Gaoa,b , Pengxiang Lic , Yitai Chengb , Ruihan Xud , James Haworthb , Stephen Lawe and Yun Yef,b,∗ a Department of Earth Science & Engineering, Imperial College London, London, SW7 2AZ, United Kingdom
arXiv:2606.24759v1 [cs.CV] 23 Jun 2026
b SpaceTimeLab, Department of Civil, Environmental, and Geomatic Engineering, University College London, London, WC1E 6BT, United
Kingdom c Department of Computing, The Hong Kong Polytechnic University, Hong Kong, China d Trinity College, University of Oxford, Oxford, OX1 3BH, United Kingdom e Department of Geography, University College London, London, WC1E 6BT, United Kingdom f Centre for Global Infrastructure Resilience, The Bartlett School of Sustainable Construction, University College London, London, WC1E 7HB, United Kingdom
ARTICLE INFO
ABSTRACT
Keywords: Autonomous driving Multimodal large language model Vision-language model Risk understanding Visual grounding Interpretable artificial intelligence
Recent multimodal large language models (MLLMs) have shown strong potential for autonomous driving scene understanding, yet existing methods still face a fundamental tradeoff between temporal reasoning and spatial precision. Models that rely on single-frame or low-resolution inputs often miss small, distant, or partially occluded hazards, while languagecentric driving models frequently provide limited grounded evidence for their explanations. To address this gap, we propose UniDrive, a unified visual-language and grounding framework for interpretable risk understanding in autonomous driving. UniDrive combines a temporal reasoning branch that models scene dynamics from multi-frame visual input with a highresolution perception branch that preserves fine-grained spatial details from the latest frame. The two branches are integrated through a gated cross-attention fusion module, enabling dynamic context to be aligned with precise spatial evidence. Based on the fused representation, UniDrive jointly generates natural-language risk descriptions and grounded bounding-box outputs for risk objects. Experiments on the DRAMA-Reasoning benchmark show that UniDrive outperforms representative image-based and video-based baselines in both captioning and risk-object grounding. In particular, UniDrive achieves the best overall performance on the validation split and demonstrates clear advantages in small-object localization, zero-shot generalization to NuScenes and BDD100K, and human-rated interpretability and trustworthiness. These results suggest that explicitly combining temporal semantics and high-resolution perception provides a stronger foundation for interpretable and safety-oriented autonomous driving systems. The code is available at https://github.com/pixeli99/unidrive-dev.
1. Introduction Autonomous driving systems are increasingly expected to operate in complex, dynamic, and safety-critical traffic environments (Che et al., 2026; Ye et al., 2026a,b). In such environments, safe driving requires more than detecting surrounding objects or producing low-level control commands. A reliable autonomous system should be able to understand which object or agent creates risk, how the risk evolves over time, where the risk-critical object is located, and why the situation requires a cautious response. This capability is particularly important in long-tail driving scenarios involving subtle hazards, distant objects, partial occlusions, vulnerable road users, uncertain intentions, and rapidly evolving interactions (Chen et al., 2024). In these cases, a system that only outputs an opaque driving decision or a coarse scene description may be insufficient for safety validation, post-hoc auditing, and communication with human users. Therefore, interpretable and visually grounded risk understanding has become a critical capability for practical autonomous driving systems (Malla et al., 2023; Zhou et al., 2024). Recent progress in vision-language models (VLMs) and multimodal large language models (MLLMs) provides a promising foundation for addressing this challenge. General-purpose multimodal models have demonstrated strong ∗ Corresponding author
[email protected] (Y. Ye)
ORCID (s): 0000-0002-3346-4640 (Y. Ye)
Gao et al.: Preprint submitted to Elsevier
Page 1 of 19
UniDrive for Interpretable Risk Understanding
capabilities in visual reasoning, instruction following, visual grounding, and video understanding (Alayrac et al., 2022; Liu et al., 2024; Peng et al., 2023; Zhang et al., 2023). Motivated by these advances, driving-oriented MLLMs have extended language-enhanced perception to risk localization, scene captioning, visual question answering, highresolution understanding, and end-to-end driving assistance (Malla et al., 2023; Xu et al., 2024; Ding et al., 2023; Sima et al., 2024; Tian et al., 2024; Hwang et al., 2024). These studies suggest that language-enhanced visual understanding can improve the transparency, generalization, and human interpretability of autonomous driving systems. Nevertheless, existing driving-oriented MLLM methods still face an important limitation for safety-critical risk understanding as they often struggle to jointly capture temporal risk evolution and fine-grained spatial evidence. Models based on single-frame reasoning or relatively low-resolution visual inputs can generate plausible descriptions of traffic scenes, but they may miss small, distant, or partially occluded risk objects that are critical for safe decision-making (Ding et al., 2023; Xu et al., 2024). Video-based models are better suited to capturing dynamic interactions and temporal context, yet their spatial grounding can remain coarse when risk objects occupy only a small region of the image (Zhang et al., 2023). Conversely, approaches that emphasize high-level language interaction, question answering, or driving interfaces may improve reasoning and communication, but they often provide limited support for fine-grained visual grounding of risk-critical targets (Sima et al., 2024; Tian et al., 2024; Wang et al., 2023). As a result, current models may produce reasonable language explanations without precise visual evidence, or localize objects without adequately explaining their dynamic risk relevance. This limitation is especially problematic because driving risk understanding is inherently both temporal and localized. A pedestrian stepping out from behind a parked vehicle, a cyclist gradually entering the ego lane, or a distant traffic participant with subtle motion cues requires the model to integrate temporal semantics with high-resolution spatial details. For safety-oriented interpretation, these two forms of information should not be treated as separate outputs. Instead, the model should align dynamic context with explicit visual evidence and generate explanations that are both semantically meaningful and spatially grounded. This requirement is closely aligned with risk-centric benchmarks such as DRAMA, where risk explanation and object-level grounding are both essential for understanding interactive driving scenarios (Malla et al., 2023). To address this gap, we propose UniDrive, a unified visual-language and grounding framework for interpretable risk understanding in autonomous driving. UniDrive combines a temporal reasoning branch that models scene dynamics from multi-frame visual input with a high-resolution perception branch that preserves fine-grained spatial details from the latest frame. The two branches are integrated through a gated cross-attention fusion module, enabling the model to dynamically align temporal context with precise spatial evidence. Based on the fused representation, UniDrive jointly generates natural-language risk descriptions and grounded bounding-box outputs for risk objects. In this way, the model not only identifies hazardous agents, but also explains the underlying reason and links that explanation to explicit visual evidence. We evaluate UniDrive on an extended risk-reasoning setting built upon DRAMA (Malla et al., 2023). The original DRAMA dataset provides risk-object annotations for interactive driving scenarios, while our extended DRAMA-Reasoning setting enriches these annotations with textual descriptions of hazardous objects, ego-vehicle intentions, and safe driving suggestions. This setting enables joint evaluation of two safety-relevant capabilities: interpretable risk captioning and object-level visual grounding. Experiments show that UniDrive consistently improves both language-based risk explanation and grounded risk-object localization compared with representative imagebased and video-based baselines. Additional evaluations on unseen driving datasets further examine the model’s zero-shot generalization, robustness under challenging conditions, and human-perceived usefulness, accuracy, and trustworthiness. The main contributions of this paper are summarized as follows: • We propose UniDrive, a unified visual-language and grounding framework for interpretable risk understanding in autonomous driving. Unlike models that only generate scene-level descriptions or ungrounded explanations, UniDrive jointly produces natural-language risk descriptions and bounding-box evidence for risk-critical objects. • We introduce a dual-branch architecture that combines temporal reasoning from multi-frame visual inputs with high-resolution spatial perception from the latest frame. A gated cross-attention fusion module is designed to align dynamic scene context with fine-grained visual evidence, thereby improving both temporal risk interpretation and object-level grounding.
Gao et al.: Preprint submitted to Elsevier
Page 2 of 19
UniDrive for Interpretable Risk Understanding
• We develop an extended DRAMA-Reasoning experimental setting by enriching risk-object annotations with explanatory descriptions, ego-vehicle intentions, and safe-action suggestions, enabling joint evaluation of risk explanation and visual grounding. • We conduct comprehensive experiments on the DRAMA-Reasoning benchmark and unseen driving datasets, including comparisons with representative MLLM baselines, zero-shot generalization tests, ablation studies, robustness analysis, efficiency analysis, qualitative analysis, and human-centered evaluation. The results demonstrate that explicitly combining temporal semantics and high-resolution spatial evidence provides a stronger foundation for interpretable and safety-oriented autonomous driving. The remainder of this paper is organized as follows. Section 2 reviews related work on multimodal reasoning and grounding, MLLMs for autonomous driving risk understanding, and benchmark-oriented safety evaluation. Section 3 presents the proposed UniDrive framework, including the temporal reasoning branch, the high-resolution perception branch, the spatio-temporal fusion module, and the unified reasoning and grounding mechanism. Section 4 reports the experimental setup and evaluation results, including comparisons with representative baselines, zero-shot generalization, ablation studies, human-centered evaluation, robustness analysis, efficiency analysis, and qualitative results. Section 5 discusses the main findings, underlying mechanisms, safety implications, limitations, and future research directions. Section 6 concludes the paper.
2. Related Work 2.1. Multimodal Reasoning and Visual Grounding Recent advances in VLMs and MLLMs have substantially improved the ability of artificial intelligence systems to connect visual perception with language-based reasoning. Representative models such as Flamingo demonstrated that large language models can be conditioned on visual inputs through cross-attention, enabling few-shot visual-language understanding across diverse tasks (Alayrac et al., 2022). LLaVA and its improved variants further showed that visual instruction tuning can produce strong multimodal conversational and reasoning abilities (Liu et al., 2024). Beyond general visual dialogue, grounded multimodal generation has received increasing attention. Kosmos-2 links language outputs to visual regions for grounded generation (Peng et al., 2023), GroundHog strengthens the coupling between language generation and dense visual understanding (Zhang et al., 2024), and SpatialRGPT highlights the importance of region-level geometry and spatial relationships for spatial reasoning (Cheng et al., 2024). Meanwhile, Video-LLaMA extends multimodal language modeling from static images to temporally structured video inputs, showing the potential of MLLMs for dynamic scene understanding (Zhang et al., 2023). These studies provide important foundations for language-enhanced perception, visual grounding, and temporal reasoning. However, general-purpose MLLMs are not specifically designed for safety-critical autonomous driving scenarios. Driving risk understanding requires more than generic object recognition, image captioning, or visual dialogue. A model should identify which object creates risk, explain why it is hazardous, and ground this explanation in the visual scene. It should also account for temporal cues such as motion, interaction, and intention changes. Therefore, although existing MLLMs establish the feasibility of multimodal reasoning and grounding, their direct application to autonomous driving remains limited without mechanisms that explicitly integrate temporal risk evolution with finegrained spatial evidence.
2.2. MLLMs for Autonomous Driving Risk Understanding Driven by the progress of general-purpose VLMs and MLLMs, recent studies have adapted multimodal language models to autonomous driving. These works extend language-enhanced perception to tasks such as scene captioning, visual question answering, risk localization, planning-oriented reasoning, and end-to-end driving assistance. DRAMA introduced a benchmark for joint risk localization and captioning, emphasizing that risk understanding in driving scenes should couple natural-language explanation with object-level localization (Malla et al., 2023). DriveGPT4 explored interpretable end-to-end autonomous driving by generating language outputs from driving videos (Xu et al., 2024), while HiLM-D highlighted the value of high-resolution visual inputs for detailed driving scene understanding (Ding et al., 2023). DriveLM formulated driving understanding as graph-based visual question answering, and DriveVLM investigated the broader convergence between autonomous driving and large vision-language models (Sima et al., 2024; Tian et al., 2024). More recent systems, such as EMMA and BEV-injected multimodal driving models, further Gao et al.: Preprint submitted to Elsevier
Page 3 of 19
UniDrive for Interpretable Risk Understanding
extend MLLMs toward end-to-end driving and bird’s-eye-view enhanced understanding (Hwang et al., 2024; Ding et al., 2024). Another related direction uses large language models as language-centric agents for decision-oriented driving. A Language Agent for Autonomous Driving models driving as a cognitive process involving perception, memory, reasoning, and action selection (Mao et al., 2023). LMDrive, Drive like a Human, and DriveMLM incorporate language or planning-oriented supervision into autonomous driving pipelines (Shao et al., 2024; Fu et al., 2024; Wang et al., 2023), while KOMA explores knowledge-driven multi-agent coordination through large language models (Jiang et al., 2024). These methods are valuable for planning, control, and interaction, but their primary objective is often decision generation rather than risk-focused visual grounding. In safety-critical perception, an explanation is incomplete if it recommends an action without identifying the responsible risk object, its location, and the temporal context in which the risk emerges. Despite these advances, existing driving-oriented MLLMs still leave an important gap for risk-centric autonomous driving. Many methods emphasize high-level semantic understanding, question answering, or planning interfaces, while the joint modeling of temporal hazard evolution and fine-grained grounded localization remains insufficiently explored. Video-based models can capture scene dynamics, but their spatial evidence may remain coarse when the risk object is small, distant, or partially occluded. High-resolution models improve visual detail, but they often provide limited reasoning about how a hazard emerges and evolves over time. In contrast, UniDrive treats driving risk understanding as a joint reasoning-grounding problem. It explicitly aligns temporal context with high-resolution spatial cues to generate both natural-language risk explanations and grounded bounding-box outputs.
2.3. Benchmarks, Safety Evaluation, and Research Gap The development of driving-oriented benchmarks and safety evaluation frameworks has accelerated research on language-enhanced autonomous driving. Existing surveys have summarized the growing landscape of MLLMs and VLMs for autonomous driving, covering perception, planning, grounding, interaction, and embodied decision-making tasks (Cui et al., 2024; Zhou et al., 2024; Yang et al., 2023; Li et al., 2025). Benchmark-oriented studies have also begun to evaluate how language models support autonomous driving under structured task settings. LaMPilot provides an open benchmark for autonomous driving with language model programs (Ma et al., 2024), while OmniTester investigates language-driven scenario testing and evaluation pipelines for autonomous vehicles (Lu et al., 2025). From a safety perspective, DriveSOTIF explores the use of multimodal large models for safety-oriented perception under the SOTIF framework, highlighting the relevance of MLLMs to perception-related safety risks (Huang et al., 2025). However, many existing evaluations still focus on scene-level question answering, planning responses, languageprogram execution, or general perception performance. These settings are valuable, but they do not fully assess whether a model can provide safety-relevant explanations that are visually verifiable. For autonomous driving, a plausible language response is not sufficient if it is not grounded in the correct risk object. A safety-oriented MLLM should explain why a specific object is risky, localize that object precisely, and remain reliable under challenging conditions such as occlusion, low visibility, small-object hazards, and distribution shifts. This paper addresses this evaluation gap by formulating autonomous driving risk understanding as a joint reasoning-grounding problem. Building on DRAMA (Malla et al., 2023), we develop an extended DRAMA-Reasoning setting that enriches risk-object annotations with explanatory descriptions, ego-vehicle intentions, and safe-action suggestions. This setting allows us to evaluate whether model outputs are not only linguistically plausible, but also grounded in the correct risk objects. Together with zero-shot transfer, robustness analysis, and human-centered evaluation, our experiments examine the reliability of UniDrive under safety-relevant conditions.
3. Methods 3.1. Overview This section presents our UniDrive approach, a unified, end-to-end model designed to perceive, reason, and act in complex driving scenarios using multi-view video inputs. As illustrated in Figure 1, our architecture is composed of two synergistic components: (a) a Temporal Reasoning Branch (T-RB) that analyzes video semantics to produce captions about risk objects, their causes, and the intended actions of the ego-car; and (b) a High-Resolution Perception Branch (P-B) that extracts fine-grained feature maps from high-resolution (HR) images to enhance perceptual accuracy, especially for small or distant objects. A core fusion module integrates these two information streams before they are processed by a large language model to generate the final output. Gao et al.: Preprint submitted to Elsevier
Page 4 of 19
UniDrive for Interpretable Risk Understanding
Input video frames: Human question: Which object is at the highest risk? Visual Encoder
Self Attn
...
❄️
evirDinU
Gated Cross Attn
🔥
Projector
Concat
Projector
🔥
Visual Encoder
❄️
High Res
Self Attn
...
Gated Cross Attn
...
Self Attn Answers: The ego car is driving on a snow-covered road, with
many parked vehicles around. A black sedan is parked to the right. Due to the slippery road, please slow down and watch for vehicles that may start up.
Figure 1: The UniDrive Architecture. Our model processes multi-view video inputs through two main pathways: a Temporal Reasoning Branch (T-RB) for semantic understanding of dynamics and a High-Resolution Perception Branch (P-B) for detailed spatial feature extraction from the most recent frame. A Spatio-Temporal Fusion module, using gated crossattention, integrates these streams. This allows the Large Language Model (LLM) to generate both descriptive captions (identifying risks, causes, and ego-car actions) and precise, grounded bounding box predictions for risk objects in an end-to-end manner.
3.2. Temporal Reasoning Branch (T-RB) The primary function of the T-RB is to process a sequence of low-resolution video frames to understand the temporal dynamics of a scene. It generates a high-level understanding of the situation, including identifying potential risks and inferring intentions. As depicted in Figure 1, the T-RB consists of a vision encoder and a language reasoning model.
3.2.1. Visual Encoder The visual encoder transforms raw video inputs into a sequence of visual tokens that the LLM can interpret. We build this encoder upon a CLIP-pretrained ViT-L/14, which remains frozen during training to preserve its strong generalization capabilities. Formally, for a given video clip 𝑉 = {𝑣1 , 𝑣2 , … , 𝑣𝐿 } with 𝐿 frames, the ViT maps each frame 𝑣𝑖 to its 𝑘-th layer feature representation, 𝑓𝑖𝑘 . This results in a set of features for the entire clip: 𝐹𝑘 = {𝑓1𝑘 , 𝑓2𝑘 , … , 𝑓𝐿𝑘 },
where
𝑓𝑖𝑘 ∈ ℝ𝑁𝑓 ×𝐷𝑓
(1)
with 𝑁𝑓 being the number of patches (tokens) per frame and 𝐷𝑓 the feature dimension. Before these features are passed to the LLM, a lightweight feature aggregator pools them along the temporal axis to create a compact representation: 𝐹 aggr = Aggregator(𝐹𝑘 )
(2)
This stage serves two critical purposes: (1) it leverages temporal prior knowledge to create a holistic video representation, and (2) it significantly reduces the number of visual tokens, ensuring the total sequence length does not exceed the LLM’s context window limit.
Gao et al.: Preprint submitted to Elsevier
Page 5 of 19
UniDrive for Interpretable Risk Understanding
3.2.2. Language Reasoning Model (LRM) We adopt a Flamingo-style architecture, which employs alternating layers of Self-Attention and Gated CrossAttention. This design is particularly adept at handling both textual and visual inputs. The Gated Cross-Attention layers allow the LLM to efficiently condition its text generation on the visual tokens 𝐹 aggr provided by the encoder. Given a textual prompt 𝑇 , the model generates a descriptive caption 𝑌 = {𝑦1 , 𝑦2 , … , 𝑦𝑂 } by maximizing the conditional probability, modeled autoregressively: 𝑃 (𝑌 |𝑇 , 𝐹 aggr ) =
𝑂 ∏
𝑃 (𝑦𝑖 |𝑦<𝑖 , 𝑇 , 𝐹 aggr )
(3)
𝑖=1
By leveraging the extensive world knowledge of the pre-trained LLM, we prompt the model to generate captions that include risk identification, causal explanations, and actionable suggestions.
3.3. High-Resolution Perception Branch (P-B) To overcome the limitations of low-resolution video inputs, which can obscure small but critical details (e.g., distant pedestrians or debris), we introduce a parallel perception branch. The P-B is designed to process a single high, typically the most recent frame in the video sequence, to capture fine-grained spatial information. resolution image 𝑣HR 𝐿 This branch utilizes a separate, dedicated vision encoder, ΦP-B , for which we also employ a Vision Transformer (ViT) architecture, chosen for its strong feature extraction capabilities on static, high-resolution images. This encoder extracts a high-fidelity feature map 𝐹 HR : 𝐹 HR = ΦP-B (𝑣HR 𝐿 ),
where
𝐹 HR ∈ ℝ𝑁HR ×𝐷HR
(4)
Crucially, these high-resolution features are not directly consumed by the LLM’s main body but are instead injected into the model via the spatio-temporal fusion module, providing precise spatial cues to ground the temporal reasoning.
3.4. Spatio-Temporal Fusion The synergy between the T-RB and P-B is unlocked by our spatio-temporal fusion mechanism. Instead of naively concatenating features, we employ a more sophisticated strategy using Gated Cross-Attention layers. In this setup, the aggregated temporal features 𝐹 aggr from the T-RB act as the queries (𝑄), while the high-resolution spatial features 𝐹 HR from the P-B serve as the keys (𝐾) and values (𝑉 ), after being passed through linear projections. Temporal features encode what dynamic risk cues to look for, so placing them as queries lets the model use motion and interaction context as a guide to selectively retrieve the most safety-relevant fine-grained details from the high-resolution spatial map. The core of the fusion is a standard cross-attention mechanism: ) ( 𝑄𝐾 𝑇 Attention(𝑄, 𝐾, 𝑉 ) = softmax √ 𝑉 (5) 𝑑𝑘 where 𝑑𝑘 is the dimension of the keys. This is integrated with a gating mechanism that dynamically balances the attended high-resolution features with the original temporal features. The final fused representation 𝐹 fused is computed as: 𝐹 fused = 𝛼 ⋅ Attention(𝑄, 𝐾, 𝑉 ) + (1 − 𝛼) ⋅ 𝑄
(6)
where 𝛼 is a learned, element-wise gating parameter. This allows the model to learn a dynamic alignment, using the temporal context (the query) to selectively attend to the most relevant high-resolution spatial details (the key-value pairs). The resulting representation is thereby enriched with both temporal context and precise spatial grounding.
3.5. Unified Reasoning and Generation
The fused spatio-temporal representation 𝐹 fused serves as the rich, contextual input for our core reasoning module (i.e., a LLM). Following the Flamingo architecture (Alayrac et al., 2022), we keep the pre-trained LM blocks frozen and interleave newly initialized Gated Cross-Attention layers at every 𝑘-th transformer block. This design is particularly adept at handling both textual and visual inputs. The Gated Cross-Attention layers allow the LLM to efficiently condition its text generation on the fused visual tokens 𝐹 fused . Gao et al.: Preprint submitted to Elsevier
Page 6 of 19
UniDrive for Interpretable Risk Understanding
Given a textual prompt 𝑇 , the model generates a descriptive caption 𝑌 = {𝑦1 , 𝑦2 , … , 𝑦𝑂 } by maximizing the conditional probability, modeled autoregressively: 𝑃 (𝑌 |𝑇 , 𝐹 fused ) =
𝑂 ∏
𝑃 (𝑦𝑖 |𝑦<𝑖 , 𝑇 , 𝐹 fused )
(7)
𝑖=1
By leveraging the extensive world knowledge of the pre-trained LLM (e.g., Llama2-7B), we prompt the model to generate captions that include risk identification, causal explanations, and actionable suggestions.
Figure 2: Detailed architecture of the Spatio-Temporal Fusion Module. Temporal features from multiple frames are aggregated and serve as queries (Q), while high-resolution spatial features are encoded to generate keys (K) and values (V) for the gated cross-attention mechanism. The gating parameter dynamically balances the contribution of attention-weighted features and residual temporal information.
3.6. Grounding Output To seamlessly integrate object localization into the generative framework, we format bounding box coordinates as plain text. Specifically, we use special delimiters to represent a bounding box in the format <box>x1, y1, x2, y2</box>. This location information is directly appended to the end of the generated caption. This elegant approach avoids the need for auxiliary regression heads or separate detection decoders, allowing the entire model to be trained end-to-end with a standard language modeling objective and simplifying the overall architecture.
3.7. Data and Training Our training process is straightforward, consisting of a single supervised fine-tuning (SFT) stage on a mixture of driving-related reasoning datasets. We start from the pre-trained weights of the vision encoders and the LLM, and we do not use reinforcement learning from human feedback (RLHF). The primary training corpus is our newly curated DRAMA-Reasoning dataset. We build upon the original DRAMA dataset, which provides risk object annotations for 17,785 interactive driving scenarios. We extend these annotations by manually adding detailed textual descriptions of the ego-vehicle’s intentions and recommended safe driving actions for each scenario. During the SFT stage, all model parameters are updated. To account for the different natures of the two branches, we set distinct initial learning rates, whereas 1 × 10−4 for the temporal reasoning branch to carefully fine-tune the pre-trained weights, and a higher learning rate of 4 × 10−4 for the perception branch components to allow them to adapt more quickly. Gao et al.: Preprint submitted to Elsevier
Page 7 of 19
UniDrive for Interpretable Risk Understanding
Algorithm 1 UniDrive: Unified Forward Pass for Perception and Reasoning Require: Video Clip = {𝑣𝑡 }𝐿 , where 𝑣𝑡 ∈ ℝ𝐻×𝑊 ×3 ; High-resolution final frame 𝑣HR ∈ ℝ𝐻 ×𝑊 ×3 ; Textual 𝐿 𝑡=1 𝑀 Prompt = {𝑝𝑖 }𝑖=1 . ′
′
′
′
′
Require: Temporal Encoder ΦTemp ∶ ℝ𝐻×𝑊 ×3 → ℝ𝑁𝑝 ×𝐷𝑓 ; Spatial Encoder ΦSpat ∶ ℝ𝐻 ×𝑊 ×3 → ℝ𝑁𝑝 ×𝐷ℎ ; Feature Aggregator ∶ {ℝ𝑁𝑝 ×𝐷𝑓 }𝐿 → ℝ𝑁𝑞 ×𝐷𝑓 ; Gated Cross-Attention Fusion Operator fuse ; Multimodal Large 𝑡=1 Language Model LLM . Ensure: Generated textual output sequence . 1: procedure U NID RIVE _F ORWARD ((), 𝑣HR , ) 𝐿
2: 3: 4: 5: 6: 7: 8: 9: 10: 11:
⊳ Phase 1: Dual-Stream Asynchronous Feature Extraction ⊳ Temporal Reasoning Branch (T-RB) processing ⨁ Let the set of frame-wise feature maps be 𝐅frames = 𝐿 Φ 𝑡=1 Temp (𝑣𝑡 ). Derive the temporally aggregated query representation: 𝑸 ← (𝐅frames ) ∈ ℝ𝑁𝑞 ×𝐷𝑓 . ⊳ High-Resolution Perception Branch (P-B) processing ′ Extract fine-grained spatial features from the final high-resolution frame: 𝐅HR ← ΦSpat (𝑣HR ) ∈ ℝ𝑁𝑝 ×𝐷ℎ . 𝐿 ⊳ Phase 2: Spatio-Temporal Manifold Fusion via Gated Attention ⊳ Project features into Key-Value space for contextual grounding Define projection matrices 𝑊𝐾 ∈ ℝ𝐷ℎ ×𝑑𝑘 , 𝑊𝑉 ∈ ℝ𝐷ℎ ×𝑑𝑣 . 𝑲 ← 𝐅HR 𝑊𝐾 ; 𝑽 ← 𝐅HR 𝑊𝑉 . ⊳ Compute the attention-weighted high-resolution context ) ( 𝐀context ← softmax
𝑸𝑲 ⊤ √ 𝑑𝑘
𝑽 ∈ ℝ𝑁𝑞 ×𝑑𝑣 .
⊳ Dynamically blend temporal and spatial information via a gating mechanism Let 𝜎 be the sigmoid function and 𝑊𝛼 be learnable gating weights. 𝜶 ← 𝜎(𝑊𝛼 [𝑸; 𝐀context ]), where [⋅; ⋅] denotes concatenation. The final fused representation is computed as a convex combination: 𝐅fused ← 𝜶 ⊙ 𝐀context + (1 − 𝜶) ⊙ (𝑸𝑊𝑄 ), where 𝑊𝑄 projects 𝑸 to dimension 𝑑𝑣 . ⊳ Phase 3: Autoregressive Generation Conditioned on Fused Multimodal Context 16: Initialize output sequence 0 = ∅ and initial hidden state ℎ0 . 17: Tokenize and embed prompt: 𝑬 = Embed(). 18: for 𝑘 = 1, … , 𝐾max do 19: ⊳ Update model state and predict next token distribution 20: 𝑃 (𝑘 |<𝑘 , 𝐅fused , 𝑬 ) = LLM (ℎ𝑘−1 , [𝐅fused ; Embed(𝑘−1 )], 𝑬 ). 21: ⊳ Select the next token via a decoding strategy (e.g., nucleus sampling) 22: 𝑦𝑘 ∼ 𝑃 (𝑘 ). 23: if 𝑦𝑘 = ⟨eos⟩ then 24: break 25: end if 26: 𝑘 ← 𝑘−1 ⊕ 𝑦𝑘 . ⊳ ⊕ denotes sequence concatenation 27: end for 28: ← Detokenize(𝐾 ). 29: return . 30: end procedure 12: 13: 14: 15:
4. Experiments and Results In this section, we conduct a comprehensive evaluation of UniDrive. Our experiments are designed to achieve three primary goals: 1) To benchmark UniDrive against state-of-the-art multimodal models on challenging driving-related tasks. 2) To perform a rigorous ablation study to dissect the contribution of each core component of our architecture. 3) To provide qualitative visualizations that offer intuitive insights into the model’s reasoning and perceptual capabilities. Gao et al.: Preprint submitted to Elsevier
Page 8 of 19
UniDrive for Interpretable Risk Understanding
We evaluate UniDrive on our newly curated DRAMA-R EASONING benchmark, covering both captioning and riskobject detection. Unless otherwise noted, all results are reported on the val split, and all hyper-parameters follow the defaults described below.
4.1. Experimental Setup Datasets. We conduct our primary experiments using the DRAMA dataset (Malla et al., 2023), which contains 17,785 two-second-long interactive driving scenarios captured at 30 FPS. While DRAMA provides crucial bounding box annotations for risk-tagged objects, it lacks the rich, explanatory language needed to train and evaluate highlevel reasoning. To address this, we have extended its annotations to create the DRAMA-Reasoning dataset. For each scenario, we manually added detailed captions under three classes, namely (1) the primary risk object and the reason it is hazardous, (2) the inferred intention of the ego-vehicle (e.g., "slowing down," "maintaining speed"), and (3) a safe driving suggestion (e.g., "prepare to brake"). This extension transforms the dataset into a comprehensive benchmark for both perception and interpretable reasoning. Further details on our annotation process are provided in the supplementary material.
Evaluation Metrics. The tasks on the DRAMA-Reasoning dataset include two main objectives: (1) a captioning
task aimed at identifying and explaining risk objects while predicting the ego-vehicle’s intended actions, and (2) a risk object detection task. We assess captioning performance using standard language generation metrics: BLEU-4 (B4), METEOR (M), CIDEr (C), and SPICE (S) . For detection performance, we measure the mean Intersection over Union (mIoU). To better understand performance across different object scales, we further categorize the IoU based on object area: small (IoUS ), medium (IoUM ), and large (IoUL ).
Implementation Details. We use Llama2-7B as the backbone LLM, initialized from its publicly available pre-trained
checkpoint. Our method is implemented in PyTorch and trained on a system with 4 NVIDIA A100 80GB GPUs. For video inputs, we uniformly sample 𝐿 = 5 frames from each clip, ensuring the final frame is always included for bounding box prediction. Input frames are resized and center-cropped to a fixed dimension of 224×224 for the temporal reasoning branch and processed at a higher resolution for the perception branch. We employ the AdamW optimizer with a cosine annealing learning rate schedule. Reflecting the different initialization states of our model’s components, we set initial learning rates of 1 × 10−4 for the pre-trained components of the reasoning branch and 4 × 10−4 for the newly added components in the high-resolution perception branch. The global batch size is set to 32.
4.2. Comparison with State-of-the-Art Methods We compare UniDrive against a suite of leading image-based and video-based MLLMs. As shown in Table 1, UniDrive establishes a new state of the art across both captioning and detection tasks.
Comparison with Image-Based Models. First, we evaluate against models that process only a single image frame.
Even when our model is constrained to a single-frame input (denoted as "Ours w/o ST"), it already demonstrates a significant advantage. It surpasses strong baselines like InstructBLIP (Dai et al., 2023) and Shikra* (Chen et al., 2023) by a large margin. For instance, our image-only model achieves an mIoU of 59.8, a +9.5 improvement over the next best competitor, Shikra*. This highlights the efficacy of our high-resolution perception branch and the architectural design that effectively grounds linguistic reasoning in fine-grained visual details. The captioning scores are also stateof-the-art, with a CIDEr score of 246.7, indicating that better spatial understanding directly translates to more accurate and relevant descriptions.
Comparison with Video-Based Models. When leveraging the full temporal context of video clips, UniDrive’s
performance advances further, underscoring the importance of temporal reasoning. Our full model significantly outperforms specialized video-language models like eP-ALM (Shukor et al., 2023) and Video-LLAMA (Zhang et al., 2023). UniDrive achieves a CIDEr score of 277.5 and a B4 score of 60.3, showcasing its superior ability to generate coherent and contextually appropriate narratives about dynamic driving events. Most critically, the introduction of temporal information boosts detection performance to an mIoU of 61.2, with a notable improvement in detecting small objects (IoUS of 31.0). This suggests that temporal cues help the model anticipate and disambiguate transient road hazards that are difficult to identify from a single snapshot. The overall AVG score of 60.8 places UniDrive significantly ahead of all other methods, confirming the powerful synergy of its spatial and temporal processing capabilities.
Gao et al.: Preprint submitted to Elsevier
Page 9 of 19
UniDrive for Interpretable Risk Understanding Table 1 Comparison with the state-of-the-art on the DRAMA-Reasoning val split. UniDrive is benchmarked against leading image-based and video-based models. For all metrics, higher is better. ‘AVG’ is the arithmetic mean of B4 and mIoU, providing a balanced view of language and perception performance. Our model sets a new state of the art in all categories. B4
Captioning M C
S
mIoU
Detection mIoU𝑆 IoU𝑀
IoU𝐿
Image
BLIP-2 LLaVA InstructBLIP Shikra* Ours w/o ST
46.1 47.5 49.9 49.8 55.2
34.3 35.2 37.9 37.7 38.1
194.7 198.6 205.0 204.7 246.7
50.7 48.3 50.9 50.7 54.3
46.3 47.2 47.8 50.3 59.8
8.1 8.0 9.1 10.4 29.8
60.2 62.1 62.2 59.5 64.3
73.7 74.2 74.5 73.8 82.1
46.2 47.4 48.9 50.1 57.5
Video
eP-ALM Video-LLAMA UniDrive (Ours)
51.4 53.9 60.3
38.0 37.8 39.6
225.1 229.5 277.5
52.8 52.6 58.1
43.2 42.8 61.2
7.2 6.9 31.0
56.8 55.3 66.5
68.8 67.9 83.7
47.3 48.4 60.8
Input
Method
AVG
Table 2 Zero-shot VQA performance on the NuScenes dataset. We report accuracy (%) on different question types.The results show that UniDrive generalizes significantly better to unseen data, especially on complex reasoning tasks, without any fine-tuning.
Method Video-LLAMA UniDrive (Ours)
Question Type Accuracy (%) Object Presence Traffic State Situation Reasoning 71.3 76.9 (+5.6)
78.5 82.3 (+3.8)
54.5 63.6 (+9.1)
Average Acc. (%) 68.1 75.3 (+7.2)
4.3. Generalization to Unseen Datasets and Scenarios A fundamental measure of a model’s utility is its ability to generalize beyond its training distribution. To rigorously test this, we evaluate UniDrive’s zero-shot performance on the NuScenes dataset (Caesar et al., 2020), a large-scale, industry-standard benchmark known for its diverse geographic locations, weather conditions, and complex traffic scenarios. This evaluation is critical as it simulates a real-world deployment scenario where the model must confront entirely new visual and contextual distributions without any prior fine-tuning. We designed a challenging Visual Question Answering (VQA) task by curating a diverse set of questions from the NuScenes validation split. These questions were crafted to probe three distinct cognitive abilities: (1) Object Presence, which tests the model’s core perceptual ability to identify specific objects (e.g., "Is there a stroller on the sidewalk?"); (2) Traffic State, which assesses its capacity to recognize critical semantic states (e.g., "What color is the traffic light for the ego-vehicle?"); and (3) Situation Reasoning, the most complex category, which demands an understanding of latent interactions, intentions, and potential risks (e.g., "Why is the car ahead of us braking?"). The results, presented in Table 2, demonstrate UniDrive’s generalization capabilities. Compared to the strong Video-LLAMA baseline, UniDrive achieves a significantly higher average accuracy of 75.3%, marking a 7.2% improvement. The performance gains are consistent across all categories. The advantage in "Object Presence" (+5.6%) can be attributed to our high-resolution perception branch, which is more adept at identifying objects in novel contexts. However, the most telling result is the substantial +9.1% lead in "Situation Reasoning". This highlights the effectiveness of our spatio-temporal fusion mechanism and the temporal reasoning branch. These components enable UniDrive to not just see objects, but to understand their dynamic relationships and infer causality, a crucial skill for safe navigation. These zero-shot results suggest that UniDrive captures transferable cues for driving scene understanding rather than dataset-specific patterns, which is encouraging for deployment under distribution shift.
Generalization to Diverse Driving Benchmarks. To further probe the model’s generalization in a zero-shot setting,
we also evaluated UniDrive on the challenging BDD100K dataset (Yu et al., 2020). We prompted the model to identify and localize the most significant risk in video clips from the validation set. As shown in Table 3, UniDrive demonstrates Gao et al.: Preprint submitted to Elsevier
Page 10 of 19
UniDrive for Interpretable Risk Understanding Table 3 Zero-shot risk object detection on the BDD100K validation set. We report mean Average Precision (mAP) at an IoU threshold of 0.5. UniDrive shows superior generalization to a new, diverse dataset. Method Video-LLAMA UniDrive (Ours)
APpedestrian
APcyclist
APcar
41.3 52.7
35.8 45.2
30.1 41.5
58.0 71.4
a strong ability to transfer its capabilities to this entirely new data distribution. It significantly outperforms the VideoLLAMA baseline in localizing critical road actors, especially vulnerable road users like pedestrians. Qualitatively, its generated risk descriptions were more context-aware and aligned with human driver intuition. This strong performance underscores that UniDrive’s architecture learns generalizable principles of risk perception rather than memorizing dataset-specific patterns.
4.4. Ablation Study To validate our design choices and understand the contribution of each technical ingredient in UniDrive, we perform a series of controlled ablations on the DRAMA-Reasoning val split. As shown in Table 4, every component proves to be integral to the model’s final performance.
Temporal Reasoning Branch (T-RB). Removing the T-RB and relying solely on the most recent frame forces the
model to act as a pure image-based system. While the high-resolution branch maintains strong detection performance (59.8 mIoU), the captioning quality drops significantly (e.g., -30.8 in CIDEr). This demonstrates that temporal context is vital for higher-level reasoning, such as anticipating future ego-car actions and explaining the evolution of a hazardous situation.
High-Resolution Perception Branch (P-B). Discarding the perception branch has the most dramatic impact on
detection, with mIoU plummeting by 13.3 points. The effect is especially pronounced for small objects (IoUS drops from 31.0 to 12.4), confirming that fine-grained visual details from the HR stream are indispensable for precise spatial localization. Captioning also suffers, as the model struggles to ground its textual descriptions in accurate visual evidence, reaffirming the need for high-fidelity perception.
Spatio-Temporal Fusion (STF). We replace our proposed gated cross-attention fusion with a naive late concatenation of features from the two branches ("w/o STF"). This change leads to a substantial drop across all metrics, with mIoU falling to 44.6. This result strongly suggests that simply presenting both temporal and spatial features to the LLM is insufficient. The explicit, learned alignment facilitated by our fusion mechanism is critical for effectively integrating temporal semantics with high-resolution spatial cues.
Box Token Grounding. To verify the effectiveness of our lightweight <box> token, we ablate it and instead rely on naive string matching to find a predicted class name in the ground truth. As expected, all detection metrics collapse to zero, while captioning quality remains almost unchanged. This confirms that our direct-to-text bounding box representation provides an efficient and fully differentiable method for injecting localization supervision without interfering with the model’s language generation capabilities. Overall, these ablations reveal that each component contributes meaningfully to UniDrive’s performance. In particular, the P-B and STF are key to accurate spatial grounding, whereas the T-RB is essential for generating temporally coherent and insightful descriptions.
4.5. Human-Centric Evaluation of Interpretability While automatic metrics measure objective accuracy, they do not capture the subjective quality and usefulness of the generated explanations for human users. To quantitatively evaluate UniDrive on the core claims of interpretability and trustworthiness, we conducted a formal user study.
Study Design. We recruited 25 participants (aged 20-32, 12 male and 13 female; all with valid driving licenses, normal or corrected-to-normal vision, and at least two years of driving experience) through campus advertisements. Gao et al.: Preprint submitted to Elsevier
Page 11 of 19
UniDrive for Interpretable Risk Understanding Table 4 Ablation results on the DRAMA-Reasoning val split. Each row represents the removal of exactly one component from the full UniDrive model. AVG is the arithmetic mean of B4 and mIoU. The results demonstrate that all components are critical for achieving optimal performance. Model Variant Full UniDrive w/o Temporal Reasoning Branch w/o High-Res Perception Branch w/o Spatio-Temporal Fusion w/o Box Token Grounding
B4
Captioning↑ M C
S
mIoU
Detection↑ mIoU𝑆 mIoU𝑀
mIoU𝐿
60.3 55.2 52.8 52.4 60.1
39.6 38.1 37.6 36.9 39.5
58.1 54.3 53.7 51.7 57.9
61.2 59.8 47.9 44.6 0.0
31.0 29.8 12.4 9.7 0.0
83.7 82.1 78.5 70.2 0.0
277.5 246.7 238.2 222.8 276.1
66.5 64.3 60.7 58.1 0.0
AVG↑ 60.8 57.5 50.4 48.5 30.1
Table 5 Results of the human-centric evaluation study. Scores show the percentage of times a model’s explanation was chosen as superior by human evaluators. Criterion Usefulness Accuracy Trustworthiness
UniDrive Win %
Baseline Win %
Tie / No Pref. %
74.7% 76.8% 82.0%
20.0% 18.8% 12.9%
5.3% 4.4% 5.1%
We showed them 30 challenging scenarios from our validation set. For each scenario, we presented the explanations from UniDrive and Video-LLAMA in a randomized, side-by-side format and asked participants to choose the superior one based on three criteria: Usefulness (provides more valuable information for a safe decision), Accuracy (more precisely describes the scene), and Trustworthiness (which AI they would trust more in their own car).
Results. The results, summarized in Table 5, show a clear and strong human preference for UniDrive’s explanations. Across all criteria, participants favored UniDrive over 74% of the time, with the highest margin in Trustworthiness (82.0%). This provides compelling evidence that the richer, more detailed, and temporally-aware explanations generated by our model are not only quantitatively better but are also perceived by humans as significantly more reliable and useful. This is a crucial step towards building human-AI trust in safety-critical autonomous systems. 4.6. Robustness, Efficiency, and Scalability Beyond accuracy on standard benchmarks, the practical viability of an autonomous driving model hinges on its robustness to adverse conditions and its computational efficiency. In this section, we analyze these critical aspects of UniDrive.
Robustness to Adverse Conditions. Driving environments are often non-ideal, with challenges like poor illumination and inclement weather that can severely degrade sensor input and cripple perception systems. To quantify UniDrive’s resilience, we partitioned the DRAMA-Reasoning validation set into challenging subsets based on their metadata: Day vs. Night and Clear vs. Rainy weather. The results are detailed in Table 6. As expected, all models exhibit some performance degradation under these challenging conditions. However, UniDrive demonstrates a significantly more graceful degradation compared to the baseline. For example, in nighttime scenarios, UniDrive’s mIoU drops by only 5.1 points (an 8% relative decrease), whereas Video-LLAMA’s performance falls by 8.6 points (a 20% relative decrease). A similar trend is observed in rainy conditions. This enhanced robustness stems from our dual-branch design. The temporal reasoning branch can infer the presence and motion of objects even when they are partially obscured or have low contrast in a single frame, while the high-resolution branch maximizes the extraction of any available visual detail. This synergy makes UniDrive less susceptible to common visual corruptions, a vital feature for ensuring safety-critical reliability. Efficiency and Scalability. While performance is paramount, any model intended for eventual deployment must operate within reasonable computational budgets. In Table 7, we analyze UniDrive’s computational footprint. Although Gao et al.: Preprint submitted to Elsevier
Page 12 of 19
UniDrive for Interpretable Risk Understanding Table 6 Robustness analysis on challenging subsets of DRAMA-Reasoning. We report key captioning (CIDEr) and detection (mIoU) metrics. UniDrive shows superior resilience and more graceful performance degradation in adverse conditions. Condition
UniDrive (Ours) mIoU↑ CIDEr↑
Video-LLAMA (Baseline) mIoU↑ CIDEr↑
Day / Clear (Base) Night Rainy
61.2 56.1 57.5
42.8 34.2 36.1
277.5 259.3 265.8
229.5 201.7 210.4
Table 7 Efficiency and resource comparison. We compare UniDrive with key baselines on model size, computational load, and inference speed. UniDrive achieves a strong balance between performance and practical efficiency. Method LLaVA-1.5 (7B) Video-LLAMA (7B) UniDrive (Ours)
Parameters (B)
GFLOPs
Speed (FPS)
7.1 7.8 8.2
795 910 980
12.1 10.5 9.8
UniDrive has a moderately larger parameter count and higher GFLOPs due to its dual-branch architecture, it is designed with scalability in mind. By keeping the core LLM frozen and employing an efficient, lightweight fusion mechanism, the additional overhead is well-managed. Its inference speed of 9.8 FPS on a single NVIDIA A100 GPU is competitive and demonstrates a favorable trade-off between its significantly advanced reasoning capabilities and its computational cost. Furthermore, its Flamingo-style cross-attention architecture is inherently more scalable for handling even longer video sequences or a greater number of camera views compared to models relying solely on self-attention, which suffer from quadratic complexity. This suggests that UniDrive scales more favorably to longer sequences or additional camera views than self-attention-only models.
4.7. Qualitative Analysis To complement our quantitative results, Figure 3 presents a qualitative comparison of UniDrive against Shikra and Video-LLaMA on three representative scenarios from the DRAMA-Reasoning val split. In scenario (a), all models correctly identify the red traffic light as the key contextual cue. However, UniDrive produces a more concise and action-oriented description, directly linking the traffic state to the appropriate ego-vehicle response. In scenario (b), the primary risk object is a white hatchback stopped ahead. Shikra misidentifies the vehicle type and incorrectly predicts the ego-action as “stop”, while Video-LLaMA identifies the stopped vehicle but also misclassifies it as a truck. UniDrive correctly identifies the stopped white van and recommends the appropriate driving response of beginning to drive. In scenario (c), the key hazard is a pedestrian crossing the road. Both Shikra and VideoLLaMA fail to identify the pedestrian, instead focusing on an irrelevant stopped vehicle. In contrast, UniDrive uniquely identifies the pedestrian as the primary risk object, correctly describes their motion, and anticipates the right-turn maneuver required by the ego-vehicle. These examples intuitively illustrate how UniDrive’s architectural innovations translate into more reliable and safety-oriented driving intelligence.
Failure Case Analysis. A complete evaluation must also examine the conditions under which UniDrive fails. In
Figure 4, we present two representative failure cases from the DRAMA-Reasoning val split. In both, the ground-truth risk object is drawn in red and the model’s prediction in purple. Notably, the two cases share a common failure mode: rather than missing the hazard altogether, UniDrive grounds a plausible but non-primary object and therefore diverges from the human-annotated primary risk. The first case (left) involves a sedan stopped at the left roadside, which the annotation designates as the primary risk because it constrains the drivable space ahead of the ego-vehicle. UniDrive instead localizes a pedestrian walking along the sidewalk and centers its explanation on this moving agent. Although the pedestrian is a reasonable object of attention, it is not the safety-critical target in this scenario. This behavior suggests that the model is biased toward
Gao et al.: Preprint submitted to Elsevier
Page 13 of 19
UniDrive for Interpretable Risk Understanding
GT: .. red traffic light….The ego car may move straight ahead. Please with care. Shikra: …red traffic light... may move straight ahead. Please with care. Video-LLaMA : …red traffic light... may move straight ahead. Please with care. Ours : …red light in view... may proceed straight. Please with care.
(a)
GT: … white hatchback stopped in …. move straight ahead. Please begin to drive. Shikra: … white truck slowing ...move straight ahead. Please stop. Video-LLaMA : …white truck stopped…. move straight ahead. Please begin to drive. Ours : … white van stopped ahead …. move straight ahead. Please begin to drive.
(b)
GT: pedestrian wearing .. is walking... make a right turn. Please remain cautious. Shikra: ..white hatchback stopped… move straight ahead, please with care. Video-LLaMA : ..white hatchback stopped ... move straight ahead, please with care. Ours : pedestrian dressed in .. is moving... to turn right. Please remain cautious.
(c)
Figure 3: Qualitative Comparison of UniDrive against a Video-LLM baseline (Video-LLAMA). We showcase three challenging scenarios from the DRAMA-Reasoning val split. The outputs include the generated caption and the predicted bounding box (in green). (a) All models correctly identify the red traffic light, but UniDrive produces a more concise and action-oriented description. (b) Shikra incorrectly identifies the vehicle type and predicts the wrong ego-action ("stop"), while UniDrive correctly identifies the stopped white van and recommends the appropriate response. (c) UniDrive uniquely identifies the pedestrian as the primary risk object and correctly anticipates the right-turn maneuver, whereas both baselines focus on an irrelevant stopped vehicle. These examples highlight our model’s enhanced capabilities in both fine-grained object recognition and safety-critical reasoning.
dynamic agents and tends to underweight static obstacles, even when the latter more directly affect the ego-vehicle’s path. The second case (right) is, in effect, the reverse. Here the ground truth marks a pedestrian located within the ego lane as the primary risk, whereas UniDrive grounds a large vehicle parked at the roadside and overlooks the smaller, more distant pedestrian. The model is drawn to a large, nearby, high-contrast object and fails to prioritize the smaller target that lies directly on the ego-vehicle’s trajectory. This case is also a useful caveat to our earlier results: although the high-resolution perception branch improves small-object localization in aggregate, accurate perception alone does not guarantee correct risk selection when several candidate hazards co-occur.
5. Discussion 5.1. Grounded Risk Reasoning The experimental results show that UniDrive improves both language-based risk explanation and object-level grounding, but the main implication goes beyond the numerical gains themselves. The consistent improvements in captioning, mIoU, small-object localization, zero-shot transfer, and human preference suggest that safety-oriented driving understanding should not be treated as either a captioning problem or a detection problem alone. In interactive driving scenes, an explanation is only safety-relevant when it can be linked to the correct risk source, and a localized object is only interpretable when its dynamic risk relevance can be explained. This finding is consistent with the motivation of DRAMA, which emphasizes the joint role of risk localization and captioning in driving scene understanding (Malla et al., 2023), but our results further show that such a joint formulation can be strengthened through explicit architectural alignment between temporal semantics and spatial evidence. The comparison with image-based and video-based baselines provides a useful way to interpret this point. Imagebased MLLMs can exploit fine-grained visual details in the current frame, but they have limited access to how a risk evolves over time. Video-based models, in contrast, are better positioned to capture motion and interaction cues, yet their spatial grounding may be weakened by low-resolution inputs, temporal compression, or limited visual-token budgets. This helps explain why UniDrive obtains improvements in both captioning and detection rather than only one side of the task. The high-resolution perception branch improves the model’s ability to identify visually subtle or small risk objects, while the temporal reasoning branch provides dynamic context for explaining why these objects matter. As a result, the generated output becomes not only linguistically plausible, but also more visually verifiable. Gao et al.: Preprint submitted to Elsevier
Page 14 of 19
UniDrive for Interpretable Risk Understanding
GT: "A silver sedan is stopped at the left roadside…slow down and keep right."
GT: "A pedestrian is walking in the ego lane ahead…please brake and yield."
Ours: "A pedestrian is walking along the right side…proceed with care."
Ours: "A dark vehicle is parked on the right…move slightly and pass."
Figure 4: Failure cases of UniDrive on the DRAMA-Reasoning val split. Ground-truth risk objects are shown in red and UniDrive’s predictions in purple. In both scenes the model grounds a plausible but non-primary hazard. (Left) The ground truth marks a sedan stopped at the left roadside, yet UniDrive localizes a pedestrian moving along the sidewalk, favoring a dynamic agent over a static obstacle. (Right) The ground truth marks a pedestrian in the ego lane, while UniDrive localizes a large parked vehicle and overlooks the smaller, more distant pedestrian on the vehicle’s path. These cases indicate that, when multiple candidate hazards co-occur, UniDrive’s risk selection can be biased by object size, motion, and proximity rather than by safety-critical priority.
The zero-shot results on NuScenes and BDD100K further indicate that the learned representation is not merely a dataset-specific mapping from DRAMA-Reasoning annotations to textual templates. The larger gains on situation reasoning and risk-object localization suggest that UniDrive captures transferable cues about object presence, motion context, and safety relevance. This is important because real-world autonomous driving systems inevitably encounter distribution shifts in weather, road layout, traffic culture, sensor viewpoint, and object appearance. Although zero-shot transfer does not prove deployment readiness, it provides evidence that grounding risk explanations in explicit visual evidence can support more robust generalization than language-only or weakly grounded reasoning.
5.2. Role of Spatio-Temporal Fusion The ablation results provide insight into the mechanism behind UniDrive’s performance. Removing the temporal reasoning branch reduces the quality of risk descriptions, even though the high-resolution branch can still maintain relatively strong detection performance. This suggests that high spatial fidelity alone is insufficient for interpreting dynamic driving risk. Many hazards are defined not only by what an object is, but also by how it moves, whether it is entering the ego lane, whether the ego vehicle is approaching it, and whether the surrounding context makes a cautious response necessary. In this sense, temporal reasoning contributes semantic and causal context rather than merely additional frames. Conversely, removing the high-resolution perception branch leads to a substantial drop in detection performance, especially for small objects. This indicates that temporal context cannot compensate for missing fine-grained spatial evidence. In driving scenes, safety-critical objects may occupy only a small image region, be partially occluded, or appear at a distance. Such cases are particularly challenging for video-based MLLMs, where spatial precision may be sacrificed for temporal coverage (Zhang et al., 2023; Xu et al., 2024). The results therefore support a complementary view: temporal information helps determine how a risk emerges, while high-resolution perception helps determine where the responsible object is located. The strongest evidence for the proposed mechanism comes from the ablation of spatio-temporal fusion. Replacing gated cross-attention with naive feature concatenation causes a clear performance collapse across both captioning and grounding metrics. This shows that simply exposing the LLM to both temporal and spatial features is not enough. The model must learn how dynamic context should attend to fine-grained spatial regions. The gated cross-attention module provides this alignment mechanism by using temporal features as queries and high-resolution spatial features as keys and values. In other words, it allows the model to ask which visual regions are relevant to the evolving risk context. Gao et al.: Preprint submitted to Elsevier
Page 15 of 19
UniDrive for Interpretable Risk Understanding
This mechanism explains why UniDrive can improve small-object grounding while also generating more coherent risk explanations. This finding also refines the interpretation of recent driving-oriented MLLMs. Prior work has shown that video inputs, high-resolution images, visual question answering, and planning-oriented language supervision can each improve different aspects of autonomous driving understanding (Ding et al., 2023; Sima et al., 2024; Tian et al., 2024; Wang et al., 2023). Our results suggest that the next step is not simply to add more modalities, frames, or tokens, but to design risk-conditioned alignment mechanisms that connect dynamic semantics with object-level evidence. For safety-critical tasks, the key question is not only whether a model can describe a scene, but whether it can align the reason for caution with the correct visual target.
5.3. Implications for Safety-Oriented Driving MLLMs In terms of model design, the results suggest that future driving MLLMs should move beyond general scene understanding and place greater emphasis on grounded risk reasoning. In general-purpose VLMs, visual grounding is often treated as a way to connect language expressions with image regions (Peng et al., 2023; Zhang et al., 2024; Cheng et al., 2024). In autonomous driving, however, grounding carries a more direct safety meaning because it determines whether an explanation refers to the object that actually matters for the ego vehicle. Therefore, grounding should not be considered an auxiliary output attached to language generation. Instead, it should be integrated into the core reasoning process, so that language explanations, risk-object localization, and driving suggestions are mutually constrained. This perspective also has implications for evaluation. Standard captioning metrics such as BLEU-4, METEOR, CIDEr, and SPICE are useful for measuring linguistic similarity, while IoU-based metrics are useful for measuring localization accuracy. However, neither type of metric alone can fully evaluate safety-oriented explanations. A fluent caption may refer to the wrong object, while a correct bounding box may still fail to explain why the object is risky. The combined evaluation adopted in this study, including captioning, grounding, object-scale analysis, zero-shot transfer, robustness testing, qualitative cases, and human-centered assessment, provides a more comprehensive view of model reliability. At the same time, the failure cases indicate that even this evaluation remains incomplete, because risk understanding also requires prioritizing among multiple plausible hazards. Future benchmarks should therefore consider risk ranking, trajectory relevance, causal relevance, and uncertainty, rather than evaluating only one primary bounding box or one reference caption. The human-centered evaluation further shows why grounded explanations are important for trustworthy autonomous driving. Participants more frequently preferred UniDrive’s explanations in terms of usefulness, accuracy, and trustworthiness. This result should not be interpreted merely as a preference for more fluent text. Rather, it suggests that explanations become more useful when they identify the risk object, describe the reason for concern, and provide a visually checkable basis for the recommended response. Such outputs are relevant to safety-oriented perception and post-hoc auditing, where an explanation must be inspectable rather than only plausible. In this sense, UniDrive can be viewed as an intermediate reasoning layer between perception and downstream decision-making. It does not replace planning or control, but it can provide grounded risk evidence that supports safety monitoring, human review, and failure analysis. The robustness results further support this safety-oriented interpretation. Under night and rainy conditions, UniDrive exhibits more graceful degradation than the video-language baseline. This suggests that combining temporal context with high-resolution spatial evidence can reduce dependence on any single fragile cue. When visual details are degraded, temporal context can still provide continuity and motion information; when temporal cues are ambiguous, high-resolution perception can still preserve object-level evidence. Such complementarity is particularly important for safety-relevant scenarios, where rare, adverse, or long-tail conditions often determine system reliability.
5.4. Limitations and Future Studies Despite these promising results, several limitations remain. First, the failure cases reveal that UniDrive can still select a plausible but non-primary risk object when multiple candidate hazards co-occur. This is not a simple object detection failure. In both representative cases, the model grounds an object that is visually reasonable, but not the one annotated as most safety-critical. This suggests a risk-prioritization problem. The model may be biased toward dynamic agents such as pedestrians, or toward large, nearby, and visually salient objects, even when another object has stronger relevance to the ego vehicle’s trajectory. Future work should therefore move beyond single-object grounding and introduce risk hierarchy, multi-risk ranking, and trajectory-aware relevance estimation. Explicit risk indicators
Gao et al.: Preprint submitted to Elsevier
Page 16 of 19
UniDrive for Interpretable Risk Understanding
such as time-to-collision, post-encroachment time, collision probability, reachable sets, or driving-corridor constraints could help distinguish visually salient objects from genuinely safety-critical ones. Second, the current DRAMA-Reasoning setting is built on primary risk-object annotations and enriched textual descriptions. This design enables joint evaluation of reasoning and grounding, but it also simplifies the complexity of real driving scenes. In practice, multiple objects may simultaneously contribute to risk at different levels, and different human annotators may disagree about which object should be considered primary. Future datasets should therefore include multiple risk candidates, risk severity levels, causal relations, and inter-annotator agreement. Such annotations would make it possible to evaluate whether a model understands the structure of risk rather than only matching one selected target. Third, UniDrive is primarily based on visual-language reasoning and does not explicitly incorporate all information used by a full autonomous driving stack. Although the model can infer many risk cues from video and high-resolution images, it does not directly model ego-motion, vehicle dynamics, traffic rules, HD maps, LiDAR/radar measurements, or downstream planning feasibility. This limits its ability to determine which object will affect the ego vehicle’s future path under different maneuvers. Future studies should integrate grounded risk reasoning with BEV representations, map priors, trajectory prediction, and planning modules. Such integration would allow the model to reason not only about what is visible, but also about what is reachable, avoidable, and operationally relevant. Finally, deployment efficiency and reliability require further investigation. UniDrive achieves a reasonable balance between performance and computational cost, but the reported inference speed on an A100 GPU should not be interpreted as sufficient evidence for real-time in-vehicle deployment. Practical systems require lower latency, stronger reliability guarantees, and extensive validation under closed-loop conditions. Future work should explore model compression, knowledge distillation, token pruning, and edge-oriented inference. More importantly, the value of grounded risk explanations should be tested in closed-loop simulation, human-in-the-loop experiments, and real-world replay studies to examine whether they improve downstream safety monitoring, decision support, and failure diagnosis.
6. Conclusion This paper presented UniDrive, a unified visual-language and grounding framework for interpretable risk understanding in autonomous driving. Motivated by the persistent trade-off between temporal understanding and spatial precision in existing driving-oriented MLLMs, UniDrive integrates a temporal reasoning branch, a high-resolution perception branch, and a gated cross-attention fusion module to align dynamic scene context with fine-grained spatial evidence. Based on this fused representation, the model jointly generates natural-language risk descriptions and grounded bounding-box outputs for risk-critical objects. Experiments on the DRAMA-Reasoning benchmark demonstrate that UniDrive achieves strong performance in both driving scene captioning and risk-object localization. Compared with representative image-based and videobased baselines, UniDrive provides more accurate grounded explanations, stronger small-object localization, and better overall reasoning-grounding consistency. Additional evaluations further show its zero-shot generalization to unseen driving datasets, improved robustness under adverse conditions, and higher human-rated usefulness, accuracy, and trustworthiness. Ablation studies confirm that temporal reasoning, high-resolution perception, and explicit spatiotemporal fusion are all essential to the final performance. Overall, the findings show that safety-oriented driving MLLMs should not be evaluated only by their ability to generate plausible language responses or detect objects in isolation. Instead, they should be able to explain why a situation is risky and ground that explanation in the correct visual evidence. By linking temporal semantics, highresolution perception, and object-level grounding within a unified generative framework, UniDrive provides a more transparent and reliable foundation for interpretable autonomous driving intelligence.
Statement 1. Conflict of Interest All authors declare no financial or non-financial competing interests. 2. Data Availability Data will be made available on reasonable request.
Gao et al.: Preprint submitted to Elsevier
Page 17 of 19
UniDrive for Interpretable Risk Understanding
3. Contributions Xiaowei Gao: Conceptualization, Methodology, Data curation, Software, Formal analysis, and Writing - original draft. Pengxiang Li: Conceptualization, Methodology, Data curation, Software, Formal analysis, and Writing original draft. Yitai Cheng: Visualization, and Writing - review & editing. Ruihan Xu: Data curation, and Writing - review & editing. James Haworth: Writing - review & editing, and Validation. Stephen Law: Writing - review & editing, and Validation. Yun Ye: Conceptualization, Methodology, Resources, Writing - review & editing, Project administration, and Supervision. All authors read and approved the final manuscript. 4. Declaration of the use of AI During the preparation of this work the authors used ChatGPT in order to improve the readability and language of the manuscript. After using this tool/service, the authors reviewed and edited the content as needed and take full responsibility for the content of the publication.
References Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al., 2022. Flamingo: A visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736. Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., Beijbom, O., 2020. nuscenes: A multimodal dataset for autonomous driving, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621–11631. Che, Y., Wong, M.O., Gao, X., Liang, H., Ye, Y., 2026. Enhancing safety in automated ports: A virtual reality study of pedestrian–autonomous vehicle interactions under time pressure, visual constraints, and varying vehicle size. Transportation Research Interdisciplinary Perspectives 37, 102041. Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., Zhao, R., 2023. Shikra: Unleashing multimodal LLM’s referential dialogue magic. arXiv preprint arXiv:2306.15195 . Chen, L., Wu, P., Chitta, K., Jaeger, B., Geiger, A., Li, H., 2024. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence 46, 10164–10183. Cheng, A.C., Yin, H., Fu, Y., Guo, Q., Yang, R., Kautz, J., Wang, X., Liu, S., 2024. Spatialrgpt: Grounded spatial reasoning in vision-language models. Advances in Neural Information Processing Systems 37, 135062–135093. Cui, C., Ma, Y., Cao, X., Ye, W., Zhou, Y., Liang, K., Chen, J., Lu, J., Yang, Z., Liao, K.D., et al., 2024. A survey on multimodal large language models for autonomous driving, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 958–979. Dai, W., Li, J., Li, D., Tiong, A., Zhao, J., Wang, W., Li, B., Fung, P.N., Hoi, S., 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36, 49250–49267. Ding, X., Han, J., Xu, H., Zhang, W., Li, X., 2023. HiLM-D: Towards high-resolution understanding in multimodal large language models for autonomous driving. arXiv preprint arXiv:2309.05186 . Ding, X., Han, J., Xu, H., Zhang, W., Li, X., 2024. Holistic autonomous driving understanding by bird’s-eye-view injected multi-modal large models. arXiv preprint arXiv:2404.07165 . Fu, D., Li, X., Wen, L., Dou, M., Cai, P., Shi, B., Qiao, Y., 2024. Drive like a human: Rethinking autonomous driving with large language models. arXiv preprint arXiv:2307.07162 . Huang, S., Shi, F., Sun, C., Zhong, J., Ning, M., Yang, Y., Lu, Y., Wang, H., Khajepour, A., 2025. Drivesotif: Advancing perception SOTIF through multimodal large language models. arXiv preprint arXiv:2505.07084 . Hwang, J.J., Xu, R., Lin, H., Hung, W.C., Ji, J., Choi, K., Huang, D., He, T., Covington, P., Sapp, B., Guo, J., Anguelov, D., Tan, M., 2024. EMMA: End-to-end multimodal model for autonomous driving. arXiv preprint arXiv:2410.23262 . Jiang, K., Cai, X., Cui, Z., Li, A., Li, Z., Li, X., Ren, H., Wang, Y., Zhao, H., Wang, Y., 2024. KoMA: Knowledge-driven multi-agent framework for autonomous driving with large language models. arXiv preprint arXiv:2407.14239 . Li, J., Li, J., Yang, G., Yang, L., Chi, H., Yang, L., 2025. Applications of large language models and multimodal large models in autonomous driving: A comprehensive review. Drones 9, 238. Liu, H., Li, C., Li, Y., Lee, Y.J., 2024. Improved baselines with visual instruction tuning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296–26306. Lu, Q., Wang, X., Jiang, Y., Zhao, G., Ma, M., Feng, S., 2025. Omnitester: Multimodal large language model driven scenario testing for autonomous vehicles. Automotive Innovation 8, 838–852. Ma, Y., Cui, C., Cao, X., Ye, W., Liu, P., Lu, J., Abdelraouf, A., Gupta, R., Han, K., Bera, A., Rehg, J.M., Wang, Z., 2024. LaMPilot: An open benchmark dataset for autonomous driving with language model programs. arXiv preprint arXiv:2312.04372 . Malla, S., Choi, C., Dwivedi, I., Choi, J.H., Li, J., 2023. DRAMA: Joint risk localization and captioning in driving, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1043–1052. Mao, J., Ye, J., Qian, Y., Pavone, M., Wang, Y., 2023. A language agent for autonomous driving. arXiv preprint arXiv:2311.10813 . Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., Wei, F., 2023. KOSMOS-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824 . Shao, H., Hu, Y., Wang, L., Song, G., Waslander, S.L., Liu, Y., Li, H., 2024. Lmdrive: Closed-loop end-to-end driving with large language models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15120–15130. Shukor, M., Dancette, C., Cord, M., 2023. ep-alm: Efficient perceptual augmentation of language models, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22056–22069.
Gao et al.: Preprint submitted to Elsevier
Page 18 of 19
UniDrive for Interpretable Risk Understanding Sima, C., Renz, K., Chitta, K., Chen, L., Zhang, H., Xie, C., Beisswenger, J., Luo, P., Geiger, A., Li, H., 2024. Drivelm: Driving with graph visual question answering, in: Computer Vision – ECCV 2024, pp. 256–274. Tian, X., Gu, J., Li, B., Liu, Y., Hu, C., Wang, Y., Zhan, K., Jia, P., Lang, X., Zhao, H., 2024. Drivevlm: The convergence of autonomous driving and large vision-language models. arXiv preprint arXiv:2402.12289 . Wang, J., Zhao, C., Liu, W., Ma, J., Sun, P., 2023. Drivemlm: Aligning multi-modal large language models with behavioral planning states for autonomous driving. arXiv preprint arXiv:2312.09245 . Xu, Z., Zhang, Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.Y.K., Li, Z., Zhao, H., 2024. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters . Yang, Z., Jia, X., Li, H., Yan, J., 2023. LLM4Drive: A survey of large language models for autonomous driving, in: NeurIPS 2024 Workshop on Open-World Agents. Ye, Y., Che, Y., Liang, H., Zhang, Y., Xu, P., 2026a. Wait or cross? Understanding the influence of behavioral tendency, trust, and risk perception on pedestrian gap-acceptance of automated truck platoons. Transportation Research Part F: Traffic Psychology and Behaviour 120, 103660. Ye, Y., Li, Z., Angeloudis, P., Wong, S., Sun, J., Liang, H., 2026b. Are eHMIs always helpful? Investigating how eHMIs interfere with pedestrian behavior on multi-lane streets: An eye-tracking virtual reality experiment. arXiv preprint arXiv:2601.03223 . Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., Darrell, T., 2020. Bdd100k: A diverse driving dataset for heterogeneous multitask learning, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2636–2645. Zhang, H., Li, X., Bing, L., 2023. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858 . Zhang, Y., Ma, Z., Gao, X., Shakiah, S., Gao, Q., Chai, J., 2024. Groundhog: Grounding large language models to holistic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14227–14238. Zhou, G., Wang, Z., Li, Y., Zhang, Y., Chen, W., 2024. Vision language models in autonomous driving: A survey and outlook. IEEE Open Journal of Intelligent Transportation Systems .
Gao et al.: Preprint submitted to Elsevier
Page 19 of 19