From Driving Videos to Simulatable Scenarios
arXiv:2606.21993v1 [cs.SE] 20 Jun 2026
Alexandre Levy1,2 , Ernest Valveny Llobet 1,2 , Antonio M. López1,2
Abstract— Autonomous vehicles (AVs) face driving scenarios ranging from routine traffic to rare events. To assess safety it is crucial to reproduce these scenarios in a controllable, repeatable, and scalable manner, with simulation playing a key role. This paper introduces D-V2S, a novel framework that automatically generates simulatable driving scenarios from driving videos. D-V2S operates in two stages: a Driving Record Analyzer (DRA) uses a vision language model (VLM) with our designed prompt to produce natural-language descriptions from input videos, capturing road layouts and dynamic traffic interactions; subsequently, a Scenario Generator (SG) uses a large language model (LLM) and our conditioning context to translate these descriptions into executable scenarios. Using simulations, we show that D-V2S generates scenarios where 90% of the relevant semantic elements of the videos are present. We also provide qualitative results demonstrating DV2S’s capability to transform real-world driving videos into simulatable scenarios. Moreover, we provide both semantic and human driven ablative analyses of D-V2S’s modules. In particular, we show how the VLM choice matters for DRA, and how our SG achieves a 75% preference rate over other state-of-the-art methods.
I. I NTRODUCTION Autonomous vehicles (AVs) are rapidly becoming a reality and encounter diverse driving scenarios, from routine traffic to rare events. To assess safety, simulating these scenarios in a controllable and scalable manner is needed. Thus, generating simulatable targeted scenarios is core to develop the underlying artificial intelligence (AI) driving systems. To guide simulatable scenario creation, engineers analyze real-world driving videos from data-collecting vehicles, AV monitoring safety interventions, or prior simulations. The customized scenarios enable reliable simulation-in-the-loop incremental development of AVs, allowing evaluations of overall driving performance or specific AI modules (perception, decision making, etc.). However, the creation of scenarios involving particular events is a descriptive programming task [1], [2]. Thus, it requires engineers with specialized knowledge and becomes time consuming. Hence, we need a more automatic approach, which is the focus of this paper. We introduce a software framework, called D-V2S, that automatically transforms driving videos into simulation-executable driving scenarios. To design D-V2S, we started from practical considerations. On one hand, we decided to leverage vision language model (VLM) technology for its high accessibility, automatic video interpretation, and continuous improvements. Moreover, we opt for using carefully designed prompts instead 1 Dept. Computer Science, Universitat Autònoma de Barcelona (UAB) 2 Computer Vision Center (CVC), UAB
Accepted for publication at the IEEE International Conference on Intelligent Transportation Systems (ITSC), 2026.
Fig. 1: Two-stage D-V2S framework for automatic scenario generation: from a driving video, the DRA module produces a natural-language description that is translated into a simulatable scenario by the SG module.
of VLM retraining, which can be costly in data and computation and requires coding. On the other hand, we decided to introduce a two-stage pipeline (Fig. 1) with a Driving Record Analysis (DRA) and Scenario Generation (SG) to enable natural-language editable outputs. Specifically, the DRA queries a VLM with our prompt to generate a naturallanguage description of the driving video (road layouts, traffic participants, and dynamic interactions); then, inspired by prior works [3], [4], the SG employs a context-conditioned large language model (LLM) to transform this description into an executable scenario, allowing future natural-language modifications to add new desired elements anytime. Overall, if we need a simulatable scenario from a video, D-V2S (DRA+SG) performs the transformation fully automatically. To conduct our research, we employ the SCENIC scripting language [2], chosen for its open-source availability, highlevel abstraction, support for randomization of scenarioparameter values (one code but multiple diverse simulations), and seamless integration with different simulators (MetaDrive, Webots, CARLA, etc.). Moreover, due to its widespread adoption and open-source nature, to run the scenarios generated by D-V2S, we use CARLA [5] as simulator. We assess the DRA by comparing three representative VLMs—LLaVA [6], Qwen-VL [7], and GPT-4o [8]—using a prompt designed to elicit natural-language descriptions of input videos, including simulation-based failure cases of a driving AI model. We evaluate them via objective semantic metrics and human judgment through elaborated surveys on the generated descriptions. GPT-4o clearly excels in accuracy and satisfaction, with statistical significance confirmed. The assessment of the SG compares it with two stateof-the-art methods, LCTGen [9] and ChatScene [4], both for scenario generation from natural-language descriptions. Generated scenarios are rendered in bird’s-eye view for comparison, using relevant descriptions of varying complexity, including some from the Crash Report Dataset [10]. Semantically, our method outperforms the others; its bird’seye view outputs were preferred by judges in 75% of cases
© 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
via a designed survey, with statistical significance confirmed. Finally, we assess D-V2S across testing videos presenting meaningful driving scenarios. Quantitatively, the generated scenarios preserve an average of 90% of relevant semantic elements from the original videos. We also showcase DV2S’s qualitative results, including the use of real-world videos from a consumer-grade camera. Hence, this paper makes the following contributions: 1) The use of a VLM with a properly designed prompt to analyze driving video records and generate descriptions of the video content in natural language. 2) The use of a LLM with a proper context-conditioned prompt that permits to translate these descriptions into executable scenarios without manual programming. 3) The use of semantic metrics and statistically validated human judgments to quantitatively demonstrate the semantic fidelity and executability of scenarios generated by D-V2S. Moreover, we also assess the performance of DRA and SG modules individually. In the remainder of the paper, Section II situates D-V2S within the state of the art, Section III details D-V2S, Section IV describes our experiments and results, and Section V presents key messages and future directions. II. R ELATED W ORKS Simulating realistic and diverse scenarios is a cornerstone of AV development. Simulators such as AirSim [11], CARLA [5], LGSVL [12], and SUMO [13] support largescale data collection and systematic evaluation of scenarios. However, they use default hand-crafted traffic rules that often fail to capture real-world traffic complexity. SceneGen [14] introduced data-driven scenario generation, extended by TrafficGen [15], with works [16], [17] formalizing extraction and synthesis from large datasets for fine-grained control. Hand-crafted rules and statistical models lack flexibility to define complex intentions and semantic relations. In contrast, LLMs enabled reasoning over natural language to generate simulatable scenarios with richer complexity. LCTGen [9] synthesizes traffic scenarios from natural language using a LLM-based interpreter for structured representations and a transformer-based generator for realistic multi-agent motions. ChatScene [4] builds scenarios from unstructured language via a SCENIC snippet database: it decomposes instructions into sub-descriptions (behavior, geometry, spawn positions), embeds and retrieves code fragments, and assembles executable SCENIC scripts to produce physically consistent scenarios. ScenicNL [18] translates crash reports into SCENIC scripts using compositional prompting, constrained decoding, and compiler-in-the-loop feedback. More recently, Text2Scenario [19] and Chat2Scenario [20] explore generating OpenSCENARIO [1] programs from language or structured inputs. Text2Scenario parses descriptions into logical forms assembled from predefined domain-specific language (DSL) fragments—reusable templates for elements like lane changes, accelerations, or yielding—producing deterministic executable scenarios. Chat2Scenario combines textual descriptions, criticality metrics, and trajectory datasets to
retrieve and assemble scenarios under predefined maneuver taxonomies. Other proposals translate videos into executable simulation code. Miao et al. [21] demonstrate converting dashcam collision clips to SCENIC scripts using a VLM. LEADE [22] extracts abstract representations from traffic videos, producing instantiated scenario programs with fixed trajectories and timing. Road2Code [23] uses multi-object tracking to recover precise vehicle trajectories, translated into SCENIC programs and evaluated via visual similarity metrics. However, small perception errors propagate directly into the generated scenario, and even minor modifications require rewriting trajectories or regenerating scripts. Another emerging direction leverages multimodal LLMs for large-scale corner-case and environment generation. AutoScenario [24] synthesizes complete driving environments—road networks, agents, behaviors— from multimodal inputs, evaluating scenarios via textual similarity between input and output descriptions to explore diversity. While effective for diverse safety-critical cases, it prioritizes descriptive alignment and diversity over preserving specific event identity or causal structure, suiting corner-case exploration but not faithful replay or regression testing of observed events. On the other hand, recent advances in AVs perception integrate LLMs/VLMs with sensor data: CarLLaVA [25] combines VLMs and LLaMA [26] for driving tasks; SimpleLLM4AD [27] provides driving descriptions; Xie et al. [28] evaluate VLMs across tasks, noting strengths and limits; CurricuVLM [29] generates curricula via reinforcement learning. However, these target perception/policy rather than scenario generation. In this paper, we move toward a fully automated form of scenario generation by letting the input videos to drive the entire process. Unlike language-conditioned generators such as LCTGen, which synthesize multi-agent traffic motions from textual descriptions, or ChatScene and ScenicNL, which translate natural-language inputs into executable SCENIC scripts through code retrieval and compositional prompting, D-V2S does not require externally provided scenario descriptions. Similarly, in contrast to Text2Scenario and Chat2Scenario, which assemble OpenSCENARIO programs from predefined DSL fragments, maneuver taxonomies, or trajectory datasets, D-V2S does not commit early to fixed behavior templates or trajectory-level instantiations. Furthermore, in contrast to approaches such as AutoScenario, DV2S explicitly preserves the identity and causal interaction structure of the observed event, i.e., focuses on semantic consistency. By operating in this semantic reconstruction regime, D-V2S bridges the gap between language-driven generation, trajectory-driven replay, and diversity-oriented synthesis. It enables interpretable, executable scenario generation suitable for debugging, benchmarking, and regression testing, thereby advancing the state of the art in real-to-simulation scenario generation.
Fig. 2: DRA: it takes a video sequence and a prompt to output a natural-language description of the video content. This video shows footage from a rear-facing camera mounted on the ego vehicle. Why did the ego vehicle become blocked? Describe the behavior of the ego vehicle and the dynamics of the surrounding actors. Additionally, what type of road section is this—an intersection, a straight road, a highway, a T-intersection, or a three-way road? Provide an answer in three sentences.
The ego vehicle was blocked by a stationary digital signboard placed on the right lane, obstructing its path. The red car ahead of the ego vehicle quickly maneuvered around the signboard to continue on its way, while the ego vehicle remained stationary. This scene takes place on a straight road where the traffic dynamics cause a temporary halt in movement. Fig. 4: The VLM answer contains traffic participants/elements (red), their actions (green), and relationships (blue).
Fig. 3: Text used in the DRA module as VLM prompt. The generic component is highlighted in blue and the videospecific component in green, and the text emphasized in blue corresponds to the desired description of the output.
III. M ETHODOLOGY D-V2S converts videos from real or simulated driving into executable scenarios in the SCENIC scripting language, while also generating a natural-language description of the video content. It operates in two sequential stages (Fig. 1): the Driving Record Analysis (DRA) and the Scenario Generation (SG). This section details DRA and SG. A. Driving Record Analysis (DRA) By using a VLM, DRA takes a driving video as input and produces a natural-language description of it as output. VLMs have input restrictions that require video sampling and frame resizing pre-processing steps. Moreover, VLMs require a prompt (Fig. 2). Too open-ended prompts (e.g., Describe what happens in this video) often produced generic or irrelevant outputs lacking spatial or causal details. Thus, we refined the prompt design by including contextual cues such as camera viewpoint and driving context (e.g., This video shows footage from a rear-facing camera mounted on the ego vehicle), and explicit questions about the event of interest (e.g., Why did the ego vehicle become blocked?). These adjustments improve the consistency and relevance of descriptions. Hence, our prompts consist of a general component common to all videos—informs the VLM that the video depicts a traffic scenario and specifies relevant output information—and a video-specific one explaining its interest. Figure 3 shows an example. Then, the VLM extracts details such as traffic participants/elements, their features, actions, relationships, etc. Figure 4 shows a GPT-4o-based (VLM) description example. B. Scenario Generation (SG) SG takes the DRA output—a natural-language traffic scenario description (e.g., Fig. 4)—and converts it into an executable SCENIC script using an LLM.
Fig. 5: As input, SG takes a fixed context and a changing (dynamic input) video description in natural language. The fixed context consists of the rules to script scenarios in SCENIC and pairs of a video description in natural language and a SCENIC script. The dynamic input describes the specific scenario to be generated and comes from the DRA.
SCENIC is a high-level probabilistic programming language that declaratively specifies behaviors, spatial layouts, and environmental conditions for driving scenarios in simulators. A SCENIC script is organized as four self-contained sections that set: 1) The simulation map to use and global parameters such as the weather conditions or the ego-vehicle model. 2) The adversarial behavior that encodes how traffic participant nearby the ego-vehicle will act: lane change, sudden braking, etc. 3) The road geometry under test: straight lane, T-junction, four-way intersection, etc. 4) The spawn position for each traffic participant. This is the structure that we aim to obtain as output of the LLM. However, as SCENIC is a relatively recent domainspecific language, there is a scarcity of publicly available scripts. Consequently, leveraging LLMs to directly generate such scripts is challenging, since these models have either not encountered this programming syntax during training or have been exposed to too few relevant examples to effectively learn its structure. In practice, when we prompted LLMs to generate SCENIC script, they frequently hallucinated nonexistent language commands or omitted required components. Consequently, many of the generated scripts failed to compile, an issue also found in parallel studies on text-toSCENIC generation [4]. To address the data scarcity without fine-tuning the LLM, we employ a lightweight prompt-engineering approach since the given LLM is provided with a fixed context that em-
ulates how a teacher start to explain a new programming language to a student. In particular, simple, diverse, and meaningful scripting examples are provided, i.e., pairs of one human-language description of the desired scenario and the SCENIC-compilable script that generates it. Moreover, the SCENIC scripting rules (i.e., the programming manual) are provided too as part of the contextual knowledge. The rules that we provide also include generic scripting pitfalls to avoid that we have collected during the development of the SG. Note that many interfaces with LLMs allow to set this kind of common information as part of a working space so that only what changes (dynamic input) must be provided each time the LLM is prompted. Therefore, our approach follows the idea of in-context learning [30] by guiding LLMs to generate SCENIC scripts that compile successfully, despite the languages limited presence in the training data (Fig. 5). IV. E XPERIMENTS We start by assessing the performance of DRA and SG individually: the former compares different VLMs for driving scene interpretation; the latter measures how well SCENIC scenarios reproduce and generalize DRA’s natural-language descriptions. Then, we assess the D-V2S pipeline (DRA + SG), quantifying original video semantics preserved in final executable scenarios. In addition, we show qualitative results.
Fig. 6: Examples of driving failures. Up row: running over pedestrians. Bottom row: vehicle crashes. TABLE I: Semantic Evaluation Metrics: Semantic Coverage Score (SCS), Hallucination Rate (HR), Semantic Preservation (SP), End-to-End Semantic Consistency (E2E-SC). Metric
Definition
SCS
|GT ∩S| |GT |
HR
1−
SP
|Sde ∩Ssc | |Sde |
E2E-SC
|GTv ∩So | |GTv |
|GT ∩S| |S|
A. Driving Record Analysis (DRA) 1) VLM: Given the available VLMs, we compared three competitive ones [28]: LLaVA [6] (LLaVA-Video-72BQwen2), Qwen-VL [7] (Qwen2-VL-72B-Instruct), and GPT4o [8] (gpt-4o via OpenAI API). LLaVA excels in opensource visual-linguistic reasoning, Qwen-VL in fine-grained visual perception, and GPT-4o represents the commercial frontier in multimodal reasoning. 2) Multimodal Query: To input video information to each VLM within its per-request image limit, we sample frames at fixed frequency and time-limit while preserving temporal coverage. These frames are presented in temporal order with the textual prompt (e.g., Fig. 3), forming a multimodal query. 3) Scenarios: While developing AI models for AVs, we iterate through training and validation cycles. Failureexposing scenarios are valuable for guiding subsequent improvements. In simulation, these arise during random or goaloriented validations, making variations of failure-causing traffic events useful for retraining and regression testing.We follow this approach for validating the selected VLMs. 4) AI Model: To perform driving tasks, we selected CIL++, a fast, well-performing vision-based end-to-end driving model available on GitHub [31]. CIL++ was trained on CARLA maps but, to our knowledge, not submitted to the CARLA Leaderboard challenge [32]. 5) Videos: Thus, we assessed CIL++ on the CARLA Leaderboard, recording videos of driving failures— specifically, seconds prior to each failure to capture preceding situations (Fig. 6). These videos were recorded from a camera behind and slightly above the ego-vehicle (driven by CIL++).
Interpretation Recall of ground-truth video elements (GT ) in the generated description (S). Fraction of generated elements (S) not in the ground-truth (GT ). Recall of semantic elements in the natural-language description (Sde ) preserved in the simulated scenario (Ssc ). Recall of ground-truth video elements (GTv ) preserved in the simulated output (So ) after D-V2S execution.
6) Experimental Setup and Evaluation Metrics: Each recorded video was processed with the three selected VLMs, yielding three descriptions per scenario. For objective semantic extraction assessment, we annotated ground-truth semantic elements per video—covering traffic participants, road topology, and actions—from which we extracted predicted semantic sets from each VLM description to compute Semantic Coverage Score (SCS) and Hallucination Rate (HR) (Table I). These metrics are deterministic and computed directly from annotated semantic sets; therefore, no hypothesis testing is required for their interpretation. Higher SCS and lower Hallucination Rate indicate better semantic grounding. This quantifies how completely and accurately each VLM captures scene structure. In addition to semantic evaluation, we conducted human evaluation with questions like those in Fig. 7. Highly experienced drivers assessed each scenario by selecting the best description of video content (Preference) and rating each answer’s adequacy on a 1–5 scale (Adequacy). They were not informed of which VLM generated each answer. To mitigate location bias, VLM answer order was randomized per question. We prepared 62 questions for distinct scenarios, randomly split into two fixed surveys; they could complete one/both online (no time limit) and skip questions. In total, we collected 360 responses across surveys.
TABLE III: VLM evaluation by the surveyed. Preference Model Qwen-VL LLaVA GPT-4o
Adequacy
Preference (%) 95% CI (%) Mean ± St.Dev. 19.17 22.22 58.61
15.43–23.55 18.23–26.80 53.46–63.58
2.53 ± 1.42 2.61 ± 1.35 3.58 ± 1.26
95% CI [2.38, 2.68] [2.47, 2.75] [3.45, 3.71]
Preference: Chi-square test: χ2 (2, N = 360) = 104.017, p = 2.589 × 10−23 , Cohen’s w = 0.538. z-test (p-value): Qwen-VL vs LLaVA: p = 0.961; Qwen-VL vs GPT-4o: p = 7.98 × 10−25 ; LLaVA vs GPT-4o: p = 6.36 × 10−21 . Adequacy: One-way ANOVA: F (2, 1077) = 67.8, p = 1.8 × 10−28 , η 2 = 0.112. Post-hoc Tukey test (p-value) / Cohen’s d: Qwen-VL vs LLaVA: p = 0.73, d = −0.06; Qwen-VL vs GPT-4o: p < 0.001, d = −0.78; LLaVA vs GPT-4o: p < 0.001, d = −0.75.
Fig. 7: Example of different answers (blue boxes) produced by the three selected VLMs, given a common prompt (yellow box) and recorded video (illustrative frame is shown here). TABLE II: VLM semantic description evaluation. Model
SCS ↑
HR ↓
LLaVA GPT-4o Qwen
0.70 ± 0.35 0.91 ± 0.18 0.69 ± 0.29
0.20 ± 0.23 0.06 ± 0.17 0.15 ± 0.22
To assess statistical significance in human judgment we compute 95% confidence intervals (CI) for true means, Chi-square tests for preference distribution differences, twoproportion z-tests for pairwise model comparisons, one-way ANOVA for adequacy score differences, Tukey post-hoc tests for specific significant pairs, and Cohen’s d for effect size (in standard deviations). These statistics are applied to both Preference and Adequacy answers. 7) Results and Discussion: Table II shows GPT-4o achieving the highest SCS (0.91), indicating nearly all elements from the annotated ground truth were detected, and with the lowest HR (0.06), indicating minimal mention to unsupported elements. Table III shows the preferences across the three VLMs: GPT-4o was preferred in 58.6% of cases (95% CI: 53.46–63.58%), Qwen-VL in 19.2%, and LLaVA in 22.2%. The CIs indicate where the true preference proportions likely lie; GPT-4o’s substantial lead is statistically significant. A chi-square test confirmed deviation from equal preference (χ2 (2, N = 360) = 104.0, p < 10−22 , Cohen’s w = 0.54). Two-proportion z-tests showed GPT-4o significantly outperforming Qwen-VL (p < 10−24 ) and LLaVA (p < 10−20 ). Table III also reports adequacy scores: GPT-4o achieved the highest mean (3.58 ± 1.26), while Qwen-VL and LLaVA received lower scores. One-way
Fig. 8: Given the description in the yellow box, we show the bird-eye-view of the generated scenarios from SG (our method), ChatScene [4] and LGTGen [9].
ANOVA confirmed significant differences among models (F (2, 1077) = 67.8, p = 1.8 × 10−28 , η 2 = 0.112). Tukey post-hoc tests showed GPT-4o rated significantly higher than both others (p < 0.001). Cohen’s d also indicated large differences of GPT-4o with the others (|d| ≈ 0.75). Considering these results, we believe that semantic metrics are consistent with human evaluation. The model achieving the highest SCS and lowest HR is also the one most strongly preferred and rated as most adequate by participants. This suggests that semantic grounding is a key determinant of perceived description quality. Overall, with statistical significance, GPT-4o is the preferred and most adequate VLM. Although the official size of GPT-4o is not publicly disclosed, it is estimated to be approximately three times larger than LLaVA and Qwen2-VL, which may explain these results. Thus, semantic metrics (SCS and HR) and VLM size serve as selection criteria for integrating future VLM models into our DRA module, ensuring alignment with human judgment. Consequently, GPT-4o is selected as the VLM for DRA in D-V2S. This matters because—though beyond this paper’s scope—improved human comprehension of descriptions enables automotive engineers to perform targeted manual modifications (via natural language) for scenario variants absent from the original video. Note that this flexibility is precisely why we designed D-V2S as a
1. The ego car is on the highway and the car in front of it suddenly decelerates.
TABLE V: Scenario generation evaluation by the surveyed.
2. The ego car that is on the left lane switches to the middle lane and another
Preference
car on the right lane switches also to the middle lane. 3. Three pedestrians crossing in the middle of the road, one going from left to right and the two others from right to left.
Fig. 9: Examples of the descriptions used as input to the SG. TABLE IV: Description-to-scenario generation evaluation. Method
SP↑
LCTGen ChatScene SG (Ours)
0.63 ± 0.35 0.53 ± 0.31 0.93 ± 0.11
two-stage pipeline with natural-language descriptions as the intermediate output.
LCTGen ChatScene Our
18.90 6.10 75.00
95% CI
13.65–25.58 1.90 ± 1.51 [1.67, 2.13] 3.35–10.86 1.82 ± 1.15 [1.64, 2.00] 67.85–81.00 4.04 ± 1.29 [3.84, 4.24]
Preference: Chi-square test: χ2 (2, N = 164) = 132.16, p = 2.0 × 10−29 , Cohen’s w = 0.90. z-test (p-value): LCTGen vs ChatScene: p = 0.001; LCTGen vs Our: p = 7.4 × 10−24 ; ChatScene vs Our: p = 1.6 × 10−36 . Adequacy: One-way ANOVA: F (2, 489) = 148.6, p = 3.8 × 10−51 , η 2 = 0.378. Post-hoc Tukey test (p-value) / Cohen’s d: LCTGen vs ChatScene: p = 0.85, d = 0.06; LCTGen vs Our: p < 0.001, d = −1.52; ChatScene vs Our: p < 0.001, d = −1.82.
TABLE VI: D-V2S Semantic Consistency.
B. Scenario Generation (SG) 1) SOTA: We consider LCTGen [9] and ChatScene [4], which translate natural-language descriptions into scenarios. LCTGen uses GPT-4 to create structured vectors, retrieve road fragments, and render traffic via transformers. ChatScene parses descriptions into behavior/geometry/spawn clauses, matches SCENIC snippets from a database, and assembles executable scripts. For our SG, we used GPT4o [8] as the LLM. Since the different outputs are not directly comparable, we rendered each as bird’s-eye views of dynamic participants’ trajectories (Fig. 8). 2) Descriptions: To evaluate SG independently of DRA, we compiled 23 scenario descriptions (e.g., Fig. 9): 15 with increasing complexity (single-vehicle to multivehicle/pedestrian) and 8 adapted from the Crash Report Dataset [10]. 3) Experimental Setup and Evaluation Metrics: SG evaluation also uses semantic metrics and human judgment with statistical validation. We employ Semantic Preservation (SP) (Table I), which quantifies the proportion of relevant semantic structures faithfully reproduced during scenario synthesis—higher values indicating stronger fidelity between natural-language descriptions and generated scenarios. To compute SP, we define semantic elements Sde (traffic participants, spatial relations, road topology, actions) from each description and extract generated elements Ssc from the simulated scenario. Additionally, we assessed perceived human alignment between descriptions and generated scenarios based on experienced drivers. We used a survey of 23 questions each time presented in random order and participants could stop anytime (no minimum answers). Scenario generation methods (Fig. 8) were anonymized and displayed in randomized positions to avoid biases. Each question required selecting the best-matching scenario (Preference) and rating each scenario’s adequacy on a 1–5 scale (Adequacy). We collected 164 responses. 4) Results and Discussion: Table IV shows that SG achieves a substantially higher SP (0.93) than LCTGen
Adequacy
Preference (%) 95% CI (%) Mean ± SD
Model
Use case
E2E-SC ↑
Collision risk avoidance on highway Insertion on highway Pedestrian crossing in urban area Left Turn at urban intersection Pull back in on urban highway
0.86 ± 0.24 0.93 ± 0.24 0.80 ± 0.29 0.93 ± 0.18 0.95 ± 0.16
Overall
0.90 ± 0.21
and ChatScene. SG preserves substantially more semantic structure than competing methods. Table V shows the preferences among surveyed: SG led clearly at 75.0%, followed by LCTGen (18.9%) and ChatScene (6.1%). The 95% CI confirms these differences are meaningful—e.g., SG’s CI (67.9–81.0%) well exceeds the others. A chi-square test revealed highly significant deviation from equal preference (χ2 (2, N = 164) = 132.2, p < 10−29 , Cohen’s w = 0.90). Two-proportion z-tests confirmed SG significantly outperformed LCTGen (p < 10−23 ) and ChatScene (p < 10−35 ). Table V also reports adequacy scores: SG received the highest (4.04±1.29), far exceeding LCTGen (1.90±1.51) and ChatScene (1.82 ± 1.15). One-way ANOVA confirmed significant differences (F (2, 489) = 148.6, p < 10−51 , η 2 = 0.378). Tukey tests showed that SG rates significantly higher than competitors (p < 0.001) and Cohen’s d confirmed large differences of SG with them (|d| > 1.5). Hence, SG outperforms LCTGen and ChatScene both semantically and according to human judgment. For instance, Fig. 8 shows that SG successfully generated a realistic multivehicle interaction: vehicles are placed respecting the spatial constraints as stated in the natural-language description, and reproduced the described dynamics with fidelity. C. D-V2S (DRA + SG) 1) Experimental Setup and Evaluation Metrics: Direct comparisons with related methods were not feasible due to the absence of reproducible implementations or complete evaluation resources in existing works [21]–[24]. To support reproducibility, we publicly release the complete D-V2S
TABLE VII: Failure taxonomy of non-executable scripts. Category Undefined symbol API hallucination Function misuse Geometric infeasibility
Description Code variable referenced but not declared Non-existent SCENIC attribute/method Missing required function parameter Invalid spawn or spatial configuration
implementation1 . We have considered the five use cases identified as especially relevant in the BERTHA Project [33], with corresponding generated videos. The use cases are: (1) collision risk avoidance on highway, (2) insertion on highway, (3) pedestrian crossing in urban area, (4) left turn at urban intersection with oncoming cars, and (5) pull back in on urban highway. For each use case we have 22 CARLA-generated videos, created by varying parameters such as vehicle speeds, spawn points for vehicles/pedestrians, and weather conditions. From these videos, D-V2S generates corresponding SCENIC scripts, which are executed in CARLA to produce output videos with a similar forwardfacing horizontal FOV as the inputs. We annotated the relevant semantic elements in both input and output videos, enabling quantitative assessment of D-V2S. Specifically, we introduce the End-to-End Semantic Consistency (E2ESC) metric (Table I), which directly compares the relevant semantic elements of input (GTv ) and corresponding output (So ) videos. 2) Results and Discussion: Table VI reports E2E-SC scores across evaluated use cases. For all of them, E2ESC is higher than 80%, with an overall value of 90%. Results show that the observed CARLA-based simulations keep a strong semantic fidelity with the input videos. Moreover, this implies that the semantic information obtained by the DRA module is effectively operationalized by the SG module across essentially diverse scenarios. D-V2S took 19.08 seconds on average to convert each input video into a SCENIC script. On the other hand, we have also analyzed failure cases. The 94% of generated SCENIC scripts compiled and executed successfully without modification. The remaining 6% had minor syntactic inconsistencies or missing parameters (see Table VII), not structural semantic errors. Failures stemmed from code formatting rather than scene misinterpretation. Qualitative results are shown in figures 10 and 11, the latter demonstrating transferability to realistic conditions using windshield-mounted GoPro footage. The third-person camera perspective is used only for interpretability, not as a framework requirement. Together, our quantitative and qualitative results demonstrate that D-V2S represents a significant advance in automatically generating semantically faithful simulation scenarios from videos, accurately reproducing the relevant content of the original footage while enabling variants via naturallanguage descriptions (output of DRA) or SCENIC script parameter value modifications (output of SG). 1 Code available at https://alexandre-levy.github.io/DV2S.github.io/.
Fig. 10: D-V2S transforms a third-person perspective video from CARLA, captured during goal-oriented autonomous driving (top-left), into a corresponding SCENIC script. The overlaid green text is the scenario-specific user question (green prompt part, Fig. 3). The top-right shows a CARLAgenerated video obtained by executing this D-V2S-generated SCENIC script using an onboard camera perspective. Bottom frames depict variants of the original scenario, generated by modifying SCENIC script parameter values such as weather conditions and specific traffic actors within meaningful intervals (which can be randomly done or as an user requirement).
Fig. 11: Left column: GoPro frames (windshield-mounted). D-V2S generates a SCENIC script from this video and prompt (green, top image). Right column: executing the script in CARLA reproduces a semantically similar scenario, using three forward-facing cameras to match GoPro’s fisheye field-of-view. Rows 1-2 show the same simulation; rows 3-4 show variants with different illumination/weather, which is straightforward to do once the SCENIC script exists.
V. C ONCLUSIONS This paper presented D-V2S, a novel two-stage framework that automatically transforms driving videos—real or simulated—into executable SCENIC scenarios. By leveraging off-the-shelf VLMs and LLMs with carefully designed prompts—no retraining required—D-V2S preserves an average of 90% of relevant semantic elements while enabling editable outputs. Ablation studies confirm GPT-4o’s superiority in DRA (semantic accuracy and human preference) and D-V2S’s SG outperforming LCTGen and ChatScene (75% preference rate). These results demonstrate D-V2S’s effectiveness for scalable generation of semantically faithful scenarios—preserving key events and interactions among participants—advancing interpretable testing for autonomous vehicles. Future work will integrate D-V2S in AI driving development loops and address current failure cases. ACKNOWLEDGMENTS Funded by the European Union under Grant Agreement 101076360 (BERTHA). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Climate, Infrastructure and Environment Executive Agency (CINEA). Neither the European Union nor the granting authority can be held responsible for them. BERTHA was particularly relevant for the definition of the driving scenarios considered in this work. A.M. López acknowledges financial support for his general research activities from ICREA under the ICREA Academia Program. All authors also acknowledge the support of the Generalitat de Catalunya through the CERCA Program and its ACCIÓ Agency for CVCs general activities. Authors acknowledge fruitful discussions on: a) foundation models carried out within ELLIOT (Funded by the European Union under Grant Agreement 101214398) and b) explainability for autonomous driving within the HAMILTON project (PID2024-157936NBI00), funded by MICIU/AEI/10.13039/501100011033 and by ERDF, EU. R EFERENCES [1] ASAM e.V., ASAM OpenSCENARIO 2.0.0, 2021. [2] D. J. Fremont, T. Dreossi, S. Ghosh, X. Yue, A. L. SangiovanniVincentelli, and S. A. Seshia, “SCENIC: a language for scenario specification and scene generation,” in ACM SIGPLAN Conf. on Programming Language Design and Implementation (PLDI), 2019. [3] S. Tang, Z. Zhang, J. Zhou, L. Lei, Y. Zhou, and Y. Xue, “LeGEND: A top-down approach to scenario generation of autonomous driving systems assisted by large language models,” in Inter. Conf. on Automated Software Engineering (ASE), 2024. [4] J. Zhang, C. Xu, and B. Li, “ChatScene: Knowledge-enabled safetycritical scenario generation for autonomous vehicles,” in Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2024. [5] A. Dosovitskiy, G. Ros, F. Codevilla, A. López, and V. Koltun, “CARLA: An open urban driving simulator,” in Conf. on Robot Learning (CoRL), 2017. [6] H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual instruction tuning,” in Conf. on Neural Information Processing Systems (NeurIPS), 2023. [7] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan et al., “Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond,” arXiv, 2023. [8] OpenAI, “GPT-4 technical report,” arXiv, 2023. [9] S. Tan, B. Ivanovic, X. Weng, M. Pavone, and P. Kraehenbuehl, “Language conditioned traffic generation,” in Conf. on Robot Learning (CoRL), 2023.
[10] NHTSA, Crash Injury Research Engineering Network (CIREN), https: //crashviewer.nhtsa.dot.gov/CIREN/SearchIndex, 2016. [11] S. Shah, D. Dey, C. Lovett, and A. Kapoor, “AirSim: High-fidelity visual and physical simulation for autonomous vehicles,” in Field and Service Robotics (FSR), 2018. [12] G. Rong, B. H. Shin, H. Tabatabaee, Q. Lu, S. Lemke et al., “LGSVL simulator: A high fidelity simulator for autonomous driving,” in Intelligent Transportation Systems Conference (ITSC), 2020. [13] P. Lopez, E. Wiessner, M. Behrisch, L. Bieker-Walz, J. Erdmann et al., “Microscopic traffic simulation using SUMO,” in Intelligent Transportation Systems Conference (ITSC), 2018. [14] S. Tan, K. Wong, S. Wang, S. Manivasagam, M. Ren, and R. Urtasun, “SceneGen: Learning to generate realistic traffic scenes,” in Int. Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. [15] L. Feng, Q. Li, Z. Peng, S. Tan, and B. Zhou, “TrafficGen: Learning to generate diverse and realistic traffic scenarios,” in Inter. Conf. on Robotics and Automation (ICRA), 2023. [16] Z. Ghodsi, S. K. S. Hari, I. Frosio, T. Tsai, A. Troccoli et al., “Generating and characterizing scenarios for safety testing of autonomous vehicles,” IEEE Trans. on Intelligent Transportation Systems, 2021. [17] W. Ding, C. Xu, M. Arief, H. Lin, B. Li, and D. Zhao, “A survey on safety-critical driving scenario generationa methodological perspective,” IEEE Trans. on Intelligent Transportation Systems, 2023. [18] K. Elmaaroufi, D. Shanker, A. Cismaru, M. Vazquez-Chanlatte, A. Sangiovanni-Vincentelli et al., “ScenicNL: Generating probabilistic scenario programs from natural language,” in Conference on Language Modeling (COLM), 2024. [19] X. Cai, X. Bai, Z. Cui, D. Xie, D. Fu et al., “Text2scenario: Textdriven scenario generation for autonomous driving test,” Automotive Innovation, 2026. [20] Y. Zhao, W. Xiao, T. Mihalj, J. Hu, and A. Eichberger, “Chat2scenario: Scenario extraction from dataset through utilization of large language model,” in Intelligent Vehicles Symposium (IV), 2024. [21] Y. Miao, G. Fainekos, B. Hoxha, H. Okamoto, D. Prokhorov, and S. Mitra, “From dashcam videos to driving simulations: Stress testing automated vehicles against rare events,” arXiv, 2025. [22] H. Tian, X. Han, Y. Zhou, G. Wu, A. Guo et al., “LMM-enhanced safety-critical scenario generation for autonomous driving system testing from non-accident traffic videos,” arXiv, 2026. [23] J. Leung, G. Tong, P. S. Duggirala, and P. Chakravarthula, “From road to code: Neuro-symbolic program synthesis for autonomous driving scene translation and analysis,” in Inter. Conf. on Neuro-symbolic Systems, 2025. [24] Q. Lu, M. Ma, Z. Wang, W. Lu, M. Ma et al., “Realistic corner case generation for autonomous vehicles with multimodal large language model,” Tsinghua Science and Technology, 2026. [25] K. Renz, L. Chen, A.-M. Marcu, J. Hnermann, B. Hanotte et al., “CarLLaVA: Vision language models for camera-only closed-loop driving,” 2024. [26] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux et al., “LLaMA: Open and efficient foundation language models,” arXiv, 2023. [27] P. Zheng, Y. Zhao, Z. Gong, H. Zhu, and S. Wu, “SimpleLLM4AD: An end-to-end vision-language model with graph visual question answering for autonomous driving,” in Inter. Conf. on Internet, Education and Information Technology (IEIT), 2024. [28] S. Xie, L. Kong, Y. Dong, C. Sima, W. Zhang et al., “Are VLMs ready for autonomous driving? an empirical study from the reliability, data, and metric perspectives,” in Inter. Conf. on Computer Vision (ICCV), 2025. [29] Z. Sheng, Z. Huang, Y. Qu, Y. Leng, S. Bhavanam, and S. Chen, “CurricuVLM: Towards safe autonomous driving via personalized safetycritical curriculum learning with vision-language models,” arXiv, 2025. [30] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan et al., “Language models are few-shot learners,” in Conf. on Neural Information Processing Systems (NeurIPS), 2020. [31] Y. Xiao, F. Codevilla, D. Porres, and A. M. López, “Scaling visionbased end-to-end autonomous driving with multi-view attention learning,” in Inter. Conf. on Intelligent Robots and Systems (IROS), 2023. [32] CARLA, Autonomous Driving Leaderboard, https://leaderboard.carla. org/, 2020. [33] BERTHA Consortium, “Deliverable 1.1. use cases for the identification of the model,” https://berthaproject.eu/wp-content/uploads/2025/ 02/Bertha WP1 D1.1 M6.pdf, 2024, accessed: 2025.