A knowledge-augmented dataset of high-risk driving scenarios with LLM annotations for autonomous driving Heye Huang1 , Jingguang Li1 , Zhiyuan Zhou2,* , Paul Liang3 , Mingyu Wu4 , Kitae Jang1 , and Jianqiang Wang5,* 1 Cho Chun Shik Graduate School of Mobility, Korea Advanced Institute of Science and Technology, Daejeon 34051, South Korea 2 College of Computer Science and Artificial Intelligence, Fudan University, Shanghai 200433, China 3 Media Lab, Massachusetts Institute of Technology, Cambridge, MA 02139, USA 4 School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China 5 School of Vehicle and Mobility, Tsinghua University, Beijing 100084, China * Correspondence ([email protected], [email protected]).
arXiv:2607.07103v1 [cs.LG] 8 Jul 2026
ABSTRACT Safe autonomous driving requires both a rapid response to common high-risk events and deeper reasoning over the rare, extreme long-tail scenarios of traffic safety. These scenarios are severely under-represented in naturalistic driving data, and existing trajectory and language-augmented datasets seldom provide high-risk event labels, semantic annotations and verifiable safety signals. Here we present K-Risk, a knowledge-augmented dataset that combines structured driving trajectories with large language model (LLM)-generated semantic annotations for safety-critical driving scenarios. K-Risk integrates 20 human-driven and autonomous-vehicle trajectory datasets from Europe, China, and the United States, covering highways, urban freeways, intersections and roundabouts. Using a unified risk-centric extraction pipeline, K-Risk curates 31,398 highrisk events, together with a 1,036-event extreme subset of near-collision cases. Each event is released as a synchronized trajectory–metadata–language triplet containing structured scenario descriptions, abnormal-behavior notifications, and, for a representative subset, LLM-generated causal risk analyses and action recommendations validated through a closed-loop simulator with iterative reflection. By combining multi-dimensional risk annotations, interpretable language supervision, and verifiable decisions, K-Risk bridges structured traffic trajectories, semantic reasoning and verifiable decision supervision, providing a standardized foundation for developing and evaluating next-generation risk-aware autonomous driving agents.
Background & Summary Autonomous driving has progressed rapidly from partial to conditional automation, yet safety in rare and complex long-tail scenarios remains the central barrier to large-scale deployment and on-road testing1 . A safe driving agent must combine two capabilities that draw on different kinds of experience: a fast and reliable response to the common high-risk events encountered every day, and a slower, deeper form of reasoning over the rare and extreme situations that sit in the long tail of traffic safety. The second capability is harder to acquire, because the corresponding scenarios occur infrequently and are therefore severely under-represented in the naturalistic driving data used to train and evaluate driving models. Reliable behavior in these high-risk edge cases is decisive not only for technical performance but also for public trust and regulatory approval2 . Agents based on large language models (LLMs) have recently emerged as a promising route to this second capability. By combining structured driving knowledge, learned experience and semantic understanding, they offer a interpretable reasoning process that interprets a scenario, reflects on prior cases and generalizes across situations. They can also be coupled with physics-based models to yield systems that are at once safer and more interpretable?, 4 . Realizing this potential, however, depends on training and evaluation data that pair real high-risk trajectories with the semantic, interpretable and verifiable signals these agents consume, and such data remain scarce. Naturalistic trajectory and perception datasets such as highD5 , inD6 , Waymo7 , nuScenes8 and Argoverse9 have substantially advanced perception, tracking and trajectory forecasting by providing precise trajectories, multimodal sensor streams and standardized benchmarks. Their records nonetheless capture mostly routine driving in structured environments, so that genuine edge cases and high-risk interactions make up only a small fraction of the data. This composition limits their value for training and evaluating autonomous driving systems under safety-critical conditions. Datasets that focus on interaction narrow this gap only partially. OnSiteVRU10 emphasizes dense urban scenarios with vulnerable road users, yet the most unpredictable behaviors, such as sudden pedestrian dart-outs, remain underrepresented; INTERACTION11 covers roundabouts, merges and near-collision
Frequency
events with semantic maps; and InterHub12 formalizes dense multi-agent interaction through the post-encroachment time and the minimum sum of absolute acceleration. Across these datasets, risk is still expressed mainly through one-dimensional proximity metrics such as time-to-collision or time headway, which oversimplify danger and fail to reflect how human drivers read a situation through multi-agent dynamics, intent and context13 . A second line of datasets adds semantic, natural language annotation to support the higher-level reasoning that LLMbased agents require. CoVLA14 pairs real-world driving video with detailed descriptions of environments and maneuvers; OmniDrive15 provides holistic vision-language annotations and counterfactual question answering; DriveLM16 structures perception, prediction and planning as graph-based question answering; and CODA-LM17 targets corner-case understanding through a categorical taxonomy. More recent efforts move toward risk and closed-loop evaluation: NuRisk18 builds an agentlevel risk visual-question-answering benchmark with quantitative risk scores over bird’s-eye-view sequences, and Impromptu VLA19 distills unstructured corner-case clips for vision-language-action fine-tuning. These datasets are rich in language, yet they center on routine driving, annotate risk coarsely, and rarely expose the causal risk analyses or verifiable safety signals needed to train and evaluate risk-aware agents14, 15 . Taken together, these datasets leave four gaps that limit the development of risk-aware driving agents, as summarized in Figure 1. First, safety-critical events are inherently rare and are seldom annotated with a consistent, multi-dimensional definition of risk. Second, existing datasets generally lack event-level semantic annotations that explain what occurred, why the situation became hazardous, and how the risk evolved over time. Third, although large language models have shown remarkable capability in reasoning and decision making, few driving datasets provide structured, machine-readable natural language descriptions that can be directly used for language-based learning or evaluation. Finally, existing datasets rarely provide verifiable decision signals that enable recommended actions to be validated in a closed-loop manner. Normal driving events
Safety-critical long-tail events
(~99.97% of recorded events)
(~< 0.03% of events, yet they dominate real-world risk)
Cut-in
Normal Cases
Camera
nuScenes Camera
NAVSIM Camera
LiDAR
LiDAR
LiDAR
Radar
Trajectory
Radar
Trajectory
knowledge-augmented, multi-source, LLM-annotated
CitySim ✓
WHAT / WHO: no event-level risk objects (ego, VRUs)
✓ ✓
WHAT RISK: risk reduced to a single threshold (TTC only)
✓ ✓
Risk: multi-dimensional risk (cut-in, hard brake, VRU conflict) Intent: agent intent & behavior modeling Context: lane topology, road geometry, surrounding agents Reasoning: LLM natural-language risk analysis Decision: human-like recommended action
WHEN / WHERE: no peak-risk frame, road-context semantics WHY: no natural-language reasoning, intent, or decision
！
Roundabout conflict
20 source datasets, 30k+ high-risk events, 3 continents
NGSIM
Waymo Camera
TTC < 2 s
K-Risk (Ours)
Normal cases-heavy, perception/trajectory only
LiDAR
VRU conflict
Edge cases (rare/high-risk/extreme…)
Existing AD Datasets
KITTI
Hard brake
Gap: normal cases-heavy, structured data
High risk data records with LLM
Scenario Descriptions
Behavior Notifications
Risk Analysis
K-Risk: a knowledge-augmented and event-level dataset with LLM-explained high-risk scenarios
Figure 1. Motivation for K-Risk. Naturalistic driving data are dominated by routine cases, whereas safety-critical long-tail events such as cut-ins, hard braking, VRU conflicts, low-TTC situations and roundabout conflicts are rare but important for autonomous driving safety. Existing datasets mainly provide perception or trajectory records for normal driving and often lack event-level risk objects, multi-dimensional risk labels, peak-risk context and natural language reasoning. K-Risk addresses these gaps by integrating 20 trajectory sources across three continents into more than 30,000 high-risk events, each paired with structured scenario descriptions, behavior notifications and, for a representative subset, LLM-generated risk analyses. To address these limitations, we introduce K-Risk, a knowledge-augmented dataset of high-risk driving scenarios that links structured vehicle trajectories with event-level semantic annotations, as illustrated in Figure 2. K-Risk is constructed from 20 trajectory sources that combine human-driven-vehicle (HV) and autonomous-vehicle (AV) data across Europe, China, and 2/22
a. Multi-Source Data Collection
b. Risk-Centric Event Extraction
highD inD/rounD
EU
US
CN
Moderate 76.8%
CitySim NGSIM
High 19.7%
Waymo Argoverse 2
Extreme 3.5%
CATS
20 sources
Highway
HV + AV
Urban freeway
OpenACC
Intersection
Raw data records
DRF top 10%
Hard maneuvers
TTC threshold
31,398 events in total
Roundabout
1,036 extreme subset (near-collision)
K-RISK c. LLM Semantic Annotation CSV trajectory JSON metadata TEXT annotation
Keep
Turn Left
d. Closed-Loop Validation LLM decision
Scenario description Behavior alerts Causal risk analysis Action recommendation
Turn Right
Accelerate
Decelerate
Memory pool Memory
Reflection
Collision rate 6%
Collision-free simulator
expert review
4.58%
4%
2.67% 1.91%
2% 0%
Trial 1
Trial 2
Trial 3
Downstream tasks
Post-training support
Risk classification · Intent prediction · Decision making
CPT · SFT · RLHF · RLVR
Figure 2. Overview of the K-Risk dataset and the LLM annotation framework. (a) Multi-source curation aggregates 20 human-driven (HV) and autonomous-vehicle (AV) trajectory sources across the United States, Europe and China, spanning highways, urban freeways, intersections and roundabouts. (b) Risk-centric event extraction screens the raw records through a driver risk field filter, calibrated hard-maneuver thresholds and a TTC and trajectory-conflict test, yielding 31,398 events graded as moderate, high or extreme, with a 1,036-event extreme near-collision subset. (c) Semantic annotation converts each event into synchronized CSV, JSON and text records carrying a scenario description and behavior notifications, with an LLM causal risk analysis and an action recommendation drawn from a five-action schema added for a representative subset. (d) Closed-loop validation checks each recommended action against a collision-free simulator with reflection, supporting downstream tasks and the post-training stages CPT, SFT, RLHF and RLVR. the United States. These sources cover highways, urban freeways, intersections, and roundabouts, allowing the dataset to reflect diverse traffic rules, road geometries, and driving conventions. The six HV sources are highD, inD, and rounD from the LevelXData collection, ExpresswayA and FreewayB from CitySim, and the I-80 dataset from NGSIM. The fourteen AV sources are the Argoverse 2 Motion Forecasting, Waymo Open Motion and Waymo Open Perception datasets, the MicroSimACC dataset, the CATS ACC, CATS Platoon and CATS UWM datasets, the Central Ohio single-vehicle and two-vehicle datasets, the OpenACC Casale, Vicolungo, AstaZero and ZalaZONE datasets, and the Vanderbilt ACC experiments. From these sources, K-Risk curates 31,398 high-risk events, including 1,036 extreme near-collision cases. Events are identified using a unified protocol that combines a driver risk field filter, calibrated behavioral thresholds, and a two-second trajectory-conflict predictor. Each event is enriched with structured scenario descriptions and abnormal-behavior notifications, while a representative subset further includes LLM-generated causal risk analyses and discrete-action recommendations validated in collision-free simulation. Stored as synchronized trajectory data, metadata, and natural language annotations, K-Risk provides risk labels, semantic context, and verifiable safety signals. Key contributions are as follows: • We introduce K-Risk, a large-scale knowledge-augmented dataset of high-risk driving scenarios built from 20 public trajectory datasets across three continents. K-Risk contains 31,398 curated high-risk events, including a 1,036-event extreme near-collision subset. • We present a unified risk annotation framework that combines physical risk modeling with LLM-based semantic reasoning, producing synchronized trajectory, metadata and natural-language representations for safety-critical driving. • We release a standardized data processing pipeline, benchmark protocols, and supporting tools for reproducible research on autonomous driving, including trajectory prediction, risk assessment, LLM-based reasoning, and decision-making. 3/22
Dataset
Viewpoint
Modality
Source
Risk definition
K-Risk (ours)
micro (BEV+ego)
trajectory +text
EU+US+CN (20 sources)
DRF top-10%/1% + |a| yes (collision- CPT + SFT + 2026 thresholds + TTC<2/3/5s + free sim + re- RLHF + RLVR 2s conflict predictor flection)
DriveLM16
micro (ego)
image+graph
CoVLA14
micro (ego)
video+text
US+SG+Sim (2 sources) JP (1 source)
OmniDrive15
micro (ego)
image+text
Impromptu VLA19
micro (ego)
video+text
NuRisk18
micro (BEV)
image+text
CODA-LM17
micro (ego)
image+text
InterHub12
micro (BEV)
trajectory
LTD/UniVLT20
macro side)
(road- image+text
none (safety implicit in P1– P3 QA) none (free-text risk captions) US+SG (1 rule-based check on simusource) lated trajectories US+EU+CN+IN 4 unstructured-scenario (8 sources) categories (no quantitative score) US+SG+Sim (3 weighted TTC + longitudisources) nal/lateral DTC per agent CN (1 source) 7-class corner-case taxonomy (no kinematic threshold) EU+US+CN (4 MSAA > 0 + PET annotasources) tion open-ended free-text hazard CN (1 source) reasoning
Closed loop
Post-training stages
Year
no
SFT
2024
no
CPT + SFT
2025
no
CPT + SFT
2025
partial (NeuroNCAP scorer) partial (CommonRoad sim) no
SFT + (partial 2025 RLVR)
no no
SFT (CoT VQA)
2025
SFT
2025
none (event index 2025 only) SFT (curriculum) 2026
Table 1. Comparison of K-Risk with representative trajectory, VQA and VLA datasets for autonomous driving. Abbreviations: EU/US/CN/JP/SG/IN = Europe/United States/China/Japan/Singapore/India; Sim = simulator generated; CPT = continued pre-training; SFT = supervised fine-tuning; RLHF = preference pairs for DPO/PPO/GRPO; RLVR = reinforcement learning with verifiable simulator reward. Full column definitions are given in the surrounding text. Table 1 compares K-Risk with representative trajectory, visual-question-answering (VQA), and vision-language-action (VLA) datasets. Two axes describe how K-Risk is positioned relative to these datasets. On the geographic axis, K-Risk combines real-world recordings from Europe, China and the United States, whereas most listed datasets are confined to a single region and the multi-region entries release event indices or question-answer pairs rather than verifiable safety signals. Aggregating 20 HV and AV trajectory sources across the three regions broadens the range of traffic rules, road geometries and driving conventions represented. On the risk and validation axes, K-Risk reports an explicit, multi-dimensional risk definition, built from a driver risk field, calibrated behavioral thresholds and a two-second trajectory-conflict predictor, together with closed-loop validation through a collision-free simulator and reflection.
Methods The K-Risk dataset was constructed from previously published trajectory data in two stages, described below. The Input data subsection lists the source datasets and their exact provenance. The annotation protocol is the set of rules and models that screens a continuous trajectory for high-risk events and assigns each retained event its semantic annotation. The data processing workflow is the end-to-end pipeline that applies this protocol to every source dataset and writes the per-event files. Throughout, the emphasis falls on how the records were produced and made reusable rather than on any single algorithmic component. Input data K-Risk is a secondary dataset compiled from twenty publicly available trajectory sources, six recorded from human-driven vehicles (HV) and fourteen from automated-driving vehicles (AV). The six HV sources are highD5 , inD6 and rounD21 from the LevelXData collection, recorded in Germany, the ExpresswayA and FreewayB subsets of CitySim22 , recorded in China and distributed through a collaboration documented in the CitySim release, and the I-80 recording of the NGSIM program23 in the United States. The fourteen AV sources are the Argoverse 2 Motion Forecasting dataset2 , the Waymo Open Motion24 and Waymo Open Perception7 datasets, the MicroSimACC dataset25 , and the OpenACC Casale, Vicolungo, AstaZero and ZalaZONE recordings26 ; the remaining AV sources, namely the CATS ACC, CATS Platoon and CATS UWM datasets, the Central Ohio single-vehicle and two-vehicle datasets, and the Vanderbilt ACC experiments, are obtained through the unified longitudinal trajectory collection of Ultra-AV27 , which documents their original providers and access points. K-Risk releases only the event-level trajectory segments derived from these sources, not the complete original recordings, which remain 4/22
available from their respective providers, listed with their repositories in the Data availability statement. Each source was used in accordance with its license and terms of use, and readers who require the full recordings obtain them from the original providers under the same terms. For each source the analysis uses the released vehicle trajectories at their native sampling rate and column schema, which the data processing workflow then converts into the common per-event representation. Annotation protocol The annotation protocol screens a continuous trajectory for safety-critical events and then enriches every retained event with a semantic layer. Screening proceeds through three filters of increasing specificity, namely a driver risk field that scores the danger of each frame, a calibrated detector of behaviorally significant interactions, and a short-horizon trajectory-conflict predictor. The semantic layer that follows pairs each event with a structured scenario description and a set of abnormal-behavior notifications, and adds an LLM-generated risk analysis for a representative subset of the extreme events. The remainder of this subsection presents the two parts in turn. The protocol first computes a driver risk field (DRF) value for each frame, which quantifies the danger experienced by the ego vehicle from three groups of factors. The first group is the ego-vehicle dynamics, comprising its speed, longitudinal and lateral acceleration, steering angle and heading. The second is the configuration of the surrounding vehicles, comprising the number of nearby agents, their classes such as car, truck, bus, pedestrian and cyclist, and their speed, acceleration and heading. The third is the spatial proximity to those vehicles, measured as Euclidean distances weighted by positional relevance across the preceding, following and adjacent lanes. Because road structure and traffic density vary across sources, the DRF threshold is set per source, and only the top 10% of frames by DRF value are retained. This first filter removes routine conditions such as constant-speed highway cruising and concentrates the subsequent analysis on frames with higher interaction complexity and elevated potential risk; the full mathematical formulation of the DRF is given in Appendix A. The second filter isolates interactions in which the surrounding vehicles exert a measurable influence on the ego vehicle, so that ego-initiated maneuvers unrelated to the environment are not retained. Four behavioral actions are treated as risk-relevant, namely hard acceleration, hard braking, left lane change and right lane change, and each is detected with a calibrated threshold, as illustrated in Figure 3. Longitudinal acceleration above +3 m/s2 is labeled hard acceleration and acceleration below −3 m/s2 hard braking. These values, approximately ±0.3g, are widely used in traffic-safety research as indicators of abrupt or emergency driving and of passenger discomfort28, 29 . Lane changes are evaluated against dataset-specific lateral-speed profiles, because the severity of a lateral maneuver depends strongly on road geometry and longitudinal speed30 . For each source the average lateral velocity during lane-change events is computed, and the threshold is set at 75% of that value, which tolerates normal variability while filtering out gradual, non-critical lane shifts. Each maneuver is classified within a detection window of 0.7 seconds, approximately the fastest documented human brake-reaction time, which keeps the detector responsive while avoiding intervals so short that sensor noise is mislabeled as behavior or so long that the immediacy of the action is lost31, 32 . An interaction is judged behaviorally significant only when such a maneuver produces a measurable escalation of risk consistent with traffic-rule interpretation. A left-preceding vehicle that changes into the ego lane and reduces the time-to-collision (TTC) below 5 seconds or the time headway below 2 seconds, for example, is flagged as critical, and more generally any behavior that reduces the TTC of the ego vehicle or of a surrounding vehicle in the current or an adjacent lane is considered significant, in line with established traffic-safety guidelines for situations that call for evasive or defensive maneuvers33 . The third filter complements TTC, which estimates the time to a potential collision under the current kinematic state but remains a one-dimensional measure. To capture the spatial dimension, the trajectories of the ego vehicle and the surrounding agents are forecast over a two-second horizon under constant-acceleration and constant-steering assumptions, a model well validated for short-term motion estimation34, 35 . A conflict is declared when two forecast paths overlap within that horizon. This combination captures imminent longitudinal hazards together with the lateral conflicts that arise during lane changes, merges and weaving, and is especially useful in complex environments such as roundabouts and unsignalized intersections where several vehicles follow intersecting paths. Scenarios with TTC below 2 s or a detected trajectory conflict within two seconds are classified as extreme-risk events and placed in a dedicated subset for evaluating safety-critical decision-making. The semantic layer then links the numerical records of each retained event to interpretable text, so that the event is usable not only for conventional trajectory modeling but also for language-based risk interpretation and decision reasoning. For every event the protocol generates two rule-based records, namely a structured scenario description and a set of abnormal-behavior notifications, and for a representative subset of the extreme events it adds a third record, an LLM-generated risk analysis. Together these records describe the road context, the traffic agents involved, the risk-relevant behaviors and the recommended response, in a form that a driving agent can parse directly or that can serve as a supervised language-model training sample. The structured scenario description supplies the contextual information required to interpret the event. It records the road layout, lane topology, speed limits and the legal maneuvers available to the ego vehicle. It then describes the ego vehicle together with up to eight surrounding agents, namely the preceding, left-preceding, right-preceding, left-alongside, right-alongside, following, left-following and right-following agents, giving for each its relative position, lane relation, velocity, acceleration, heading and 5/22
102 Hard acc, overtaking, THW = 1.4s
Hard brake, |a| > 3 m/s², TTC = 1.6 s
103
c. Safety-critical interaction extraction
b. Behavior detection
Longitudinal acceleration (m/s²)
a. Candidate frames (post-DRF)
Longitudinal behavior 6
Preceding braking 2s conflict
hard acc
3
+3
0 -3
Lateral acceleration (m/s²)
101
-2
-1
0
1
High risk
2
ego
102 Ego
103
Lateral behavior
DRF top 1%, |a| > 5 m/ s² or TTC < 3 s
Unprotected intersection
VRU dart-out
VRU
2
lane change
1
Extreme risk
0 -1
time (s)
-2 -2
-1
0
1
101
2
0.7 s window Filter normal driving retain interaction-rich frames
DRF top 10%, |a| > 3 m/ s² or TTC < 5 s
-3 hard brake time (s)
-6
Ego Ego
Adjacent cut-in
Moderate risk
Ego
0.7 s windows
Cut-in, TTC = 0.9 s
d. Risk events collection
Detect risk maneuvers hard acc, hard brake, left / right lane change
non-conflict vehicle
conflict vehicle
TTC < 2 s or 2 s trajectory conflict
Driver risk field
High 19.7%
Moderate 76.8%
Extract neighbor maneuvers that measurably reduces ego TTC
Conflict point
Ego
Ego
Extreme 3.5% !
Collect severity-graded events
Mod 76.8 risk maneuver
VRU
High 19.7
Figure 3. Detection of risk-relevant maneuvers and the resulting graded events. (a) The driver risk field retains interaction-rich candidate frames and removes routine cruising; the ego vehicle and its surrounding agents are shown with their TTC and time headway. (b) Each maneuver is detected within a 0.7 s window: longitudinal acceleration beyond ±3 m/s2 marks hard acceleration or hard braking, and a peak in lateral velocity marks a lane change. (c) An interaction is kept as safety-critical only when a neighboring vehicle measurably reduces the ego TTC, for example through a forecast trajectory conflict. (d) Retained events are graded as moderate (76.8%), high (19.7%) or extreme (3.5%) risk by the calibrated thresholds, illustrated by preceding braking, adjacent cut-in, unprotected intersection and VRU dart-out cases.
vehicle class where available. This representation mirrors the way a human driver observes surrounding traffic while preserving machine-readable state variables for model training and evaluation. The abnormal-behavior notifications are derived from the event-extraction results and attach concise risk cues to the description. Each agent receives behavior labels when it triggers a predefined risk-relevant pattern such as hard braking, hard acceleration, an abrupt lane change, a short time-to-collision or a trajectory overlap. When such behavior produces a meaningful interaction with the ego vehicle, for example a cut-in from a left-preceding vehicle that reduces the time-to-collision below a safety threshold, the corresponding warning is attached to the event record. Isolated abnormal behaviors such as excessive speed or close-proximity following are recorded even when no explicit interaction is detected. Additional cautionary messages are added when vulnerable road users or large vehicles, namely pedestrians, cyclists, trucks, buses or motorcycles, appear in the surrounding context, because such agents impose stricter safety margins, larger blind zones or different right-of-way considerations. These notifications reduce the chance that a language model overlooks important but sparse safety signals, and they let the dataset represent mixed-traffic interactions in a way compatible with both traffic-safety analysis and language-based scenario understanding. For the representative subset, an LLM generates an event-level risk analysis from the structured description and the abnormal-behavior notifications. The model receives the lead-in phase, the peak-risk frame and the following phase, together with the surrounding-agent states and the rule-based risk reminders. The generated text explains what led to the hazardous state, why the event is safety-critical, how the interaction evolves over time and which response is recommended. The response is constrained by a predefined action schema, namely keeping the current state, changing lane to the left or right, accelerating or decelerating. This schema links the textual reasoning to a discrete decision label that can be evaluated in downstream experiments. To improve reliability, the LLM-generated records were reviewed by domain experts for factual correctness, contextual consistency and action validity, and the feedback was used to refine the prompt template, adjust the risk reminders and correct ambiguous descriptions. The same generation procedure is released with the dataset, so that the LLM layer can be reproduced and extended to additional events. The final semantic annotation therefore combines rule-based risk cues, structured traffic context and natural language reasoning, producing event records that support risk interpretation, behavior modeling, 6/22
Extr 3.5%
decision prediction and language-based reasoning. Data processing workflow The data processing workflow applies the annotation protocol to each source dataset and writes the resulting per-event files, proceeding through lane-topology construction, surrounding-vehicle augmentation, event extraction with risk grading, and annotation enrichment, as shown in Figure 4. The same workflow runs on every source, so that the heterogeneous upstream recordings yield events in a single, consistent format. b. Risk event extraction
c. Semantic enrichment
DRF filter
Hard maneuver
TTC & conflict
Scene description
keep top 10% risk frames
hard brake / acc / lane change within a 0.7 s window
TTC < 5 s · 3 s · 2 s trajectory conflict
Raw data millions of frames · from 20 sources
a. Lane context augmentation
road layout · agent states
Behavior notifications hard brake · cut-in · VRU alert expert review
3s 2s peak-risk lead-in frame resolution
Curated events
Lane dictionary • adjacent-lane map • legal maneuvers
Reconstruct topology • attach to surrounding dynamic agents
LLM risk analysis cause · evolution · safe action
Ego
d. Standardized file output CSV trajectory structured physical metrics
Moderate 76.8%
High 19.7%
Extreme 3.5%
JSON metadata roles, lanes, behaviors, risk
TXT / JSONL narrative description, analysis, decision
Figure 4. Data processing workflow of K-Risk. Raw records from the 20 sources first pass through lane context augmentation, which reconstructs the lane topology and attaches up to eight surrounding agents to each ego vehicle (a). Risk event extraction then applies the driver risk field filter, the hard-maneuver detector and the TTC and trajectory-conflict test, and grades the retained events as moderate (76.8%), high (19.7%) or extreme (3.5%) risk, with each event window comprising a 3 s lead-in, a peak-risk frame and a 2 s resolution phase (b). Each curated event is enriched with a scenario description and behavior notifications, and a representative subset additionally receives an LLM risk analysis with expert review in the loop (c), and written to a standardized three-part file: a CSV of trajectories, a JSON of metadata and a text narrative (d). On loading the raw trajectory data from a source, the workflow first builds a structured dictionary that encodes the lane topology. Most datasets provide lane identifiers but not the positional relationships between lanes, such as adjacent-lane mappings, so this step reconstructs the spatial configuration of the road network and makes it possible to locate vehicles relative to the ego vehicle. Using this dictionary, the workflow appends up to eight surrounding-vehicle identifiers to each ego-vehicle instance, covering the preceding, following, left- and right-preceding, left- and right-following, and left- and right-alongside positions. This representation mirrors the situational awareness of a driver and provides the role structure on which the semantic annotation depends. The workflow then extracts events and grades their severity. It applies the DRF filter to discard routine, uneventful segments and detects hard maneuvers over the 0.7-second window using the calibrated thresholds of the annotation protocol. A frame is flagged as risk-relevant only when it shows both a significant behavioral indicator, such as hard acceleration, braking or an abrupt lane change, and a temporal-proximity measure below a safety threshold, such as a low TTC or a projected trajectory conflict. Each risk-relevant frame then receives a severity grade. A frame is graded high risk when its DRF lies within the top 1% of the source distribution, the magnitude of its acceleration or braking exceeds 5 m/s2 , or its TTC falls below 3 s, and it is graded moderate risk when the DRF lies within the top 10%, the acceleration or braking magnitude exceeds 3 m/s2 , or the TTC falls below 5 s; extreme-risk frames, identified by a TTC below 2 s or a trajectory conflict within two seconds, are graded separately as defined in the annotation protocol. Around each graded frame the workflow extracts an event window comprising the 3 seconds preceding and the 2 seconds following it, which captures the full temporal context in three phases: a lead-in phase that exposes anticipatory behavior before the risk peak, a peak-risk frame that represents the highest-risk state, and a resolution phase that records the actual response of the human driver or autonomous system. This segmentation lets downstream models learn both risk-escalation patterns and the causal relationship between decisions and their outcomes. 7/22
For each extracted event the workflow finally appends the multi-layer annotation defined by the protocol and writes the event to a standardized three-part file schema. The annotation comprises the structured scenario description and the abnormal-behavior notifications for every event, together with the LLM-generated risk analysis for the representative subset described above. The CSV file holds the per-event trajectory records, including position, velocity, heading and acceleration for every relevant agent; the JSON file encodes the symbolic event metadata, including agent roles, lane relationships, scenario context, interaction structure, behavioral labels and the assigned severity grade; and the text file, in JSON Lines format, holds the natural language narrative, comprising the environment description and, where available, the risk analysis and the recommended decision. This layered structure supports conventional supervised learning for motion prediction and trajectory forecasting as well as language-based reasoning and decision-making, and the precise field definitions of each file are listed in Appendix B.
Data Records Each data record in the K-Risk dataset is a multi-modal representation that combines per-event trajectory segments, structured metadata and natural language semantic descriptions, with visualization snapshots provided for inspection. This design supports analysis across diverse tasks, including behavior modeling, risk-sensitive decision-making and LLM-based prompt engineering.
Visual context
d. Visualization
Peak-risk f169
Lead-in f139 6
6
10 7
5
6
10
6
6
10
7
11
5
4
5 11
4
7
ego vehicle conflict vehicle
11
4
• High-risk scenario visualization
Semantics Metadata
truck
c. TEXT 1. Collision Warning: High 2. For each vehicle in the
b. JSON
Raw data
Resolution f194
6
a. CSV
collision risk detected.
surrounding vehicle list:
Evasive maneuvers are severely restricted due to an aggressive following vehicle (11) and rightlane blockage (6, 10).
Following Vehicle (11): Status: Accelerating above average and yawing left. Constraint: Sudden braking and left lane changes are not allowed......
Ø abc"frame": 169, Ø "id": 7, Ø "precedingId": 0, Ø "followingId": 0, Ø "leftPrecedingId": 0, Ø "leftAlongsideId": 11,
Ego Vehicle
Ø "leftFollowingId": 0, Ø "leftFollowingId": 0, Ø "rightPrecedingId": 6, Ø "rightAlongsideId": 0, Ø "rightFollowingId": 10, Ø "laneId": 5,
3. Post-Pruning Actions:
4. Decision Making: Changing
After considering all the surrounding vehicles, the only safe action available is: IDLE
right is blocked by trucks (6/10) ...... Given the current speed (34.5 m/s) and constraints, the safest and only viable action is to hold
(Maintain speed).
Conflict Vehicle
the current speed.
Ø "frame": 169, Ø "id": 11, Ø "precedingId": 0, Ø "followingId": 0, Ø "leftPrecedingId": 0, Ø "leftAlongsideId": 0,
Ø "leftFollowingId": 0, Ø "leftFollowingId": 0, Ø "rightPrecedingId": 0, Ø "rightAlongsideId": 7, Ø "rightFollowingId": 0, Ø "laneId": 4,
frame
id
x
y
xV
yV
xAcc
lane
ttc
yaw_left
class
139
7
336.33
21.71
34.32
0.07
0.16
5
59.2
false
Car
169
7
377.59
21.80
34.50
0.12
0.15
5
0.00
false
Car
169
11
344.82
23.74
24.87
-1.17
0.67
5
0.00
true
Car
• Scenario description • Risk warning • Risk analysis • Action output
• Structured data with scenario description augmentation
• Raw trajectory data with physical metrics
stored as one synchronized triple (CSV + JSON + TXT)
Event Structure As shown in Figure 5, every event is extracted from a continuous trajectory and segmented into a lead-in phase, a peak-risk frame and a resolution phase. The peak-risk frame corresponds to the highest-risk point, such as the moment of minimum time-to-collision, while the lead-in and resolution phases provide the temporal context necessary for interaction analysis.
Figure 5. Layered structure of a single K-Risk event, stored as one synchronized triple. The CSV layer holds the raw per-frame trajectories of the ego and conflict vehicles, including position, velocity, acceleration, lane, TTC and behavioral flags (a). The JSON layer encodes the structured metadata, with agent roles, lane relationships and the risk level (b). The text layer provides the natural language annotation, comprising the imminent-collision warning, the surrounding-agent analysis, the post-pruning actions and the final decision (c). Visualization snapshots of the lead-in, peak-risk and resolution phases support inspection (d). Each event is stored in three synchronized formats to ensure compatibility across research tasks. The CSV file contains the per-event trajectory records for all participating agents, including their spatial coordinates, velocity components, acceleration values, lane identifiers and heading. The JSON file encodes the structured metadata, capturing lane topology, agent roles, detected behaviors, time-to-collision estimates and the assigned risk level. The text file provides the semantic annotation in natural language, comprising the environment description and, for the representative subset, the detailed risk analysis and the recommended decision. The complete field definitions of each file are listed in Appendix B. 8/22
File organization and access K-Risk is hosted in the public repository described in the Data availability statement. The release is organized into three top-level directories. The trajectory_data directory holds the per-event trajectory segments that underlie each retained event, the event_annotations directory holds the extracted per-event records, and the llm_analysis directory holds the natural language scenario descriptions and the corresponding LLM responses. Within trajectory_data and event_ annotations the events are separated into human-driven (HV) and automated-driving (AV) groups, then into one folder per source dataset. The human-driven sources are further split into risk-level folders, for example highd_normal_risk and highd_high_risk, and the near-collision cases are collected in a separate extreme folder. Each per-event triple shares the event identifier encoded in its file name, so that the CSV, JSON and text records of one event can be matched across directories. The complete original recordings are not redistributed; they remain available from their providers and are cited as input sources in the Methods Input data subsection and listed in the Data availability statement2, 5–7, 21–27 .
Data Overview The complete K-Risk dataset contains 31,398 curated events, systematically categorized by their source datasets and assigned risk levels, with the distributions shown in Figure 6a. Although the events are drawn from the 20 upstream trajectory sources, the high-risk yield is uneven: a small number of large naturalistic recordings dominate, because high-risk events are sparse in many sources and especially rare in the autonomous-vehicle recordings. The bulk of the events therefore originate from a handful of sources, with highD contributing the largest share at 8,807 events, followed by FreewayB at 5,304, inD at 4,684 and ExpresswayA at 4,519; NGSIM, the roundabout recordings and the aggregated autonomous-vehicle sources add a further 2,761, 2,753 and 2,570 events, respectively. Each human-driven event is assigned one of three risk levels, moderate (76.8%), high (19.7%) and extreme (3.5%), with the percentages taken over the mutually exclusive human-driven events, and the extreme fraction corresponds to the 1,036-event extreme subset, while the autonomous-vehicle events are released without a human-driver risk stratification. These classifications follow the calibrated thresholds of the annotation protocol, which combine time-to-collision metrics, driver risk field percentiles and the detection of hard maneuvers such as abrupt braking or lane changes. Figure 6 further reports the agent composition and the per-scenario speed and time-to-collision distributions, which together summarize the heterogeneous traffic participants and the range of severity levels present in the release.
Technical Validation We validate the K-Risk dataset along four dimensions: the diversity of its scenarios and participants, its emphasis on high-risk edge cases, its compatibility with LLM-based agents for decision-making, and the experimental performance of such an agent on the extreme subset. Diversity of scenarios and participants
To reflect real-world complexity, K-Risk spans diverse road structures, participant types, and risk-relevant behaviors. Across the human-driven events, the driving environments include high-speed highways (40.1%), urban freeways (34.1%), intersections (16.2%), and roundabouts (9.5%), as shown in Figure 6a. Behavioral variability is also broad. In the three sources that provide frame-level behavior labels, braking, acceleration, and lane changes are each marked on 105 to 106 frames, as shown on the logarithmic scale of Figure 6c. Unlike datasets focused mainly on passenger vehicles, K-Risk includes vulnerable road users (VRUs) and heavy-duty vehicles. Among 53,295 unique agents, 18.5% are non-car road users (Figure 6d): VRUs, including pedestrians, cyclists, and motorcyclists, account for 7.9%, while heavy or special vehicles, including trucks, buses, vans, and trailers, account for a further 10.6%. This diversity allows K-Risk to support interaction-rich, mixed-traffic, and safety-critical evaluations of autonomous systems. Emphasis on high-risk edge cases
K-Risk prioritizes edge scenarios where collision likelihood is elevated. Compared with the original datasets, K-Risk exhibits markedly lower time-to-collision (TTC) values, obtained through calibrated filtering thresholds (for example, TTC < 5 s and top-percentile DRF scores). Because these thresholds enter the extraction protocol, the shift toward low TTC is expected by construction; the joint speed and TTC density in Figure 6b illustrates its magnitude, showing that relative to the original HighD records the K-Risk events concentrate in the near-collision region below the TTC = 5 s line while retaining a realistic speed spread rather than collapsing onto the threshold, which supports the evaluation of time-critical decision-making algorithms. The dataset also preserves coverage across all speed regimes. As the per-scenario speed distributions in Figure 6e show, urban intersection and roundabout scenarios concentrate at low speed, whereas the highway and freeway scenarios extend beyond 40 m/s, supporting analysis of high-speed merges, lane changes, and emergency maneuvers. This range allows planning and control systems to be tested under both congested and high-speed conditions. 9/22
a
c
d
b
e
Figure 6. Composition and safety-critical properties of K-Risk. (a) Curated events per source, stacked by risk level (moderate, high, extreme) and grouped by driving environment; the autonomous-vehicle (AV) subset carries no human-driver risk stratification. (b) Joint ego-speed × time-to-collision (TTC) density for K-Risk (filled) versus the original HighD data (blue contours), with speed and TTC marginals; K-Risk concentrates below the TTC = 5 s line (dashed). (c) Behaviour-labelled frame counts per source (log scale) for braking, acceleration and lane change. (d) Agent composition over 53,295 unique agents (top), with the non-car classes resolved by count (bottom). (e) Per-scenario ego-speed distributions (ridgeline, peak-normalised, ordered by median; the tick marks the median).
Compatibility with LLM-based agents
K-Risk is accompanied by a closed-loop LLM annotation framework for scalable and interpretable decision reasoning, illustrated in Figure 7. In this framework a high-risk scenario is converted into a structured description of the ego and surrounding vehicles, the legal maneuvers, and the interaction context. The LLM generates risk-aware decisions, such as “Turn Right” or “Decelerate,” with justifications based on spatial relations and predicted hazards. All actions are validated through collision-free simulation, and the feedback is fed back to refine the prompts. Validated outcomes form a reusable memory pool that supports few-shot learning for LLM-based driving agents. This design makes the dataset compatible with closed-loop experiments and allows decision-making to be evaluated under safety-critical conditions. A complete walk-through of one event from this framework, covering the system prompt, the structured scenario description fed to the LLM, the rule-based risk reminders produced by the risk filters, and the LLM-generated risk analysis together with the chosen action identifier, is provided in Appendix C. 10/22
a. Structured Input
b. LLM Risk Reasoning
d. Reflection-driven Refinement
c. Decision Recommendation
LLM Output
Keep
Turn Left
Accelerate
Scenario description Abnormal-behavior alerts
Turn Right
Store validated data records
Decelerate
collision-free simulator
Causal risk Lane topology + legal maneuvers hard brake
Ego + 8 agents · kinematics · TTC
cut -in
TTC <5 s
VRU
expert review
Fail (Unsafe)
Pass (safe)
prompt refinement
self-check& few-shot
reflection feedback · prompt refinement
e. Event Visualization (example: expresswayA_high_risk_1)
f. Closed-loop Evaluation
Lead-in (−3 s)
Peak-risk (TTC < 5 s)
Resolution (+2 s)
Frame 5720
Frame 5740
Frame 5760
Human-driver baseline: 5% (14 collisions）
816 ego vehicle
ego vehicle
822 812
conflict vehicle
ego vehicle
819
819 819
Warning
812
conflict vehicle
822
High risk
822
conflict vehicle
812
Collision rate
5%
4.58% prompt v1
3%
prompt v2
2%
1 T1
Turn right → safe
Time
Collision
2.67% 1.91% prompt v3
2 T2
rate
58% ↓ （from T1 to T3）
3 T3
262 extreme-risk scenarios
Figure 7. Closed-loop LLM annotation framework. A structured input describes the ego vehicle, up to eight surrounding agents, the lane topology and legal maneuvers, and the TTC (a). The LLM agent reasons over the scenario description, the abnormal-behavior notifications and the causal risk cues (b) and recommends an action from the five-action schema (c). The action is checked against a collision-free simulator, and a failure triggers reflection that refines the prompt and yields validated records through expert review and self-check (d). A real ExpresswayA event is shown across its lead-in, peak-risk and resolution phases (e), and the collision rate on the 262 extreme-risk evaluation events is 4.58%, 2.67% and 1.91% over three trials, shown together with the 5% human-driver reference (f). This event is the worked example detailed in the stages below. Closed-loop usability of the extreme subset
The extreme subset was further evaluated to examine whether K-Risk can support closed-loop, safety-critical decision testing. We selected 262 extreme-risk scenarios that include severe near-collision interactions and 14 recorded human-driver collisions. For each scenario, an LLM-based decision-making agent received the structured scene description, abnormal-behavior notifications, and causal risk cues, and then selected one action from the five-action schema. The selected action was evaluated by a collision-free simulator. Failed cases were returned to the agent as reflection feedback, and the refined prompt was used in the next trial. The collision rate decreased from 4.58% in the first trial to 2.67% in the second trial and 1.91% in the third trial, corresponding to a 58.3% relative reduction from Trial 1 to Trial 3 (Table 2). The non-zero collision rate across all trials indicates that the subset is not composed of routine driving cases, but instead contains scenarios that remain challenging even after iterative refinement. More importantly, each tested event produces reusable validation records, including the initial decision, simulator outcome, failure reason, reflection prompt, and refined decision. These records allow K-Risk to serve not only as a collection of high-risk trajectories, but also as a stress-test resource for evaluating risk-aware decision-making, prompt refinement, and preference-based learning. Trial Human driver Trial 1 Trial 2 Trial 3
Evaluated scenarios
Collisions
Safe outcomes
Collision rate (%)
262 262 262 262
14 12 7 5
248 250 255 257
5.34 4.58 2.67 (−41.7%) 1.91 (−58.3%)
Table 2. Closed-loop validation on the K-Risk extreme subset.
11/22
Family
Task
K-Risk asset used
Suggested benchmark / scorer internal macro-F1; CODA-LM17
Risk-level classification (Moder- JSON severity grade ate/High/Extreme) Risk-cue VQA (which agent, why) TXT risk explanations → QA pairs Causal / counterfactual reasoning three-phase lead-in/critical/following slice
Lingo-Judge36 ; DriveLM16 ; DVBench37 OmniDrive15 counterfactual QA
Prediction
Intent prediction Trajectory forecasting Risk / TTC forecasting
JSON behavior labels CSV trajectory frame-level TTC in CSV/JSON
DRAMA-X38 ADE/FDE on Argoverse 2, WOMD InterHub PET metric12
Decision & action
Discrete action prediction (5-class) Closed-loop driving Plan justification Preference / reflection learning
TXT decision recommendation 1,036-event extreme subset as stress test TXT risk analyses as references trial-1 vs trial-3 chosen/rejected pairs
internal accuracy / macro-F1 NeuroNCAP39 ; Bench2ADVLM40 BLEU / ROUGE / CIDEr; Lingo-Judge36 DPO41 / SimPO42 training; pairwise win-rate
Evaluation & audit
Interpretable safety evaluation Responsibility analysis Rule-compliance audit
1,036 events extreme subset JSON agent roles + TXT causal chain JSON legal-action schema vs LLM output
collision rate, infraction count expert inter-rater agreement rule-violation rate
Perception & reasoning
Data augmentation
Scenario augmentation (counterfactual CSV trajectory + TXT description as seed variants) Curriculum construction for RL three risk levels as difficulty stages
CrashAgent43 ; ChatScene44 internal RL curriculum
Table 3. Downstream tasks that K-Risk supports, grouped into five families. The “K-Risk asset” column indicates which of the per-event CSV, JSON and text files, or the extreme subset and trial pairs, provides supervision; the “Suggested benchmark/scorer” column lists external evaluators or established metrics that are compatible with the K-Risk release.
Usage Notes K-Risk is released as a set of per-event triples together with the extreme subset, and the file schema in Appendix B is intended to let others reuse the data without re-running the construction pipeline. The three files per event support complementary modes of use: the CSV trajectories feed conventional supervised learning for motion prediction and trajectory forecasting, while the JSON metadata and natural language text support language-based risk interpretation and decision-making. Table 3 summarizes the downstream tasks that the release supports, grouped into five families, and indicates which asset each task consumes together with a suggested external benchmark or scorer where one exists. Each suggested evaluator corresponds to a published benchmark or metric that can be applied to the K-Risk release. These evaluators include NeuroNCAP39 for closed-loop driving, CODA-LM17 for corner-case classification, Lingo-Judge36 and DVBench37 for risk-cue question answering, DriveLM16 for graph-structured perception, prediction and planning, DRAMA-X38 for fine-grained intent prediction, OmniDrive15 for counterfactual reasoning, and the post-encroachment-time metric of InterHub12 for risk forecasting. Beyond these analysis tasks, the structure of K-Risk aligns with the standard stages used to adapt a general-purpose LLM into a driving agent 41, 42, 45, 46 . The structured natural language descriptions can serve as a continued pre-training corpus that exposes a base model to driving-specific vocabulary, road-rule context, interaction patterns and quantitative agent states. For the subset that carries an LLM annotation, the description, risk analysis and decision jointly form a supervised fine-tuning sample that links the five-action schema to a causal explanation of the driving risk. Closed-loop validation further produces trial-level outputs, with the collision rate falling from 4.58% in the first trial to 1.91% in the third trial on the extreme cases. These first-trial and third-trial outputs naturally form rejected and chosen response pairs that can support preference optimization, such as Direct Preference Optimization41 or Simple Preference Optimization42 , without additional human labeling. The collision-free simulator used to validate each annotation can also provide a verifiable reward signal for reinforcement learning 46, 47 , while the explicit five-action schema enables constrained decoding at inference time. In this way, K-Risk can support a complete adaptation pipeline from domain exposure and supervised alignment to preference learning and reward-based refinement for risk-aware driving agents. Two properties of the release should be kept in mind when reusing the data. High-risk events involving autonomous vehicles are comparatively rare, because the automated-driving sources exhibit smoother driving behavior, so the autonomous-vehicle component is smaller than the human-driven component. The risk definition rests on calibrated thresholds over physical and behavioral indicators rather than on a single universal standard, and it does not encode probabilistic uncertainty or multi-agent intent.
12/22
Data availability The K-Risk dataset is deposited in a public repository and is available at https://github.com/benmagnifico/ K-Risk (code and processed records) and at the Figshare record https://doi.org/10.6084/m9.figshare. 32896772. The release contains the event-level trajectory segments, the risk annotations and metadata indices, the natural language semantic descriptions and LLM responses, and the benchmark splits, organized into the trajectory_data, event_annotations and llm_analysis directories described in the Data Records section. K-Risk is a secondary dataset compiled from publicly available human-driven (HV) and automated-driving (AV) trajectory sources, which are cited here as input data and remain available from their original providers; they are not redistributed as part of K-Risk. The six HV sources are highD5 (https://www.highd-dataset.com), inD6 (https://www. ind- dataset.com) and rounD21 (https://www.round- dataset.com) from the LevelXData collection, the ExpresswayA and FreewayB subsets of CitySim22 (https://github.com/UCF-SST-Lab/UCF-SST-CitySim1Dataset), and the I-80 recording of NGSIM23 (https://ops.fhwa.dot.gov/trafficanalysistools/ngsim. htm). The fourteen AV sources are the Argoverse 2 Motion Forecasting dataset2 (https://www.argoverse.org/ av2.html), the Waymo Open Motion24 and Waymo Open Perception7 datasets (https://waymo.com/open/), the MicroSimACC dataset25 (https://github.com/microSIM-ACC/ICE), the OpenACC Casale, Vicolungo, AstaZero and ZalaZONE recordings26 (https://data.jrc.ec.europa.eu/dataset/9702c950-c80f-4d2f-982f44d06ea0009f), and the CATS ACC, CATS Platoon and CATS UWM datasets, the Central Ohio single-vehicle and two-vehicle datasets, and the Vanderbilt ACC experiments, obtained through the unified Ultra-AV collection27 (https: //github.com/CATS-Lab/Filed-Experiment-Data-ULTra-AV).
Code availability The data processing pipeline that builds K-Risk is openly available in the same repository (https://github.com/ benmagnifico/K-Risk). The repository documents the steps used to convert each source recording into the per-event triple and to reproduce the figures and statistics reported here. The event-extraction components implement the annotation protocol of the Methods, namely the driver risk field computation with its per-source top-percentile thresholds, the hardmaneuver detector over the 0.7-second window, and the time-to-collision and two-second trajectory-conflict test that assign the severity grades. The source-specific scenario generators (generate_highd.py, generate_expresswayA.py, generate_freewayB.py, generate_ind.py, generate_round.py, generate_ngsim.py and generate_ AV.py) build the lane context, attach the surrounding agents and write the structured scenario descriptions; prompts.py holds the system prompts and the five-action schema; llm_test.ipynb sends each scenario description to the LLM and stores the generated risk analysis and action; the closed-loop validation component forwards each recommended action to the collision-free simulator and records the per-trial outcomes and the reflection prompts, from which the first-trial and third-trial pairs are written; and compute_statistics.py and plot_overview.py compute the per-source and risk-level statistics and produce the composition figure. The code is written in Python 3 and relies on the standard scientific stack (pandas, numpy and matplotlib) together with an LLM client for the annotation step. The code is released for research use, and access to it carries no further restriction.
References 1. Sun, P. et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2446–2454 (2020). 2. Wilson, B. et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493 (2023). 3. Fu, D. et al. Drive like a human: Rethinking autonomous driving with large language models. In arXiv preprint arXiv:2307.07162 (2023). Demonstrates LLMs’ reasoning, interpretation, and memorization abilities in long-tail driving scenarios. 4. Zhou, Z. et al. Safedrive: Knowledge- and data-driven risk-sensitive decision-making for autonomous vehicles with large language models (2024). 2412.13238. 5. Krajewski, R., Bock, J., Kloeker, L. & Eckstein, L. The highd dataset: A drone dataset of naturalistic vehicle trajectories on german highways for validation of highly automated driving systems. In 2018 21st International Conference on Intelligent Transportation Systems (ITSC), 2118–2125 (IEEE, 2018). 6. Bock, J. et al. The ind dataset: A drone dataset of naturalistic road user trajectories at german intersections. In 2020 IEEE Intelligent Vehicles Symposium (IV), 1929–1934 (IEEE, 2020). 13/22
7. Sun, P. et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2446–2454 (2020). 8. Caesar, H. et al. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 11621–11631 (2020). 9. Chang, M.-F. et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 8748–8757 (2019). 10. Yan, Z., Li, J., Hang, P. & Sun, J. Onsitevru: A high-resolution trajectory dataset for high-density vulnerable road users. arXiv preprint arXiv:2503.23365 (2025). 11. Zhan, W. et al. Interaction dataset: An international, adversarial and cooperative motion dataset in interactive driving scenarios with semantic maps. In arXiv preprint arXiv:1910.03088 (2019). 12. Jiang, X. et al. Interhub: A naturalistic trajectory dataset with dense interaction for autonomous driving. arXiv preprint arXiv:2411.18302 (2024). 13. Mokhtarian, H. & Farooq, B. Time-to-collision and collision probability in traffic safety analysis: A review. Accid. Analysis & Prev. 165, 106528 (2022). 14. Arai, H. et al. Covla: Comprehensive vision-language-action dataset for autonomous driving. arXiv:2408.10845 (2024).
arXiv preprint
15. Wang, S. et al. Omnidrive: A holistic vision-language dataset for autonomous driving with counterfactual reasoning. arXiv preprint arXiv:2504.04348 (2025). 16. Sima, C. et al. DriveLM: Driving with graph visual question answering. In Proceedings of the European Conference on Computer Vision (ECCV) (2024). 17. Li, Y. et al. Automated evaluation of large vision-language models on self-driving corner cases (CODA-LM). In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (2025). 18. Gao, Y. et al. NuRisk: A visual question answering dataset for agent-level risk assessment in autonomous driving. arXiv preprint arXiv:2509.13869 (2025). 19. Chi, H. et al. Impromptu VLA: Open weights and open data for driving vision-language-action models. arXiv preprint arXiv:2505.23757 (2025). 20. Huang, W. et al. Towards safe mobility: Large-scale traffic data (LTD) and the UniVLT traffic vision-language transformer. arXiv preprint arXiv:2604.22260 (2026). 21. Krajewski, R., Moers, T., Bock, J., Vater, L. & Eckstein, L. The rounD dataset: A drone dataset of road user trajectories at roundabouts in Germany. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 1–6 (IEEE, 2020). Dataset available at https://www.round-dataset.com. 22. Zheng, O. et al. CitySim: A drone-based vehicle trajectory dataset for safety-oriented research and digital twins. Transp. Res. Rec. 2678, 606–621 (2024). Dataset available at https://github.com/UCF-SST-Lab/UCF-SST-CitySim1-Dataset. 23. Federal Highway Administration. Next generation simulation (NGSIM) vehicle trajectories and supporting data. U.S. Department of Transportation, Intelligent Transportation Systems Joint Program Office (2006). I-80 recording, Emeryville, CA. https://ops.fhwa.dot.gov/trafficanalysistools/ngsim.htm. 24. Ettinger, S. et al. Large scale interactive motion forecasting for autonomous driving: The Waymo open motion dataset. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 9710–9719 (2021). 25. Yang, M. et al. Microsimacc: an open database for field experiments on the potential capacity impact of commercial adaptive cruise control (acc). Transp. A: Transp. Sci. 1–30 (2024). 26. Makridis, M., Mattas, K., Anesiadou, A. & Ciuffo, B. Openacc. an open database of car-following experiments to study the properties of commercial acc systems. Transp. Res. Part C: Emerg. Technol. 125, 103047 (2021). 27. Zhou, H., Ma, L., Liang, S., Li, X. & Qu, X. A unified longitudinal trajectory dataset for automated vehicle. Sci. Data 11, 1123 (2024). Dataset available at https://github.com/CATS-Lab/Filed-Experiment-Data-ULTra-AV. 28. Ishikawa, H. An analysis of vehicle acceleration behavior in urban traffic flow. Transp. Res. Part A: Policy Pract. 25, 265–278 (1991). 29. Liu, H., Wang, W. & Li, K. Traffic safety and comfort evaluation based on longitudinal acceleration thresholds. Accid. Analysis & Prev. 80, 220–229 (2015). 14/22
30. Milanés, V., Pérez, J., Godoy, J. & Onieva, E. Lane-change fuzzy control in autonomous vehicles for lateral maneuvers. IEEE Transactions on Intell. Transp. Syst. 11, 132–141 (2010). 31. Green, M. How long does it take to stop? methodological analysis of driver perception-brake times. Transp. Hum. Factors 2, 195–216 (2000). 32. Treiber, M. & Kesting, A. Traffic Flow Dynamics: Data, Models and Simulation (Springer, 2013). 33. Federal Highway Administration. Surrogate safety measures from traffic simulation models: Final Report. U.S. Department of Transportation, FHWA-RD-03-050 (2003). Reference inherited from the original K-Risk manuscript; placeholder pending verification. 34. Schöller, C., Aravantinos, V., Lay, S. & Knoll, A. What the constant velocity model can teach us about pedestrian motion prediction. IEEE Robotics Autom. Lett. 5, 1696–1703 (2020). 35. Gilles, T., Schöller, C., Duthon, P., Stanciulescu, B. & Nashashibi, F. A review on motion prediction and risk assessment for intelligent vehicles. IEEE Transactions on Intell. Veh. 6, 585–598 (2021). 36. Marcu, A.-M. et al. LingoQA: Video question answering for autonomous driving. arXiv preprint arXiv:2312.14115 (2023). 37. Zeng, T., Wu, L., Shi, L., Zhou, D. & Guo, F. Are vision LLMs road-ready? A comprehensive benchmark for safety-critical driving video understanding (DVBench). In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD) (2025). 38. Godbole, M., Gao, X. & Tu, Z. DRAMA-X: A fine-grained intent prediction and risk reasoning benchmark for driving. arXiv preprint arXiv:2506.08188 (2025). 39. Ljungbergh, W. et al. NeuroNCAP: Photorealistic closed-loop safety testing for autonomous driving. In Proceedings of the European Conference on Computer Vision (ECCV) (2024). 40. Zhang, T. et al. Bench2ADVLM: A closed-loop benchmark for vision-language models in autonomous driving. arXiv preprint arXiv:2510.15793 (2025). 41. Rafailov, R. et al. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems (NeurIPS) (2023). 42. Meng, Y., Xia, M. & Chen, D. SimPO: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems (NeurIPS) (2024). 43. Wang, H. et al. CrashAgent: Crash report driven scenario reconstruction with multi-agent large language models. arXiv preprint arXiv:2505.18341 (2025). 44. Zhang, J., Xu, C. & Li, B. ChatScene: Knowledge-enabled safety-critical scenario generation for autonomous vehicles. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024). 45. Ouyang, L. et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS) (2022). 46. Lambert, N. et al. Tülu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124 (2024). 47. Shao, Z. et al. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 (2024).
Acknowledgements This research was conducted by KAIST as part of a joint research project under the Korea Institute of Science and Technology Information (KISTI) R&D program, “Development of the Next-Generation Integrated Wired/Wireless Communication Gateway (X-Gateway).”
Author contributions H.H. conceived and supervised the project. H.H., J.L., and Z.Z. designed the K-Risk dataset and its construction protocol. J.L. extracted high-risk events and organized the dataset. Z.Z. curated the semantic annotations and implemented the data processing pipeline. M.W. conducted data preprocessing, and visualization. P.L., K.J., and J.W. provided technical guidance and revised the manuscript. All authors contributed to the manuscript and approved the final version. 15/22
Competing interests The authors declare no competing interests.
16/22
A Mathematical Formulation of the Driver Risk Field (DRF) The DRF assigns to every frame a scalar that quantifies the danger the ego vehicle perceives from its own motion and from the surrounding agents. K-Risk adopts the speed- and steering-adaptive DRF of SafeDrive4 , which extends an earlier model of the risk a human driver perceives ahead of the vehicle. The construction has four steps: a kinematic prediction of the path the ego vehicle would follow, a Gaussian field laid along that path, a cost assigned to each agent according to the severity of a potential conflict, and an accumulation of the cost-weighted field over the scenario. The remainder of this appendix presents the four steps in turn; the resulting scalar is the value stored per frame as total_risk (highD) or risk_value (the other sources). The ego vehicle is described by a kinematic car model with position (x, y), heading φ and steering angle δ . Under a constant steering angle, the vehicle follows a circular arc whose radius is Predicted path.
R=
L , tan δ
where L is the wheelbase. The heading is taken as φ = heading · π/180, and where a source does not log the steering angle directly, δ is inferred from the change in heading and a steering ratio Sr that maps the steering-wheel angle to the road-wheel angle. The radius R fixes the center of the turning circle (xc , yc ) from the vehicle position and heading, and the arc length s is measured along the predicted path from the vehicle to a grid point. The risk contributed by one vehicle at a spatial grid point (X,Y ) is modeled as a torus with a Gaussian cross-section centered on the predicted arc: Gaussian field along the path.
(X − xc )2 + (Y − yc )2 − R2 R(X,Y ) = a(s) exp − 2σ 2
2 ! ,
where the height a(s) and the width σ both depend on the arc length s. The height is largest near the vehicle and decreases along the predicted path; following SafeDrive it is a parabolic function of s whose peak is set by a steepness parameter p and whose reach grows with speed through a look-ahead time tla . The width broadens with distance and with steering, and differs between the inner and outer sides of a curve: σi = (m + ki |δ |) s + c,
i ∈ {1 (inner), 2 (outer)}.
Here c = wcar /4 sets the width at the vehicle, so that ±2σ covers about 95% of the lateral spread, m sets the width during straight driving (δ = 0), and k1 and k2 broaden the inner and outer sides as the steering magnitude |δ | grows. The asymmetry between k1 and k2 reflects driving styles such as curve-cutting or overshooting. The field is therefore parameterized by p, tla , m, q 2 2 c, k1 and k2 , and depends only on the ego state (ve , φ , δ ), with ve = vx + vy . Each agent in the scenario is assigned a cost weight α that reflects the severity of a potential conflict with it, set by the agent class so that large or vulnerable road users such as trucks, buses, pedestrians and cyclists receive higher weights than passenger cars. The cost map is combined with the field by element-wise multiplication and summed over the grid, giving the quantified perceived risk Event cost and risk accumulation.
QPR =
α ·R .
∑
grid points
A classical DRF covers only the forward half-plane. To assess risk in dense, interaction-rich traffic, the field is extended to a full 360◦ view by adding the contributions of leading and following agents: Omnidirectional accumulation.
QPRtotal = QPRfront + QPRrear , QPRfront =
∑
grid points
αfront · Rego ,
QPRrear =
∑
αego · Rrear ,
grid points
where Rego is the field of the ego vehicle weighted by the cost of the agents ahead, and Rrear is the field of the following agents weighted by the cost of the ego vehicle. The per-frame scalar QPRtotal is the DRF value to which the annotation protocol applies its top-percentile thresholds. 17/22
B K-Risk Event Schema This appendix documents the field-level schema of a K-Risk event. Each event is stored as a synchronized triple, comprising a CSV file of raw trajectory records, a JSON file of per-frame metadata, and a text file of natural language annotation, all sharing the event identifier encoded in the file name. Table 5 lists the event-level descriptors carried by the file name and folder, Table 4 the per-frame trajectory fields, and Table 6 the components of the natural language annotation. Each event retains the native column names of its source dataset rather than being renamed to a single shared vocabulary, so the per-frame fields are described by their role together with the column names used across sources; fields that are absent in a given source are simply not present in its files. Field role
Column name(s) across sources
frame index agent identifier
frame; frame_id; Frame_ID Frame index within the recording. id; car_id; trackId; Vehicle_ Identifier of the agent in the frame. ID x, y; car _ center _ x, car _ Planar position of the agent center (m). center _ y; xCenter, yCenter; Local_X, Local_Y width, height; length; Vehicle length and width (m). v_Length, v_Width speed; v_Vel Scalar speed of the agent (m/s). xVelocity, yVelocity; vx, vy; Longitudinal and lateral (or x and y) velocity (m/s). lonVelocity, latVelocity xAcceleration, yAcceleration; Longitudinal (and lateral) acceleration (m/s2 ). acceleration; lonAcceleration, latAcceleration; v_Acc heading; course Heading or course angle of the agent (deg). yaw_rate Rate of change of heading, CitySim sources (deg/s). laneId; lane_id; Lane_ID Lane occupied by the agent. vehicle_class; class; v_Class Agent class (car, truck/bus, pedestrian, cyclist, . . . ); absent for CitySim. dhw, thw; Space_Headway, Time_ Distance and time headway to the preceding vehicle (m, s). Headway ttc; TTC; TTC_0_1s, TTC_1_2s Time-to-collision estimate (s). collision _ 0 _ 1s, collision _ Conflict flags within the next one and two seconds and the conflicting-agent identifiers, 1 _ 2s, collision _ ids _ 0 _ CitySim sources. 1s, collision_ids_1_2s total_risk; risk_value Driver risk field value for the frame. Identifiers of the up to eight surrounding agents by spatial role (preceding, following, precedingId, followingId, . . . ; preceding _ id, following _ left/right-preceding, left/right-following, left/right-alongside). id, . . . acc _ high, brake _ high, yaw _ Boolean flags raised by the annotation protocol; the highD and CitySim sources. left, yaw _ right; acc _ label, brake _ label, left _ turn_label, right_turn_label
position
size speed velocity components acceleration
heading yaw rate lane agent class spacing time-to-collision trajectory conflict
risk value surrounding agents
behavioral flags
Description (unit)
Table 4. Per-frame trajectory fields in the CSV file and the JSON per-frame records. Because each event preserves the native column names of its source dataset, the fields are listed by role together with the corresponding names used across highD, CitySim (ExpresswayA, FreewayB), LevelXData (inD, rounD), and NGSIM sources, separated by semicolons. Fields not recorded by a given source are omitted from its event files rather than imputed, thereby preserving the original information without introducing artificial values. The CitySim sources additionally store the four bounding-box vertex coordinates of each vehicle, enabling accurate reconstruction of vehicle orientation, footprint, and spatial occupancy at each frame. This unified field mapping facilitates consistent interpretation of heterogeneous trajectory records while retaining the native data representation of each source dataset.
18/22
Field
Type
Description
event_id
string
source_dataset
string
ego_id risk_level frame_start, frame_end relation
integer categorical integer
Full event identifier, given by the file name (e.g. expresswayA_track_1_car_135_frame_640_to_735 or highd_01_113_precedingId_yaw_right_frame_2205_to_2312). Upstream source from which the event was extracted, given by the file-name prefix and folder (highd, ind, round, expresswayA, freewayB, ngsim). Identifier of the ego vehicle within the source recording. Severity grade given by the containing folder: normal (moderate), high or extreme. First and last frame of the extracted event window, spanning the lead-in, peak-risk and resolution phases.
categorical
behavior
categorical
Spatial role of the interacting agent relative to the ego vehicle (preceding, following, left/right-preceding, left/right-following, left/right-alongside); present in the highD file names. Triggering behavior of the interaction (acc_high, brake_high, yaw_left, yaw_right); present in the highD file names.
Table 5. Event-level descriptors encoded in the file name and the containing folder of each event. The road structure (highway, urban freeway, intersection or roundabout) follows from source_dataset, and the peak-risk frame lies inside the event window.
Component
Description
scenario description
Structured natural language summary of the road layout, lane topology, legal maneuvers, and the dynamic states of the ego vehicle and surrounding agents, generated per frame of the event window. Rule-based notifications attached to the description, keyed by the spatial relation and the triggered behavioral flag, that flag abnormal behavior and the presence of vulnerable road users. LLM-generated explanation of what produced the hazard, why it is safety-critical, how it evolves, and which response is recommended. Discrete decision drawn from the five-action schema (1 IDLE, 2 Turn Left, 3 Turn Right, 4 Acceleration, 5 Deceleration).
risk reminders risk analysis action_id
Table 6. Components of the natural language annotation stored in the text (JSON Lines) file.
C Worked Example of the Closed-Loop LLM Annotation Pipeline This appendix expands the schematic in Figure 7 into a concrete walk-through of one extreme-risk event from the K-Risk release. The event is drawn from the ExpresswayA source, in which ego vehicle 134 travels eastbound on lane 10 of an urban freeway. During the event window, preceding vehicle 113 in the same lane brakes sharply and nearly stops, causing the time-to-collision to drop to approximately 1.6 s. This rapid closing interaction places the ego vehicle in an imminent rear-end conflict, and the event is therefore included in the extreme subset and graded as extreme risk by the annotation protocol. Every fragment shown below is reproduced from the corresponding event files in the release, including the trajectory records, textual description, risk analysis and decision recommendation. Ellipses (. . .) indicate omitted frames or fields that are not essential to the walk-through. Before the field-by-field walk-through, Figure 8 and Figure 9 provide a graphical overview of the same closed-loop annotation process applied to an ExpresswayA event. In this example, ego vehicle 822 travels eastbound on lane 4, while the left-adjacent vehicle 819 accelerates and cuts toward the ego lane, driving the time-to-collision down to approximately 1.2 s. This interaction illustrates how a short temporal window is converted into a structured high-risk event with aligned textual, visual and trajectory records. Figure 8 shows how one such event is organized into the synchronized triple defined in Figure 5. The text layer records the environmental description, the risk analysis and the recommended decision. The visualization layer presents the preceding, critical and resolution frames of the event window, making the temporal evolution of the conflict directly inspectable. The JSON layer stores the per-frame trajectory records of the ego vehicle and its surrounding agents, including their positions, speeds, accelerations, headings and lane information where available. Figure 9 traces the corresponding annotation loop end to end. The loop begins with the structured scenario description and the system prompt, proceeds through the LLM-based risk analysis and action selection, and ends with a reflection step that uses validated outcomes to improve the consistency of the released annotations. The five stages that follow then examine a distinct ExpresswayA event, in which ego vehicle 134 encounters the sharply braking preceding vehicle 113. This case is read field by field to instantiate the same closed-loop annotation logic on a concrete extreme-risk scenario. 19/22
The scenario generator prepends a domain-specific system prompt that fixes the legal-action schema. The five action identifiers in this prompt are the schema referenced in the Usage Notes and in Table 3. Stage 1: System prompt and action schema.
You are ChatGPT, ..., now acting as a mature driving assistant. Your task is to make the decision that assures the safety. You have access to the same information as a real human driver, ... This scenario takes place at an expressway with exits. Your available actions include: 1. IDLE: Remain in the current lane with the current speed (Action ID: 1) 2. Turn Left: Change to the lane on the left of the current lane (Action ID: 2) 3. Turn Right: Change to the lane on the right of the current lane (Action ID: 3) 4. Acceleration: Increase vehicle speed (Action ID: 4) 5. Deceleration: Reduce vehicle speed (Action ID: 5) A clear action ID must be chosen at the end of your reasoning process.
The raw labeled trajectory stores per-frame kinematics together with the eight-surrounding-vehicle role columns (preceding_id, following_id, left_preceding_id, . . ., right_following_id), the behavioral flags (acc_label, brake_label, left_turn_label, right_turn_label) and the conflict fields (collision_in_2s, Time to Collision) used by the annotation protocol. The event-extraction step slices this trajectory into a list of per-frame, per-agent records and writes the JSON file shown below; only the peak-risk frame 769 for ego 134 and a subset of its surrounding agents are reproduced here. Stage 2: Trajectory record (CSV → JSON event slice).
[ { "car_id": 134, "frame_id": 769, "car_center_x": 134.71, "car_center_y": 55.94, "speed": 7.08, "course": 0.32, "lane_id": 10, "vx": 7.08, "vy": 0.04, "acceleration": -0.38, "yaw_rate": -0.21, "preceding_id": 113, "following_id": null, "left_preceding_id": 131, "left_alongside_id": 133, "left_following_id": 137, "right_preceding_id": null, "acc_label": false, "brake_label": false, "left_turn_label": false, "right_turn_label": false, "collision_in_2s": true, "collision_id": 113, "Time to Collision": 1.6 }, { "car_id": 113, "frame_id": 769, "car_center_x": 149.22, ..., "speed": 0.03, "acceleration": -5.17, "brake_label": true, "collision_in_2s": true, "collision_id": 134, "Time to Collision": 1.45 }, { "car_id": 131, "frame_id": 769, ... }, { "car_id": 133, "frame_id": 769, ... }, ... /* frames 679, 684, ..., 799 follow */ ]
The generator converts each retained frame into the structured English description that is later fed to the LLM. The Safety Reminder lines are produced from the rule-based risk filters of the annotation protocol, keyed by the spatial relation and the triggered condition, here a short time-to-collision and a sharply decelerating preceding vehicle. The excerpt below shows the peak-risk frame at frame = 769, when vehicle 113 has almost stopped directly ahead of the ego vehicle. Stage 3: Structured scenario description and rule-based risk reminders.
Frame 769: Ego Vehicle 134 is currently traveling in lane 10, a west-east lane. The available legal maneuvers for this lane include: Move straight to lane 11, change to left lane 8. - Position: (134.71, 55.94) m - Speed: 7.08 m/s, Heading: 0.32 degrees Vehicle 113 is driving directly ahead in your lane (lane 10). - Position: (149.22, 55.49) m, Speed: 0.03 m/s Vehicle 131 is ahead in your adjacent left lane (lane 8): 4.33 m/s. Vehicle 133 is laterally aligned in your adjacent left lane (lane 8): 5.01 m/s. Safety Reminders: - Preceding vehicle 113 is decelerating sharply. Immediately decelerate to maintain a safe distance. - [TTC Risk Assessment] Vehicle 113 poses a collision risk within 1-2 seconds. Estimated TTC: 1.6.
The structured description above is sent to the LLM together with the system prompt of Stage 1. The response below is reproduced from the generated annotation. It illustrates the three artefacts that K-Risk extracts per event: a free-form risk explanation, a decision rationale grounded in the eight-surrounding-vehicle roles, and a single legal action ID drawn from the schema of Stage 1. Stage 4: LLM risk analysis and chosen action ID.
20/22
Imminent collision warning: - Preceding vehicle 113 is directly ahead in the same lane, almost stopped (0.03 m/s), while the ego vehicle is moving at 7.08 m/s; TTC is about 1.6 s. - Any action that maintains or increases speed (IDLE, Acceleration) is unsafe. Surrounding-vehicle analysis: - Left-preceding vehicle 131 (lane 8): closing TTC below 5 s, so turning left is not allowed. - Left-alongside vehicle 133 (lane 8): a vehicle alongside on the left blocks a left lane change. - No legal right lane exists in this lane. After pruning unsafe actions, only deceleration remains. Given the imminent rear-end risk with a sharply decelerating vehicle ahead, and with both left-lane options blocked, the only safe and legal maneuver is to decelerate as quickly and smoothly as possible. Chosen Action ID: 5
(Deceleration: reduce vehicle speed)
The chosen action identifier is forwarded to the constant-acceleration, constant-steering simulator described in the annotation protocol, which rolls every relevant agent forward for two seconds under the predicted action. If any forecast pair of trajectories overlaps within this two-second horizon, the trial is flagged as a near-collision, the diagnostic message together with the new state are appended to the prompt, and the LLM is re-invoked. Iterating this loop three times produced the first, second and third trial collision rates of 4.58%, 2.67% and 1.91% on the 262 extreme-risk evaluation events (Table 2). The first-trial and third-trial outputs of every such event are persisted side by side, yielding the preference pairs that the Usage Notes use as direct supervision for preference optimization without further human labeling. Stage 5: Closed-loop simulation and reflection.
Text File
K-RISK
**Risk Analysis **: Vehicle 819 accelerates sharply while cutting into the right lane, creating a critical hazard. This maneuver endangers two vehicles: Vehicle 812 (right-following): Faces sudden intrusion ahead, TTC = 1.2s. Vehicle 822 (target lane): Vulnerable from behind, TTC = 0.8s. The abrupt lane change with minimal buffer heightens the risk of rearend or multi-vehicle collisions. **Decision Making for Vehicle 822**: -A potential cut-in from Vehicle 819 makes turning left, accelerating, or staying idle unsafe. -Turning right appears safe, as the adjacent lane is clear. -Decelerating is also safe, increasing the gap to Vehicle 819. -After applying safety rules, the viable options are Turn Right (ID: 3) and Decelerate (ID: 5). -Final Decision: Turn Right for maximum safety and smooth traffic flow.
**Frame 5740 Environmental Description**: =========================================================== Ego Vehicle 822 is currently traveling in lane 4, classified as a east-west direction lane. The available legal maneuvers for this lane include: Move straight, change to left lane 5, change to right lane 3. The vehicle is located at coordinates (231.84, 28.02), moving at a speed of 11.81 m/s. The vehicle's heading angle is 179.60 degrees. Vehicle 803 is driving directly ahead in your lane: lane 4 — a east-west lane. The legal actions for this lane include: Move straight, change to left lane 5, change to right lane 3. The vehicle is located at (185.06, 27.86), moving at a speed of 14.24 m/s. The vehicle's heading angle is 179.77 degrees. Vehicle 819 ... Vehicle 827 ... Safety Reminders: Left preceding vehicle 819 is shifting toward your lane. Prepare for a potential cut-in. Left preceding vehicle 819 is accelerating. Stay alert for overtaking.
├── Event_Annotations/ │ ├── ExpresswayA_01.json │ ├── highD_02.json │ ├── Waymo_03. json
├── LLM_Text/ │ ├── event_descriptions.txt │ ├── risk_explanations.txt │ ├── decision_reasoning.txt Event Visualization Preceding Frame 5720
Critical Frame 5740
Following Frame 5760
822 819
812
t=0
822
819
812
812
t=1
t=2
JSON File frame_id
car_id
center_x
center_y
5720 5720 5740 5740 5740 5760 5760 5760
819 812 822 819 812 819 822 812
230.5 217.83 227.88 219.67 210.52 206.95 219.84 203.15
30.65 31.76 28.14 29.98 31.76 29.09 28.14 31.88
speed acceleration yaw_rate
course
lane_id
length
vx
vy
preceding_id
following_id
left_preceding_id
right_following_id
ttc
14.33 10.85 12.2 17.26 11.03 19.89 12.44 11.43
179.77 179.39 179.69 181.14 179.53 180 179.61 179.68
5 5 4 5 5 4 4 5
4.63 4.69 4.63 4.87 4.52 4.75 4.58 4.52
-14.33 -10.85 -12.2 -17.26 -11.03 -19.89 -12.44 -11.43
0.06 0.12 0.07 -0.34 0.09 0 0.09 0.06
812 801 803 812 801 803 819 801
827 819 829 827 819 822 829 827
N/A N/A 819 N/A N/A 812 812 N/A
822 822 826 822 822 826 826 822
3.64 N/A 18.55 1.47 N/A 6.84 1.73 20.2
3.3 0.29 1.2 3.93 0.14 3.94 0.6 1.35
0.36 -0.77 0.21 2.86 -0.02 -1.94 -0.39 0.03
Figure 8. Worked example of a single K-Risk event, drawn from the ExpresswayA source. The text layer holds the environmental description, the LLM risk analysis and the decision for the ego vehicle (top left). Time-stamped snapshots show the preceding, critical and resolution frames of the event window (middle). The JSON layer lists the per-frame trajectory records, position, speed, acceleration, heading, lane and the eight surrounding-vehicle role identifiers, for the ego and its neighbors (bottom).
21/22
Frame 5710, Ego Vehicle 822: ======================================================= Ego Vehicle 822 is currently traveling in lane 4, classified as a east-west direction lane. The available legal maneuvers for this lane include: Move straight, change to left lane 5, change to right lane 3. At the current timestamp, the vehicle is located at coordinates (239.55, 28.02), moving at a speed of 11.64 m/s. The vehicle's heading angle is 179.52 degrees.
Textual Description
Vehicle 819 is laterally aligned in your adjacent left lane: lane 5 — a east-west lane. The legal actions for this lane include: Move straight, change to right lane 4. The vehicle is located at (235.31, 30.93), moving at a speed of 13.19 m/s. The vehicle's heading angle is 179.55 degrees. Vehicle 812 is ahead in your adjacent left lane: lane 5 — a east-west lane ... Vehicle 829 ... Vehicle 803 ... Vehicle 827 ....
Wrong
System Message ======================================================= You are ChatGPT, a large language model trained by OpenAI. You are now acting as a mature driving assistant. And your task is to make the decision that assures the safety.
Correct
**Decision Making:** - Given the potential cut-in from the left-preceding vehicle (ID: 819), turning left or accelerating would be unsafe.
You have access to the same information as a real human driver, which includes the following: 1. Description of the environment 2. Your own vehicle's position, dynamic information, and legally permissible actions in that lane. 3. Vehicles in your current lane and adjacent lanes, including their position, dynamic information, and legally permissible actions in that lane.
Reflection
- Decelerating is a safer choice, as it increases the longitudinal gap between the ego vehicle and vehicle 819. However, this action may lead to a less smooth driving experience and could also affect the following vehicle (ID: 829). Thus, deceleration is considered the baseline action. - Turning right is also a safe option, as there is no known vehicle in the right lane, suggesting it is clear.
This scenario takes place at an expressway. The positive x-coordinate points east, while the positive y-coordinate points south. The top-left corner is the origin (0,0). The velocity components follow the same convention: a negative vx indicates movement toward the west, while a negative vy indicates movement toward the north. Your available actions include: 1. IDLE: Remain in the current lane with the current speed 2. Turn Left: Change to the lane on the left of the current lane 3. Turn Right: Change to the lane on the right of the current lane 4. Acceleration: Increase vehicle speed 5. Deceleration: Reduce vehicle speed
Collision Free Test
K-Risk
Data LLM Annotation
- Therefore, the optimal action is to turn right, as it ensures safety while maintaining a smooth driving experience. Final Decision: Turn Right (Change to the right lane)
Input Backin Input
Risk Analysis
Risk Analysis: Left alongside vehicle 819 is accelerating. Avoid lane changes to the left. Left alongside vehicle 819 is turning right. Increase caution and prepare for evasion.
Figure 9. Worked example of the closed-loop LLM annotation on an ExpresswayA event. The structured scenario description of ego vehicle 822 and its neighbors, together with the domain-specific system prompt and its five-action schema, form the LLM input (left). The trajectory visualization marks the critical frame and the TTC of about 1.2 s, with the safe and unsafe action regions indicated (top right). The LLM produces a risk analysis and a decision rationale and selects a single action, here a right lane change away from the accelerating left-adjacent vehicle 819 (right). The chosen action is checked in simulation, and the reflection step feeds the validated risk analysis and decision back into the K-Risk release (bottom).
22/22