IEEE INTERNET OF THINGS JOURNAL
1
Large Language Model Enhanced Differentiable Trajectory Planning for IoT-Enabled Autonomous Driving
arXiv:2607.10438v1 [cs.RO] 11 Jul 2026
Shihao Zhang, Jing Yang, Ziyu Song, Zheng Lin, Sunil Prajapat, Zhaochen Xia, Hemant Ghayvat, Haitao Ding, Lip Yee Por, Ashok Kumar Das
Abstract—Autonomous driving planning is a key component of IoT-enabled intelligent transportation systems, requiring vehicles to generate safe, efficient, and executable trajectories in complex urban environments from multi-source contextual information. While imitation learning (IL) has shown promise on large-scale datasets, IL-based planners still suffer from limited coverage of complex long-tail interactions, weak consistency with downstream constrained refinement, and insufficient use of high level scene semantics under real time constraints. To address these issues, this paper proposes a large language model (LLM) enhanced differentiable trajectory planning framework for IoT-enabled autonomous driving. Specifically, we introduce a surrounding agent centric data augmentation strategy to reorganize surrounding agent trajectories as additional planning supervision, thereby improving the training distribution without collecting additional raw data. We further design a complexity-aware asynchronous LLM-based semantic enhancement module to extract scene-related high-level semantic features with controlled online overhead. In addition, a differentiable optimization module is incorporated to refine generated trajectories with explicit residual penalties while backpropagating optimization gradients to the upstream planner. Experiments show that the proposed method achieves the best overall scores of 83.63 and 78.29 on the nuPlan closed-loop nonreactive and reactive Hard20 benchmarks, respectively, and CARLA-ROS tests further verify its online deployment and real time closed-loop execution capability. Index Terms—Imitation Learning, Differentiable Optimization, Large Language Model, Trajectory Planning, Connected Autonomous Driving.
I. I NTRODUCTION
This work was supported by the China Automobile Industry Innovation and Development Joint Fund under Grant No. U1864206, and China Shenzhen Major Science and Technology Projects under Grant No. ZDCY20250901101003004. (Corresponding author: Haitao Ding.) Shihao Zhang, Ziyu Song, Zhaochen Xia and Haitao Ding are with the State Key Laboratory of Automotive Simulation and Control, Jilin University, Changchun 130000, China (e-mail: [email protected]; [email protected]; [email protected]; [email protected]). Jing Yang and Por Lip Yee are with the Center of Research for Cyber Security and Network (CSNET), Faculty of Computer Science and Information Technology, Universiti Malaya, 50603 Kuala Lumpur, Malaysia (e-mail: [email protected]; [email protected]). Zheng Lin is with the Interdisciplinary Centre for Security, Reliability and Trust (SnT), University of Luxembourg, Luxembourg (e-mail: [email protected]). Sunil Prajapat and Hemant Ghayvat are with the IMT, Department of Humanities and Technology, Roskilde University, Roskilde, Denmark (e-mail: [email protected]; [email protected]). Ashok Kumar Das is with the Center for Security, Theory and Algorithmic Research, International Institute of Information Technology, Hyderabad 500 032, India, and also with the Department of Computer Science and Engineering, College of Informatics, Korea University, 145 Anam-ro, Seongbuk-gu, Seoul 02841, South Korea (e-mail: [email protected], [email protected]).
UTONOMOUS driving systems are a key component of IoT-enabled intelligent transportation services [1]–[4], where vehicles integrate multi-source contextual information, including surrounding traffic participants, maps, traffic states, and navigation instructions, to generate safe, efficient, and socially compliant trajectories in complex environments [5]– [7]. Traditional rule-based planners rely on manually designed rules and objective functions, which often limits their adaptability and generalization in dynamic and uncertain traffic scenarios. To address these limitations, researchers have explored model-based approaches, including imitation learning (IL) [8], reinforcement learning (RL) [9]–[11], and hybrid methods [12], improving the flexibility of autonomous driving systems in complex environments. Among these approaches, imitation learning has attracted considerable attention because it can leverage expert demonstrations to learn complex driving behaviors, such as lane changing, obstacle avoidance, and interactions with surrounding traffic participants. Several IL-based planning models [13]–[15] have achieved promising performance on large scale real world datasets such as nuPlan [16]. However, existing IL-based methods still face several critical challenges in practical deployment for complex urban mobility services [17]. First, imitation learning relies heavily on expert driving data during training. Although high quality public datasets such as nuPlan and the Waymo Open Dataset [18] are available, highly interactive, complex, and rare scenarios remain relatively scarce. From a data centric IoT intelligent transportation perspective, this long-tail imbalance may limit the robustness and generalization of learning-based planners in safety critical urban mobility services. Our statistical analysis of approximately 2 × 107 nuPlan scenarios further confirms this issue (in Fig. 1): simple scenarios dominate the dataset, whereas complex and rare scenarios account for only a small proportion. Since prior studies have also suggested that data quality can be more critical than data volume for model performance [19], enriching complex traffic behavior samples becomes important for improving the practical utility of autonomous driving planning models. Second, in highly interactive traffic environments, pure imitation learning models often struggle to maintain stable and robust performance, and their generated trajectories still leave room for improvement in safety, feasibility, and comfort [20]– [22]. To improve controllability, existing methods commonly combine IL planners with rule-based post processing, filtering, or hybrid planning strategies [23], [24]. However, these mechanisms are usually applied at the output stage and can-
A
IEEE INTERNET OF THINGS JOURNAL
Fig. 1: Scenario distribution statistics of the nuPlan dataset. The dataset shows a typical long-tail distribution: Simple scenarios dominate, while constrained, complex, and especially rare scenarios are less frequent but critical for evaluating robustness and generalization. SST: Simple Scenario Types; CIST: Constrained Interaction Scenario Types; CST: Complex Scenario Types; RST: Rare Scenario Types.
not sufficiently feed safety boundaries, kinematic constraints, or interaction risks back into the upstream planner during training. This may lead to inconsistency between generated candidate trajectories and downstream constrained refinement objectives. In addition, many existing methods loosely couple prediction and planning, making it difficult for prediction results to fully support planning decisions and for planning objectives to guide trajectory generation. Although unified prediction and planning frameworks can implicitly model interactions, they may still lack sufficient stability and robustness in safety critical scenarios [25]. Furthermore, imitation learning models are prone to performance degradation in out of distribution long-tail scenarios due to limited generalization capability. Large language models (LLMs) [26]–[30] have shown promising commonsense reasoning and zero-shot generalization ability, offering a new opportunity for enhancing scene understanding in complex driving scenarios. However, directly applying LLMs to autonomous driving planning remains challenging. Critical traffic information, such as road topology, dynamic interactions, and fine-grained spatiotemporal context, is difficult to represent fully in compact text. Moreover, large parameter scale and inference overhead of LLMs further make direct high frequency deployment difficult for real time closed-loop planning in IoTenabled intelligent transportation systems [26]. To address the above challenges, we develop a unified ILbased planning framework by integrating surrounding agent centric data augmentation, complexity-aware asynchronous LLM-based semantic enhancement, and residual-based differentiable optimization. The main contributions of this work are summarized as follows: 1) We propose an LLM enhanced IL-based planning framework with differentiable optimization. The proposed
2
framework leverages a complexity-aware asynchronous LLM to provide high level scene semantic guidance, while a differentiable optimization module imposes explicit constraints on the output trajectories and backpropagates optimization gradients to the upstream planning model, thereby improving scene understanding and closed-loop planning performance in complex scenarios. 2) We develop a surrounding agent centric data augmentation strategy. By reorganizing the real trajectories of surrounding traffic participants in complex scenarios as additional planning supervision, the proposed strategy increases the proportion of high value training samples, which improves the training distribution and enhances the model’s generalization capability in complex scenarios without collecting additional raw data. 3) Comprehensive experiments on the nuPlan closed-loop benchmarks demonstrate that the proposed method achieves the best overall scores of 83.63 and 78.29 on the nuPlan nonreactive and reactive challenges on the Hard20 split, respectively. In addition, experiments on the real time CARLA-ROS software-in-the-loop (SIL) platform further verify the online deployment feasibility and real time closed-loop execution capability of the proposed framework. II. RELATED WORK A. Data Augmentation for Autonomous Driving Existing data augmentation methods for autonomous driving can be broadly categorized into perception level, scene level, and trajectory level approaches. Perception level methods increase input diversity through image transformations, noise perturbations, or weather variations [31]. Scene level methods enrich traffic scenarios through simulation reconstruction, digital twins, or generative models, including LLM-based scenario generation [32]. Trajectory level methods reuse expert trajectories, behavioral patterns, or interaction processes as planning-related demonstrations [33]. Despite these advances, non-ego motion records contained in real driving logs remain underexploited as planning demonstrations. Our surrounding agent centric data augmentation strategy therefore reindexes selected surrounding vehicles as planning subjects, allowing one traffic episode to provide multiple agent centric training samples without additional data collection. B. IL-based Planning Existing IL-based planners mainly fall into three categories. The first category consists of pure IL-based planners, which directly map scene representations to planned trajectories such as PlanT and PlanTF [13], [15]. The second category focuses on interaction-aware or joint prediction and planning methods, which explicitly model the coupling between ego decisions and surrounding agent behaviors to handle complex dynamic scenarios, as exemplified by GameFormer [34]. The third category [23], [24] combines IL with output-stage trajectory enhancement. In this line of work, post processing evaluates
IEEE INTERNET OF THINGS JOURNAL
or ranks candidate trajectories after inference without modifying the trajectory generation process, whereas trajectory refinement directly adjusts the geometry or state sequence of a selected trajectory to improve performance of the trajectory. Inspired by recent progress in IL-based planning [24], [25], [35], [36], we employ a residual-based differentiable optimizer to refine the selected initial ego trajectory using the predicted trajectories of surrounding agents as conditional inputs. Since the optimization process is differentiable, gradients associated with the residual objectives can be backpropagated through the solver to the upstream planning network, allowing the optimization objectives to guide initial trajectory generation during training. C. LLM for Autonomous Driving Existing studies can be broadly grouped into three directions. First, some works combine natural language instructions with multimodal perception inputs for closed-loop or goaloriented driving. For example, LMDrive integrates multimodal sensor inputs and language instructions for closed-loop endto-end driving, LeGo-Drive uses language guided goal representations for closed-loop trajectory generation, and DriveLM formulates driving scene understanding as graph-based visual question answering for perception, prediction, and planning reasoning [37]–[39]. Second, another line focuses on rule understanding, decision reasoning, and interpretable driving cognition. Driving with Regulation incorporates traffic laws, regulations, and safety requirements into driving decision making through retrieval augmentation and language reasoning, while Reason2Drive provides chain-based reasoning data for interpretable autonomous driving [40]. Third, survey studies such as LLM4AD summarize representative LLMbased paradigms and show their potential in scene understanding, rule reasoning, and decision support [41]. These studies indicate that LLMs are more suitable as sources of high level knowledge and semantic information, complementing downstream decision making and planning modules rather than replacing conventional planners. Nevertheless, applying LLMs [42]–[46] to autonomous driving remains challenging because high level language semantics do not directly match the continuous and geometry-constrained representations required for trajectory level planning, and closed-loop LLM inference still incurs considerable real time overhead. Recent LLM-assisted driving studies, including LMDrive [37] and AsyncDriver [47], motivate using language models as high level semantic feature providers for real time planning. Following this direction, this paper introduces an asynchronous LLM-based scene-associated feature extraction module for an IL-based real time planner. A lightweight scene complexity recognition mechanism further adaptively schedules semantic feature updates according to traffic density, interaction risk, navigation changes, and short term scene variations. The extracted semantic features are fused with scene encodings and coupled with differentiable trajectory refinement, enabling scene level semantic guidance to support trajectory proposal generation and optimization-aware refinement while keeping online computation under control.
3
III. METHODOLOGY A. Problem Formulation This paper studies the decision making and motion planning problem for autonomous driving in urban traffic environments. Let the autonomous vehicle be denoted by A0 , and the other t A traffic participants by {Ai }N i=1 . Let si denote the state of the ith traffic participant at time step t, and let TH and TF represent the historical observation horizon and the future planning horizon, respectively. At the current time step t = 0, given the historical state sequence S = {sti | i = 0, . . . , NA , t = −TH , . . . , 0}, the map scene context M , and the high level language information I, the model is required to output a set of NT multimodal initial planning trajectories for the ego vehicle, T T0 = {τ0n }N n=1 , together with their corresponding confidence scores, while simultaneously predicting the trajectories of surrounding traffic participants over the next TF time steps. Subsequently, the model further refines the ego trajectory using a differentiable optimizer. As illustrated in Fig. 2, the proposed framework combines IL-based planning, asynchronous LLM-based semantic enhancement, and differentiable optimization. Map, ego, and surrounding agent features are first encoded by a Transformer encoder into a unified scene representation. The system prompt, navigation instructions, and encoded scene representation are then organized as multimodal LLM inputs to extract scene-associated instruction features, which are adapted and asynchronously updated before being injected into the planning module. The planning decoder generates multimodal initial trajectories and confidence scores, while the prediction decoder estimates future trajectories of surrounding agents. Finally, the differentiable nonlinear optimizer refines the highest confidence ego trajectory using predicted agent trajectories and residual-based cost terms. Since the optimizer is differentiable, planning-related gradients can be propagated back to the upstream planner during training, enabling joint learning of trajectory generation and optimization-aware refinement. B. Surrounding Agent Centric Data Augmentation IL-based planning typically uses only the ego vehicle’s future trajectory as supervision, which limits the learned behavior prior to a single planning subject in each scene. To increase the coverage of complex interactions, we propose a surrounding agent centric data augmentation strategy that reuses the real trajectories of surrounding traffic participants as additional planning supervision. The key idea is to reinterpret the same traffic scene from multiple agent centric perspectives while preserving its original semantics and interaction structure. Candidate surrounding vehicles are collected within a predefined screening radius and prioritized according to interaction relevant behaviors, including lane changing, intersection traversal or turning, low time to collision (TTC) interactions, high lateral acceleration, and high magnitude speed behaviors. To reduce noise from perception errors or incomplete tracking, we retain only target vehicles with valid observations, temporally continuous trajectories, and physically plausible motion patterns. After filtering, the scene is recentered around each retained target vehicle, so that
IEEE INTERNET OF THINGS JOURNAL
4
Asynchronous LLM-based Scene-Associated Feature Extraction LLM Multi-Modal Input System Prompt
... ...
Scene-Associated Instruction Feature
IL-based Planning with Differentiable Optimization
Complexity-aware Asynchronous Inference Controller
LLM Key & Value
Polyline Qlat Encoder Lateral & Longitudinal SelfAttn Qlon Longitudinal queries
...
Transfomer Encoder
...
...
Navigation Prompt
...
Map Features
Ego Features
Feature Adapter
LLM (Llama2-13B)
Traj & Probs, MLP
LLM CrossAttn Query Query2Scene CrossAttn Key & Value
Adaptive Gate ×Ldec
Initial Traj & Probs
...
Cost Functions
Agent Features
Predictions Decoder
Predictions
×Lenc
Final Plan Differentiable Nonlinear Optimizer
Fig. 2: Overview of the proposed planning framework.
a single traffic episode can provide multiple training samples with different planning subjects. After selecting a target vehicle, the original scene is reformulated in a local coordinate system centered on that vehicle. Let pc and θc denote the current position and heading of the target vehicle, and let p denote the position of an arbitrary scene element in the global coordinate system. Its representation in the target-centered coordinate system is given by p̃ = R(−θc )(p − pc )
(1)
where R(−θc ) denotes the two-dimensional rotation matrix determined by the target vehicle heading. Through this transformation, map elements, surrounding traffic participants, and the target vehicle state are consistently mapped into the new local frame, thereby reindexing the same scene as a new planning sample. To maintain feature consistency with the downstream planner, missing dynamic states are approximated from trajectory sequences. Let θt and vt denote the heading angle and speed of the target vehicle at time step t, respectively. The yaw rate and longitudinal acceleration are approximated as ωt ≈
θt+1 − θt−1 , 2∆t
at ≈
vt+1 − vt−1 2∆t
(2)
where ∆t is the time interval between two consecutive time steps. In this way, surrounding agent trajectories are converted into additional planning samples that enrich complex behavior patterns without collecting new raw data.
C. Asynchronous LLM-based Scene-Associated Feature Extraction To enhance the planner’s ability to capture complex traffic scenes, navigation intent, and high level semantic constraints, we introduce an asynchronous LLM-based scene-associated feature extraction module for the downstream planning backbone, following the asynchronous semantic feature extraction paradigm [47]. In this module, the LLM encodes the system prompt, navigation instructions, and structured scene representation into hidden semantic features. These features provide high level scene and instruction guidance to the planning decoder, while trajectory candidates, confidence scores, and final optimized trajectories are produced by the IL-based planner and the differentiable optimizer. Since a general purpose pretrained large language model cannot be directly applied to autonomous driving planning, we adopt a lightweight adaptation strategy to align the language model with scene-semantic understanding and navigation instruction modeling, while introducing an asynchronous inference mechanism to reduce its online computational overhead. Specifically, the high level language input I consists of a system prompt Isys and navigation instructions Inav . The system prompt follows the fixed driving role template: “Role: You are now an autonomous driving driver. I will provide you with the environment information, including ego vehicle information, surrounding agent information, and map information. Please extract scene-associated instruction features based on the given environmental information and routing instructions.” The navigation instructions are generated from the reference path or route through a rule-based procedure and contain maneuver
IEEE INTERNET OF THINGS JOURNAL
5
commands with distance information. The structured scene representation zscene , extracted by the scene encoder from ego vehicle states, surrounding agent features, and map features, is formatted together with the tokenized system prompt and navigation instructions to form the multimodal LLM input xllm = Φ(Isys , Inav , zscene )
(3)
where zscene denotes the encoded scene representation and Φ(·) denotes the multimodal input construction process, including language tokenization, scene feature formatting, and the input alignment required by the LLM interface. In this way, the structured scene representation is converted into LLMcompatible contextual input, allowing the LLM to jointly encode navigation instructions and traffic scene information. The large language model then encodes the multimodal input and outputs the hidden feature sequence Hllm = {h0 , h1 , . . . , hNtok −1 } ∈ RNtok ×Dllm
(4)
where Ntok denotes the number of valid input tokens after multimodal input construction and Dllm denotes the hidden dimension of the LLM. Following the scene-associated instruction feature extraction strategy, the hidden feature of the last valid token is selected as the compact semantic representation: fsem = hNtok −1 (5) The feature adapter then maps fsem into the plannercompatible feature space: Fsem = Linear(fsem )
(6)
where Fsem ∈ RDp and Dp = 128 in our implementation. The adapted feature captures navigation intent, scene context, and high level behavioral preferences associated with the current driving task, and is used as the scene-associated instruction feature for the downstream planning decoder. Because LLM inference is computationally expensive, invoking it synchronously at every planning step would substantially increase online latency and reduce the practicality of high frequency closed-loop planning. Existing studies have shown that scene complexity and criticality in automated driving can be reasonably characterized by interpretable indicators such as traffic density, interaction intensity, topological complexity, and surrogate safety measures including TTC [48]. Meanwhile, event-triggered update strategies have been shown to reduce unnecessary computation in real time autonomous driving systems while maintaining comparable control performance [49]. Motivated by these observations, we replace the fixed asynchronous update strategy with a rulebased complexity-aware asynchronous scheduler. Specifically, a lightweight scene complexity estimator is introduced on top of the scene encoder outputs and several readily available geometric and interaction statistics. At time step t, the scene complexity score is defined as Ct = α1 N̄t + α2 N̄tconf + α3
1 TTCmin + ϵttc t
¯t + α4 Iint (t) + α5 Inav (t) + α6 ∆
(7)
where N̄t denotes the normalized number of neighboring agents, N̄tconf denotes the normalized number of potentially denotes the normalinteractive or conflicting agents, TTCmin t ized minimum TTC between the ego vehicle and surrounding agents, and ϵttc is a small positive constant used to avoid numerical instability, Iint (t) indicates whether the ego vehicle is currently located in a topologically complex area such as an intersection, merge, or turning region, Inav (t) indicates whether the current navigation command involves a high level ¯t maneuver change such as turning or lane changing, and ∆ measures the normalized short term scene variation, such as changes in surrounding agent states or traffic light conditions. All continuous complexity-related factors are normalized to comparable ranges before computing Ct , and the coefficients αi are set to equal values in the default configuration to assign the above factors the same prior importance. This setting keeps the scheduler interpretable and avoids introducing additional trainable parameters into the asynchronous update policy. Based on the estimated complexity score, the semantic feature reuse length is adaptively adjusted according to a rulebased scheduling policy: Kmin , Ct ≥ τh , (8) Ktsem = Kmid , τl ≤ Ct < τh , Kmax , Ct < τl . where τh and τl denote the high and low complexity thresholds, respectively, and Kmin < Kmid < Kmax denote the corresponding numbers of planning frames for reusing the most recent semantic feature. In our implementation, τl = 0.35 and τh = 0.65 are empirically selected according to the validation split statistics and the online latency budget, while Kmin = 3, Kmid = 9, and Kmax = 29 frames are used to balance semantic freshness and online inference efficiency. Here, Ktsem is measured in frames, while ∆t is reserved for the physical sampling interval used in trajectory state differentiation. In this way, the LLM is invoked more frequently in highly interactive or rapidly changing scenes, while in simple or stable scenes the most recent semantic feature is reused for a longer period. Let the k-th semantic update time be tk , let the corresponding (k) semantic feature be fsem , and let Kksem be the reuse length selected at this update time. The semantic feature used by the planning module can then be written as t (k) fˆsem = fsem ,
tk ≤ t < tk + Kksem
(9)
This complexity-aware scheduling strategy improves the trade-off between semantic richness and online efficiency. On the one hand, it avoids unnecessary LLM invocations in low complexity scenarios where high level semantics evolve slowly. On the other hand, it allows the semantic enhancement module to respond more promptly in high complexity scenarios involving dense interactions, topological changes, or increased collision risk. After the asynchronously updated semantic feature is obtained, it is injected into the downstream planning module and jointly used with scene encoding features for multimodal trajectory generation and surrounding agent behavior modeling. In this way, the planner can better capture high level semantic constraints and behavioral preferences
IEEE INTERNET OF THINGS JOURNAL
6
associated with the current driving task, thereby improving planning performance in complex and long-tail traffic scenarios. D. IL-based Planning with Differentiable Optimization Within the proposed framework, we build a unified decisionmaking and planning module that combines IL-based planning with differentiable optimization. The IL planner generates initial ego trajectories and predicts surrounding agent behaviors from scene encoded features and asynchronously updated scene-associated instruction features. The differentiable optimizer then refines the selected ego trajectory using predicted agent trajectories and explicit residual costs, enabling joint learning of trajectory generation and constraint-aware refinement. Following [24], the planning decoder represents multimodal driving behaviors using lateral and longitudinal queries. Reference-line polyline features are encoded as lateral queries, while longitudinal behaviors are modeled by learnable longitudinal query vectors. The fused queries are further processed by Query2Scene CrossAttn and LLM CrossAttn, where the former models road geometry, surrounding agents, and scene context, and the latter injects navigation intent and high level scene semantics from the asynchronous LLM feature. To prevent the semantic branch from excessively interfering with the original planning backbone, we introduce an adaptive gate to control the semantic injection strength. Let the query at the l-th decoder layer be q l , the scene feature be Fscene , and the adapted semantic feature be Fsem . The query update is formulated as q l+1 = CrossAttn(q l , Fscene , Fscene ) + g l · CrossAttn(q l , Fsem , Fsem ),
(10)
where g l is the learnable gate at the l-th layer. The gate is initialized to zero, allowing the model to first rely on the original scene branch and then gradually learn the contribution of semantic guidance. The enhanced decoder features are fed into trajectory and score prediction heads to generate multimodal initial trajectories and confidence scores. After initial trajectory generation and surrounding agent prediction, the selected ego trajectory is refined by a differentiable F optimizer. The ego trajectory is denoted by u = {ut }Tt=1 , where ut = {xt , yt , θt , vt }. In our implementation, TF = 80, and the optimizer operates on 80 × 4 trajectory variables. The candidate trajectory predicted by the upstream IL planner is converted into the (x, y, θ, v) representation and used as the optimizer initialization. Given the predicted surrounding agent trajectories ŝ, reference-line information, and the current scene state, the final trajectory is obtained by solving 1X 2 ∥ωi ci (u, ŝ)∥2 , (11) u∗ = arg min u 2 i where ci (·) is the i-th residual term and ωi is its weight. The residuals encode driving efficiency, comfort, safety, and kinematic feasibility as differentiable soft penalties. During
the inner optimization, ŝ is fixed as the conditional prediction input and is updated at the next planning step. The optimizer contains four categories of residual terms. First, for driving efficiency and reference-line consistency, the speed residual is cspeed = vt − vlimit,t , t
(12)
where vlimit,t is the local speed limit matched from the reference line. The position and heading residuals are clane-xy,t = pt − pref clane-θ,t = wrap θt − θtref , (13) t , ref where pt = (xt , yt ), and pref are the nearest matched t and θt reference-line position and heading. The heading residual is evaluated only at selected time steps to reduce optimization dimensionality and improve stability. Second, for comfort, longitudinal acceleration and jerk are constrained. With at = (vt+1 − vt )/∆t and jt = (at+1 − at )/∆t, the residuals are
cacc,t = 0.01|at | + max(at − amax , 0) + max(amin − at , 0), cjerk,t = 0.01|jt | + max(|jt | − jmax , 0), (14) 2 2 where amax = 2.40 m/s , amin = −4.05 m/s , and jmax = 3 4.13 m/s , following the nuPlan criteria [16]. The small L1 terms suppress unnecessary acceleration and jerk fluctuations. Third, for safety, valid surrounding agents entering the local risk corridor of the ego trajectory are selected in the Frenet frame. The corridor is determined by longitudinal relation and lateral offset thresholds around the reference line, covering forward path-overlap conflicts and lateral or crossing agents entering the ego driving corridor. For each checked time step, the nearest or highest-risk target is used to define Lego + Lobj +5.0, 2 (15) is the Euclidean distance to the selected conflict where dnear t target. The size term accounts for vehicle lengths, and the additional 5.0 m provides a conservative urban safety buffer. Rear-end risks are mainly handled by the upstream interactionaware prediction and planning module, closed-loop feedback, and safety-related evaluation metrics. This selective design balances collision-risk coverage and online efficiency. Finally, for kinematic feasibility, planar consistency residuals are defined using a discrete bicycle model: csafety,t = max(dsafe −dnear , 0), t t
dsafe = t
cxkin,t = (xt+1 − xt ) − vt cos θt ∆t, cykin,t = (yt+1 − yt ) − vt sin θt ∆t.
(16)
The heading rate and curvature are computed as θ̇t = wrap(θt+1 − θt )/∆t and κt = θ̇t / max(vt , 1.0). The steering angle is δt = arctan(Lκt ), with L denoting the wheelbase. The steering and steering-rate residuals are csteer,t = clip(δt ) and c∆steer,t = (δt+1 − δt )/∆t, which suppress infeasible steering actions and rapid steering variations. The residual weights are generated by a lightweight cost weight network. The residual forms and physical thresholds are manually specified according to driving priors, while the network adjusts the relative importance of driving efficiency,
IEEE INTERNET OF THINGS JOURNAL
7
comfort, safety, and kinematic feasibility. A fixed categorylevel scaling factor is also used to emphasize safety-related terms, preserving interpretability while allowing end-to-end weight adaptation. For optimization, we employ the Levenberg–Marquardt solver in Theseus [50]. The solver is initialized by the upstream predicted trajectory. At each iteration, residuals and Jacobians are computed through automatic differentiation, and trajectory variables are updated by solving the damped normal equation. We set the maximum number of LM iterations to 5, the step size to 0.2, and the absolute error tolerance to 1 × 10−3 . The initial damping value is 1 × 10−1 , and adaptive and ellipsoidal damping are enabled with a stability parameter of 1 × 10−6 . During training, the executed LM iterations are retained in the computation graph, allowing gradients from the planning loss and optimization cost to propagate through residual evaluation, linear solving, and trajectory updates to the upstream planner. E. Learning Process We adopt a staged training strategy. For the LLM component, we follow the alignment assistance loss in [47] for pretraining and fine tuning. After this stage, the LLM backbone is frozen during planner training, and the extracted scene-semantic features are used by the downstream planning module. The alignment assistance loss is formulated as Lalign = L1 (x̃va , xva ) + CE(x̃dec , xdec ) + CE(x̃traf , xtraf ) + BCE(x̃adj , xadj ) + BCE(x̃chg , xchg ) (17) where x̃va , x̃dec , x̃traf , x̃adj , and x̃chg denote the predictions of ego velocity and acceleration, future velocity decision, traffic light state, adjacent lane existence, and future lane change demand, respectively, and xva , xdec , xtraf , xadj , and xchg are the corresponding labels. For the planning module, the endpoint of the expert trajectory is projected onto the reference line to determine the target reference line and longitudinal query, yielding the target supervision trajectory τ̂ and its associated one-hot distribution π0∗ [24]. The supervision trajectory is then passed through the differentiable optimizer to obtain the optimized trajectory τ ∗ , which is supervised by the expert trajectory τ gt . The planning and confidence supervision terms are defined as Lplan = Lsmooth (τ ∗ , τ gt ), 1
(18)
Lcls = CE(π0 , π0∗ )
(19)
where π0 denotes the predicted confidence distribution over multimodal ego trajectory proposals, and π0∗ denotes the target one-hot distribution. The IL supervision term is Limi = Lplan + Lcls . ∗
(20)
Since τ remains differentiably connected to the initial trajectory proposal, the gradient of Lplan is propagated through the executed LM iterations to the upstream planning decoder and the cost weight generation network. For surrounding agent prediction, we follow the prediction loss in PLUTO. Let P1:NA denote the output of the prediction
gt decoder and P1:N denote the corresponding ground truth A future trajectories. The prediction loss is gt Lpred = Lsmooth (P1:NA , P1:N ). 1 A
(21)
Following [25], we also incorporate the overall optimization cost Lcost , which is computed from the manually structured residual terms with weights produced by the cost weight generation network. The overall objective used in the staged training pipeline is L = λalign Lalign + λpred Lpred + λimi Limi + λcost Lcost
(22)
where λalign , λpred , λimi , and λcost are the corresponding loss weights. IV. EXPERIMENT A. Experimental Setup To evaluate the proposed method from both benchmark and online system perspectives, experiments are conducted on the nuPlan closed-loop benchmark and a CARLA-ROS platform. The nuPlan benchmark provides reproducible quantitative comparison under the official closed-loop evaluation protocol, while the SIL platform validates online deployment and closed-loop execution under interactive simulation and cross module communication. For standardized benchmarking, we evaluate the proposed method on the nuPlan dataset following its official closed-loop protocol. The closed-loop score considers collision avoidance, drivable area compliance, TTC margin, speed limit compliance, ride comfort, and route progress, providing a comprehensive assessment of planning performance in complex urban traffic environments [16]. We compare the proposed method with representative baselines under the same evaluation setting, including IDM [51], UrbanDriver [14], GCPGP [52], PDM-Closed [23], PlanTF [15], Diffusion Planner [53], PLUTO [24], and AsyncDriver [47]. Beyond the benchmark evaluation, we conduct system level validation on a CARLA-ROS platform, whose communication architecture is shown in Fig. 3. CARLA provides the interactive traffic environment and publishes multi-source messages to ROS through Carla Ros Bridge, including IMU, GNSS, vehicle states, object states, and navigation information. The planner generates trajectories from these inputs, and the tracking controller converts them into control commands for CARLA execution. The LLM module and the real time planner are implemented as separate ROS nodes: the former publishes LLM features, while the latter performs trajectory planning inference upon receiving them. This decoupled design supports real time SIL validation. B. Implementation Details This subsection presents the main implementation details and key hyperparameter settings. The model is trained on an NVIDIA RTX 4090 GPU with a batch size of 128 for 25 epochs. The learning rate is linearly warmed up to 1 × 10−3 during the first 3 epochs and then decayed using a cosine schedule. For the LLM branch, we follow implementation
IEEE INTERNET OF THINGS JOURNAL
8
Carla Ros Bridge /Vehicle status
/GNSS Position
Map Info Process
/IMU
Dynamics info
Attitude info
/Objects
/Plan
Objects info
Carla Simulator
Navigation info
Planner
AsyncLLM
Lightweight Map
LLM Feature
Control cmd
Controller PID+LQR
Planning Trajectory
Realtime Planner
Map Engine Map info
Fig. 3: SIL platform communication architecture. TABLE I: Key hyperparameter settings of the proposed method. Parameter Historical observation horizon TH Future planning horizon TF Number of surrounding traffic participants NA Target vehicle screening radius (data augmentation) Dimension of instruction feature Fsem Low complexity threshold τl High complexity threshold τh Minimum semantic feature reuse length Kmin Medium semantic feature reuse length Kmid Maximum semantic feature reuse length Kmax
Value 20 80 20 50 m 128 0.35 0.65 3 frames 9 frames 29 frames
in [47] and adapt a LLaMA series language model with low rank adaptation (LoRA) using instruction style samples constructed from nuPlan training scenarios [28]. The original LLM backbone is frozen, while the LoRA parameters, feature adapter, adaptive gate, and downstream planning modules are trainable. The LLM branch runs in half precision during inference. In the loss function, the weights of all main task losses are set to 1.0, while the weight of the optimizationrelated auxiliary loss is set to 1 × 10−2 . The remaining implementation hyperparameters are summarized in Table I. V. RESULTS AND DISCUSSION A. Comparison with Baselines Table II reports the comparison results on the nuPlan closedloop nonreactive challenge benchmark. The proposed method achieves the best overall score of 83.63, while also attaining the highest values on the safety-related metrics of Coll. and TTC, reaching 95.97 and 85.14, respectively. At the same time, it maintains strong performance on Drivable and Prog. These results indicate that the gain does not come from optimizing a single metric in isolation, but from a better overall balance among safety, trajectory feasibility, and driving efficiency. This can be attributed to three aspects. First, the surrounding agent centric data augmentation strategy increases the coverage of complex interactions and high value behavior
samples, thereby improving the model’s ability to learn from long-tail scenarios. Second, the differentiable optimization module jointly incorporates speed constraints, reference line consistency, safety margins, and kinematic constraints into trajectory refinement, and improves the alignment between trajectory generation and final execution objectives through endto-end training. Third, the asynchronous LLM module further complements the planner with navigation intent and scenesemantic information, enabling the generated trajectories to better satisfy task requirements in complex scenarios. The behavior of representative baselines further reflects the influence of different design choices. PLUTO performs strongly when equipped with post processing, whereas PLUTO w/o refine. drops substantially from 80.82 to 74.26, indicating that output stage refinement plays an important role in its closed-loop performance. AsyncDriver improves planning through asynchronous semantic enhancement, but its lack of explicit trajectory refinement limits its gains in safetyrelated and feasibility-related metrics. Diffusion Planner shows competitive trajectory generation ability, but it does not explicitly incorporate downstream residual-based optimization during planning. These comparisons suggest that combining high quality augmented supervision, semantic guidance, and differentiable trajectory refinement is beneficial for achieving more balanced closed-loop performance. Table III presents the comparison results on the nuPlan closed-loop reactive challenges benchmark, where surrounding traffic participants can respond to the ego vehicle. The proposed method achieves the best overall score of 78.29 and obtains the best performance on Drivable, Direct., and TTC, with scores of 98.99, 99.58, and 88.44, respectively. Although PLUTO and PLUTO w/o refine. show slightly higher collisionrelated scores, the proposed method maintains competitive collision performance while achieving a better overall balance among drivable area compliance, driving direction consistency, TTC safety, and closed-loop progress. These results indicate that the proposed framework maintains robust behavior under dynamic interactions, benefiting from interaction-aware training samples, semantic guidance, and optimization-aware trajectory refinement. B. Qualitative Results To further analyze the behavior of the proposed method in complex interactive scenarios, Fig. 4 compares the proposed method with AsyncDriver in two representative left turn scenarios. The figure shows the closed-loop evolution from 0 s to 15 s, where the first and third rows correspond to the proposed method, and the second and fourth rows correspond to AsyncDriver. In the first scenario, the target lane is occupied during the left turn. The proposed method adjusts its trajectory while preserving the left turn intention and completes a lane change during the turning process, thereby avoiding prolonged stopping and maintaining trajectory continuity. In contrast, AsyncDriver remains near the intersection for a longer period and fails to complete the maneuver adjustment in time. This difference indicates that the proposed method benefits not only
IEEE INTERNET OF THINGS JOURNAL
9
TABLE II: Evaluation on nuPlan closed-loop nonreactive challenges on Hard20 split. The best results are highlighted in bold, while the second best results are underlined with an underline for clear distinction. Score: average final score. Drivable: drivable area compliance. Direct.: driving direction compliance. Comf.: ego is comfortable. Prog.: ego progress along expert route. Coll.: no ego at fault collisions. Lim.: speed limit compliance. TTC: time to collision within bound. Method
Score
Drivable
Direct.
Comf.
Prog.
Coll.
Lim.
TTC
IDM GC-PGP UrbanDriver PDM-closed PlanTF PLUTO w/o refine.* AsyncDriver Diffusion Planner PLUTO Ours
56.16 47.02 51.67 64.18 72.56 74.26 74.27 75.47 80.82 83.63
86.40 85.66 88.60 95.69 94.48 97.76 92.47 95.59 97.05 97.43
98.53 96.32 95.96 99.10 96.69 98.69 98.92 98.90 98.53 98.90
88.60 86.02 97.79 77.06 94.85 82.09 91.40 91.54 73.89 84.24
65.82 51.96 73.99 68.20 84.21 76.99 79.63 89.74 90.43 89.87
82.54 76.65 76.65 87.81 87.50 92.73 92.11 87.31 94.30 95.97
96.87 98.57 89.07 99.57 97.11 98.61 97.96 96.44 97.70 97.38
69.48 72.06 69.85 73.47 80.51 85.05 82.80 77.57 83.08 85.14
TABLE III: Evaluation on nuPlan closed-loop reactive challenges on Hard20 split. The best results are highlighted in bold, while the second best results are underlined with an underline for clear distinction. Method
Score
Drivable
Direct.
Comf.
Prog.
Coll.
Lim.
TTC
IDM GC-PGP UrbanDriver PDM-closed PLUTO w/o refine.* PlanTF AsyncDriver Diffusion Planner PLUTO Ours
62.26 44.29 49.06 62.26 60.91 60.34 64.89 68.50 76.88 78.29
84.19 85.29 82.72 84.19 97.01 94.49 94.62 95.22 97.39 98.99
98.53 96.88 95.59 98.53 98.32 97.43 98.82 98.90 98.32 99.58
87.87 88.60 98.52 87.87 88.05 94.49 81.36 84.93 89.55 84.41
69.60 46.51 80.18 69.60 59.57 64.41 67.31 76.72 74.67 75.97
84.38 83.82 69.85 84.38 93.47 91.54 85.48 86.95 94.59 93.22
96.52 98.63 86.14 96.52 98.36 97.77 98.09 97.38 98.58 97.88
72.43 78.68 63.97 72.43 87.68 86.40 73.48 79.41 87.69 88.44
from asynchronous semantic guidance, but also from differentiable trajectory refinement, which incorporates reference line consistency, safety margins, and kinematic feasibility to generate more executable adjustment trajectories. In the second scenario, straight going vehicles continuously pass through the intersection during the left turn. AsyncDriver adopts a conservative waiting strategy and starts turning only after the intersection is nearly cleared. By contrast, the proposed method first moves moderately into the intersection, maintains the yielding relationship, and completes the turn once a suitable passing opportunity appears. This behavior shows that the proposed method can make dynamic rather than purely static yielding decisions. Such behavior benefits from surrounding agent centric data augmentation, high level semantic guidance, and constraint aware trajectory refinement, enabling higher passing efficiency while maintaining safety. Beyond the nuPlan benchmark, the CARLA-ROS experiments further validate the online closed-loop execution capability of the proposed method, as shown in Fig. 5. Across the evaluated scenarios, the planner ROS node runs at 5 Hz, achieving an 80% success rate, a collision rate below 5%, and an average traversal time of less than 30 s. In the unprotected left turn scenario, the ego vehicle performs a preparatory lane change, yields to the oncoming straight going vehicle, and completes the turn after the conflict clears. In the right turn scenario, the ego vehicle handles right lane occupation and waits for the oncoming straight going vehicle before completing the maneuver. These results show that the proposed method can handle lane occupation, dynamic yielding, and turning maneuvers in continuous execution. This capability
benefits from complexity-aware semantic enhancement and differentiable trajectory refinement, which jointly improve task awareness, safety, and trajectory executability on the SIL platform. C. Ablation Studies To analyze the contribution of key components and the sensitivity of important design choices, we conduct a series of ablation studies. All ablation studies in this subsection are conducted under the same nuPlan closed-loop reactive Hard20 setting, with only the target component or parameter changed in each study. 1) Effects of Each Component: Table IV reports the stepwise ablation results of the key components. Adding the surrounding agent centric data augmentation strategy improves the score from 60.43 (M0) to 65.60 (M1), mainly with gains in Progress and Collisions, showing that reusing surroundingagent trajectories helps learn interaction-related behaviors. Introducing the differentiable optimizer further increases the score to 75.19 (M2), with notable improvements in Progress, Collisions, and Speed. Although Comfort and TTC decrease slightly, the overall gain indicates that residual-based refinement improves closed-loop efficiency, safety, and executability. M3∗ adds the complexity-aware asynchronous LLM module without the adaptive gate and achieves 76.45, confirming the benefit of scene-associated semantic features. The full model M3 reaches 78.29 and outperforms M3∗ on Drivable, Direct, Progress, and TTC, suggesting that the adaptive gate reduces excessive semantic interference and enables controllable use of high-level guidance.
IEEE INTERNET OF THINGS JOURNAL
10
t=0
t=3
t=6
t=9
t = 12
t = 15
Scenario1 Ours
Scenario1 AsyncDriver
Scenario2 Ours
Scenario2 AsyncDriver
Ego vehicle
Planning trajectory
Surrounding vehicles
Pedestrians
Fig. 4: Qualitative comparison of closed-loop planning results on representative scenarios from the nuPlan Hard20 split. Snapshots are sampled from 0 s to 15 s. Compared with AsyncDriver, the proposed method generates more route consistent and drivable trajectories in complex intersection and turning scenarios, showing improved closed-loop robustness under dynamic interactions. TABLE IV: Ablation study results on the effects of each component. Model
Description
Score Drivable Direct. Comfort Progress Collisions Speed TTC
M0 M1 M2 M3∗ M3
IL Planner M0 + Data Aug. M1 + Diff. Opt. M2 + AsyncLLM w/o adaptive gate M2 + AsyncLLM (Ours)
60.43 65.60 75.19 76.45 78.29
96.85 97.58 95.89 95.22 98.99
TABLE V: Ablation study results on different LLM update schedules. Model
Interval
Synchronous LLM every step Fixed high frequency LLM 3 frames Fixed medium frequency LLM 9 frames Fixed low frequency LLM 29 frames Complexity-aware AsyncLLM adaptive
98.23 98.39 99.08 99.52 99.58
88.19 87.50 83.45 84.69 84.41
58.87 62.93 75.48 75.49 75.97
93.50 94.15 95.22 90.91 93.22
98.48 98.26 99.53 98.32 97.88
88.58 88.71 84.19 83.73 88.44
TABLE VI: Ablation analysis of the surrounding agent centric data augmentation strategy
Score Runtime (ms)
Group
Setting
Score
80.46 79.61 76.74 75.33 78.29
Selection rule Selection rule
Random selection Interaction-aware selection
61.35 65.60
Screening radius Screening radius Screening radius Screening radius Screening radius
r = 15 m r = 25 m r = 50 m r = 75 m r = 100 m
60.89 62.54 65.60 65.83 66.13
477 237 155 126 172
2) Analysis of Data Augmentation Design: Table VI analyzes two key design choices in the surrounding agent centric data augmentation strategy. The default setting corresponds to M1 in Table IV. For target agent selection, interaction-aware
selection improves the score from 61.35 to 65.60 compared with random selection, indicating that agents associated with
IEEE INTERNET OF THINGS JOURNAL
11
Scenario1 Bounding Box
Scenario1 RGB Camera
Scenario2 Bounding Box
Scenario2 RGB Camera
Fig. 5: Qualitative closed-loop results on the CARLA-ROS platform, shown from both bounding box and RGB camera views. In representative urban scenarios, the proposed planner continuously generates executable trajectories under online feedback and maintains stable behavior when interacting with surrounding traffic participants, demonstrating its real time deployment feasibility in the SIL environment. TABLE VII: Ablation analysis of the complexity-aware scheduler. Setting
Score Runtime (ms)
Default scheduler Low thresholds (0.25, 0.55) High thresholds (0.45, 0.75) w/o TTC factor w/o interaction factor w/o navigation change factor w/o scene variation factor
78.29 78.83 75.63 70.74 72.57 75.21 76.46
172 224 132 146 138 155 163
lane changing, intersection traversal or turning, small TTC interactions, high lateral acceleration, and high magnitude speed behaviors provide more informative planning supervision. For the screening radius, increasing the radius from 15 m to 50 m improves the score from 60.89 to 65.60, showing that a small radius may miss useful interaction-relevant agents. Further enlarging the radius to 75 m or 100 m brings only marginal gains, with scores of 65.83 and 66.13, respectively. Therefore, we use 50 m as the default radius to balance interaction coverage, sample relevance, and augmentation cost. These results show that the effectiveness of the proposed augmentation strategy mainly comes from selecting interaction-relevant surrounding agent trajectories rather than simply increasing the screening range.
3) Effects of Different LLM Update Schedules: Table V compares different LLM update schedules. Synchronous inference achieves the highest score of 80.46 but requires 477 ms, while fixed-frequency updates reduce runtime at the cost of decreasing performance as the interval increases. The proposed complexity-aware AsyncLLM achieves a score of 78.29 with a runtime of 172 ms, outperforming the fixed medium- and low-frequency variants with substantially lower overhead than synchronous and fixed high-frequency inference. These results demonstrate that adaptive semantic updates provide a practical balance between semantic freshness and online efficiency. 4) Ablation Analysis of the Complexity-Aware Scheduler: Table VII shows that lowering the thresholds from (0.35, 0.65) to (0.25, 0.55) slightly improves the score from 78.29 to 78.83, but increases runtime from 172 ms to 224 ms. Conversely, higher thresholds (0.45, 0.75) reduce runtime to 132 ms but lower the score to 75.63. Removing any complexity factor also degrades performance, with the largest drops caused by excluding the TTC and interaction factors. These results confirm the importance of risk- and interaction-aware cues, while navigation-change and scene-variation cues provide complementary information. Overall, the default scheduler provides a practical balance between planning performance and online efficiency. 5) Effects of Residual Categories: Table VIII analyzes the three adjustable residual categories of efficiency, comfort, and safety, while the kinematic residuals are kept in all variants
IEEE INTERNET OF THINGS JOURNAL
12
TABLE VIII: Ablation analysis of different residual categories in the differentiable optimizer. Model
Removed residual category
Score
Drivable
Direct.
Comf.
Prog.
Coll.
Lim.
TTC
R0 R1 R2 R3
None, full residual set Efficiency residuals Comfort residuals Safety residuals
75.19 60.83 66.93 65.69
95.89 96.85 94.35 97.89
99.08 98.23 97.18 97.90
83.45 88.19 50.01 80.01
75.48 59.70 79.01 94.97
95.22 93.90 93.55 74.74
99.53 98.33 95.64 94.39
84.19 88.19 85.48 69.47
to maintain basic trajectory feasibility. The full residual set achieves the best overall score of 75.19. Removing the efficiency residuals (R1) reduces the score to 60.83 as Progress drops from 75.48 to 59.70, indicating overly conservative behavior. Removing the comfort residuals (R2) lowers Comfort to 50.01, confirming their role in maintaining motion smoothness. Without the safety residuals (R3), Progress increases to 94.97, whereas Collisions and TTC decrease to 74.74 and 69.47, respectively, revealing a substantial loss of interaction safety. These results demonstrate the complementary roles of the adjustable residual categories and the benefit of jointly balancing efficiency, comfort, and safety under basic kinematic feasibility constraints. VI. C ONCLUSIONS This paper presented a unified autonomous driving planning framework for IoT-enabled intelligent transportation scenarios by integrating surrounding agent centric data augmentation, asynchronous LLM-based semantic enhancement, and differentiable optimization into an IL-based planner. The proposed framework improves planning from three complementary aspects: interaction-oriented data augmentation, sceneaware semantic guidance, and optimization-aware trajectory refinement. Extensive experiments on the nuPlan closed-loop nonreactive and reactive benchmarks show that the proposed method achieves superior overall performance compared with representative baselines. Ablation studies and CARLA-ROS experiments further verify the effectiveness, efficiency, and online deployment potential of the proposed framework. Future work will extend the framework in three directions. First, the current framework is mainly validated with structured intermediate representations and simulation-based closed-loop evaluation, while its robustness to real-world perception noise, localization drift, and system uncertainty requires further study. Second, although the asynchronous semantic module reduces LLM inference overhead through complexity-aware scheduling, the current rule-based scheduler may have limited adaptability in more open and diverse traffic environments. Third, the differentiable optimization module mainly uses soft residual penalties for safety, comfort, efficiency, and basic vehicle kinematics. In particular, the safety residual adopts a selective local risk formulation for online efficiency, and the kinematic residuals use a lightweight bicycle model formulation for trajectory refinement. Future work will investigate richer multi-directional and multi-agent interaction modeling, especially for rear-end and dense interaction cases, as well as dynamic vehicle constraints for aggressive or high-speed maneuvers. We will also explore more adaptive semantic scheduling, multimodal semantics and V2X integration, and
higher fidelity hardware-in-the-loop and real vehicle validation. R EFERENCES [1] X. Gong, P. Lyu, and B. Wang, “Cooperative motion planning and decision making for cavs at roundabouts: A data-efficient learning-based iterative optimization method,” IEEE Internet of Things Journal, vol. 11, no. 19, pp. 32 205–32 220, 2024. [2] J. Yang, X. Xu, M. A. Khan, G. B. Brahim, J. Baili, L. Y. Por, and C. Li, “Explainable deep reinforcement learning for anomaly detection in iot-enabled metaverse healthcare: Toward trustworthy cyber threat intelligence,” Research, vol. 9, p. 1245, 2026. [3] Z. Lin, O. Aouedi, W. Ni, S. Chatzinotas, and X. Chen, “Gapsl: A gradient-aligned parallel split learning on heterogeneous data,” arXiv preprint arXiv:2603.18540, 2026. [4] Z. Fang, Z. Lin, S. Hu, H. Cao, Y. Deng, X. Chen, and Y. Fang, “IC3M: In-Car Multimodal Multi-Object Monitoring for Abnormal Status of Both Driver and Passengers,” arXiv preprint arXiv:2410.02592, 2024. [5] Z. Song, W. Li, B. Yang, H. Ding, and Y. Huang, “Enhanced control for handling and stability of 4wd electric vehicles with uncertain stability margins,” IEEE Transactions on Transportation Electrification, 2024. [6] Z. Song, Z. Lin, Y. Hu, Y. Deng, J. Yang, S. Prajapat, Z. Fang, Y. Zhang, L. Y. Por, H. Ding et al., “Safety-critical scenarios for autonomous driving: A survey of methods, benchmarks, and verification pipelines,” 2018. [7] S. Zhang, J. Yang, Z. Song, Z. Xia, N. Kumar, L. Y. Por, T. R. Gadekallu, and H. Ding, “Reliable lane-level localization for autonomous vehicles with enhanced hmm-based map matching and lane-line assistance,” IEEE Transactions on Vehicular Technology, 2026. [8] Z. Song, H. Ding, L. Jamel, J. Yang, M. A. Khan, J. M. Gorriz, J. Baili, and L. Y. Por, “Smart-city spatiotemporal data-driven trajectory prediction for autonomous vehicles via attention mechanisms and self-supervised learning,” IEEE Transactions on Consumer Electronics, 2025. [9] J. Yang, L. Fang, Z. Liu, Z. A. Shaikh, A. Ksibi, Z. Song, S. Prajapat, and L. Y. Por, “Qugrid-fed: Quantum-assisted federated iov energy and security management for sustainable urban transportation systems,” IEEE Communications Magazine, 2026. [10] T. Duan, Z. Zhang, S. Guo, D. Huang, Y. Zhao, Z. Lin, Z. Fang, D. Luan, H. Cui, and Y. Cui, “Leed: A highly efficient and scalable llm-empowered expert demonstrations framework for multi-agent reinforcement learning,” arXiv preprint arXiv:2509.14680, 2025. [11] Z. Zhang, T. Duan, Z. Lin, D. Huang, Z. Fang, Z. Sun, L. Xiong, H. Liang, H. Cui, Y. Cui et al., “Robust deep reinforcement learning in robotics via adaptive gradient-masked adversarial attacks,” Proc. IROS, 2025. [12] D. Lee and M. Kwon, “Episodic future thinking with offline reinforcement learning for autonomous driving,” IEEE Internet of Things Journal, vol. 12, no. 11, pp. 17 012–17 023, 2025. [13] K. Renz, K. Chitta, O.-B. Mercea, A. Koepke, Z. Akata, and A. Geiger, “Plant: Explainable planning transformers via object-level representations,” arXiv preprint arXiv:2210.14222, 2022. [14] O. Scheel, L. Bergamini, M. Wolczyk, B. Osiński, and P. Ondruska, “Urban driver: Learning to drive from real-world demonstrations using policy gradients,” in Conference on Robot Learning. PMLR, 2022, pp. 718–728. [15] J. Cheng, Y. Chen, X. Mei, B. Yang, B. Li, and M. Liu, “Rethinking imitation-based planners for autonomous driving,” in 2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 14 123–14 130. [16] H. Caesar, J. Kabzan, K. S. Tan, W. K. Fong, E. Wolff, A. Lang, L. Fletcher, O. Beijbom, and S. Omari, “nuplan: A closed-loop mlbased planning benchmark for autonomous vehicles,” arXiv preprint arXiv:2106.11810, 2021.
IEEE INTERNET OF THINGS JOURNAL
[17] J. Yang, V. Govindarajan, M. Y. H. Al-Shamri, H. Aldossary, A. Ksibi, Z. A. Shaikh, L. Y. Por, and K. Qi, “Neuroagent-x: A self-evolving cognitive agent for securing consumer iot systems against ai-enabled anomalies and adversarial threats,” IEEE Transactions on Consumer Electronics, vol. 71, no. 4, pp. 12 226–12 235, 2025. [18] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine et al., “Scalability in perception for autonomous driving: Waymo open dataset,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 2446–2454. [19] E. Bronstein, S. Srinivasan, S. Paul, A. Sinha, M. O’Kelly, P. Nikdel, and S. Whiteson, “Embedding synthetic off-policy experience for autonomous driving via zero-shot curricula,” in Conference on Robot Learning. PMLR, 2023, pp. 188–198. [20] Z. Huang, X. Mo, and C. Lv, “Multi-modal motion prediction with transformer-based neural network for autonomous driving,” in 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 2605–2611. [21] H. Gao, Y. Qin, C. Hu, Y. Liu, and K. Li, “An interacting multiple model for trajectory prediction of intelligent vehicles in typical road traffic scenario,” IEEE transactions on neural networks and learning systems, vol. 34, no. 9, pp. 6468–6479, 2021. [22] X. Mo, Z. Huang, Y. Xing, and C. Lv, “Multi-agent trajectory prediction with heterogeneous edge-enhanced graph attention network,” IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 7, pp. 9554–9567, 2022. [23] D. Dauner, M. Hallgarten, A. Geiger, and K. Chitta, “Parting with misconceptions about learning-based vehicle motion planning,” in Conference on Robot Learning. PMLR, 2023, pp. 1268–1281. [24] J. Cheng, Y. Chen, and Q. Chen, “Pluto: Pushing the limit of imitation learning-based planning for autonomous driving,” arXiv preprint arXiv:2404.14327, 2024. [25] Z. Huang, H. Liu, J. Wu, and C. Lv, “Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving,” IEEE transactions on neural networks and learning systems, vol. 35, no. 11, pp. 15 222–15 236, 2023. [26] Z. Lin, G. Qu, Q. Chen, X. Chen, Z. Chen, and K. Huang, “Pushing large language models to the 6g edge: Vision, challenges, and opportunities,” IEEE Communications Magazine, vol. 63, no. 9, pp. 52–59, 2025. [27] Z. Fang, Z. Lin, Z. Chen, X. Chen, Y. Gao, and Y. Fang, “Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Models,” IEEE Trans. Mobile Comput., 2025. [28] Z. Lin, Y. Zhang, Z. Chen, Z. Fang, X. Chen, P. Vepakomma, W. Ni, J. Luo, and Y. Gao, “Hsplitlora: A heterogeneous split parameterefficient fine-tuning framework for large language models,” IEEE Transactions on Mobile Computing, 2026. [29] T. Duan, Z. Zhang, Z. Lin, S. Guo, X. Guan, G. Wu, Z. Fang, H. Meng, X. Du, J.-Z. Zhou et al., “LLM-Driven Stationarity-Aware Expert Demonstrations for Multi-Agent Reinforcement Learning in Mobile Systems,” arXiv preprint arXiv:2511.19368, 2025. [30] Z. Fang, Z. Lin, S. Hu, Y. Ma, Y. Tao, Y. Deng, X. Chen, and Y. Fang, “Hfedmoe: Resource-aware heterogeneous federated learning with mixture-of-experts,” arXiv preprint arXiv:2601.00583, 2026. [31] M. Yang, L. S. Ewe, W. K. Yew, S. Deng, and S. K. Tiong, “A survey of data augmentation techniques for traffic visual elements,” Sensors, vol. 25, no. 21, p. 6672, 2025. [32] X. Li, E. Liu, T. Shen, J. Huang, and F.-Y. Wang, “Chatgpt-based scenario engineer: A new framework on scenario generation for trajectory prediction,” IEEE Transactions on Intelligent Vehicles, vol. 9, no. 3, pp. 4422–4431, 2024. [33] H. Mirkhani, B. Khamidehi, and K. Rezaee, “Augmenting safety-critical driving scenarios while preserving similarity to expert trajectories,” in 2024 IEEE Intelligent Vehicles Symposium (IV). IEEE, 2024, pp. 2085– 2090. [34] Z. Huang, H. Liu, and C. Lv, “Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 3903–3913. [35] B. Amos and J. Z. Kolter, “Optnet: Differentiable optimization as a layer in neural networks,” in International conference on machine learning. PMLR, 2017, pp. 136–145. [36] B. Amos, I. Jimenez, J. Sacks, B. Boots, and J. Z. Kolter, “Differentiable mpc for end-to-end planning and control,” in Advances in Neural Information Processing Systems, vol. 31. Curran Associates, Inc., 2018, pp. 8289–8300. [37] H. Shao, Y. Hu, L. Wang, G. Song, S. L. Waslander, Y. Liu, and H. Li, “Lmdrive: Closed-loop end-to-end driving with large language models,”
13
in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 15 120–15 130. [38] P. Paul, A. Garg, T. Choudhary, A. K. Singh, and K. M. Krishna, “Lego-drive: Language-enhanced goal-oriented closed-loop end-to-end autonomous driving,” in 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2024, pp. 10 020–10 026. [39] C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beißwenger, P. Luo, A. Geiger, and H. Li, “Drivelm: Driving with graph visual question answering,” in European conference on computer vision. Springer, 2024, pp. 256–274. [40] M. Nie, R. Peng, C. Wang, X. Cai, J. Han, H. Xu, and L. Zhang, “Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving,” in European Conference on Computer Vision. Springer, 2024, pp. 292–308. [41] C. Cui, Y. Ma, S.-Y. Park, Z. Yang, Y. Zhou, P. Liu, J. Lu, J. Peng, J. Zhang, R. Zhang et al., “Llm4ad: Large language models for autonomous driving—concept, review, benchmark, experiments, and future trends,” Proceedings of the IEEE, 2026. [42] Z. Lin, X. Hu, Y. Zhang, Z. Chen, Z. Fang, X. Chen, A. Li, P. Vepakomma, and Y. Gao, “SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Models,” arXiv preprint arXiv:2407.00952, 2024. [43] G. Qu, Q. Chen, W. Wei, Z. Lin, X. Chen, and K. Huang, “Mobile edge intelligence for large language models: A contemporary survey,” IEEE Communications Surveys & Tutorials, 2025. [44] Z. Lin, W. Wei, Z. Chen, C.-T. Lam, X. Chen, Y. Gao, and J. Luo, “Hierarchical Split Federated Learning: Convergence Analysis and System Optimization,” IEEE Trans. Mobile Comput., 2025. [45] L. Fang, Z. Xiaowen, S. Weixing, and W. Yonggang, “Dynamic pathspeed planning algorithm for autonomous driving on structured roads,” Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering, vol. 238, no. 10-11, pp. 3172–3193, 2024. [46] S. Lyu, Z. Lin, G. Qu, X. Chen, X. Huang, and P. Li, “Optimal resource allocation for u-shaped parallel split learning,” in 2023 IEEE Globecom Workshops (GC Wkshps), 2023, pp. 197–202. [47] Y. Chen, Z.-h. Ding, Z. Wang, Y. Wang, L. Zhang, and S. Liu, “Asynchronous large language model enhanced planner for autonomous driving,” in European Conference on Computer Vision. Springer, 2024, pp. 22–38. [48] J. Li, R. Zong, Y. Wang, and W. Deng, “Complexity evaluation for urban intersection scenarios in autonomous driving tests: Method and validation,” Applied Sciences, vol. 14, no. 22, p. 10451, 2024. [49] Z. Zhou, C. Rother, and J. Chen, “Event-triggered model predictive control for autonomous vehicle path tracking: Validation using carla simulator,” IEEE Transactions on Intelligent Vehicles, vol. 8, no. 6, pp. 3547–3555, 2023. [50] L. Pineda, T. Fan, M. Monge, S. Venkataraman, P. Sodhi, R. T. Chen, J. Ortiz, D. DeTone, A. Wang, S. Anderson et al., “Theseus: A library for differentiable nonlinear optimization,” Advances in Neural Information Processing Systems, vol. 35, pp. 3801–3818, 2022. [51] M. Treiber, A. Hennecke, and D. Helbing, “Congested traffic states in empirical observations and microscopic simulations,” Physical Review E, vol. 62, no. 2, pp. 1805–1824, 2000. [52] M. Hallgarten, M. Stoll, and A. Zell, “From prediction to planning with goal conditioned lane graph traversals,” in 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2023, pp. 951–958. [53] Y. Zheng, R. Liang, K. Zheng, J. Zheng, L. Mao, J. Li, W. Gu, R. Ai, S. E. Li, X. Zhan et al., “Diffusion-based planning for autonomous driving with flexible guidance,” arXiv preprint arXiv:2501.15564, 2025.