Compact Latent Coordination for Autonomous Vehicles at Unsignalized Intersections
arXiv:2607.21488v1 [cs.LG] 23 Jul 2026
Gil Lifshits Ben-Gurion University of the Negev [email protected]
Igal Bilik Ben-Gurion University of the Negev [email protected]
Gilad Katz Ben-Gurion University of the Negev [email protected] July 24, 2026
Abstract
vehicle coordination.
Coordinating autonomous vehicles at unsignalized intersections remains a critical challenge for multiagent reinforcement learning (MARL) systems, which typically struggle with combinatorial action spaces, reliance on privileged information, or rigid agent designs. We propose Master-Agent Proto-plan System (MAPS), a hierarchical deep reinforcement learning (DRL) architecture in which a centralized Master agent generates a compact, continuous embedding, denoted as proto-plan, that encodes a global coordination strategy. Decentralized Worker agents integrate this embedding with local observations to execute vehicle-specific control, decoupling strategic intent from tactical execution and enabling independent optimization of each module. As a proof-ofconcept evaluation of this coordination mechanism, we test MAPS across 72 intersection configurations in HighwayEnv. MAPS achieves collision-free navigation while significantly reducing average travel time, outperforming state-of-the-art baselines. The learned proto-plans further exhibit robust generalization: a system trained with three agents achieves a 94% success rate when deployed zero-shot to five-agent scenarios, confirming that proto-plan-based hierarchical learning provides a promising framework for multi-
1
Introduction
Coordinating multiple autonomous vehicles (AVs) at unsignalized intersections is a central challenge for intelligent transportation: vehicles must jointly navigate complex, dynamic scenarios while guaranteeing collision-free operation and maintaining throughput [Wang et al.(2024)]. The difficulty of this multiagent coordination problem has driven extensive research in multi-agent deep reinforcement learning (MADRL) [Chen et al.(2024)], spanning valuedecomposition methods [Huang et al.(2023)], graphbased representations [Cai et al.(2022)], policy optimization [Peng et al.(2023), Xu et al.(2022)], and hierarchical frameworks [Al-Sharman et al.(2022), Liu et al.(2025), Zhao et al.(2024)]. Despite notable progress, existing approaches share several limitations. First, many exhibit limited generalization, as they are trained and evaluated on simplified layouts that do not reflect the real-world diversity of intersections. Second, state-of-the-art methods often depend on privileged information, such as future trajectories, game-theoretic priors, rule-based safety layers, or expert demonstrations, that may be unavailable in practical deployments. Third, most 1
methods employ fixed, discrete action spaces consisting of small sets of predefined maneuvers, which scale combinatorially with the number of agents and require redesign when vehicle capabilities change. We address these limitations with master-agent proto-plan system (MAPS), a hierarchical DRL architecture that replaces explicit, per-vehicle action assignments with proto-plans: learned, continuous embedding vectors that encode high-level coordination strategies. A centralized Master agent observes the global traffic state and produces a single protoplan; decentralized Worker agents each combine this proto-plan with their own local observations to select vehicle-specific actions. This decomposition decouples coordination from control, keeps communication overhead constant at O(d) regardless of fleet size, and allows each module to be updated independently. Experiments across 72 intersection configurations show that MAPS achieves zero collisions during evaluation while reducing average travel time to 7.8 steps, which is a 38% improvement over the best baseline. The architecture further demonstrates robust zeroshot transfer: trained with only three active agents, it attains a 94% success rate when deployed to fiveagent scenarios without fine-tuning. Our contributions are as follows: • We introduce MAPS, a hierarchical DRL architecture that replaces rigid coordination commands with continuous proto-plan embeddings, keeping action and communication complexity at O(d) as fleet size grows. • We demonstrate zero-shot generalization through incremental training: a model trained on three agents transfers directly to five-agent deployment, confirming that the learned proto-plans capture transferable coordination strategies. • We show that effective multi-vehicle coordination requires only readily available kinematic state (positions and velocities), with no dependence on privileged information or expert demonstrations. Scope. This work introduces the proto-plan coordination mechanism as a foundational advancement in hierarchical multi-agent systems. Our primary contribution is the architectural demonstration that continuous, learned embeddings provide a more robust and scalable coordination interface than traditional
discrete command structures. By isolating the multiagent coordination challenge from the complexities of raw perception and low-level vehicle dynamics, we provide a rigorous validation of the proto-plan’s efficacy under controlled simulation conditions. This design choice allows us to confirm that the observed performance gains, including collision-free navigation and zero-shot transfer, are intrinsic to the hierarchical architecture itself. While this work serves as a definitive proof-of-concept for the MAPS framework, the architecture is designed for modular extensibility to high-fidelity environments and complex sensor suites, as discussed in Section 5.4.
2
Related Work
2.1
Multi-Agent Learning
Reinforcement
MARL extends single-agent RL to settings with multiple decision-makers facing partial observability, non-stationary dynamics, and cooperation– competition tradeoffs. Recent work targets scalability, safety, and coordination in large-scale systems [Liu et al.(2024)]. Model-based approaches include Ma et al. [Ma et al.(2024)], who topologically decouple global dynamics for local model learning while approximating global value information, improving sample efficiency at scale. For safety-critical settings, Zhang et al. [Zhang et al.(2024b)] enforce joint constraints via local trust-region updates over κ-hop neighborhoods, enabling decentralized training without centralized critics, while Hsu and Pajic [Hsu and Pajic(2025)] provide regret guarantees for safe cooperative MARL with function approximation. Graph-based representations encode structural priors effectively. GNN-based methods support resilient multi-robot coordination under agent failures and communication disturbances [Weil et al.(2024)]. In traffic management, decentralized graph-based MARL with traffic digital twins improves signal timing in large networks [Wang et al.(2025a)], and EECG [Peng et al.(2025)] integrates GNNs with curiosity-driven exploration and evolutionary opti2
mization for improved credit assignment in partially observable tasks. These approaches address important coordination aspects but typically operate within fixed action-space structures requiring agents to reason over peer-scaling representations. Our work is complementary: rather than improving the learning algorithm, we introduce a hierarchical communication mechanism via the proto-plan embedding, compressing coordination into a constant-size signal that allows more effective application of existing RL algorithms.
strates risk-aware constrained optimization for dense intersection safety. These hierarchical methods typically issue discrete commands or explicit sub-goals, yielding action spaces that grow combinatorially with fleet size and require redesign for new behaviors. Similarly, peer-to-peer learned communication methods like CommNet [Sukhbaatar et al.(2016)] and TarMAC [Das et al.(2019)] require each agent to send and receive messages, with per-agent aggregation scaling with peer count. MAPS departs from both paradigms: the Master produces a continuous protoplan embedding encoding coordination intent in fixed-dimensional latent space. This avoids combi2.2 MARL for Autonomous Driving natorial scaling, requires no predefined command voApplying MARL to autonomous driving demands cabulary, keeps Worker input size constant regardless safety, efficiency, and real-time responsiveness. Xu et of fleet size, and, as shown in Section 5.1, enables al. [Xu et al.(2022)] propose DDPG-based algorithms zero-shot transfer to larger agent populations. with meta-exploration and twin-delayed variants for intersection navigation, highlighting reward engineering for multi-objective optimization. In value decom- 3 Proposed Approach position, QMIXwD [Huang et al.(2023)] integrates self-generated demonstrations for improved early ex- 3.1 Overview ploration, while VN-MADDPG [Zhang et al.(2024a)] extends MADDPG with variable-noise and We formulate multi-vehicle intersection coordination importance-sampling for more efficient multi-vehicle as a hierarchical Markov decision process comprislearning. Graph-based models effectively capture ing two levels (Figure 1): a centralized Master vehicle interactions. DQ-GAT [Cai et al.(2022)] agent that observes the global traffic state and gend leverages bird’s-eye-view maps and graph attention erates a proto-plan embedding zt ∈ R representing for complex spatial relationships, and Spatharis and the desired coordination strategy, and decentralBlekas [Spatharis and Blekas(2024)] propose collab- ized Worker agents, each controlling an individorative frameworks with route agents and collision ual vehicle, that combine the proto-plan with local terms for scalable SUMO coordination. For policy observations to select vehicle-specific actions. The optimization, Peng et al. [Peng et al.(2023)] intro- Master’s action space is a continuous latent space duce curriculum PPO with stage-decaying clipping rather than an enumeration of per-vehicle commands, N across difficulty levels, and Xu et al. [Xu et al.(2022)] avoiding the |A| combinatorial scaling of joint accombine TD3 with LSTM-based motion prediction tion spaces while keeping communication overhead at O(d) regardless of fleet size. for smoother trajectories. Hierarchical architectures, particularly relevant to our work, decouple strategic from tactical decision- 3.2 Problem Formulation making. Al-Sharman et al. [Al-Sharman et al.(2022)] separate high-level behavioral planning from low- We model the problem as a decentralized parlevel control via SAC-MPC hierarchy. MA-GA- tially observable Markov decision process (DecDDPG [Liu et al.(2025)] augments MADDPG with POMDP) with hierarchical control, defined by the multi-head attention and level-k game priors, incor- tuple ⟨N , S, {Oi }, {Ai }, T , {Ri }, γ⟩, where N = porating a safety inspector for cooperative CAV be- {1, . . . , N } is the set of vehicles, S is the global state havior. SafeR-ADAIM [Zhao et al.(2024)] demon- space (positions and velocities), Oi and Ai are the 3
3.3
State Space Representation
3.3.1
Master State Space
The Master observes a global state sM formed by t concatenating the kinematic information of all N vehicles: 4N sM , t = [x1 , y1 , vx1 , vy1 , . . . , xN , yN , vxN , vyN ] ∈ R (2) where (xi , yi ) and (vxi , vyi ) denote the position and velocity of vehicle i in a global frame centered at the intersection. Vehicles are ordered by approach direction (N, E, S, W) and then by descending proximity to the intersection center. The input layer accommodates a fixed maximum of Nmax vehicles. When only k < Nmax vehicles are active, the remaining slots are zero-padded: sM t = x1 , y1 , vx1 , vy1 , . . . , xk , yk , vxk , vyk , 04(Nmax −k) . (3)
Figure 1: Hierarchical MARL framework. The Mas- 3.3.2 Worker State Space ter agent observes global state sM t and generates a Each Worker i observes a local state combining its proto-plan embedding zt . Each Worker i receives zt own kinematics with the proto-plan: along with its local observation to produce vehiclei = [xi , yi , vxi , vyi , zt ] ∈ R4+d , (4) sW specific actions ait . t where zt ∈ Rd is the proto-plan embedding (d = 4 in our experiments). Workers receive no direct oblocal observation and action spaces for Worker i, servations of other vehicles; they rely entirely on the T : S × A1 × · · · × AN → ∆(S) is the transition proto-plan for coordination information. This keeps function, Ri : S × Ai → R is the reward for agent i, each Worker’s input size constant at 4 + d regardless and γ ∈ [0, 1) is the discount factor. of fleet size, improving both scalability and privacy. The objective is to find policies {πi }N i=1 maximizing the expected cumulative discounted reward: J =E
" T X t=0
γt
N X
3.4
Action Space Mechanism
3.4.1
Master Action Space
# Ri (st , ait ) ,
(1)
i=1
and
Proto-Plan
The Master’s action space is defined as AM = Rd . Given the global state sM t , the Master policy outputs a proto-plan embedding:
subject to each Worker i accessing only its local observation oit ∈ Oi rather than the full state st ∈ S. The Master bridges this information gap by observing the global state and communicating coordination intent through the proto-plan, realizing a centralized-training-with-decentralizedexecution (CTDE) paradigm [Lowe et al.(2017), Wang et al.(2025b)].
θM M zt = πM (st ),
(5)
θM where πM is parameterized by neural network weights θM , with a tanh output layer bounding zt ∈ (−1, 1)d .
4
Unlike conventional hierarchical RL that issues dis- 3.5 Reward Structure crete high-level commands such as “vehicle 1 yields, 3.5.1 Worker Reward Function vehicle 2 proceeds”, the proto-plan encodes coordination strategies as dense vectors in a continuous space. Each Worker receives a reward encouraging safe, efThe semantics of individual dimensions emerge from ficient traversal: end-to-end training rather than manual specification. +Rsuccess if vehicle i crosses successfully While this mechanism shares surface sim- Wi rt = −Rcollision if vehicle i is involved in a collision ilarity with learned communication proto −Rstep otherwise (per time step) cols [Sukhbaatar et al.(2016), Das et al.(2019)], (7) the design differs fundamentally. First, proto-plan with R = 50, R = 300, and R = 5. success collision step communication is asymmetric and one-to-many: a The asymmetric magnitudes (R ≫ R > collision success single Master broadcasts a fixed-size vector to all Workers, whereas CommNet and TarMAC employ Rstep ) enforce a strict priority hierarchy: safety first, symmetric peer-to-peer messaging with message then efficiency, then throughput [Hua et al.(2025), complexity scaling with fleet size. Second, the Zheng and Gu(2025)]. We forego reward normalizaproto-plan is not a message about any individual tion to maintain a steep value gradient, ensuring the agent; it is a compressed global coordination strategy, collision penalty strictly dominates cumulative step conceptually closer to a learned option or subgoal in costs. the hierarchical RL sense. Third, Workers are fully decoupled from fleet composition: they receive the 3.5.2 Master Reward Function same d-dimensional input regardless of active agents, whereas peer-to-peer protocols require each agent to The Master’s reward aggregates individual Worker aggregate messages from a variable number of peers. rewards via a minimum operator: ( This architectural separation means the Master mini∈Wactive (t) rtWi if Wactive (t) ̸= ∅ M absorbs all coordination complexity, allowing Worker (8) Rt = 0 otherwise policies to remain fleet-size-agnostic and enabling the zero-shot transfer demonstrated in Section 5.3. where Wactive (t) = {i ∈ N | fi (t) = 0} is the set of vehicles that have not yet reached their destination. This maximin objective forces the Master to maximize the worst-case individual outcome, preventing 3.4.2 Worker Action Space coordination strategies that sacrifice any single vehicle. Replacing min with mean aggregation reduces Each Worker operates with a discrete action space SR to 84% during training and 80% during evaluaA = {accelerate, decelerate}, where each ac- tion. W
tion modifies the target speed by ∆vtarget = ±5 m/s. A low-level controller then computes the physical acceleration required to reach the new target speed. The control frequency is 1 Hz; no “maintain speed” option is provided.
3.5.3
An episode terminates when (i) all vehicles reach their destinations, (ii) a collision occurs, or (iii) the 50-step limit is reached.
At each time step, the Worker policy selects an action based on its local state: θW Wi ait = πW (st ).
Episode Termination
3.6
Network Architecture
Both Master and Worker (6) trained using Proximal Policy 5
agents are Optimization
(PPO) [Schulman et al.(2017)]. Each Worker module consists of separate policy and value networks with four fully-connected layers (64, 32, 16, 8 neurons; ReLU activations). The policy network outputs a categorical distribution over the two actions via softmax; the value network outputs a scalar estimate. The Master uses three-layer networks (128, 256, 128 neurons) to accommodate the higherdimensional global input (4Nmax features), with a tanh output layer producing the d-dimensional proto-plan. All Worker agents share parameters θW , reducing learnable parameters and encouraging generalizable control strategies. Since Workers operate on local, ego-centric observations, a single policy can map diverse situational contexts to unified driving behaviors.
1. Phase 1: Train with 1 learning agent until convergence. 2. Phase 2: Introduce 2 additional agents (3 total) and continue training. The resulting model is then evaluated on 5-agent configurations without fine-tuning; results are reported in Section 5.
Algorithm 1 Alternating Master-Worker Training Input: Configuration E, environment E, Master policy πM , Worker policy πW Output: Trained policies πM , πW ; collision count c; training statistics
1: N ← number of vehicles in E
3.7
Training Process
2: Initialize rollout buffers BM for Master and BW
for Workers
Training proceeds through alternating optimization of the Master and Worker modules, as formalized in Algorithm 1. In the initial cycle (cycle 0), both policies are updated jointly to establish preliminary coordination. Subsequent cycles alternate between freezing the Master (Workers adapt to current protoplan representations) and freezing the Workers (the Master learns to generate more effective proto-plans given Workers’ current behavior). This reduces the non-stationarity inherent in multi-agent optimization: each module trains against a fixed counterpart, yielding more reliable gradient estimates and stable convergence. Experience is collected into a dedicated rollout buffer for the Master and individual buffers for each Worker. The complete set of training hyperparameters is provided in Table 2. 3.7.1
3: Initialize collision count c ← 0 4: Initialize statistics container S 5: for cycle = 0 to E.NUM CYCLES − 1 do 6: for episode = 1
to E.EPISODES PER CYCLE do 7: (rep , τ ) ← ExecuteEpisode(πM , πW , E) 8: Store trajectory τ in appropriate buffers 9: if collision detected in τ then 10: c←c+1 11: end if 12: Record episode statistics in S 13: if cycle = 0 then 14: UpdatePPO(πM , BM ); UpdatePPO(πW , BW ) {Joint} 15: else if cycle mod 2 = 0 then 16: UpdatePPO(πM , BM ) {Master only} 17: else 18: UpdatePPO(πW , BW ) {Workers only} 19: end if 20: E.Reset() 21: end for 22: end for 23: return πM , πW , c, S
Incremental Training for Generalization
To evaluate the transferability of learned proto-plans, we employ an incremental training protocol. The Master’s input layer is sized for Nmax = 5 agents from the outset, with unused slots zero-padded. Training proceeds in two phases: 6
4
Experimental Setup
4.2.3
4.1
Simulation Environment
At the start of each episode, five vehicles are positioned along their respective approach lanes at distances ranging from 0 to 75 m from the intersection entry point, all initialized at 20 m/s. Each vehicle’s approach direction and intended trajectory are specified by the scenario definition. Of the five vehicles, three are controlled by MAPS learning agents, and two follow constant-speed trajectories (20 m/s) to simulate non-cooperative background traffic. Assignments are fixed per scenario. Figure 2 illustrates an example initial configuration and the resulting conflict zone.
Vehicle Initialization and Agent Assignment
We evaluate our approach using the HighwayEnv simulator [Leurent(2018)], an open-source reinforcement learning environment for autonomous driving research. Its modular architecture permits customization of state representations, action spaces, and reward functions, making it well-suited to our hierarchical framework. We deliberately employ HighwayEnv ’s simplified kinematics as a feature: by abstracting away perception pipelines and detailed vehicle dynamics, the simulator isolates the multiagent coordination challenge, providing a controlled testbed where performance differences can be attributed to coordination architecture rather than 4.2.4 Training and Evaluation Protocol confounding factors. Training is conducted over 900 episodes using the alternating optimization procedure of Algorithm 1, with scenarios sampled uniformly at random from 4.2 Scenario Configuration the 72 configurations. Evaluation is performed deterministically, removing all exploration noise across 4.2.1 Intersection Scenarios 100 episodes drawn uniformly from the same scenario distribution. Deterministic evaluation tests whether We constructed 18 base scenarios varying in approach the learned policy has converged to a robust coordidirection (N, S, E, W), initial distance from the internation strategy, and uniform sampling ensures covsection, and turning intention (straight, left, right). erage across the full range of traffic patterns. All Each base scenario is rotated through all four cardinal methods use a fixed random seed for deterministic reorientations (0°, 90°, 180°, 270°), yielding 72 unique producibility; we discuss implications in Section 5.4. configurations. This rotation augmentation prevents overfitting to specific approach directions and ensures generalization across symmetric traffic pat- 4.3 Baseline Methods terns. Table 2 (see Appendix) summarizes the training configuration used by our approach in all experiments. We compare MAPS against two recent methods rep4.2.2 Scenario Design Principles resenting distinct paradigms for multi-agent intersecScenarios were hand-designed to satisfy two proper- tion coordination: value decomposition with demonties. First, solvability: each scenario admits at least strations and actor-critic with adaptive exploration. one collision-free coordination strategy, validated by executing rule-based policies. Second, non-triviality: all scenarios feature inherently conflicting trajectories that cannot be resolved by simple reactive behaviors such as fixed yielding. The high collision rates of both baselines on these scenarios (Table 1) empirically confirm that non-trivial coordination is required.
QMIXwD [Huang et al.(2023)]. This baseline addresses exploration challenges by integrating learning from demonstrations into the QMIX valuedecomposition framework. A pre-training stage leverages both expert demonstrations and selfgenerated interaction data to mitigate distributional 7
and SafeR-ADAIM [Zhao et al.(2024)], were not included because they rely on additional privileged mechanisms—level-k game-theoretic priors and safety inspector modules, or risk-aware constrained optimization with domain-specific safety layers—that are external to the core learning architecture. Since our central claim concerns the proto-plan coordination mechanism itself, we compare against methods that, like MAPS, rely solely on learned coordination from kinematic observations, ensuring performance (a) Initial configuration: (b) Conflict zone: vehicles differences are attributable to architectural design five vehicles approach from converge at the intersec- rather than supplementary safety modules. all directions with varying distances to the intersection.
tion center, requiring coordinated timing to avoid collisions.
4.4
Evaluation Metrics
We assess performance using four complementary Figure 2: Example scenarios from the HighwayEnv metrics capturing safety, efficiency, and learning staenvironment, illustrating initial vehicle placement bility: and the resulting high-density conflict zone. • Success Rate (SR): The percentage of episodes in which all vehicles cross the intersecshift. Following the original methodology, our imtion without collision: SR = Nsuccess /Ntotal × plementation employs a loss combining a supervised 100%. The complementary collision rate is margin term, a T D(λ) loss, and L2 regularization. CR = 100% − SR. Episodes reaching the 50The demonstration dataset consists of 10% expert step time limit without all vehicles completing trajectories from a pre-trained greedy policy and 90% traversal are counted as failures. self-generated data. • Training Collision Count (CC): The total number of collision VN-MADDPG [Zhang et al.(2024a)]. This PK events across all training episodes, CC = baseline extends the MADDPG actor-critic framek=1 ck where ck ∈ {0, 1}. This tracks learning safety, where lower values indiwork with variable-noise exploration and importancecate fewer dangerous experiences during trainsampling mechanisms to improve learning efficiency ing. in continuous multi-agent action spaces. • Cumulative Episode Reward (R): The total PT reward accumulated over an episode, R = t=1 rt . Higher values indicate better combined safety and efficiency.
Fair comparison. All methods were implemented following the original papers, received identical observations, and were trained for the same 900 episodes on the same scenario distribution. Baseline hyperparameters follow the original publications.
• Average Travel Time (AT T ): The mean number of simulation steps for active learning agents to traverse the intersection: AT T = P|Wactive | 1 τi , where τi is the travel time i=1 |Wactive | for vehicle i. This reflects traversal efficiency: inefficient coordination leads to longer episodes, while early collisions artificially reduce step counts.
Baseline selection rationale. We selected baselines representing two dominant paradigms: value decomposition with demonstrations (QMIXwD) and actor-critic with adaptive exploration (VNMADDPG). Other hierarchical methods from Section 2, such as MA-GA-DDPG [Liu et al.(2025)] 8
for QMIXwD. The safety advantage extends to training. MAPS incurs only 21 collisions during 900 training episodes, an 84–85% reduction relative to VN-MADDPG (132) and QMIXwD (147), indicating that the hierarchical architecture learns safer behaviors more rapidly. A two-proportion z-test confirms these reductions are statistically significant (p < 0.001) for both baselines.
5.1.2
Efficiency Performance
MAPS demonstrates superior traversal efficiency: vehicles require an average of 7.8 simulation steps Figure 3: Training dynamics comparing MAPS and to cross the intersection, compared to 12.7 for VNbaselines: (a) success rate, (b) collision rate, (c) cuMADDPG and 13.2 for QMIXwD, a 38% reduction mulative episode reward, and (d) average travel time. over the best baseline. This gain stems from MAPS Moving average window size: 20. learning proactive coordination strategies that minimize unnecessary yielding, with the proto-plan mechTable 1: Comparison of collision counts and efficiency anism enabling Workers to anticipate coordination across methods. requirements rather than react to imminent conflicts. Approach
Collisions Collisions Avg Steps (Train - 900) (Eval - 100) (Evaluation)
5.1.3
Learning Dynamics
Figure 3 reveals several notable patterns. MAPS converges to near-optimal performance within approximately 180 episodes, after which its learning curves exhibit markedly lower variance than either baseline. VN-MADDPG and QMIXwD show persistent oscillations in both success rate and reward throughout 5 Evaluation Results training, suggesting difficulty maintaining consistent coordination. Despite sharing the same 900-episode 5.1 Main Results training budget, MAPS achieves substantially better final outcomes, indicating that the hierarchical deTable 1 summarizes the comparative results, and Figcomposition yields a more tractable learning problem ure 3 illustrates the learning dynamics during trainthan flat multi-agent architectures. ing. MAPS is the only approach to achieve collisionfree evaluation while recording the lowest average travel time. MAPS (ours) VN-MADDPG QMIXwD
5.1.1
21 132 147
0 35 31
7.8 12.7 13.2
Safety Performance
5.2
Ablation Studies
5.2.1
Contribution of the Master Agent
MAPS achieves zero collisions during evaluation To isolate the contribution of the learned proto-plan, (100% success rate), compared to 35 collisions (65% we hold Worker policies fixed and manipulate only SR) for VN-MADDPG and 31 collisions (69% SR) the Master’s output during inference. We evaluate 9
three conditions: zttrained = πM (sM t ),
(9)
ztrandom ∼ U(−1, 1)d , ztzero = 0d ,
(10) (11)
where πM is the trained Master policy. Under all conditions, Workers use the same trained policy πW , i with sW constructed by concatenating kinematic obt servations with the manipulated zt . Figure 4 reports cumulative episode rewards over 400 evaluation episodes. The trained Master achieves the highest returns (mean reward: 56.6), random proto-plans yield intermediate performance (25.7), and the zero-output condition collapses to strongly negative rewards (−304.7), a difference of over 360 reward points. These results demonstrate that the proto-plan channel is essential for coordination and that Workers have learned to condition their behavior on its content. 5.2.2
Figure 4: Cumulative episode reward under three proto-plan conditions with Workers fixed. The trained Master significantly outperforms random and zero baselines, confirming that learned proto-plans encode essential coordination information.
Proto-Plan Embedding Analysis
To interpret the information encoded by the Master, we project the d-dimensional proto-plan vectors collected during evaluation to two dimensions using PCA and label each timestep by a safety predicate (dangerous if any inter-vehicle distance falls below 5 m). Figure 5 reveals that proto-plans occupy distinct regions for safe versus dangerous states, indicating that the Master has learned to compress global interaction risk into a compact representation. The first two principal components capture 76% of the total variance (PC1: 63%, PC2: 13%). 5.2.3
Sensitivity to Embedding Dimension
We evaluated the architecture across embedding dimensions d ∈ {2, 4, 8, 16}. As shown in Figure 6, d = 4 yields the best performance (80.0% SR), compared to 70.0% for d = 2, 78.0% for d = 8, and 72.0% for d = 16. Note on evaluation protocols. The success rates reported in this ablation study differ from those in Table 1 because the two experiments use different training configurations. Figure 6 uses a single-phase
Figure 5: PCA projection of proto-plan vectors, colored by safety predicate (dangerous: any inter-vehicle distance < 5 m). Proto-plans encode safety-relevant context that Workers use for coordination decisions.
training setup to isolate the effect of embedding dimension, whereas Table 2 reports the final incrementally trained model (Phase 1 → Phase 2).
10
Figure 6: Impact of proto-plan embedding dimension on success rate. Performance peaks at d = 4 (80.0%), indicating that a compact 4-dimensional vector provides the optimal balance between representational capacity and optimization stability.
Analysis of training dynamics confirms that d = 4 yields the most stable reward distribution. Figures 7 and 8 provide direct optimization and reward evidence. Figure 7 shows that d = 4 achieves the most balanced convergence across both Master and Worker value losses, with low final losses and reduced latestage oscillations. Figure 8 shows that d = 4 also maximizes reward quality with the strongest central tendency and robust dispersion, while higherdimensional settings (especially d = 16) exhibit degraded central tendency and frequent outliers. Together, these results indicate that a 4-dimensional proto-plan is the optimal operating point: it is stable to optimize, converges reliably to low-loss solutions, and yields the strongest policy-level returns. This pattern reflects a trade-off between representational capacity and learnability: d = 2 provides insufficient capacity to encode necessary coordination context, while d ≥ 8 introduces optimization difficulty, leading to noisier policies. The result supports the central thesis that effective fleet coordination can be compressed into a very low-dimensional signal, making the proto-plan mechanism practical even under bandwidth-constrained conditions.
Figure 7: Master and Worker value-loss trajectories across embedding dimensions. The d = 4 setting shows the most balanced and stable joint convergence across modules, with low final losses and reduced late-stage oscillations.
5.3
Zero-Shot Transfer Fleet Sizes
to
Unseen
To evaluate whether the learned proto-plans capture transferable coordination strategies, we test the model trained with three active agents (Section 3.7.1) on five-agent configurations without finetuning. Across 100 test episodes, the system achieves a 94% success rate, demonstrating robust general-
11
dynamics. This choice is intentional: it isolates the multi-agent coordination mechanism from confounding factors, providing a controlled testbed for validating the proto-plan architecture. However, validation in higher-fidelity environments (e.g., SUMO or CARLA) is needed to confirm that coordination benefits transfer when richer dynamics and sensor noise are present.
Figure 8: Reward distribution by embedding dimension. The d = 4 policy exhibits the strongest central reward profile and robust dispersion, while higherdimensional settings, especially d = 16, show degraded central tendency and increased outlier behavior. ization to larger fleet sizes. The 6 failed episodes result from collisions in high-density or turning scenarios, with no significant pattern. This result can be attributed to three architectural properties: (1) the proto-plan embedding conveys coordination information in a format agnostic to the specific number of agents; (2) Worker parameter sharing ensures that control policies generalize across vehicle instances; and (3) the maximin reward structure encourages strategies that do not depend on particular agent configurations. Together, these properties indicate that proto-plan-based coordination generalizes beyond the training regime without requiring additional learning.
5.4
Action space. The current Worker action space is limited to binary longitudinal speed control (accelerate/decelerate). While this is a simplification, the architecture is agnostic to the Worker action space: continuous acceleration, lateral maneuvers, or richer discrete action sets can be substituted by modifying only the Worker policy head, without changes to the Master or the proto-plan mechanism. Evaluating such extensions is a priority for future work.
Fleet size. While the architecture demonstrates zero-shot transfer from three to five agents, the current Master uses a fixed-size concatenated input (4Nmax features), which imposes a hard maximum. Scaling to substantially larger fleets would benefit from permutation-invariant input processing, such as attention-based or set-based architectures for the Master, which is a natural extension that preserves the proto-plan broadcast mechanism. The present study validates the core coordination mechanism; large-scale deployment remains an open question.
Limitations and Design Scope
Baseline scope. We compare against two representative flat MARL methods. Hierarchical baselines that incorporate privileged mechanisms (game priors, safety inspectors) were excluded to ensure that Simulation fidelity. Our evaluation uses High- performance differences reflect the coordination arwayEnv, a kinematic simulator that abstracts away chitecture rather than supplementary modules (see perception noise, sensor latency, and detailed vehicle Section 4.3 for details). We identify several limitations that reflect deliberate scoping decisions and directions for future work.
12
6
Conclusions Work
and
Future
works. IEEE Transactions on Intelligent Transportation Systems 23, 11 (2022), 21102–21112. doi:10.1109/TITS.2022.3189917
We presented MAPS, a hierarchical DRL architecture for multi-vehicle coordination at unsignalized inter- [Chen et al.(2024)] Kaixin Chen, Bing Li, Rongqing Zhang, and Xiang Cheng. 2024. Autonomous sections. The architecture uses a continuous protoIntersection Management with Heterogeneous plan embedding generated by a centralized Master Vehicles: A Multi-Agent Reinforcement Learnand consumed by decentralized Workers, decoupling ing Approach. In 2024 IEEE Intelligent Vehistrategic coordination from tactical control while cles Symposium (IV). 2255–2260. doi:10.1109/ avoiding combinatorial action space scaling. EvalIV55156.2024.10588863 uations across 72 intersection configurations showed collision-free navigation with 38% reduced travel time versus the best baseline, and ablation studies con- [Das et al.(2019)] Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike firmed that proto-plans encode meaningful coordiRabbat, and Joelle Pineau. 2019. TarMAC: Tarnation information. The architecture achieved 94% geted Multi-Agent Communication. In Proceedsuccess in zero-shot generalization from three to five ings of the 36th International Conference on Maagents. chine Learning (Proceedings of Machine LearnThe key insight is that multi-vehicle coordination ing Research, Vol. 97), Kamalika Chaudhuri compresses into a 4-dimensional continuous signal, and Ruslan Salakhutdinov (Eds.). PMLR, 1538– enabling deployment in communication-constrained 1546. https://proceedings.mlr.press/v97/ and bandwidth-limited V2X systems. Future work das19a.html includes: (1) extending proto-plans to encode longhorizon strategies, amortizing communication cost across multiple time steps; (2) exploring hierarchical [Hsu and Pajic(2025)] Hao-Lun Hsu and Miroslav Pajic. 2025. Safe Cooperative Multi-Agent Rescaling through mid-level masters that relay refined inforcement Learning with Function Approximaplans from higher nodes to local Workers; and (3) tion. In Proceedings of the 7th Annual Learning investigating heterogeneous fleets via vehicle-classfor Dynamics & Control Conference (Proceedspecific Worker modules, leveraging modularity to inings of Machine Learning Research, Vol. 283). tegrate new agent types without retraining. PMLR, 1353–1364. https://proceedings. mlr.press/v283/hsu25a.html
References
[Hua et al.(2025)] Min Hua, Xinda Qi, and Dong Chen. 2025. Multi-Agent Reinforcement Learn[Al-Sharman et al.(2022)] Mohammad Al-Sharman, ing for Connected and Automated Vehicles Rowan Dempster, Mohamed A. Daoud, MahControl: Recent Advancements and Future moud Nasr, Derek Rayside, and William Melek. Prospects. IEEE Transactions on Automation 2022. Self-Learned Autonomous Driving at Science and Engineering 22 (2025). Unsignalized Intersections: A Hierarchical Reinforced Learning Approach for Feasible DecisionMaking. doi:10.36227/techrxiv.20770486.v1 [Huang et al.(2023)] Chang Huang, Junqiao Zhao, Hongtu Zhou, Hai Zhang, Xiao Zhang, and Preprint. Chen Ye. 2023. Multi-agent Decision-making at [Cai et al.(2022)] Peide Cai, Hengli Wang, Yuxiang Unsignalized Intersections with Reinforcement Sun, and Ming Liu. 2022. DQ-GAT: Towards Learning from Demonstrations. In 2023 IEEE Safe and Efficient Autonomous Driving With Intelligent Vehicles Symposium (IV). IEEE, 1– Deep Q-Learning and Graph Attention Net8. doi:10.1109/IV55152.2023.10186792 13
[Leurent(2018)] Edouard Leurent. 2018. An [Peng et al.(2023)] Zengqi Peng, Xiao Zhou, Yubin Environment for Autonomous Driving DecisionWang, Lei Zheng, Ming Liu, and Jun Ma. Making. https://github.com/eleurent/ 2023. Curriculum Proximal Policy Optimization highway-env. with Stage-Decaying Clipping for Self-Driving at Unsignalized Intersections. In 2023 IEEE 26th [Liu et al.(2024)] Dingbang Liu, Fenghui Ren, Jun International Conference on Intelligent TransYan, Guoxin Su, Wen Gu, and Shohei Kato. portation Systems (ITSC). IEEE, 5027–5033. 2024. Scaling Up Multi-Agent Reinforcement doi:10.1109/ITSC57777.2023.10422594 Learning: An Extensive Survey on Scalability Issues. IEEE Access 12 (2024), 94610–94631. [Schulman et al.(2017)] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg doi:10.1109/ACCESS.2024.3410318 Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 [Liu et al.(2025)] Jiaqi Liu, Peng Hang, Xiaoxiang (2017). Na, Chao Huang, and Jian Sun. 2025. Cooperative Decision-Making for CAVs at Unsignalized Intersections: A MARL Approach With At- [Spatharis and Blekas(2024)] Christos Spatharis and Konstantinos Blekas. 2024. Multiagent reintention and Hierarchical Game Priors. IEEE forcement learning for autonomous driving in Transactions on Intelligent Transportation Systraffic zones with unsignalized intersections. tems 26, 1 (Jan. 2025), 443–455. doi:10.1109/ Journal of Intelligent Transportation Systems TITS.2024.3503092 28, 1 (2024), 103–119. doi:10.1080/15472450. 2022.2109416 [Lowe et al.(2017)] Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, [Sukhbaatar et al.(2016)] Sainbayar Sukhbaatar, and Igor Mordatch. 2017. Multi-agent actorarthur szlam, and Rob Fergus. 2016. critic for mixed cooperative-competitive enviLearning Multiagent Communication with ronments. Advances in neural information proBackpropagation. In Advances in Neural cessing systems 30 (2017). Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and [Ma et al.(2024)] Chengdong Ma, Aming Li, Yali R. Garnett (Eds.), Vol. 29. Curran AsDu, Hao Dong, and Yaodong Yang. 2024. Efsociates, Inc. https://proceedings. ficient and Scalable Reinforcement Learning for neurips.cc/paper_files/paper/2016/file/ Large-Scale Network Control. Nature Machine 55b1927fdafef39c48e5b73b5d61ea60-Paper. Intelligence 6, 9 (2024), 1006–1020. doi:10. pdf 1038/s42256-024-00879-7 [Wang et al.(2024)] Bowen Wang, Xinle Gong, Yafei [Peng et al.(2025)] Kexing Peng, Pengyi Li, and Wang, Peiyuan Lyu, and Sheng Liang. 2024. CoJianye Hao. 2025. Enhancing Graph-based ordination for Connected and Autonomous VehiCoordination with Evolutionary Algorithms cles at Unsignalized Intersections: An Iterative for Episodic Multi-agent Reinforcement LearnLearning-Based Collision-Free Motion Planning ing. In Proceedings of the 24th InternaMethod. IEEE Internet of Things Journal 11, tional Conference on Autonomous Agents and 3 (2024), 5439–5454. doi:10.1109/JIOT.2023. Multiagent Systems (AAMAS 2025). Inter3306572 national Foundation for Autonomous Agents and Multiagent Systems, Detroit, MI, USA, [Wang et al.(2025a)] Kang Wang, Zhishu Shen, Zhen 1623–1631. https://www.ifaamas.org/ Lei, Xianhui Liu, and Tiehua Zhang. 2025a. Proceedings/aamas2025/pdfs/p1623.pdf Toward Multi-Agent Reinforcement Learning 14
Based Traffic Signal Control Through Spatio- [Zhao et al.(2024)] Rui Zhao, Yun Li, Kui Wang, Temporal Hypergraphs. IEEE Transactions on Yuze Fan, Fei Gao, and Zhenhai Gao. 2024. Mobile Computing 24, 9 (2025), 8258–8271. Centralized Cooperation for Connected Audoi:10.1109/TMC.2025.3556243 tonomous Vehicles at Intersections by Safe Deep Reinforcement Learning. IEEE Transactions on [Wang et al.(2025b)] Ye Wang, Jingjing Wang, RuiMobile Computing 23, 12 (2024), 12830–12847. jie Zhu, Hang Fu, Jianrui Chen, and C. L. Philip doi:10.1109/TMC.2024.3417441 Chen. 2025b. Facilitating Multiagent Coordination Relying on Graph Information Representa- [Zheng and Gu(2025)] Zhi Zheng and Shangding Gu. 2025. Safe Multiagent Reinforcement Learning tion. IEEE Transactions on Neural Networks With Bilevel Optimization in Autonomous Drivand Learning Systems 36, 10 (2025), 17929– ing. IEEE Transactions on Artificial Intelligence 17940. doi:10.1109/TNNLS.2025.3575196 6, 4 (2025), 829–840. doi:10.1109/TAI.2025. 10752922 [Weil et al.(2024)] Jannis Weil, Zhenghua Bao, Osama Abboud, and Tobias Meuser. 2024. Towards Generalizability of Multi-Agent Reinforcement Learning in Graphs with Recurrent Message Passing. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems (AAMAS ’24). 1919–1927. [Xu et al.(2022)] Shu-yuan Xu, Xue-mei Chen, Zijia Wang, Yu-hui Hu, and Xin-tong Han. 2022. Decision-Making Models for Autonomous Vehicles at Unsignalized Intersections Based on Deep Reinforcement Learning. In 2022 7th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM). IEEE, 672–677. doi:10.1109/ICARM54641.2022.9959664 [Zhang et al.(2024a)] Hao Zhang, Yu Du, Shixin Zhao, Ying Yuan, and Qiuqi Gao. 2024a. VN-MADDPG: A Variable-Noise-Based MultiAgent Reinforcement Learning Algorithm for Autonomous Vehicles at Unsignalized Intersections. Electronics 13, 16 (2024), 3180. doi:10. 3390/electronics13163180 [Zhang et al.(2024b)] Lijun Zhang, Lin Li, Wei Wei, Huizhong Song, Yaodong Yang, and Jiye Liang. 2024b. Scalable Constrained Policy Optimization for Safe Multi-agent Reinforcement Learning. In Advances in Neural Information Processing Systems, Vol. 37. doi:10.52202/ 079017-4400 NeurIPS. 15
A
The Hyperparameters Our Proposed Approach
of
Table 2: Training Hyperparameters Parameter
Value
Algorithm Optimizer Loss Function Discount Factor (γ) Learning Rate (Master & Agent) Batch Size Rollout Buffer Size (N ST EP S) Clip Range (PPO)
PPO Adam MSE 0.99 0.005 32 64 0.2
Additional Configuration Episodes Per Cycle Total Cycles Gradient Update Frequency Max Episode Time Agent State Input Size proto-plan Embedding Dimension Action Space Size Number of Agents Seed Deep learning framework RL library Hardware Approximate training time
300 3 Every 2 Episodes 50 Steps 8 4 2 5 42 PyTorch StableBaselines3 CPU 20m
Reward Structure Target Reached Reward Collision Reward Starvation Reward
50 -300 -5
16