Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks

arXiv:2608.20858v1 [eess.SY] 21 Aug 2026

Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks Giray Onur, Azita Dabiri, Bart De Schutter Delft Center for Systems and Control, Delft University of Technology, Delft, 2628CD, The Netherlands

Abstract Transportation networks, in particular multi-class transportation networks (i.e., networks with mixed vehicle types), are complex systems that are challenging to control. Recently, Deep Reinforcement Learning (DRL), which learns control policies from interactions with the environment, and Model Predictive Control (MPC), which uses a system model to optimize control inputs, have been increasingly utilized for transportation network control. However, nonlinear system dynamics and high-dimensional state spaces in large-scale networks limit DRL’s learning capacity under time-constrained training and increase MPC’s computation time, hindering real-time implementation with limited computational resources. Moreover, MPC depends on an accurate network model, which is often unavailable for complex systems such as multi-class transportation networks. This paper proposes a novel DRL-MPC framework for multi-class transportation networks that divides control authority between DRL and MPC, combining DRL’s fast online computation and model independence with MPC’s built-in optimization and constraint-handling capabilities. In the hierarchical framework, MPC operates at the higher level and determines low-frequency control inputs whose slower update rate accommodates its high computation time, while DRL operates at the lower level and determines high-frequency control inputs using its fast online deployment. The framework is evaluated on a multi-class freeway network against a hierarchical MPC controller and a hybrid statefeedback-MPC controller, including scenarios with model mismatch and noisy traffic demands. Results show that the proposed framework outperforms the hybrid state-feedback-MPC controller, substantially reduces online computa-

tion time compared with the hierarchical MPC controller, and provides more effective constraint enforcement under model mismatch. Keywords: deep reinforcement learning, model predictive control, transportation network management, multi-class transportation networks 1. Introduction Rapid and often unregulated growth in modern cities has led to significant challenges, one of the most critical being traffic congestion, which negatively impacts mobility and leads to various environmental, social, and economic issues, such as increased pollution, fuel consumption, noise, accidents, and longer travel times. These challenges have spurred a surge in the use of advanced control methods, such as Model Predictive Control (MPC) and Deep Reinforcement Learning (DRL), to improve the efficiency of transportation networks. MPC has been widely applied to transportation networks (e.g., controlling ramp metering rates, variable speed limits, and route guidance) due to its ability to handle constraints and to optimize long-term objectives (Mayne et al., 2000; Rawlings et al., 2018) (e.g., reducing the total time spent (TTS) or the fuel consumption). At each control step, MPC predicts the future evolution of the system using a prediction model over a finite horizon, solves an optimization problem to compute the optimal control input trajectory, applies only the first control input of the computed trajectory, and repeats this procedure at the next control step in a receding-horizon fashion. MPC approaches have been applied to the control of single-class (Todorović et al., 2022; van de Weg et al., 2019) and multi-class freeway networks (Liu et al., 2017; Pasquale et al., 2017), as well as to urban transportation networks (Pham and Ahn, 2023; Sirmatel and Geroliminis, 2018). However, the performance of MPC strongly depends on the accuracy of the network model and can degrade significantly in the presence of model mismatch. In addition, the computational complexity of solving MPC problems in real time with limited computational resources poses a major challenge, particularly when dealing with nonlinear, high-dimensional systems such as large-scale multi-class transportation networks (Castañeda et al., 2022; Siri et al., 2021). Moreover, since the MPC controller then relies on a nonlinear optimizer to compute the control inputs, the resulting solutions are often suboptimal due to the nonconvex nature of the optimization problem, which is often miti2

gated by using a multi-start approach (Hegyi et al., 2005), resulting however in longer computation time. DRL, on the other hand, has emerged as an effective model-free approach that can learn control policies directly from interactions with the environment (Arulkumaran et al., 2017; Mnih et al., 2013), making it particularly appealing in scenarios with limited model knowledge or uncertain system dynamics, as it does not require an explicit system model. This has led to the application of DRL in various transportation network management tasks, including ramp metering control (Belletti et al., 2018; Han et al., 2022) and variable speed limit control (Gao et al., 2024; Kušić et al., 2020) in freeway networks, along with traffic signal control in urban networks (Liang et al., 2019; Wu et al., 2020a). Despite its potential, DRL faces several challenges, including poor sample efficiency and difficulty in handling hard constraints (Dulac-Arnold et al., 2021). Poor sample efficiency indicates the need for many interactions with the environment to learn a good policy, which can make policy learning in transportation networks time consuming, as these interactions must often be obtained through computationally intensive simulations or costly field data collection. Furthermore, hard operational constraints such as vehicle queue length limits are not inherently ensured for standard DRL frameworks (Haydari and Yilmaz, 2022). The complementary strengths and limitations of MPC and DRL have motivated the development of hybrid control frameworks that integrate both approaches. In existing hybrid approaches, MPC is often employed as a safety filter to ensure constraint satisfaction while preserving the model-free nature of DRL (Zanon and Gros, 2021), or DRL is used to adjust MPC’s control inputs or parameters to compensate for inaccuracies in the prediction model (Gros and Zanon, 2020). Similar strategies have also been applied to mitigate model errors in ramp metering control within freeway networks (Airaldi et al., 2025; Sun et al., 2024b). However, despite overall progress, integrating DRL and MPC remains a developing area with a limited number of practical applications. In transportation network control, updating different control inputs at different frequencies has been a common strategy, since it limits the computational burden of recomputing every input at the highest frequency. For example, in the framework proposed by Sun et al. (2023), the ramp metering rates are controlled at a high frequency, while the target densities of the ramp meters are controlled at a lower frequency. As another example, in the framework proposed by Pasquale et al. (2017), the ramp metering rates 3

are controlled at a high frequency, whereas the vehicle splitting rates are controlled at a lower frequency. In these multi-frequency frameworks, the high-frequency control inputs must be computed sufficiently fast for realtime control, which is challenging for computationally demanding methods such as MPC, particularly in large-scale transportation networks. The main contribution of the current paper is an innovative integrated DRL-MPC framework that divides control authority and control inputs between DRL and MPC. In the proposed framework, MPC operates at a low frequency for control measures whose slower update rate accommodates MPC’s high computation demand, while DRL handles high-frequency control measures by leveraging its short deployment time. In addition, we illustrate the proposed approach on a multi-class transportation network, which, to the best of our knowledge, has not previously been considered for DRL-MPC. To evaluate the proposed framework, we apply it to a benchmark multiclass freeway network where the framework regulates vehicle splitting rates and ramp metering rates, and we assess its performance under noisy vehicle demands and model mismatch in the MPC prediction model. The results show that the proposed DRL-MPC framework significantly improves constraint handling compared to a fully MPC-based hierarchical controller under model mismatch, while achieving substantially faster online computation. The rest of this paper is organized as follows: Section 2 reviews related work on MPC and DRL for transportation network control and discusses existing hybrid DRL-MPC methods. Section 3 presents the proposed DRLMPC framework and its hierarchical structure, including the low-level DRL agent and the high-level MPC controller. Section 4 describes the training procedure of the DRL agent within the DRL-MPC framework, with the deterministic and stochastic actor-critic algorithms used for training detailed in Appendix B. Section 5 presents the case study on a benchmark multi-class freeway network and compares the proposed framework with benchmark controllers. Finally, Section 6 concludes the paper. 2. Related work Several control methods have been proposed to improve the efficiency and resilience of freeway and urban transportation networks in the literature (Majstorović et al., 2023; Siri et al., 2021). Among them, MPC and DRL have emerged as leading approaches for traffic management due to their appealing features. This section reviews relevant work on the application of MPC and 4

DRL for transportation network control, followed by a discussion of hybrid DRL-MPC methods. 2.1. MPC for transportation network control MPC has been widely adopted in transportation network control due to its capability to optimize over a prediction horizon while explicitly considering system dynamics and constraints. In freeway transportation networks, Hegyi et al. (2005) applied MPC for the optimal coordination of variable speed limits and ramp metering to reduce the TTS. Deo et al. (2009) extended this approach to multi-class freeway networks, coordinating these measures to manage different vehicle types. Similarly, Liu et al. (2017) deployed MPC in a multi-class freeway setting to simultaneously minimize the TTS, fuel consumption, and emissions, accounting for the dynamics of both cars and trucks. To address the computational burden of managing large-scale freeway networks using MPC, Lin et al. (2011) approximated the nonlinear model of the transportation network by a mixed-integer linear model, while Todorović et al. (2022) developed a distributed MPC scheme for ramp metering and variable speed limit control. In urban transportation networks, Lin et al. (2013) applied MPC to reduce both the travel delays and the emissions of various exhaust gases. Pham et al. (2022) proposed a distributed MPC controller for urban settings to improve traffic conditions and to ensure smooth operation across roads and intersections, achieving reduced computation times relative to centralized approaches. Despite their demonstrated performance, MPC-based approaches face notable limitations. Centralized MPC, as in (Deo et al., 2009; Hegyi et al., 2005; Lin et al., 2013; Liu et al., 2017), becomes computationally demanding for large-scale networks, limiting its real-time applicability. Although model simplification techniques such as approximating a nonlinear system with a mixed-integer linear formulation, as done in (Lin et al., 2011), can reduce computation time, they introduce a trade-off between computational efficiency and control performance, since the model approximation may lead to inaccurate predictions of the transportation network dynamics. Distributed MPC strategies (Pham et al., 2022; Todorović et al., 2022) mitigate the computational burden but remain dependent on accurate prediction models, making them vulnerable to model mismatch and disturbances. Therefore, developing MPC approaches that combine computational efficiency with robustness to model errors remains a challenge in advancing transportation 5

network control. 2.2. DRL for transportation network control DRL has gained significant attention in transportation network control due to its ability to learn control policies directly from experience without requiring explicit models of the environment (Haydari and Yilmaz, 2022). Wang et al. (2022) proposed a centralized framework that coordinates multiple ramp metering and variable speed limit controllers using an actor-criticbased DRL method to minimize the TTS, demonstrating that managing medium-scale networks with multiple on-ramps and off-ramps is feasible without complex multi-agent architectures. Han et al. (2022) introduced a physics-informed reinforcement learning strategy that combines historical and model-generated synthetic data, enhancing learning efficiency but requiring a reliable model and large datasets for training. Sun et al. (2023) developed an adaptive parameterized control scheme using deep Q-learning to tune the parameters of a state-feedback ramp metering controller online, allowing it to react to changing weather conditions. Wu et al. (2020b) proposed a DRL-based variable speed limit control approach trained with multiple reward components including the TTS, bottleneck speed, emergency braking, and emissions to improve both efficiency and safety, demonstrating generalization across different driver behaviors. Jiang et al. (2025) employed a proximal policy optimization-based method for coordinated ramp metering and route guidance within a heterogeneous-agent reinforcement-learning framework, targeting lower TTS and higher mean travel speed. Kolat et al. (2023) applied a multi-agent deep Q-learning algorithm to control traffic signals in an urban network, reducing fuel consumption and average travel time, and validated its effectiveness in a microscopic traffic simulator. Despite their potential, DRL-based approaches face several challenges, including long training times, sample inefficiency, and the lack of mechanisms to guarantee satisfaction of state constraints, such as upper bounds on the queue lengths. Therefore, developing DRL methods that are data-efficient, constraint-aware, and capable of achieving near-optimal performance remains an active area of research for scalable and reliable transportation network control. 2.3. Combined DRL-MPC approaches To exploit the complementary strengths of MPC and DRL, several hybrid frameworks have been proposed (Dabiri et al., 2025; Reiter et al., 2026). 6

One common approach is to train a DRL agent to adjust the MPC prediction model in order to improve model accuracy (Gros and Zanon, 2020), as MPC relies heavily on accurate representations of system dynamics for predictions. Another widely used strategy is to parametrize the cost function and constraints of the MPC controller and to train a DRL policy to modify them, thereby enhancing MPC performance in uncertain environments (Hewing et al., 2020). An alternative approach involves pre- or post-processing the actions generated by a DRL policy using MPC (Reiter et al., 2026). In the pre-processing case, MPC serves as a reference generator for the DRL policy, whereas in post-processing, MPC acts as a safety filter, taking the DRL policy’s action as an initial guess and ensuring that the final control input satisfies the constraints. While these integration approaches have demonstrated promising performance gains over standalone DRL-based or MPCbased approaches, most existing studies do not consider hierarchical control structures in which different control inputs are updated at different frequencies, and there are fewer demonstrations on networked transportation systems, especially under multi-class dynamics. Furthermore, integrating DRL and MPC remains an emerging research area, and the approaches described above represent only a subset of the possible methods for combining these techniques. In the context of transportation network control, Sun et al. (2024b) proposed a novel hierarchical DRL-MPC framework. In this framework, a highlevel MPC component operates at a low frequency to provide a baseline control input, while a DRL component operates at a high frequency to modify the output of MPC in real time. Simulation results demonstrated that this framework outperformed standalone MPC and DRL methods in terms of TTS and constraint satisfaction in the presence of model uncertainties and external disturbances. Sun et al. (2024a) proposed another framework that integrates DRL and MPC to address different sources of model mismatches. This framework uses DRL to adapt all components of a parametrized MPC scheme (e.g., the objective function, state-feedback control law, optimization settings, and system model) to manage model mismatches caused by changing environments and disturbances. In a freeway network control case study, the framework’s ability to adjust state-feedback control laws under model mismatches and varying weather conditions was tested. Simulation results showed that the proposed framework reduced computational complexity relative to conventional MPC while achieving better control performance than DRL-based controllers. 7

Subsequently, Airaldi et al. (2025) used a DRL policy to adjust selected parameters of the prediction model, objective function, and cost function of the MPC controller to control ramp metering in freeway networks. Simulation results demonstrated that their approach improved traffic-control performance and satisfied constraints even when the MPC controller used a mismatched prediction model and initially poorly tuned parameters. Despite the promising results of these approaches (Airaldi et al., 2025; Sun et al., 2024a,b), their focus has largely been on adapting MPC outputs and parameters via DRL to address model uncertainties in single-class transportation networks. As such, the potential of DRL-MPC integration for multi-class transportation networks in settings where different control measures are updated at different frequencies remains underexplored, leaving opportunities for further research. 3. The DRL-MPC framework The proposed DRL-MPC framework divides control inputs between DRL and MPC. The framework has a hierarchical structure where the MPC controller operates at the high level, running at a low frequency, and the DRL agent operates at the low level, running at a high frequency (see Figure 1). Consequently, the DRL agent can compute high-frequency control inputs quickly due to its fast online deployment capabilities, whereas the MPC controller has sufficient time to calculate low-frequency control inputs. The framework aims to minimize a given objective function, such as the TTS in a transportation network. Accordingly, as explained in the following section, the MPC cost function and the DRL reward function are designed to complement each other in achieving a common objective. Moreover, constraint violations, such as exceeding the vehicle queue length limits, are incorporated into the DRL reward function as penalty terms, allowing the DRL agent to learn to take actions that aim to respect the constraints imposed by the MPC controller. Note that in the framework, the MPC controller uses the trained DRL agent, whose policy is held fixed during deployment, to obtain high-frequency control inputs required by MPC for predicting the future states of the transportation network (see Figure 1(a)). This allows the MPC controller to predict the future states of the transportation network over its prediction window, even though the control input of the lower level is not determined by the MPC controller but determined by the DRL agent. 8

Figure 1: (a) Block diagram illustrating the proposed DRL-MPC framework that divides control inputs between the DRL agent and the MPC controller. (b) Time scales for the low-frequency control inputs of the high-level MPC controller and the high-frequency control inputs of the low-level DRL agent.

In the following sections, the DRL agent, the MPC controller, and the training methods for the DRL-MPC framework are explained in detail. The key notations used for the DRL-MPC framework are summarized in Table 1. 3.1. Low-level DRL agent In the proposed framework, the DRL agent determines the high-frequency control inputs, such as ramp metering rates in a freeway network (Pasquale et al., 2017). Let kl represent the control step counter for the low-level DRL agent, with a control time interval of length Tl . The DRL agent receives the network states, xrl (kl ), at every step kl . The states of the DRL agent are defined as ⊤ xrl (kl ) = [x̄⊤ (kl ), d¯ (kl )]⊤ , (1) ¯ l ) are the normalized states of the transportation netwhere x̄(kl ) and d(k work (e.g., densities, outflows, and velocities of vehicles on road segments) and the normalized current traffic demands, respectively. Normalization is applied by scaling the network states and traffic demands by constant gains

9

Table 1: Key notations used for the DRL-MPC framework

Notation

Definition

k kl kh T Tl Th ml mh Np x(k) xrl (kl ) ul (kl ) ūl (k) uh (kh ) ūh (k) x̂(k) M max keps πθ Qϕ D n r(kl ) γ τ

Network sampling step counter Low-level (DRL) control step counter High-level (MPC) control step counter Network sampling time interval Low-level control sampling time interval High-level control sampling time interval Ratio of low-level control and network sampling time intervals Ratio of high-level control and network sampling time intervals Prediction horizon (in high-level steps) State vector of the transportation network at sampling step k State vector of the DRL agent at control step kl Low-level control input computed at control step kl Low-level control input applied at network sampling step k High-level control input computed at control step kh High-level control input applied at network sampling step k Predicted future network state at sampling step k Total number of training episodes Maximum simulation steps per episode DRL actor policy network with parameters θ DRL critic value network with parameters ϕ Experience replay buffer Number of steps to look ahead for the reward Reward received at control step kl Discount factor Smoothing factor for Polyak averaging

10

to bring these variables to similar orders of magnitude, thereby enhancing the efficiency of the learning process. The relationship between the DRL control sampling time interval and the network sampling time interval is described as Tl = ml T,

k = ml kl ,

ml ∈ N+ ,

ml ≥ 1,

where k is the network sampling step counter, T is the network sampling time interval length, and N+ denotes the set of positive integers. The DRL agent computes the high-frequency control inputs every Tl time units as
ul (kl ) ∼ πθ · | xrl (kl ) ,

ul (kl ) ∈ Ul ,

where πθ is the DRL policy network with parameters θ, Ul is the constraint set for the high-frequency control inputs, and ul ∼ πθ (· | ·) implies sampling from a policy distribution1 . Then, the high-frequency control inputs that are applied to the transportation network during the control sampling time interval Tl are calculated as ūl (k) = ul (kl ),

k ∈ {ml kl , ml kl + 1, ..., ml kl + ml − 1},

meaning that the control input ul (kl ) is applied over a time interval of duration Tl . 3.2. High-level MPC controller The MPC controller computes the low-frequency control inputs, such as vehicle splitting rates in a freeway network (Pasquale et al., 2017). The MPC controller employs a receding-horizon strategy by computing the control sequence ⊤  ⊤ ⊤ ũh (kh ) = u⊤ h (kh ), uh (kh + 1), . . . , uh (kh + Np − 1) over a prediction horizon of Np high-level time steps, each of duration Th , with a step counter kh , where Th = mh T,

k = mh kh ,

1

mh ∈ N+ , mh ≥ ml .

In the case of a deterministic policy, this corresponds to sampling from a Dirac delta distribution where the entire probability mass is concentrated at the single output value of the policy network.

11

Figure 2: Relationship between the network sampling step counter k, low-level control step counter kl , and high-level control step counter kh for ml = 3 and mh = 6.

To ensure temporal synchronization, mh is selected as an integer multiple2 of ml , guaranteeing that MPC and DRL control input updates align (see Figure 2). The predicted future states of the freeway network are denoted as  ⊤ x̃(kh ) = x̂⊤ (mh kh + 1), x̂⊤ (mh kh + 2), . . . , x̂⊤ (mh kh + mh Np ) , where x̂(k) is the predicted future state of the freeway network at sampling step k. The states are predicted at the network sampling steps to evaluate the objective function accurately, even though the control inputs are updated only at the high-level control steps. The MPC controller computes the control inputs over the prediction hori2

Setting mh = ml is also possible and the framework remains applicable; in this case, however, the hierarchical structure (in which the MPC controller runs at a low frequency and the DRL agent runs at a high frequency) is no longer present.

12

zon by solving the following optimization problem: mh Np

min ũh (kh ),x̃(kh )

X

JTTS x̂(mh kh + l)

l=1 Np −1

+ ω uh

X

∥uh (kh + j) − uh (kh + j − 1)∥22

j=0

s.t. x̂(mh kh + l + 1) =
ˆ h kh + l) , F x̂(mh kh + l), ûh (mh kh + l), ûl (mh kh + l), d(m for l ∈ {0, 1, . . . , mh Np − 1}, (2a) x̂(mh kh ) = x(mh kh ), (2b) ûh (mh (kh + o) + l) = uh (kh + o), for l ∈ {0, 1, . . . , mh − 1}, o ∈ {0, 1, . . . , Np − 1}, (2c)
ûl (ml (kl + o) + l) ∼ πθ · | x̂rl (kl + o) , mh Np − 1}, (2d) for l ∈ {0, 1, . . . , ml − 1}, o ∈ {0, 1, . . . , ml x̂(mh kh + l) ∈ X , l ∈ {1, . . . , mh Np }, uh (kh + j) ∈ Uh , j ∈ {0, . . . , Np − 1}. (2e) In (2), the first term in the objective function, JTTS , represents the TTS, and the second term represents the quadratic penalty on fluctuations between consecutive high-level control inputs, with penalty weight ωuh , where ∥·∥2 denotes the Euclidean norm. Constraint (2a) represents the prediction model dynamics, where F represents the multi-class METANET model (see ˆ Appendix A) and d(k) denotes the estimated traffic demands. Constraint (2b) sets the initial condition of the predicted state trajectory to the measured state at the current high-level step. Constraint (2c) maps the MPC control inputs from high-level to low-level control steps using a zero-order hold strategy. Constraint (2d) specifies that the low-level control inputs used in the prediction are generated by the DRL policy πθ based on the predicted DRL agent state x̂rl (k) (see Figure 1(a)). Finally, (2e) collects the state and high-level input constraints, where X and Uh denote the corresponding constraint sets, respectively. After solving (2), the MPC controller applies only the first control input uh (kh ) to the system during the high-level control step, following a receding13

horizon strategy: ūh (k) = uh (kh ),

k ∈ {mh kh , mh kh + 1, ..., mh kh + mh − 1},

and repeats this procedure at every Th time units. Note that (2) leads to a nonlinear and nonconvex optimization problem for the multi-class METANET model (Liu et al., 2017), which necessitates the use of a nonlinear solver such as multi-start sequential quadratic programming, simulated annealing, or genetic algorithms (Floudas and Pardalos, 1996). 4. Training the DRL-MPC framework We train the DRL component of the DRL-MPC framework using data samples collected while the DRL agent operates at the low level together with the MPC controller operating at the high level in the DRL-MPC architecture (see Figure 3). The collected data samples are then used to update the DRL policy, so that the learned policy accounts for the effect of the MPC controller within the DRL-MPC framework. We consider two actor-critic algorithms for training the DRL agent within the DRL-MPC framework: Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al., 2016) and Soft Actor-Critic (SAC) (Haarnoja et al., 2018a), given their success in many control benchmarks (Wang et al., 2024), including freeway traffic control (Airaldi et al., 2025; Lu et al., 2025). DDPG learns a deterministic policy, whereas SAC learns a stochastic policy. This allows us to compare the performance of the proposed DRL-MPC framework when the DRL agent is trained using either a deterministic or a stochastic actor-critic algorithm. Both DDPG and SAC utilize off-policy learning, which enhances sample efficiency by allowing the reuse of past experience tuples (xrl , ul , r, x+ rl ) + stored in a replay buffer D, where r is the collected reward and xrl is the next observed state after taking the action ul . This sample efficiency is particularly beneficial for the proposed DRL-MPC framework, as training involves computing high-level control inputs through MPC, which increases simulation time and, consequently, the time required for each sample. In both algorithms, the following reward function is used: ! ml X
r(kl ) = − JTTS x(ml kl + l) + ωul ∥ul (kl ) − ul (kl − 1)∥22 + P rl (kl ) l=1

(3) 14

Figure 3: Block diagram illustrating the training process of the DRL-MPC framework using the DDPG algorithm.

where the first term coincides with the TTS term used by the MPC controller in (2), and the second term represents the quadratic penalty on fluctuations between consecutive low-level control inputs, with penalty weight ωul . The term P rl is a positive state constraint penalty term used to penalize violations of the state constraints in (2e), such as queue length constraints at the mainstream and on-ramp origins, since the DDPG and SAC implementations used in this study do not directly enforce hard state constraints. Note that the reward is defined as the negative of the TTS and penalty terms, so that maximizing the cumulative reward in DRL corresponds to minimizing these terms, aligning the objective of the DRL policy with that of the MPC controller. The overall training procedure is summarized in Algorithm 1, and the detailed formulations of how DDPG and SAC are utilized for training the DRL-MPC framework are provided in Appendix B.

15

5. Case study The proposed DRL-MPC framework is evaluated in a case study for freeway traffic control on a multi-class network with two vehicle classes and two control measures, where a vehicle splitting rate (via a variable message sign for route guidance) is updated at a low frequency and two ramp metering rates are updated at a higher frequency. The performance of the DRL-MPC framework is compared with that of a hierarchical MPC controller, a statefeedback-MPC controller, and the no-control case. Robustness of the controllers is assessed under prediction model mismatch in the MPC prediction models and under noisy demand profiles. The remainder of this section is organized as follows: Section 5.1 describes the experimental setup and simulation scenarios, Section 5.2 summarizes the controllers and evaluation metrics, Section 5.3 reports training results, and Section 5.4 presents the case study results and compares the controllers based on the evaluation metrics. The source code for this case study is available at https://github.com/GirayOnur/Sharing-Control-Authority-Between -DRL-and-MPC. 5.1. Experimental setup We consider a benchmark freeway network (see Figure 4) with two vehicle classes. The network consists of a mainstream stretch of three segments with four lanes and a mainstream origin O1 , and the mainstream stretch splits into two routes: a primary route with on-ramp origin O2 and a secondary route with on-ramp origin O3 . Each route consists of three segments with two lanes, and both routes merge at the single destination D1 . Each on-ramp has one lane. The free-flow capacity of all origins is 2000 veh/h/lane, and the queue length limits are 200 vehicles at O1 and 100 vehicles at each on-ramp origin. Each segment is 1 km long, the network sampling time interval is 10 s, and the case study simulation duration is 2.5 h. The multi-class transportation network shown as the controlled system in Figure 1(a) is simulated using the multi-class METANET model (see Appendix A). The parameter values used for simulations and the perturbed parameter values used by MPC under model mismatch are reported in Table 2. The nominal demand profiles and the estimated demands used for MPC predictions in the mismatched cases are shown in Figure 5. These demand profiles are selected to induce severe congestion in the absence of

16

Figure 4: The benchmark freeway network with one variable message sign for route guidance and two metered on-ramps.

effective ramp metering and vehicle splitting rate control, providing a challenging operating condition for evaluating traffic controller performance. To evaluate robustness to demand variability, we generate noisy demand profiles by adding zero-mean Gaussian noise to the nominal demand profiles, with origin- and class-specific standard deviations: (σ1,1 , σ1,2 , σ2,1 , σ2,2 , σ3,1 , σ3,2 ) = (200, 50, 40, 10, 40, 10), and then smoothing the resulting demands using a third-order low-pass Butterworth filter with a normalized cutoff frequency of 0.1. We evaluate all controllers under four scenarios that combine demand noise and prediction model mismatch. The scenarios are defined as follows: • Scenario 1: The simulator of the multi-class network uses the original values of the parameters and the nominal demand profiles. The MPC-based controllers use the original values of the parameters in their prediction model and assume the nominal demands as demand predictions. • Scenario 2: The simulator of the multi-class network uses the original values of the parameters and noisy demand profiles. The MPC-based controllers use the original values of the parameters in their prediction model and assume the noisy demands as demand predictions. • Scenario 3: The simulator of the multi-class network uses the original values of the parameters and the nominal demand profiles as in Scenario 1. The MPC-based controllers use the perturbed values of the parameters in their prediction model and assume the estimated demands as demand predictions. 17

Table 2: Original and perturbed values of multi-class freeway network parameters in the case study.

Parameter T τc κc ηc Lm Lveh c σc am,1 am,2 free vm,1 free vm,2 max ρm ρcrit m

Unit

Original Value

Perturbed Value

[s]a [s]a [veh/km/lane] [km2 /h] [km] [m] [km/h] [km/h] [veh/km/lane] [veh/km/lane]

10 18 40 60 1 5 0.0122 1.800 2.023 110 83.33 180 33.5

10 18 40 60 1 5 0.0122 1.950 2.317 102 75.33 165 38.5

a

The tabulated values of T and τc are expressed in seconds and converted to hours when used in the METANET equations.

(b)

(a)

Figure 5: The vehicle demand profiles used in the case study for the (a) mainstream origin and (b) on-ramp origins. The solid curves show the nominal demands, and the dashed curves show the estimated demands used for MPC predictions in the mismatch cases.

18

• Scenario 4: The simulator of the multi-class network uses the original values of the parameters and noisy demand profiles as in Scenario 2, while the MPC-based controllers use the perturbed values of the parameters in their prediction model and assume the estimated demands as demand predictions as in Scenario 3. To ensure that the scenario starts from a congested operating condition, we first simulate the network for 10 min under the no-control setting, with a fixed vehicle splitting rate of 0.5 and the ramp metering rates set to 1. All controllers are activated after this initialization period, and performance metrics are evaluated over the remaining controlled interval of 2.5 h. The vehicle splitting rate and the two ramp metering rates are bounded in the interval [0, 1]. The ramp metering rates are updated every Tl = 60 s, while the vehicle splitting rate is updated every Th = 300 s. The same vehicle splitting rate and ramp metering rates are applied to both vehicle classes. In each scenario, the hierarchical MPC framework and the no-control case are each evaluated 10 times. SF-MPC is evaluated 10 times for each of five independently tuned PI-ALINEA parameter sets, while DDPG-MPC and SAC-MPC are each evaluated 10 times for each of five independently trained agents. This yields 50 evaluations each for SF-MPC, DDPG-MPC, and SAC-MPC per scenario. For each of the 10 evaluation runs, all control frameworks use the same random seed. This seed is used to generate the multi-start initializations for the MPC-based frameworks. In Scenarios 2 and 4, the same seed is also used to generate the noisy demand profiles. All case study evaluations are performed in Matlab R2025a on a PC with a 12th Gen Intel Core i3-12100F CPU and 16 GB RAM. 5.2. Compared control frameworks The proposed DRL-MPC framework is evaluated against a hierarchical MPC controller, a state-feedback-MPC controller, and a no-control case, with the details of each approach described below. All controllers are evaluated on the same freeway network, have access to the same network state measurements, and are subject to identical control input bounds. 5.2.1. DRL-MPC framework The DRL-MPC framework assigns vehicle splitting rate control to the MPC controller and ramp metering rate control to the DRL agent.

19

MPC computes the control inputs by solving the optimization problem (2) using the Matlab fmincon function with the sequential quadratic programming (SQP) algorithm. A multi-start strategy with 5 initializations is used to balance the trade-off between control performance and computation time. To achieve sufficient numerical accuracy while maintaining computational efficiency, the optimality tolerance, step tolerance, and constraint tolerance are set to 10−2 . While training the DRL agent, the MPC controller uses a soft penalty for queue length limit violations instead of hard constraints. This choice avoids infeasibility at early training stages, when the DRL policy may not yet regulate the ramp metering rates effectively and the MPC controller may not be able to compute a split rate that satisfies the queue length limits. Specifically, the penalty  2 X P mpc (k) = ωw max{0, wo (k) − womax } (4) o∈{O1 ,O2 ,O3 }

is included as an additional term in the objective function of (2), where ωw PC in (4) is the queue length penalty weight, wo (k) = c=1 wo,c (k) is the total queue length at network sampling step k at origin o, and womax is the queue length limit at origin o. Although the soft penalty term in (4) is used during training of the DRL agent, during deployment in the case study evaluation the MPC controller enforces queue length limits as hard constraints in (2). In (2), the queue length penalty weight is set to ωw = 1 and the vehicle splitting rate smoothing weight to ωuh = 2.0, with mh = 30 and ml = 6 corresponding to the update intervals Th = 300 s and Tl = 60 s relative to the network sampling time interval of 10 s. The MPC controller uses a prediction horizon3 Np = 2. During training, the DRL reward is calculated using (3) with ωul = 0.4, obtained by scaling ωuh according to the ratio between ml and mh . The reward in (3) is multiplied by a constant scaling factor of 1/30 before being stored in the replay buffer, as scaling the reward signal has been shown to improve learning performance and stability of off-policy actor-critic algorithms (Haarnoja et al., 2018a). The queue violation term P rl in (3) is implemented 3

Although small in the number of high-level steps, this horizon spans 600 s due to the long high-level control sampling time interval, Th , corresponding to 10 low-level control steps or 60 network sampling steps.

20

using the maximum total queue length within each low-level control step. Let w̄o (kl ) = maxl∈{1,...,ml } wo (ml kl + l) denote the maximum total queue length at origin o ∈ {O1 , O2 , O3 } over Tl . Then,   X w̄o (kl ) rl I{w̄o (kl ) > womax }, (5) P (kl ) = ps + ws o∈{O1 ,O2 ,O3 }

where I{·} is the indicator function (i.e., it equals 1 when its condition holds and 0 otherwise), ps = 10 is a constant violation penalty, and ws = 100 is a queue scaling factor. The DRL agent is trained using the DDPG and SAC algorithms described in Appendix B.1 and Appendix B.2, respectively. Both DRL variants use the same observation and action definitions. The observation is an 81-dimensional vector comprising the following values: for each of the nine road segments, two class-specific mean speeds, two class-specific densities, the total density, and two class-specific outflows; and for each of the three origins, two class-specific queue lengths, two class-specific outflows, and two current class-specific demands. The DRL action is a 2-dimensional vector corresponding to the two ramp metering rates. The observation is normalized by a constant scaling vector selected according to the magnitude of each component to improve training stability: speed, density, and queue-related states are normalized by dividing by 100, and flow-related states and demand values are normalized by dividing by 1000. The network architectures and the training hyperparameters used for the DDPG and SAC agents in the case study are reported in Appendix B.3. We run the training in Matlab R2025a on the DelftBlue supercomputer (Delft High Performance Computing Centre (DHPC), 2024), using 32 CPU cores with 2100 MB memory per CPU core, and asynchronous parallel training with the Matlab Reinforcement Learning Toolbox. For each scenario defined in Section 5.1, five independent agents are trained for DDPG and five for SAC. 5.2.2. Hierarchical MPC framework The hierarchical MPC controller consists of two coupled MPC controllers operating at different time scales: a high-level MPC controller that computes the vehicle splitting rates and a low-level MPC controller that computes the ramp metering rates. The high-level MPC controller uses the same formulation and parameter settings as in Section 5.2.1, but differs in how low-level 21

inputs are treated within the prediction process. In the proposed DRL-MPC framework, predicted ramp metering inputs are generated by evaluating the DRL policy during the high-level MPC prediction. In the hierarchical MPC controller, the high-level MPC prediction uses the most recent ramp metering trajectory computed by the low-level MPC controller, shifted by one step with the terminal value duplicated, to predict the effect of ramp metering, thereby avoiding the prohibitive computational cost of solving additional low-level MPC problems inside the high-level MPC problem. The low-level MPC controller computes the ramp metering rates ul . The controller uses ml = 6 simulation steps per low-level control step, matching the low-level control step of the DRL-MPC framework in Section 5.2.1, and a low-level prediction horizon Np,l = 10, defined in low-level control steps and chosen so that the low-level prediction spans the same time interval as the high-level MPC prediction. Its objective is similar to (2), but the smoothing term penalizes squared changes in ul instead of uh , using the same ωul as the DRL agents in Section 5.2.1. In its prediction, the vehicle splitting rate inputs are taken from the most recent high-level MPC trajectory, shifted by one step with the terminal value duplicated. Both MPC controllers enforce the queue length limits as hard constraints. The two MPC problems are solved using the Matlab fmincon function with the SQP algorithm and the same tolerances as in Section 5.2.1. The highlevel and low-level MPC controllers use a multi-start strategy with 5 and 20 initializations, respectively, tuned to balance the trade-off between computation time and optimality. 5.2.3. State-feedback-MPC framework The state-feedback-MPC (SF-MPC) controller consists of a high-level MPC controller that computes the vehicle splitting rates and a low-level state-feedback controller, PI-ALINEA (Wang and Papageorgiou, 2006), that computes the ramp metering rates. PI-ALINEA is a widely used state-feedback ramp metering method that updates the ramp metering rate using proportional and integral feedback of the measured bottleneck density. In particular, for each on-ramp, PIALINEA updates the ramp metering rate as

ul (kl + 1) = ul (kl ) + KR ρ̄ − ρb (kl ) − KA ρb (kl ) − ρb (kl − 1) , where ul denotes the ramp metering rate, ρb (kl ) is the lane-averaged density measured at the bottleneck location (i.e., the link directly downstream of the 22

on-ramp), whereas the desired operating density ρ̄ and the gains KR and KA are the PI-ALINEA parameters. The high-level MPC controller uses the same formulation and parameter settings as in Section 5.2.1. Predicted ramp metering inputs required by the high-level MPC prediction are generated by running the PI-ALINEA controller inside the prediction model, similar to running the DRL policy within the high-level MPC prediction in the proposed DRL-MPC framework. Separate PI-ALINEA parameters are tuned for the two on-ramps using the Bayesian optimization function of the Matlab Statistics and Machine Learning Toolbox. For each scenario defined in Section 5.1, Bayesian optimization is run five times independently, yielding five tuned parameter sets (see Appendix C), for a fair comparison with the five independent DRL training runs per scenario. Each run uses the same DelftBlue computing setting as for DRL training, with parallel evaluations enabled and a total runtime matched to the average DRL training time. Based on preliminary trial runs, the search ranges of the PI-ALINEA parameters are restricted to effective regions for the case study. Specifically, for each on-ramp r ∈ {1, 2}, the parameter ranges are set as KR,r ∈ [0, 4], KA,r ∈ [0, 15], and ρ̄r ∈ [30, 100]. The Bayesian optimization objective is computed by running the state-feedbackMPC controller over the simulation period and evaluating the soft objective cost (SOC) defined in (11) below, with the only difference that the control input change penalty considers only the ramp metering inputs and not the vehicle splitting rate, since the vehicle splitting rate is included in the highlevel MPC cost function and its variation is minimized by the high-level MPC controller. 5.2.4. No-control case The no-control case applies a constant vehicle splitting rate of 0.5 and sets both ramp metering rates to 1, i.e., vehicles are evenly split between the two downstream routes and on-ramp flows are not restricted. Note that this strategy yields reasonable performance in the considered case study due to the symmetric network structure (see Figure 4) and the similar demand profiles at the two on-ramps (see Figure 5). 5.2.5. Evaluation metrics The compared control frameworks are evaluated using the TTS, the queue length constraint violations, the total input variation (TIV), the SOC, and the total computation time as metrics. 23

Traffic efficiency is assessed by the TTS, obtained by accumulating the per-step cost JTTS (x(k)) over the simulation period. The per-step TTS cost is defined as   C X X X X  JTTS (x(k)) = T ρm,i,c (k) Lm λm + wo,c (k) , (6) c=1

m∈M i∈Im

o∈{O1 ,O2 ,O3 }

where M is the set of links, Im is the set of segments in link m, ρm,i,c (k) ∈ x(k) is the density of class c in segment i of link m at network sampling step k, wo,c (k) ∈ x(k) is the queue length of class c at origin o at network sampling step k, and Lm and λm denote the segment length and the number of lanes in link m, respectively (see Appendix A for details). Constraint handling is assessed using the queue length limit exceedance at origin o ∈ {O1 , O2 , O3 } at network sampling step k, defined as ∆wo (k) = max(0, wo (k) − womax ) .

(7)

The total queue length constraint violation over the simulation period is defined as X X ∆wtot = ∆wo (k), (8) o∈{O1 ,O2 ,O3 } k∈K

where K denotes the set of simulation time step indices. Thus, ∆wtot is reported with number of vehicles as units. The maximum queue length constraint violation over the simulation period is defined as ∆wmax =

max

o∈{O1 ,O2 ,O3 }, k∈K

∆wo (k).

(9)

The total input variation accumulates the magnitude of the changes in the applied control inputs over the simulation period and is defined as X TIV = ∥u(k) − u(k − 1)∥2 , (10) k∈K\{0}

where u(k) stacks the vehicle splitting rate and the two ramp metering rates applied to the network at network sampling step k. A larger TIV indicates more aggressive variations in the vehicle splitting rate or in the ramp metering rates across consecutive control steps. 24

The SOC aggregates TTS, a quadratic penalty on consecutive input differences, and the soft quadratic queue-length-violation penalty into a single composite metric: X SOC = TTS + ωu ∥u(k) − u(k − 1)∥22 k∈K\{0}

+

X

X

∆wo (k)2 ,

(11)

o∈{O1 ,O2 ,O3 } k∈K

where the value of ωu is set as ωul /ml since the input trajectories u(k) are evaluated at the network sampling time step rather than at the low-level control step. Computational efficiency is assessed by the total computation time required to compute the control inputs over the simulation period. 5.3. Training results Figure 6 reports the learning curves of the five DDPG and five SAC agents, trained with independent random seeds in each of the four scenarios. The plotted returns retain the 1/30 reward scaling used during training and are smoothed using a 10-episode moving average before computing the acrossseed statistics, making the learning trends easier to discern. Each curve shows the per-episode return, defined as the undiscounted sum of rewards collected within an episode. In all four scenarios, the learning curves of all agents converge within the training budget, and the across-seed min–max ranges narrow as training progresses. Note that, since the plotted returns are obtained during training under exploration, they do not reflect the deployment performance of the converged policies. The deployment performance of the agents within the DRL-MPC framework is reported in the next section. 5.4. Deployment results The deployment results include two variants of the proposed DRL-MPC framework: DDPG-MPC, which deploys the deterministic DDPG actor, and SAC-MPC, which deploys the trained SAC actor deterministically by computing the ramp metering rates from the mean of its output distribution, following the observation of Haarnoja et al. (2018a) that deterministic evaluation of a trained SAC policy often achieves better performance. Per scenario, the hierarchical MPC framework and the no-control case are assessed 25

Scenario 1

-500

0

DDPG SAC

-1000

-1500 1000

2000

Cumulative Reward

Cumulative Reward

0

Scenario 2

-500

DDPG SAC

-1000

-1500 1000

3000

Scenario 3

-500

0

DDPG SAC

-1000

-1500 1000

2000

Cumulative Reward

Cumulative Reward

0

2000

3000

Episode

Episode

Scenario 4

-500

DDPG SAC

-1000

-1500 1000

3000

2000

3000

Episode

Episode

Figure 6: Cumulative reward curves of the five DDPG and five SAC agents under four scenarios. Bold curves show the mean cumulative reward across the five agents and shaded areas show the min–max range.

over the 10 evaluation simulations, each using a different random seed, while SF-MPC and each DRL-MPC variant are assessed over 50 runs, obtained by repeating these same 10 evaluation simulations for the five tuned PI-ALINEA parameter sets and the five independently trained DRL agents, respectively. Figure 7 reports the TTS obtained by each control framework in the four scenarios. In Scenario 1 and Scenario 2, where the MPC-based controllers have access to a perfect model of the network dynamics, the hierarchical MPC framework achieves the lowest median TTS, with a reduction of about 10% relative to the no-control case. DDPG-MPC, SAC-MPC, and SF-MPC obtain similar median TTS values in these two scenarios, with reductions of about 5% relative to the no-control case. Under prediction model mismatch (Scenario 3 and Scenario 4), the median TTS advantage of the hierarchical MPC framework shrinks to about 4% relative to the no-control case, less than half of the reduction achieved when the prediction model is matched. DDPG-MPC and SAC-MPC remain competitive with the hierarchical MPC framework in terms of median TTS in both mismatch scenarios, with SAC-MPC trailing DDPG-MPC by a small margin. The TTS spread of DDPG-MPC widens noticeably in Scenario 4, 26

Scenario 1

Scenario 2 3000

TTS (veh"h)

TTS (veh"h)

3000

2800

2600

2800

2600

2400

2400

PC PC PC PC trol Con SF-M ier. M PG-M AC-M S H No DD

No

PC PC PC PC trol Con SF-M ier. M PG-M AC-M S H DD

Scenario 3

Scenario 4 3000

TTS (veh"h)

TTS (veh"h)

3000

2800

2600

2400 No

t C on

2800

2600

2400 rol

C C C C MP M P G -M P C -M P S F - H ie r . P SA DD

No

PC PC PC PC trol Con SF-M ier. M PG-M AC-M S H DD

Figure 7: Total Time Spent (TTS) comparison for all control frameworks under the four evaluation scenarios.

where one of the five trained agents attains substantially worse performance than the remaining four, which inflates the across-seed range. This behavior is also observed in the remaining performance metrics (see Figures 8–11) and is consistent with the higher variability across random training seeds reported for DDPG relative to SAC in (Haarnoja et al., 2018a). Figures 8 and 9 report the total and the maximum queue length constraint violations, respectively. In Scenario 1 and Scenario 2, the hierarchical MPC framework keeps the queue length constraint violations close to zero, since it explicitly accounts for the queue length limits during optimization using a prediction model that matches the simulated network. DDPG-MPC also keeps queue length limit violations small: the total queue length constraint violation is close to zero in Scenario 1 and remains low in Scenario 2, while the median maximum queue length constraint violation is substantially smaller than for SF-MPC, SAC-MPC, and the no-control case. When the prediction model is accurate, DDPG-MPC closely matches the constraint-handling performance of the hierarchical MPC framework. Under prediction model mismatch (Scenario 3 and Scenario 4), the differences between the control frameworks become more pronounced. The hierarchical MPC framework can no longer keep the queue length constraint 27

Scenario 1

#104

#104 8

"wtot (veh)

"wtot (veh)

8 6 4 2

6 4 2

0

0

PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie No DD

#104

Scenario 4

8

"wtot (veh)

8

"wtot (veh)

PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie DD

No

Scenario 3

#104

6 4 2

6 4 2

0 No

Scenario 2

0

t C on

rol

MP SF-

PC PC PC r. M PG-M AC-M S H ie DD

PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie DD

C

No

Figure 8: Total queue length constraint violation comparison for all control frameworks under the four evaluation scenarios.

Scenario 1

Scenario 2 400

"wmax (veh)

"wmax (veh)

400 300 200 100

300 200 100

0 No

t C on

0 rol

C C C C MP M P G -M P C -M P S F - H ie r . P SA DD

No

PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie DD

Scenario 3

Scenario 4

300 200 100

400

"wmax (veh)

"wmax (veh)

400

300 200 100

0 PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie No DD

0 No

PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie DD

Figure 9: Maximum queue length constraint violation comparison for all control frameworks under the four evaluation scenarios.

28

Scenario 1

Scenario 2 100

TIV

TIV

100

50

50

0 No

Co

0 l ntro

C C C C MP M P G -M P C -M P S F - H ie r . P SA DD

No

PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie DD

Scenario 3

Scenario 4 100

TIV

TIV

100

50

50

0 No

t C on

0 rol

C C C C MP M P G -M P C -M P S F - H ie r . P SA DD

No

PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie DD

Figure 10: Total input variation (TIV) comparison for all control frameworks under the four evaluation scenarios.

violations close to zero in the simulated network, since its constraint handling operates on a mismatched prediction model. SF-MPC also incurs larger median total and maximum queue length constraint violations. DDPG-MPC, in contrast, keeps the median total and maximum queue length constraint violations small, showing that the learned low-level ramp metering policy provides useful robustness against errors in the high-level MPC prediction model. SAC-MPC, however, incurs larger median total and maximum queue length constraint violations than DDPG-MPC in both mismatch scenarios. Figure 10 reports the TIV values. SF-MPC generates large TIV values in most scenarios, consistent with the known tendency of PI-ALINEA-based state-feedback ramp metering controllers to produce substantial variations in the ramp metering rates (Airaldi et al., 2025). The hierarchical MPC framework and DDPG-MPC produce smoother control trajectories, with DDPG-MPC remaining close to the hierarchical MPC framework in terms of median TIV in all scenarios. SACMPC attains the smallest TIV among the controlled methods, which may be attributed to the entropy regularization employed during SAC training. By promoting stochastic exploration throughout training, entropy regularization may yield learned mean actions that vary less aggressively between 29

#107 10

2.5

Scenario 1

#107

#10 6

2.5

8

Scenario 2 #10 6

8

2

6

1.5

2 0

1 0.5

1.5 1

2 0

0.5

0 No

6 4

4

SOC

SOC

2

0

Co

l ntro

#107 2.5

10

C C C C M P r. M P G -M P C -M P SFP SA H ie DD

No

PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie DD

Scenario 3

#107

#10 6

Scenario 4

2.5

8

1.5 1

2

6 4

SOC

SOC

2

2 0

0.5

1.5 1 0.5

0 No

t C on

0 rol

C C C C M P r. M P G -M P C -M P SFP SA H ie DD

No

PC PC PC tro l -M P C C on r. M PG-M AC-M SF S H ie DD

Figure 11: Soft objective cost (SOC) comparison, where SOC combines TTS, inputvariation penalty, and the soft quadratic queue-length-violation penalty.

consecutive control steps when deployed deterministically. Figure 11 reports the SOC values. In Scenario 1 and Scenario 2, the hierarchical MPC framework attains the lowest median SOC by minimizing the TTS while keeping the queue length constraint violations close to zero and the input variations small, owing to a prediction model that matches the simulated network. DDPG-MPC obtains a lower median SOC than SAC-MPC and SF-MPC in these two scenarios. The higher SOC values of SAC-MPC and SF-MPC are mostly driven by their larger queue length constraint violations, owing to the quadratic queue-length-violation penalty in the SOC. Under prediction model mismatch, DDPG-MPC achieves the lowest median SOC among all controlled methods in both Scenario 3 and Scenario 4, whereas SAC-MPC attains higher median values. This indicates that the learned DDPG ramp metering policy compensates more effectively for mismatch in the high-level MPC prediction model. Table 3 reports the mean and standard deviation of the total control framework computation time over the simulation period across all experimental runs. Here, the no-control case is omitted, since it involves no control input 30

Table 3: Mean and standard deviation of the total control framework computation time (s) across all experimental runs for each control framework and scenario.

Controller

Scenario 1

Scenario 2

Scenario 3

Scenario 4

SF-MPC Hier. MPC DDPG-MPC SAC-MPC

30.5 ± 1.9 1778.9 ± 3.7 55.8 ± 3.5 76.9 ± 2.6

29.0 ± 2.2 1736.7 ± 5.1 52.9 ± 2.3 78.8 ± 3.1

27.7 ± 1.0 1638.2 ± 33.0 54.6 ± 2.8 62.4 ± 1.8

27.9 ± 1.2 1659.3 ± 39.2 54.3 ± 2.3 64.9 ± 2.1

computation. The hierarchical MPC framework requires by far the largest computation time, around 28 minutes across the four scenarios, since it solves a nonlinear optimization problem at both the high-level and the low-level control time scales. The two DRL-MPC variants have computation times that are close to each other, in the order of a minute, which corresponds to approximately a thirtyfold reduction relative to the hierarchical MPC framework. SF-MPC is the fastest control framework, since the PI-ALINEA state-feedback controller requires less computation time than a forward pass through the DRL policy network. Overall, the comparison shows that DDPG-MPC provides the most favorable balance among the proposed DRL-MPC variants and the considered baselines: it preserves a competitive median TTS across the four scenarios, substantially reduces the median queue length constraint violations under prediction model mismatch relative to SF-MPC and the hierarchical MPC framework, and reduces the online computation time by approximately a factor of thirty compared with the hierarchical MPC framework. Although SAC-MPC produces smoother control trajectories and shows lower variability across training seeds, it incurs larger queue length constraint violations and higher SOC values than DDPG-MPC. A possible explanation for this performance difference is the distinct policy-learning mechanisms of the two algorithms: DDPG learns a DRL policy using deterministic policy gradients, whereas SAC learns a DRL policy using an entropy-regularized objective that balances expected return and policy entropy.

31

6. Conclusions This paper has proposed a novel DRL-MPC framework that divides the control inputs between DRL and MPC, with an application to the control of multi-class transportation networks. By assigning the low-frequency control inputs to MPC and the high-frequency control inputs to DRL, the proposed framework combines the built-in optimization and constraint-handling capabilities of MPC with the fast online computation and model independence of DRL. The proposed framework has been evaluated on a benchmark multi-class freeway network with one vehicle splitting rate control input, computed by the high-level MPC controller, and two ramp metering rate control inputs, computed by the low-level DRL agent. Across the scenarios that combine prediction model mismatch and noisy traffic demands, the proposed framework has been shown to offer a favorable trade-off between traffic efficiency, constraint handling, input smoothness, and online computation time, and to be more resilient to prediction model mismatch than a hierarchical MPC framework and a state-feedback-MPC framework. Beyond the considered multi-class transportation network control case study, the proposed DRL-MPC framework could also be adapted to other control problems involving multiple control time scales, such as robotic motion control (Rosolia et al., 2022), where reference trajectories are updated at a low frequency while actuator commands are updated at a high frequency, and power-electronic converter control (Xue et al., 2024), where motor-current references are updated at a low frequency while semiconductor switching commands are applied at a high frequency. Future work will consider evaluating the proposed framework across different application areas. Another direction will be to extend the proposed framework to the control of large-scale transportation networks by combining distributed MPC with multi-agent DRL. Appendix A. Multi-class METANET model The METANET model (Kotsialos et al., 1999) is a macroscopic traffic flow model widely used for simulating freeway networks (Airaldi et al., 2025; Chanfreut et al., 2021), including multi-class freeway networks (Liu et al., 2017), due to its good balance between computational efficiency and modeling accuracy. In the current work, we employ the multi-class METANET model 32

both to simulate the real multi-class freeway network and as the prediction model within the MPC controller. In the multi-class METANET model (Liu et al., 2017), a freeway network consists of links representing freeway stretches with approximately uniform characteristics, connected by nodes representing junctions, ramps, or locations of major geometric changes. Each segment is described by the traffic density ρm,i,c (k), mean speed vm,i,c (k), and outflow qm,i,c (k) of each vehicle class. Here, m is the link index, i is the segment index, c ∈ {1, 2, . . . , C} is the vehicle class index, C is the number of vehicle classes, and k is the network sampling step counter. The number of segments in link m is denoted by Nm . The density ρm,i,c (k) is calculated as ρm,i,c (k) =

Lveh c ρact (k), veh m,i,c L1

where ρact m,i,c (k) is the actual number of vehicles of class c per unit length and per lane, and Lveh and Lveh represent the typical vehicle lengths of vehicle c 1 class c and the base class (taken as class 1), respectively. The segment outflow for vehicle class c is calculated as qm,i,c (k) = ρm,i,c (k)vm,i,c (k)λm , where λm is the number of lanes in link m. The segment density for vehicle class c is calculated as  T  qm,i−1,c (k) − qm,i,c (k) , ρm,i,c (k + 1) = ρm,i,c (k) + Lm λm where T is the network sampling interval and Lm is the length of the segments in link m. The fraction of the traffic volume of class c in a segment is calculated as ξm,i,c (k) =

ρm,i,c (k) , ρm,i (k)

where the total density of the segment is defined as ρm,i (k) =

C X c=1

33

ρm,i,c (k).

The segment mean speed is calculated as
T Ṽc (ρm,i (k), Ξm,i (k)) − vm,i,c (k) τc
T vm,i,c (k) vm,i−1,c (k) − vm,i,c (k) + Lm T ηc ρm,i+1 (k) − ρm,i (k) − , Lm τ c ρm,i (k) + κc

vm,i,c (k + 1) =vm,i,c (k) +

(A.1)

where Ξm,i (k) = (ξm,i,1 (k), ξm,i,2 (k), . . . , ξm,i,C (k)) and τc , ηc , and κc are model parameters. The term Ṽc (ρm,i (k), Ξm,i (k)) represents the interpolated speed-density relationship of vehicle class c, and it is calculated as 

Ṽc (ρm,i (k), Ξm,i (k)) = min Vc (ρm,i (k)),

C X

 ξm,i,γ (k)Vγ (ρm,i (k)) .

γ=1

This interpolation relies on Vc (ρm,i (k)), which denotes the desired speed of the drivers of class c for the given density and is calculated as  a   ρm,i (k) m,c 1 free Vc (ρm,i (k)) = vm,c exp − , am,c ρcrit m free where vm,c , am,c , and ρcrit m are model parameters. Links are connected at nodes, where the flow of each entering link is distributed among the leaving links according to a per-class splitting rate. Let out Lin n and Ln denote the sets of links that enter and leave node n, respectively. The total flow of vehicles of class c that enters node n is calculated as X Qn,c (k) = qµ,Nµ ,c (k), µ∈Lin n

and the inflow of vehicles of class c into the first segment of a leaving link m ∈ Lout n is calculated as qm,1,c (k) = βn,m,c (k) Qn,c (k), where βn,m,c (k) P ∈ [0, 1] is the splitting rate of class c at node n towards leaving βn,m,c (k) = 1. When node n has more than one leaving link m, with m∈Lout n 34

link, the virtual downstream density used in (A.1) for the last segment of an entering link m ∈ Lin n is calculated as P ρ2µ,1 (k) µ∈Lout n , ρm,Nm +1 (k) = P ρµ,1 (k) µ∈Lout n where ρµ,1 (k) is the total density of the first segment of leaving link µ. The queue length at an origin (e.g., a mainstream source or an on-ramp) is updated as wo,c (k + 1) = wo,c (k) + T (do,c (k) − qo,c (k)), where do,c (k) and qo,c (k) are the demand and the outflow of vehicles of class c at origin o at network sampling step k, respectively. For an on-ramp, qo,c (k) is calculated as " des (k) qo,c des Qo ro (k), qo,c (k) = min qo,c (k), PC des (k) q o,γ γ=1  max # des qo,c (k) ρµ − ρµ,1 (k) , PC des Qo max − ρcrit ρ q (k) µ µ o,γ γ=1 where µ is the index of the link to which the on-ramp is connected, Qo is the free-flow capacity of on-ramp origin, ro (k) ∈ [0, 1] is the ramp metering rate and ρcrit at network sampling step k, and ρmax µ are the maximum and critical µ des densities of link µ, respectively. The term qo,c (k) denotes the desired origin outflow for class c at origin o and is calculated as des qo,c (k) = do,c (k) +

wo,c (k) . T

For mainstream sources, qo,c (k) is calculated as " des qo,c (k) des qo,c (k) = min qo,c (k), PC Qmain , o des q (k) o,γ γ=1  max # des qo,c (k) ρ − ρ (k) µ,1 µ , PC des Qmain o max − ρcrit ρ q (k) µ µ o,γ γ=1 where Qmain is the free-flow capacity of mainstream origin aggregated over o the lanes of the link it feeds. 35

For links with an on-ramp, the mean speed of the first segment is adjusted to account for the speed drop caused by merging phenomena by adding the term T σc qo (k)vm,1,c (k) − Lm λm (ρm,1 (k) + κc ) PC to (A.1), where qo (k) = γ=1 qo,γ (k) is the total outflow from the on-ramp and σc is a model parameter. Appendix B. Training algorithms This appendix details the two actor-critic algorithms used to train the DRL agent within the DRL-MPC framework, namely the Deep Deterministic Policy Gradient (DDPG) and the Soft Actor-Critic (SAC) algorithms, together with the overall training procedure and the network architectures and hyperparameters used for the two algorithms in the case study of Section 5. Appendix B.1. Training the framework using DDPG The DDPG algorithm (Lillicrap et al., 2016) is an off-policy actor-critic approach designed for environments with continuous action spaces. It learns a deterministic policy through an actor network πθ , parameterized by θ, which maps the observed state to a DRL action, while employing a critic network Qϕ , parameterized by ϕ, which estimates the expected discounted return for a state–action pair under the current policy (see Figure 3). To enhance learning stability, DDPG also maintains a target actor network πθ′ and a target critic network Qϕ′ , whose parameters are updated gradually, ensuring smoother changes compared to the main actor and critic networks. All transitions (xrl , ul , r, x+ rl ) are collected from the environment and stored in a replay buffer, removing the oldest transitions once the buffer reaches its capacity. The actor and the critic are updated using a mini-batch of N transitions sampled uniformly at random from the replay buffer. Sampling from the replay buffer reduces temporal correlation between training samples and enables data reuse, thereby improving learning stability and sample efficiency. For more details, the interested reader is referred to (Lillicrap et al., 2016). Sun et al. (2024b) showed that using the n-step temporal difference (TD) return, which sums the rewards over the next n steps using a set of n consecutive transitions sampled from the replay buffer, instead of the traditional 36

one-step TD target improves the learning efficiency and enhances the quality of actions taken in systems with time delays, such as transportation networks. Therefore, we use the n-step TD method to compute the target value, which serves as the learning reference by representing the estimated total discounted reward, as
′ y(kl ) = rn (kl ) + γ n Qϕ′ xrl (kl + n), uθl (kl + n) , ′

where uθl (kl + n) = πθ′ (xrl (kl + n)) and the n-step reward is calculated as rn (kl ) =

n−1 X

γ o r(kl + o).

(B.1)

o=0

If an episode terminates within the n-step window, only the rewards collected up to termination are included in the target. The loss for the critic network is calculated as N 2 1 X y(i) − Qϕ (xrl (i), ul (i)) , L(ϕ) = N i=1

(B.2)

where the index i runs over the data points of a mini batch of size N that is randomly sampled from the replay buffer. The parameters of the critic, ϕ, are updated by minimizing the loss function (B.2), e.g., using the adaptive moment estimation (Adam) optimizer (Kingma and Ba, 2015). Then, the parameters of the actor, θ, are subsequently updated by maximizing the estimated return, Qϕ , based on the policy gradient (Lillicrap et al., 2016). The parameters of the target actor, θ′ , and the target critic, ϕ′ , are updated using Polyak averaging as θ′ ← τ θ + (1 − τ )θ′ , ϕ′ ← τ ϕ + (1 − τ )ϕ′ , where τ is the smoothing factor, with τ ≪ 1, so that the target values change slowly, improving learning stability (Lillicrap et al., 2016). To enhance exploration efficiency, noise is added to the DRL actions, e.g., using the Ornstein-Uhlenbeck process (Uhlenbeck and Ornstein, 1930), which generates temporally correlated noise and can be particularly beneficial in control settings with inertia-like dynamics, such as those encountered in transportation network control. 37

Algorithm 1 Training algorithm for the DRL-MPC framework Input: Initial neural-network parameters (θ, θ′ , ϕ, and ϕ′ for DDPG; max max ˆ keps θ, ϕ1 , ϕ2 , ϕ′1 , and ϕ′2 for SAC), mh , ml , d, , M , N , n, Nw , Ne , Nmb , Nt , and τ 1: Initialize critic network(s) and actor network 2: Initialize target networks 3: Initialize experience replay buffer D 4: Initialize critic-update counter kϕ ← 0 5: for episode = 1 to M do 6: Receive initial transportation network state x(0) 7: Initialize step counter k ← 0 8: Observe initial DRL state xrl max 9: while k < keps do 10: if k (mod mh ) = 0 then 11: kh ← k/mh 12: Obtain uh (kh ) by solving (2) 13: end if 14: ūh (k) ← uh (kh ) 15: if k (mod ml ) = 0 then 16: kl ← k/ml 17: Sample ul (kl ) ∼ πθ (· | xrl (kl )); for DDPG, add exploration noise 18: end if 19: ūl (k) ← ul (kl ) 20: x(k + 1) ← F (x(k), ūh (k), ūl (k), d(k)) 21: if (k + 1) (mod ml ) = 0 then 22: Calculate reward r(kl ) using (3) 23: Observe new DRL state xrl (kl + 1) 24: Store transition (xrl (kl ), ul (kl ), r(kl ), xrl (kl + 1)) in D 25: end if 26: k ←k+1 27: end while 28: if |D| ≥ Nw then max 29: Nmb ← min{Nmb , ⌊|D|/N ⌋} 30: for each of Nmb mini-batches in each of Ne epochs do 31: Sample N transitions from D, each containing n steps 32: Update critic(s) by minimizing loss L(ϕ) 33: kϕ ← kϕ + 1 34: Update actor by maximizing the policy objective 35: Update the entropy weight for SAC 36: if kϕ (mod Nt ) = 0 then 37: Update the target critic(s) and the target actor using Polyak averaging 38: end if 39: end for 40: end if 41: end for Output: Trained actor policy πθ

38

Appendix B.2. Training the framework using SAC The SAC algorithm (Haarnoja et al., 2018a) is an off-policy actor-critic approach that can handle continuous action spaces. SAC learns a stochastic policy that optimizes a combination of the expected return and the policy entropy. The policy entropy quantifies the uncertainty of the policy given a particular state, with higher entropy encouraging broader exploration of the action space. By jointly maximizing the expected cumulative discounted reward and the entropy, SAC aims to achieve a balance between exploration and exploitation. SAC employs two critic networks and uses the minimum of their Q-value estimates to reduce positive overestimation bias. SAC utilizes a replay buffer and updates the actor and critics using mini-batches, as in DDPG. While SAC uses target critic networks in a manner similar to DDPG, it does not maintain a separate target actor network. Instead, the target critics evaluate actions generated directly by the actor πθ . For more details, the interested reader is referred to (Haarnoja et al., 2018a). Similar to DDPG, the target value is computed via the n-step TD method using (B.1) as follows: 
y(kl ) = rn (kl ) + γ n min Qϕ′ψ xrl (kl + n), ul (kl + n) ψ∈{1,2}
 −α ln πθ ul (kl + n) | xrl (kl + n) , where ψ is the index of the target critic, α is the entropy loss weight, and
ul (kl + n) ∼ πθ · | xrl (kl + n) . The entropy loss weight is updated by minimizing the following loss function: N

1 X − α ln πθ ul (i) | xrl (i) − αH , L(α) = N i=1 where H < 0 is the target entropy used to adapt the entropy weight, with values closer to zero promoting a more stochastic policy (Haarnoja et al., 2018a). The parameters ϕj of each critic, j ∈ {1, 2}, are updated by minimizing the loss function in (B.2), with ϕ replaced by ϕj and y given by the SAC target defined above, using the Adam optimizer. Then, the parameters of the actor, θ, are subsequently updated using the policy gradient to maximize the entropy-regularized estimated return, defined as the sum of the minimum Q-value estimated by the two critics and the policy entropy weighted by α 39

(Haarnoja et al., 2018a). The parameters of the target critics are updated using Polyak averaging as ϕ′1 ← τ ϕ1 + (1 − τ )ϕ′1 , ϕ′2 ← τ ϕ2 + (1 − τ )ϕ′2 . The training procedure for the DRL-MPC framework is summarized in Algorithm 1. Appendix B.3. Network architectures and hyperparameters The DDPG actor and critic network architectures and hyperparameters are adopted from Sun et al. (2024b), where they were shown to be effective for DDPG-based ramp metering control. The actor uses two hidden layers of width 256 (ReLU activations), followed by a tanh output layer and an affine scaling to bound each action (i.e., each ramp metering rate) to [0, 1]. The critic uses separate state and action paths with hidden layer widths 256 and 128, respectively. The concatenated features are passed through fully connected layers of widths 256 and 128 with ReLU activations, followed by a scalar output. The target actor and target critic use the same network architectures. The SAC agent uses two critic networks with the same architecture as the DDPG critic, and the target critics use the same architecture. The SAC actor consists of a shared feature extractor with a hidden layer of width 256 (ReLU activations), followed by separate mean and standard deviation heads, each with a hidden layer of width 256. The standard deviation output uses a softplus activation to enforce nonnegative values. Sampled actions are mapped to the action bounds using the same tanh and affine scaling used for the DDPG actor. The hidden layer width of 256 is chosen to match that of the DDPG actor. Each agent is trained for M = 3500 episodes. The DDPG and SAC agents use a replay buffer of length 2 · 105 , a mini-batch size N = 512, discount factor γ = 0.99, target smoothing factor τ = 10−2 , and n-step TD targets with n = 10, following Sun et al. (2024b). The actor and critic learning rates are set to 10−3 with gradient threshold 1, and DDPG exploration uses an Ornstein-Uhlenbeck process with initial standard deviation 0.3 and decay rate 5 · 10−6 , as in Sun et al. (2024b). For SAC, the entropy weight is initialized as α = 1 and is tuned during training using the target entropy H = −2, set as the negative of the action 40

dimension, similar to Haarnoja et al. (2018b). Learning is performed at the end of each episode once the replay buffer contains at least Nw = 512 samples. max Each learning iteration comprises Ne = 1 epoch with at most Nmb = 100 mini-batches, depending on the number available in the replay buffer, i.e., max , ⌊|D|/N ⌋}. Polyak updates with τ = 10−2 are applied Nmb = min{Nmb every Nt = 10 critic updates to the target actor and critic for DDPG and to the two target critics for SAC. Appendix C. Tuned PI-ALINEA parameters

4

3

3

2

10

2

5

0

0

0

15

100

100

5

;72 [veh/km/lane]

1

;71 [veh/km/lane]

1

10

KA;2

15

KA;1

4

KR;2

KR;1

Figure C.12 reports the PI-ALINEA parameters obtained from the five independent Bayesian optimization runs per scenario described in Section 5.2.3.

80 60 40

80 60 40

0 S1

S2

S3

Scenario

S4

S1

S2

S3

Scenario

S4

S1

S2

S3

S4

Scenario

Figure C.12: Tuned PI-ALINEA parameters across the four scenarios. Each marker corresponds to one of the five independently tuned parameter sets per scenario, and the solid horizontal tick denotes the per-scenario mean.

Acknowledgements This research has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No. 101018826, ERC Advanced Grant CLariNet).

41

References Airaldi, F., De Schutter, B., Dabiri, A., 2025. Reinforcement learning with model predictive control for highway ramp metering. IEEE Trans. Intell. Transp. Syst. 26, 5988–6004. doi:10.1109/TITS.2025.3549227. Arulkumaran, K., Deisenroth, M.P., Brundage, M., Bharath, A.A., 2017. Deep reinforcement learning: A brief survey. IEEE Signal Process. Mag. 34, 26–38. doi:10.1109/MSP.2017.2743240. Belletti, F., Haziza, D., Gomes, G., Bayen, A.M., 2018. Expert level control of ramp metering based on multi-task deep reinforcement learning. IEEE Trans. Intell. Transp. Syst. 19, 1198–1207. doi:10.1109/TITS.2017.2725 912. Castañeda, K., Sánchez, O., Herrera, R., Mejía, G., 2022. Highway planning trends: A bibliometric analysis. Sustainability 14, 5544. doi:10.3390/su 14095544. Chanfreut, P., Maestre, J.M., Camacho, E.F., 2021. Coalitional model predictive control on freeways traffic networks. IEEE Trans. Intell. Transp. Syst. 22, 6772–6783. doi:10.1109/TITS.2020.2994772. Dabiri, A., He, K., Shi, S., Sun, D., Lago, J., De Schutter, B., 2025. Integrating learning-based and MPC-based control for PWA systems: Challenges and opportunities, in: Garone, E., Kolmanovsky, I., Nguyen, T.W. (Eds.), Nonlinear and Constrained Control: Applications, Synergies, Challenges and Opportunities. Springer, Cham. volume 496 of Lecture Notes in Control and Information Sciences, pp. 131–149. doi:10.1007/978-3-031-826 81-8_6. Delft High Performance Computing Centre (DHPC), 2024. DelftBlue supercomputer (Phase 2). https://www.tudelft.nl/dhpc/ark:/44463/Del ftBluePhase2. Accessed 12 August 2026. Deo, P., De Schutter, B., Hegyi, A., 2009. Model predictive control for multiclass traffic flows. IFAC Proc. Vol. 42, 25–30. doi:10.3182/20090902-3 -US-2007.0017. Dulac-Arnold, G., Levine, N., Mankowitz, D.J., Li, J., Paduraru, C., Gowal, S., Hester, T., 2021. Challenges of real-world reinforcement learning: 42

definitions, benchmarks and analysis. doi:10.1007/s10994-021-05961-4.

Mach. Learn. 110, 2419–2468.

Floudas, C.A., Pardalos, P.M. (Eds.), 1996. State of the art in global optimization: Computational methods and applications. volume 7 of Nonconvex Optimization and Its Applications. Kluwer Academic Publishers, Dordrecht. doi:10.1007/978-1-4613-3437-8. Gao, H., Jia, H., Wu, R., Huang, Q., Tian, J., Liu, C., Wang, X., 2024. Variable speed limit control for mixed traffic flow on highways based on deep-reinforcement learning. J. Transp. Eng. Part A Syst. 150, 04023147. doi:10.1061/JTEPBS.TEENG-8116. Gros, S., Zanon, M., 2020. Data-driven economic NMPC using reinforcement learning. IEEE Trans. Autom. Control 65, 636–648. doi:10.1109/TAC.20 19.2913768. Haarnoja, T., Zhou, A., Abbeel, P., Levine, S., 2018a. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor, in: Proceedings of the 35th International Conference on Machine Learning (ICML), vol. 80, pp. 1861–1870. Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., Levine, S., 2018b. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 doi:10.485 50/arXiv.1812.05905. Han, Y., Wang, M., Li, L., Roncoli, C., Gao, J., Liu, P., 2022. A physicsinformed reinforcement learning-based strategy for local and coordinated ramp metering. Transp. Res. Part C Emerg. Technol. 137, 103584. doi:10 .1016/j.trc.2022.103584. Haydari, A., Yilmaz, Y., 2022. Deep reinforcement learning for intelligent transportation systems: A survey. IEEE Trans. Intell. Transp. Syst. 23, 11–32. doi:10.1109/TITS.2020.3008612. Hegyi, A., De Schutter, B., Hellendoorn, H., 2005. Model predictive control for optimal coordination of ramp metering and variable speed limits. Transp. Res. Part C Emerg. Technol. 13, 185–209. doi:10.1016/j.trc.20 04.08.001. 43

Hewing, L., Wabersich, K.P., Menner, M., Zeilinger, M.N., 2020. Learningbased model predictive control: Toward safe learning in control. Annu. Rev. Control Robot. Auton. Syst. 3, 269–296. doi:10.1146/annurev-con trol-090419-075625. Jiang, Z., Qi, Z., Wang, L., Zhu, Z., Lee, D.H., 2025. A dynamic cooperative ramp metering and navigation guidance approach based on heterogeneousagent reinforcement learning. IEEE Internet Things J. 12, 31256–31271. doi:10.1109/JIOT.2025.3572177. Kingma, D.P., Ba, J., 2015. Adam: A method for stochastic optimization, in: Proceedings of the 3rd International Conference on Learning Representations (ICLR). Kolat, M., Kővári, B., Bécsi, T., Aradi, S., 2023. Multi-agent reinforcement learning for traffic signal control: A cooperative approach. Sustainability 15, 3479. doi:10.3390/su15043479. Kotsialos, A., Papageorgiou, M., Messner, A., 1999. Integrated optimal control of motorway traffic networks, in: Proceedings of the 1999 American Control Conference (ACC), pp. 2183–2187. doi:10.1109/ACC.1999.786 339. Kušić, K., Ivanjko, E., Gregurić, M., Miletić, M., 2020. An overview of reinforcement learning methods for variable speed limit control. Appl. Sci. 10, 4917. doi:10.3390/app10144917. Liang, X., Du, X., Wang, G., Han, Z., 2019. A deep reinforcement learning network for traffic light cycle control. IEEE Trans. Veh. Technol. 68, 1243– 1253. doi:10.1109/TVT.2018.2890726. Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., Wierstra, D., 2016. Continuous control with deep reinforcement learning, in: Proceedings of the 4th International Conference on Learning Representations (ICLR). Lin, S., De Schutter, B., Xi, Y., Hellendoorn, H., 2011. Fast model predictive control for urban road networks via MILP. IEEE Trans. Intell. Transp. Syst. 12, 846–856. doi:10.1109/TITS.2011.2114652.

44

Lin, S., De Schutter, B., Xi, Y., Hellendoorn, H., 2013. Integrated urban traffic control for the reduction of travel delays and emissions. IEEE Trans. Intell. Transp. Syst. 14, 1609–1619. doi:10.1109/TITS.2013.2263843. Liu, S., Hellendoorn, H., De Schutter, B., 2017. Model predictive control for freeway networks based on multi-class traffic flow and emission models. IEEE Trans. Intell. Transp. Syst. 18, 306–320. doi:10.1109/TITS.2016. 2573306. Lu, Y., Li, C., Yu, H., Wang, H., 2025. Soft actor-critic-based regional traffic signal control in a connected environment and its application in priority signal control. J. Intell. Transp. Syst. doi:10.1080/15472450.2025.2532 724. Majstorović, Ž., Tišljarić, L., Ivanjko, E., Carić, T., 2023. Urban traffic signal control under mixed traffic flows: Literature review. Appl. Sci. 13, 4484. doi:10.3390/app13074484. Mayne, D.Q., Rawlings, J.B., Rao, C.V., Scokaert, P.O.M., 2000. Constrained model predictive control: Stability and optimality. Automatica 36, 789–814. doi:10.1016/S0005-1098(99)00214-9. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., Riedmiller, M., 2013. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 doi:10.48550/arXiv.1312.5602. Pasquale, C., Sacone, S., Siri, S., De Schutter, B., 2017. A multi-class modelbased control scheme for reducing congestion and emissions in freeway networks by combining ramp metering and route guidance. Transp. Res. Part C Emerg. Technol. 80, 384–408. doi:10.1016/j.trc.2017.04.007. Pham, V.H., Ahn, H.S., 2023. Distributed stochastic MPC traffic signal control for urban networks. IEEE Trans. Intell. Transp. Syst. 24, 8079– 8096. doi:10.1109/TITS.2023.3262580. Pham, V.H., Sakurama, K., Mou, S., Ahn, H.S., 2022. Distributed control for an urban traffic network. IEEE Trans. Intell. Transp. Syst. 23, 22937– 22953. doi:10.1109/TITS.2022.3208565.

45

Rawlings, J.B., Mayne, D.Q., Diehl, M., 2018. Model Predictive Control: Theory, Computation, and Design. 2nd ed., Nob Hill Publishing, Madison, WI. Reiter, R., Hoffmann, J., Reinhardt, D., Messerer, F., Baumgärtner, K., Sawant, S., Boedecker, J., Diehl, M., Gros, S., 2026. Synthesis of model predictive control and reinforcement learning: Survey and classification. Annu. Rev. Control 61, 101045. doi:10.1016/j.arcontrol.2026.101045. Rosolia, U., Singletary, A., Ames, A.D., 2022. Unified multirate control: From low-level actuation to high-level planning. IEEE Trans. Autom. Control 67, 6627–6640. doi:10.1109/TAC.2022.3184664. Siri, S., Pasquale, C., Sacone, S., Ferrara, A., 2021. Freeway traffic control: A survey. Automatica 130, 109655. doi:10.1016/j.automatica.2021.10 9655. Sirmatel, I.I., Geroliminis, N., 2018. Economic model predictive control of large-scale urban road networks via perimeter control and regional route guidance. IEEE Trans. Intell. Transp. Syst. 19, 1112–1121. doi:10.1109/ TITS.2017.2716541. Sun, D., Jamshidnejad, A., De Schutter, B., 2023. Adaptive parameterized control for coordinated traffic management using reinforcement learning. IFAC-PapersOnLine 56, 5463–5468. doi:10.1016/j.ifacol.2023.10.1 98. Sun, D., Jamshidnejad, A., De Schutter, B., 2024a. Adaptive parameterized model predictive control based on reinforcement learning: A synthesis framework. Eng. Appl. Artif. Intell. 136, 109009. doi:10.1016/j.engapp ai.2024.109009. Sun, D., Jamshidnejad, A., De Schutter, B., 2024b. A novel framework combining MPC and deep reinforcement learning with application to freeway traffic control. IEEE Trans. Intell. Transp. Syst. 25, 6756–6769. doi:10.1109/TITS.2023.3342651. Todorović, U., Frejo, J.R.D., De Schutter, B., 2022. Distributed MPC for large freeway networks using alternating optimization. IEEE Trans. Intell. Transp. Syst. 23, 1875–1884. doi:10.1109/TITS.2020.3028850. 46

Uhlenbeck, G.E., Ornstein, L.S., 1930. On the theory of the Brownian motion. Phys. Rev. 36, 823–841. doi:10.1103/PhysRev.36.823. Wang, C., Xu, Y., Zhang, J., Ran, B., 2022. Integrated traffic control for freeway recurrent bottleneck based on deep reinforcement learning. IEEE Trans. Intell. Transp. Syst. 23, 15522–15535. doi:10.1109/TITS.2022.31 41730. Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., Miao, Q., 2024. Deep reinforcement learning: A survey. IEEE Trans. Neural Netw. Learn. Syst. 35, 5064–5078. doi:10.1109/TNNLS.2022.3207346. Wang, Y., Papageorgiou, M., 2006. Local ramp metering in the case of distant downstream bottlenecks, in: Proceedings of the 2006 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 426–431. doi:10.1109/IT SC.2006.1706778. van de Weg, G.S., Hegyi, A., Hoogendoorn, S.P., De Schutter, B., 2019. Efficient freeway MPC by parameterization of ALINEA and a speed-limited area. IEEE Trans. Intell. Transp. Syst. 20, 16–29. doi:10.1109/TITS.201 8.2790167. Wu, T., Zhou, P., Liu, K., Yuan, Y., Wang, X., Huang, H., Wu, D.O., 2020a. Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks. IEEE Trans. Veh. Technol. 69, 8243–8256. doi:10.1 109/TVT.2020.2997896. Wu, Y., Tan, H., Qin, L., Ran, B., 2020b. Differential variable speed limits control for freeway recurrent bottlenecks via deep actor-critic algorithm. Transp. Res. Part C Emerg. Technol. 117, 102649. doi:10.1016/j.trc.20 20.102649. Xue, C., Ding, L., Quan, Z., Li, Y., 2024. Multirate modeling and predictive control for WBG-device-based high-switching-frequency power converters. IEEE Trans. Ind. Electron. 71, 93–103. doi:10.1109/TIE.2023.3250765. Zanon, M., Gros, S., 2021. Safe reinforcement learning using robust MPC. IEEE Trans. Autom. Control 66, 3638–3652. doi:10.1109/TAC.2020.302 4161.

47

Related documents

• Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks#662052
• Sharing the Control Authority Between Deep Reinforcement Learning and Model Predictive Control: Application to Multi-Class Transportation Networks#661002
• Composite-Gradient Learning for Shared Control Authority Between Deep Reinforcement Learning and Model Predictive Control#1011422
• REFINEPPO: Learning Continuous Control Policies by Iterative Action Refinement#1035078
• Fading Expert Guidance: Bridging Model-Based and Learning-Based Control for Abortable Autonomous Overtaking#1128485