1
Augmented Lagrangian Multiplier Network for State-wise Safety in Reinforcement Learning
arXiv:2605.00667v1 [cs.LG] 1 May 2026
Jiaming Zhang, Yujie Yang, Yao Lyu, Shengbo Eben Li*, Liping Zhang*
return is frequently inadequate for real-world deployments. For instance, autonomous vehicles cannot risk collisions simply to minimize travel time. Therefore, it is necessary to incorporate cost signals and feasibility constraints into the training process to ensure policy safety [1]. Traditionally, safe RL is formulated as a Constrained Markov Decision Process (CMDP), defining safety requirements by bounding the expected cumulative cost [4]. However, this paradigm only bounds the cumulative cost of an entire trajectory, failing to guarantee safety at every individual state. An agent may still violate constraints at specific steps while keeping its total trajectory cost below the permissible threshold. To overcome this limitation, recent methods adopt state-wise constraints to enforce feasibility at every state [5]–[7]. These constraints provide a more rigorous guarantee, ensuring the agent remains confined to the safe region at every time step [8]. To solve safe RL problems, existing approaches primarily combine RL algorithms with constrained optimization techniques, among which the Lagrange multiplier method is the most widely used. For example, algorithms such as TRPOLag, PPO-Lag, and SAC-Lag embed classic RL methods into Lagrangian formulation to solve CMDPs [4], [9]. RCPO directly integrates the constraint function into the reward to form a Index Terms—safe reinforcement learning, state-wise con- Lagrangian composite signal, and employs a multi-timescale straints, augmented Lagrangian, parameterized multiplier, super- update scheme with theoretical convergence guarantees [10]. vised regression To restrict the probability of constraint violations under extreme scenarios, the Lagrangian framework is combined I. I NTRODUCTION with Conditional Value-at-Risk (CVaR) and chance constraints EINFORCEMENT learning has demonstrated remarkable in [11]. This work derives the policy gradient for CVaR performance in solving complex sequential decision- MDPs and develops specific actor-critic algorithms. These making and optimal control problems. Its enormous potential methods convert the original formulation into a primal-dual has driven widespread adoption across various fields, including minimax problem to balance task performance and safety autonomous driving [1], embodied artificial intelligence [2], and requirements. Optimization proceeds with an alternating update large language models [3]. Fundamentally, RL enables agents to scheme: the policy optimizes the Lagrangian objective to achieve optimal performance by learning policies that maximize maximize returns alongside penalized costs, while the multiplier expected cumulative reward. However, optimizing solely for updates via dual gradient ascent to dynamically calibrate the penalty scale. Consequently, this multiplier increases to This study is supported by National Natural Foundation of China enforce safety compliance when constraint thresholds are with 12571323, National Science and Technology Major Project (No exceeded, and decreases to prioritize reward maximization 2025ZD1606200), Beijing Natural Science Foundation with L257002 and NSF China with 92582205. It is also partially supported by SunRising AI Lab. once safety is guaranteed. While these methods effectively * Corresponding authors: Liping Zhang and Shengbo Eben Li. solve trajectory-based CMDPs using a single scalar multiplier, Jiaming Zhang and Liping Zhang are with the Department of Mathematical they are fundamentally ill-equipped for state-wise constraints. Sciences, Tsinghua University, Beijing, 100084, China (e-mail: jiamingEnforcing safety across a continuous state space introduces an [email protected], [email protected]). Yujie Yang and Yao Lyu are with the School of Vehicle and infinite number of constraints, which theoretically requires an Mobility, Tsinghua University, Beijing, 100084, China (e-mail: infinite number of distinct multipliers. A naive workaround is to [email protected], [email protected]). Shengbo Eben Li is with the School of Vehicle and Mobility & College of AI, mathematically aggregate these infinitely many constraints into Tsinghua University, Beijing, 100084, China (e-mail: [email protected]). a single global condition, such as summing the constraint
Abstract—Safety is a primary challenge in real-world reinforcement learning (RL). Formulating safety requirements as state-wise constraints has become a prominent paradigm. Handling statewise constraints with the Lagrangian method requires a distinct multiplier for every state, necessitating neural networks to approximate them as a multiplier network. However, applying standard dual gradient ascent to multiplier networks induces severe training oscillations. This is because the inherent instability of dual ascent is exacerbated by network generalization—local overshoots and delayed updates propagate to adjacent states, further amplifying policy fluctuations. Existing stabilization techniques are designed for scalar multipliers, which are inadequate for state-dependent multiplier networks. To address this challenge, we propose an augmented Lagrangian multiplier network (ALaM) framework for stable learning of state-wise multipliers. ALaM consists of two key components. First, a quadratic penalty is introduced into the augmented Lagrangian to compensate for delayed multiplier updates and establish the local convexity near the optimum, thereby mitigating policy oscillations. Second, the multiplier network is trained via supervised regression toward a dual target, which stabilizes training and promotes convergence. Theoretically, we show that ALaM guarantees multiplier convergence and thus recovers the optimal policy of the constrained problem. Building on this framework, we integrate soft actor-critic (SAC) with ALaM to develop the SAC-ALaM algorithm. Experiments demonstrate that SAC-ALaM outperforms state-of-the-art safe RL baselines in both safety and return, while also stabilizing training dynamics and learning well-calibrated multipliers for risk identification.
R
2
violations across all states, and penalize it with a single stabilize the training of parameterized multipliers. Our main scalar multiplier. However, this aggregation essentially assigns contributions are summarized as follows. a uniform penalty across the entire state space, completely • We introduce an augmented Lagrangian formulation with discarding the state-specific nature of the constraints. This a quadratic penalty for constraint violations. This design homogeneity significantly increases optimization difficulty and compensates for delayed multiplier responses in convenusually yields suboptimal behaviors in practice: the policy tional methods through instantaneous feedback, thereby becomes overly conservative in safe regions while remaining mitigating policy oscillations. Meanwhile, the quadratic unsafe in hazardous states. penalty establishes local convexity around the optimum. To genuinely satisfy state-wise constraints, the algorithm This structural refinement improves the optimization must maintain distinct multipliers for different states. In landscape and alleviates the ill-conditioned saddle-point continuous state spaces, this is practically achieved by emgeometries in the original Lagrangian formulation, preventploying parameterized approximation functions, transforming ing policy divergence and therefore leading to substantially the theoretical infinite multipliers into a neural network. This more stable training. multiplier network takes the state as input and dynamically • We show that direct gradient ascent on multiplier network outputs a dedicated penalty for that specific state. For instance, parameters, as commonly used in existing methods, does FAC trains a multiplier network to assign specific penalties to not realize true dual ascent in function space because individual states [12]. To accommodate local safety constraints, nonlinear parameterization distorts the effective update this research defines a state-wise Lagrange function. System step. To address this issue, we train the multiplier network optimization then proceeds through the primal-dual procedure via supervised regression toward an analytical dual target. over this formulation: the policy minimizes the Lagrangian, This mechanism more faithfully captures the desired and the multiplier parameters adjust via dual gradient ascent. multiplier function and yields improved training stability However, empirical results in FAC exhibit noticeable training and convergence behavior. oscillations, a drawback explicitly acknowledged by the authors • We provide rigorous theoretical analysis showing that as a primary limitation of their method. In fact, this instaALaM guarantees multiplier convergence under standard bility is a broadly recognized challenge even in conventional assumptions. Based on this result, we further establish Lagrangian-based safe RL algorithms that rely on a single scalar that any limit point of the induced policy sequence is multiplier [13], [14]. The root cause lies in the dynamics of both feasible and optimal for the constrained problem. standard dual gradient ascent: the adjustments to the multiplier These results provide a principled foundation for stable inherently lag behind the rapid shifts in the policy’s safety multiplier network training in safe RL. behavior. This persistent delay traps the multiplier in a cycle Finally, extensive experiments demonstrate that SAC-ALaM of insufficient penalization and severe overshooting, alter- achieves state-of-the-art performance in balancing task return nately leading to constraint violations and overly conservative and safety compliance. In addition, it learns well-calibrated policies. To overcome this, the PID-Lagrangian approach state-dependent multipliers for reliable local risk identification models the multiplier update as a control problem, replacing and exhibits strong generalization to unseen scenarios. basic gradient ascent with a proportional-integral-derivative controller to dampen these oscillations [13]. Nonetheless, it II. P RELIMINARIES introduces extra hyperparameters, lacks theoretical guarantees, and frequently fails to outperform standard gradient ascent A. Reinforcement learning with state constraints in practice [15]. Alternatively, APPO employs the augmented RL is typically formulated as a Markov Decision Process Lagrangian method, which adds a quadratic penalty to enable (MDP) represented by M = ⟨X , A, f, r, γ, µ⟩. Here, X and A closed-form multiplier updates [16]. While APPO has been denote the state and action spaces. f : X ×A → X specifies the validated on scalar multipliers, our experiments reveal that its state transition dynamics, where x t+1 = f (xt , at ) defines the efficacy in mitigating oscillations within multiplier networks is next state given the current state x and action a . Furthermore, t t limited and often sacrifices task performance. This inadequacy r : X ×A → R is the reward function, γ ∈ [0, 1] is the discount arises because introducing a parameterized network intrinsically factor, and µ represents the initial state distribution. The policy aggravates the multiplier’s oscillation problem, driven by two in the MDP considered in this paper is a deterministic function main factors. First, network generalization can erroneously that maps a state to an action: π : X → A. Under this policy, over-penalize safe regions near hazards. Second, policy updates the expected cumulative reward J(π) is defined as: induce distribution shifts that trigger catastrophic forgetting, "∞ # X mistakenly erasing valid penalties for still-hazardous states. To t J(π) = Ex0 ∼µ γ r(xt , π(xt )) . date, achieving stable training for multiplier networks under t=0 state-wise constraints remains an open challenge. To address this challenge, we propose an augmented La- This function is the maximization objective in standard RL. grangian multiplier network (ALaM) framework for stable However, optimizing solely for performance is insufficient learning of state-wise multipliers in safe reinforcement learning. for real-world applications; the safety of the policy must also be By integrating soft actor-critic (SAC) with ALaM, we further guaranteed. We formalize this feasibility requirement through develop SAC-ALaM, a practical safe RL approach. To the best state-wise constraints h(x) ≤ 0, where h is a function whose of our knowledge, our approach is the first to successfully zero sublevel set defines the safe region. Since RL involves
3
sequential decision-making, we enforce this constraint at every time step, i.e., h(xt ) ≤ 0, ∀t ≥ 0. Consequently, the safe RL problem is formulated as max
J(π)
s.t.
h(xt ) ≤ 0,
π
∀x0 ∼ µ, t ≥ 0.
(1)
B. Feasibility function Directly enforcing state-wise constraints is computationally intractable, as it involves an infinite number of conditions spanning the continuous state space and the infinite time horizon. To overcome this, we adopt the feasibility function [8], [17], which aggregates the infinite-horizon constraints into a single condition determined by the initial state. We formally define it as follows: Definition 1 (Feasibility Function). F π : X → R is a feasibility function for policy π if: 1) For any initial state x0 = x, the condition F π (x) ≤ 0 holds if and only if h(xt ) ≤ 0 for all t ≥ 0. 2) There exists a risky self-consistency operator T π : RX → RX such that: (i) T π F π = F π . (ii) T π is a monotone operator. (iii) T π is a γ-contraction under the infinity norm. That is, for any F1 , F2 ∈ RX , ∥T π F1 − T π F2 ∥∞ ≤ γ∥F1 − F2 ∥∞ , where γ ∈ [0, 1) is the discount factor. As a concrete example, the cost value function is a standard feasibility function: ∞ X F π (x) = γ t c(xt ), ∀x ∈ X , t=0
where x0 = x, xt+1 = f (xt , π(xt )), and c(x) = 1h(x)>0 is the indicator function of h. Since c(x) is non-negative, F π (x) ≤ 0 holds if and only if the state x is long-term feasible under policy π. The corresponding risky self-consistency operator is T π F π (x) = c(x) + γF π (f (x, π(x))). Other commonly used feasibility functions include the constraint decay functions and Hamilton-Jacobi reachability functions (e.g., see [18] for a detailed review). By utilizing this feasibility function, we effectively bypass the computationally prohibitive infinite-horizon conditions while strictly guaranteeing state safety. Consequently, the original safe RL problem can be equivalently reformulated as follows: max J(π) π (2) s.t. F π (x) ≤ 0, ∀x ∈ supp(µ). Here, supp(µ) = {x ∈ X : µ(x) > 0} denotes the support of the initial state distribution µ. While the feasibility function converts the infinite-horizon constraints into singlestep conditions, the formulation still imposes an infinite number of constraints across the state space. In contrast, the standard CMDPs evaluate only the expected cost under the distribution µ, and reduce the safety requirement to a single global constraint. Therefore, the state-wise constrained framework provides a more rigorous definition of safety.
C. Lagrange multiplier network Under the standard CMDP setting, safe RL aims to maximize expectedPreturns and restrict expected costs, i.e., Jc (π) := ∞ Ex0 ∼µ [ t=0 γ t c(xt , π(xt ))] ≤ d. Notably, c herein denotes the environmental cost signal, which is distinct from the feasibility indicator function defined previously. Within the Lagrange multiplier paradigm, the dual problem with a scalar multiplier λ ≥ 0 takes the following standard structure: max min L(θ, λ) = −J(πθ ) + λ(Jc (πθ ) − d). λ≥0
θ
The algorithm updates parameters through an alternate optimization mechanism. Specifically, the policy minimizes the Lagrangian function, and the multiplier adjusts via the dual gradient ascent step: θ ← argmin L(θ, λ),
λ ← λ + αλ (Jc (π) − d).
θ
This setup aggregates the risks over the entire state distribution into a single global constraint, which fails to guarantee the local safety of specific states. To establish genuine state-wise safety, the system must satisfy an infinite number of local constraints. In continuous state spaces, it is impractical to maintain an infinite number of independent multipliers. Therefore, recent advanced methods introduce a parameterized multiplier network to approximate the theoretical multipliers [12]. Based on this network, the framework reformulates the Lagrangian to accommodate the state-wise cost value constraints: L(θ, w) = −J(πθ ) + Ex∼µ λw (x)(vcπθ (x) − d). Under this formulation, the multiplier parameters w adjust via dual gradient ascent based on constraint violations. The corresponding primal-dual updates are: θ ← argmin L(θ, w), θ
w ← w + αw Ex∼B [(vcπθ (x) − d)∇w λw (x)] .
(3)
Although the multiplier network theoretically equips the algorithm with the capacity to control local risks at each state, the transition from theoretical scalar multipliers to a neural network reshapes the optimization landscape. This is because the gradients of the neural network distort the exact theoretical step size within the functional space. Furthermore, during the alternate update of policy and multiplier parameters, the generalization of multiplier parameters amplifies the oscillations intrinsic to traditional primal-dual methods. This instability constitutes the exact problem that this research aims to resolve. III. AUGMENTED L AGRANGIAN MULTIPLIER NETWORK To overcome the instability in multiplier network optimization, we introduce the augmented Lagrangian method (ALM) [19]. ALM augments the standard Lagrange function with a quadratic penalty for constraint violations, which compensates for the inherent response delays and parameter overshoots in traditional methods via instantaneous penalty feedback, thereby mitigating algorithmic oscillations. Furthermore, ALM establishes local convexity around the optimum and reshapes the optimization landscape into a stable basin. This structural
4
refinement resolves the ill-conditioned saddle-point geometries in the Lagrangian formulation, prevents the fluctuations caused by policy divergence and therefore improves training stability. Motivated by these advantages, we develop the augmented Lagrangian multiplier network method to solve problem (2). A. Augmented Lagrangian function To construct the augmented framework for problem (2), we first address the continuum of state-wise inequality constraints. We introduce a state-dependent, non-negative slack function p(x) ≥ 0 to convert these inequalities into equivalent equality constraints: max
J(π)
s.t.
F π (x) + p(x) = 0,
π,p
p(x) ≥ 0, ∀x ∈ supp(µ).
Unlike the standard Lagrangian, this augmented formulation incorporates a quadratic penalty. Through the association of a multiplier function λ : X → R with these equality constraints, we formulate the augmented Lagrangian function as follows: Z L(π, p, λ, ρ) = −J(π) + λ(x)(F π (x) + p(x)) dx Z (4) ρ π 2 + (F (x) + p(x)) dx. 2 where the integration is performed over the state space X with respect to the distribution µ, and ρ > 0 denotes the penalty factor. Similar to the standard Lagrangian method, we construct the primal-dual formulation with a dynamic penalty ρ, which translates into the saddle-point problem: (π, p, λ) = max min L(π, p, λ, ρ). λ
π,p
To obtain the optimal policy, ALaM iteratively advances the overall training process through three coupled mechanisms: optimization of the primal variables (the policy and the slack function), update for the multiplier function, and an adaptive adjustment scheme for the penalty factor. B. Primal variables update At the k-th iteration, given the current multiplier λk and penalty factor ρk , we update the primal variables by solving the following joint minimization problem: π k+1 , pk+1 = argmin L π, p, λk , ρk . (5) π,p≥0
Since the objective function is quadratic with respect to the slack variable p(x), we can analytically eliminate it, reducing the formulation (5) to an optimization problem depending only on π. That is, for any fixed state x, the subproblem for p(x) is given by: pk+1 (x) = argmin λk (x)(F π (x)+p(x))+ p(x)≥0
ρk π (F (x)+p(x))2 , 2
which admits a closed-form solution: k λ (x) pk+1 (x) = max − k − F π (x), 0 . ρ
Substituting the expression back into L(π, p, λk , ρk ) produces a simplified augmented Lagrangian: k Z λ (x) π k k k L(π, λ , ρ ) = −J(π) + λ (x) max − k , F (x) dx ρ 2 k k Z ρ λ (x) + dx. max − k , F π (x) 2 ρ Consequently, the policy update rule reduces to the subproblem: π k+1 = argmin L π, λk , ρk . (6) π
C. Multiplier update We initiate the multiplier update through the derivation of the theoretical update rule within the functional space. An optimal solution for the original problem must fulfill the Karush-KuhnTucker (KKT) stationarity condition: Z −∂J(π) + λ(x)∂F π (x) dx = 0, (7) where ∂ denotes the derivative with respect to π. Concurrently, at the k-th iteration, the newly obtained optimal solutions π k+1 and pk+1 from the augmented Lagrangian subproblem (5) satisfy their stationarity optimality condition: − ∂J(π k+1 )+ Z k+1 k+1 λk (x) + ρk F π (x) + pk+1 (x) ∂F π (x) dx = 0. (8) To align the augmented condition (8) with the KKT condition (7), the multiplier function must be updated to match the k+1 effective multiplier term λk + ρk (F π + pk+1 ). Substituting the derived closed-form solution for the optimal slack variable pk+1 , we obtain the dual target: n o k k π k+1 λk+1 (x) = max λ (x) + ρ F (x), 0 , ∀x ∈ supp(µ). target Because the state space X contains an infinite continuum of constraints, a discrete point-wise update for every multiplier is computationally intractable. This necessitates the representation of the multiplier function via a neural network. Through this parameterization, we reformulate the dual update as a supervised regression task. Specifically, the network is trained to match the theoretical target by minimizing the mean squared error objective: Z 2 Lλ = λ(x) − λk+1 dx. (9) target (x) D. Penalty factor update Unlike traditional multiplier methods that alternate between policies and multipliers, ALaM dynamically adjusts the penalty factor to provide instantaneous feedback on constraint violations. This adaptive implementation serves as a crucial compensation for delayed dual response, and thereby suppresses algorithmic oscillations. To illustrate this mechanism, consider the scenario where constraints are active. Under such conditions, the closed-form update of the dual variable simplifies to: λk+1 (x) = λk (x) + ρk F π
k+1
(x),
5
which implies Fπ
k+1
1 (x) = k λk+1 (x) − λk (x) . ρ
Therefore, the constraint violation at any state x scales inversely with the penalty factor ρk . This indicates that we can reduce infeasibility by increasing the penalty parameter. To quantify the overall constraint violation, we monitor the following metric at each iteration: sZ 2 vk = pk+1 (x) + F πk+1 (x) dx s =
Z
max F
π k+1
λk (x) (x), − k ρ
2 dx.
by a neural network over the compact space X , it is inherently bounded and thus square-integrable. Furthermore, because neural networks possess sufficient expressive power, we develop our theoretical analysis under the assumption of exact function approximation. The update rules are summarized as follows: π k+1 = argmin L(π, λk , ρk ), π n o k+1 k+1 λ (x) = max 0, λk (x) + ρk F π (x) , ∀x, ρk+1 = min{σρk , ρmax }, if v k > 1/ρk . Since the sequence {ρk } geometrically increases and is bounded by ρmax , it remains constant at a terminal value ρ̄ after finite iterations. Our subsequent convergence analysis relies on the following assumptions.
If v k > 1/ρk , we amplify the penalty factor via ρk+1 ← σρk , where σ > 1 is a scaling coefficient. Classical augmented Assumption 1. The convergence requires the following condiLagrangian theory [19] shows that this method possesses tions: exact penalty properties. Once ρ exceeds a finite threshold, 1) J(π) and the mapping π 7→ F π are bounded over the the optimization of the augmented objective recovers the policy space. optimal solution of the original constrained problem. We utilize 2) J(π) and π 7→ F π are strongly continuous with respect this property to improve the penalty factor update scheme. to π. Specifically, we restrict the penalty parameter to an upper bound 3) J(π) is concave, and F π (x) is convex with respect to π ρmax through the assignment ρk+1 ← min{σρk , ρmax }. This for all x ∈ X . clipping mechanism safely prevents gradient explosion during Several remarks regarding these assumptions are in order. training while effectively guaranteeing constraint satisfaction. Assumption 1 naturally holds in physical systems, where hardware and energy limits bound the returns and costs. AsIV. C ONVERGENCE ANALYSIS sumption 2 imposes standard continuity to ensure convergence In this section, we establish the theoretical convergence under the weak topology [20, Definition 26.1]. Finally, while guarantees for ALaM. We assume that the state space X is the literature typically achieves convexity by mapping RL compact, and the action space A is a compact and convex set. problems into the occupancy measure space [9], extending These assumptions are well-justified by practical continuous this transformation to our setting poses significant analytical control scenarios, such as finite sensor ranges and actuator challenges due to the infinite-dimensional dual variable and saturation. The Lagrangian L(π, λ) and the dual functional the state-wise constraints. Thus, Assumption 3 serves as a d(λ) are defined as: necessary condition to guarantee the existence of a global Z π L(π, λ) = −J(π) + λ(x)F (x) dx, d(λ) = inf L(π, λ). saddle point. π Since the action space A is compact, any deterministic For a rigorous analysis, we formulate the dual problem policy π : X → A is bounded. Combined with the fact that in a Hilbert space. Let H = L2 (X ) be the Hilbert space the compact state space X possesses a finite measure, this of square-integrable functions defined on X , with its closed ensures that the policy space Π is uniformly bounded within 2 convex positive cone denoted by H+ = {λ ∈ H | λ(x) ≥ the Hilbert space L (X ). We now consider the Lagrange dual problem: 0, R ∀x ∈ X }. Equipped with the standard inner 2product ⟨f, g⟩ = max d(λ), and the induced norm ∥f ∥ = ⟨f, f ⟩, the f (x)g(x) dx X λ∈H+ standard and augmented Lagrangian can be rewritten as: which is equivalent to solving the unconstrained minimization L(π, λ) = −J(π) + ⟨λ, F π ⟩ , problem: min Φ(λ), (10) and λ λ where Φ(λ) = −d(λ) + δH+ (λ). Here, δH+ denotes the L(π, λ, ρ) = −J(π) + λ, max − , F π ρ indicator function of H+ , mapping to 0 if λ ∈ H+ and 2 ρ λ π ∞ otherwise. Finally, let T = ∂Φ denote the subdifferential + max − , F operator of Φ. 2 ρ 1 1 2 2 = −J(π) + ∥max {0, λ + ρF π }∥ − ∥λ∥ . Lemma 1. T is a maximal monotone operator. Specifically, 2ρ 2ρ an operator T is monotone if ⟨y1 − y2 , λ1 − λ2 ⟩ ≥ 0 for any Throughout the analysis, we assume that the multiplier sequence remains square-integrable, i.e., {λk } ⊂ H. This condition naturally holds in practice: since the multiplier is parameterized
y1 ∈ T (λ1 ) and y2 ∈ T (λ2 ), and it is maximal in that there exists no other monotone operator T ′ satisfying Graph(T ) ⊊ Graph(T ′ ), where Graph(T ) = {(λ, y) ∈ H × H : y ∈ T (λ)}.
6
Proof. By Rockafellar’s theorem [21, Theorem A], the subdif- Since the subdifferential ∂δH+ coincides with the normal cone ferential of a proper, lower semi-continuous (l.s.c.), and convex NH+ for convex indicator functions, it follows that: functional is guaranteed to be a maximal monotone operator. k+1 λk − λk+1 We verify these properties for Φ. Since H+ is a closed convex ∈ −F π + ∂δH+ (λk+1 ). k ρ set, its indicator functional δH+ is convex and l.s.c. The negative dual functional −d(λ) = − inf π L(π, λ) = supπ {−L(π, λ)} Invoking Lemma 2, we have is the point-wise supremum of a family of affine functions in λ. λk − λk+1 ∈ ∂(−d)(λk+1 ) + ∂δH+ (λk+1 ) = T (λk+1 ), Such a supremum is convex and l.s.c. Their sum Φ = −d+δH+ k ρ preserves both properties. By Assumption 1, since J(π) and F π are bounded over Π, Φ(λ) is proper. Therefore, we conclude which can be rearranged as λk ∈ (I + ρk T )(λk+1 ). Since T is maximal monotone, Minty theorem [22, Theorem 21.1, that T = ∂Φ is maximal monotone. Proposition 23.7] ensures that the resolvent J k = (I + ρk T )−1 πk k Lemma 2. For any index k ≥ 1, we have −F ∈ ∂(−d)(λ ). is everywhere defined and single-valued. Thus, the update corresponds to λk+1 = J k λk . Proof. For k ≥ 0, the update rule π k+1 = Let λ̂ ∈ Ω⋆ be an arbitrary optimal solution, meaning 0 ∈ argminπ L(π, λk , ρk ) implies the following first-order T (λ̂). By the monotonicity of T , we have stationarity condition: k λ − λk+1 k+1 − 0, λ − λ̂ ≥ 0, k+1 k k 0 ∈ ∂π L π ,λ ,ρ ρk 1 2 = ∂π −J(π) + k max 0, λk + ρk F π which implies ⟨λk − λk+1 , λk+1 − λ̂⟩ ≥ 0. Thus, expanding 2ρ k+1 π=π the squared norm yields: D n o E k k π k+1 π = ∂π −J(π) + max 0, λ + ρ F ,F ∥λk − λ̂∥2 = ∥λk − λk+1 + λk+1 − λ̂∥2 π=π k+1 D E k+1 = −∂π J π k+1 + ∂π λk+1 , F π = ∥λk − λk+1 ∥2 + ∥λk+1 − λ̂∥2 (14) + 2⟨λk − λk+1 , λk+1 − λ̂⟩ = ∂π L π k+1 , λk+1 . (11) ≥ ∥λk − λk+1 ∥2 + ∥λk+1 − λ̂∥2 . By Assumption 2, both −J(π) and F π are convex in π, L(π, λ) is also convex. Consequently, Eq. (11) ensures that π k+1 ∈ Rearranging the terms, we obtain ∥λk+1 − λ̂∥2 ≤ ∥λk − λ̂∥2 − ∥λk − λk+1 ∥2 . This establishes that the sequence {∥λk − λ̂∥2 } argminπ L(π, λk+1 ), evaluating the dual function as: is monotonically decreasing. Being bounded below by zero, it k+1 d λk+1 = L π k+1 , λk+1 = −J π k+1 + λk+1 , F π . must converge, which in turn guarantees that {λk } is bounded. (12) Taking the sum of Eq. (14) over k, we have For any λ ∈ H+ , the definition of the dual function provides: ∞ X ∥λk − λk+1 ∥2 ≤ ∥λ0 − λ̂∥2 < ∞, k+1 d(λ) = inf L(π, λ) ≤ −J π k+1 + λ, F π , (13) k=0 π
which necessitates that limk→∞ ∥λk −λk+1 ∥2 = 0. Since {λk } is bounded in the reflexive Hilbert space L2 (X ), the EberleinŠmulian theorem [23, Theorem 3.18] guarantees the existence of a weakly convergent subsequence λkj ⇀ λ̄. Moreover, since k+1 kj kj +1 → 0, it follows that λkj +1 ⇀ λ̄ as well. Let By the definition of the subdifferential, this implies −F π ∈ ∥λ − λ k ∥k+1 λ −λ k+1 y = . With the penalty parameter ρk bounded ∂(−d)(λk+1 ), which completes the proof. ρk away from zero, the convergence ∥λk − λk+1 ∥ → 0 ensures We are now positioned to establish the convergence of ALaM. that y kj +1 → 0. Since the dual problem (10) is convex, a multiplier λ⋆ is optimal We now assert that 0 ∈ T (λ̄). Since T is maximal monotone, if and only if 0 ∈ ∂Φ(λ⋆ ). Let Ω⋆ = {λ ∈ H : 0 ∈ ∂Φ(λ)} any (λ, y) ∈ Graph(T ) satisfies denote the set of optimal dual solutions. The following theorem ⟨y − y kj +1 , λ − λkj +1 ⟩ ≥ 0. demonstrates that the sequence of multiplier converges weakly to a point in Ω⋆ . Because y kj +1 → 0 and λkj +1 ⇀ λ̄, the inner product is continuous with respect to this mixed convergence. Taking the Theorem 1. The sequence {λk } converges weakly to an limit as j → ∞ yields: ⋆ ⋆ optimal dual solution λ ∈ Ω . k+1 ⟨y, λ − λ̄⟩ ≥ 0. (15) Proof. The dual update rule λk+1 = max 0, λk + ρk F π k+1 / T (λ̄), we could construct a new operator T ′ with is equivalent to the projection λk+1 = PH+ λk + ρk F π . If 0 ∈ ′ ′ Let NH+ (λ) = {y ∈ H : ⟨y, λ − λ⟩ ≤ 0, ∀λ ∈ H+ } be the Graph(T ′ ) = Graph(T ) ∪ {(λ̄, 0)}. From Eq. (15), T ′ is also monotone and Graph(T ) ⊊ Graph(T ′ ), which is a normal cone of H+ at λ. The projection implies contradiction with the maximality of T . Therefore, 0 ∈ T (λ̄), k+1 λk + ρk F π − λk+1 ∈ NH+ (λk+1 ). meaning λ̄ ∈ Ω⋆ . substituting Eq. (12) into Eq. (13), we obtain k+1 −d(λ) ≥ −d λk+1 + − F π , λ − λk+1 .
7
In summary, for any λ̂ ∈ Ω⋆ , {∥λk − λ̂∥} is monotonically decreasing with a lower bound, and every weak sequential cluster point of {λk } belongs to Ω⋆ . By Opial lemma [24, Theorem 5.5], {λk } weakly converges to a point in Ω⋆ . This concludes the proof. The next theorem establishes the convergence of the policy sequence {π k } and guarantees that any of its weak sequential cluster points is optimal for the original constrained problem. Theorem 2. Any weak sequential cluster point of the policy sequence {π k } is a globally optimal policy of problem (2). Proof. Because the policy space Π is a bounded subset of the reflexive Hilbert space L2 (X ), the Eberlein-Šmulian theorem guarantees the existence of a weakly convergent subsequence π kj ⇀ π̄. We first verify π̄ ∈ Π by proving that the policy space Π = {π ∈ L2 (X ) | π(x) ∈ A a.e.} is weakly closed. Since the action space A is compact and convex, Π is inherently convex. To establish strong closedness, consider any sequence {πn } ⊂ Π satisfying ∥πn − π∥ → 0. By the Riesz theorem [20, Theorem 4.9], there exists a subsequence converging to π a.e. Since A is a closed set, this pointwise limit must also reside within A a.e., ensuring π ∈ Π. Thus, Π is strongly closed. By Mazur’s lemma [20, Theorem 3.7], a strongly closed and convex subset of a Hilbert space is inherently weakly closed. Consequently, the weak convergence π kj ⇀ π̄ guarantees π̄ ∈ Π. Next, we evaluate the KKT conditions for the limit point. Given the strong continuity of F and the limit ∥λkj −1 − kj −1 −λkj λkj ∥ → 0 established in Theorem 1, it follows that λ ρkj −1 + kj
F π → F π̄ . Recall that this sequence belongs to NH+ (λkj ) and NH+ (λ) is a maximal monotone operator. Applying a maximal monotonicity argument identical to that in Theorem 1, we obtain F π̄ ∈ NH+ (λ⋆ ). Since NH+ (λ⋆ ) is a cone, it implies ρ̄F π̄ ∈ NH+ (λ⋆ ). This is equivalent to λ⋆ + ρ̄F π̄ − λ⋆ ∈ NH+ (λ⋆ ), leading to the projection λ⋆ = PH+ (λ⋆ + ρ̄F π̄ ) = max{0, λ⋆ + ρ̄F π̄ }.
This implies π̄ ∈ argminπ∈Π L(π, λ⋆ , ρ̄), i.e., 0 ∈ ∂π L(π̄, λ⋆ , ρ̄) = ∂π L(π̄, λ⋆ ). Overall, the pair (π̄, λ⋆ ) fulfills all KKT conditions. Given the convexity of problem (2), these conditions are sufficient, ensuring π̄ is indeed a globally optimal feasible policy. V. P RACTICAL IMPLEMENTATIONS In this section, we present the practical implementation of ALaM in the context of deep RL. Taking advantage of the enhanced policy exploration and training robustness offered by soft actor-critic (SAC) [25], we incorporate it as the foundational algorithm for reward maximization within the ALaM framework; the resulting algorithm is denoted as SACALaM. We set the feasibility function as the cost value function. The architecture involves training several neural networks (with subscripts denoting their learnable parameters): a policy network πθ , two Q-networks Qϕ1 , Qϕ2 , two cost Q-networks Qcψ1 , Qcψ2 , and a multiplier network λw . Correspondingly, we maintain four target Q-networks parametrized by ϕ̄1 , ϕ̄2 , ψ̄1 , and ψ̄2 . To update the policy, we aim to solve the subproblem: Z λw (x) θk+1 = argmin −J(πθ ) + λw (x) max − , Fψ (x) dx ρ θ 2 Z ρ λw (x) + max − , Fψ (x) dx, 2 ρ where Fψ (x) is the state-wise cost value of policy πθ . In the SAC framework, we replace the standard objective J(π) with the soft Q-function, and estimate the integral using empirical expectations over sampled transitions. Combining these components, the policy loss is constructed as: Lπ (θ) =Ex∼B,a∼πθ α log πθ (a|x) − min Qϕi (x, a) i∈{1,2} λw (x) +Ex∼B λw (x) max − , F̂ψ (x) ρ " 2 # ρ λw (x) +Ex∼B max − , F̂ψ (x) , 2 ρ (16)
Therefore, for any x ∈ X , if λ⋆ (x) > 0, it necessitates λ⋆ (x) = where α denotes the temperature. To prevent risk underestiλ⋆ (x) + ρ̄F π̄ (x), forcing F π̄ (x) = 0; conversely, if λ⋆ (x) = 0, mation, the cost value estimation relies on the clipped double it requires λ⋆ (x)+ρ̄F π̄ (x) ≤ 0, ensuring F π̄ (x) ≤ 0. Thus, the Q-learning trick [26]. In practice, we compute F̂ψ (x) using primal feasibility F π̄ (x) ≤ 0, the dual feasibility λ⋆ (x) ≥ 0, Monte Carlo sampling with K actions drawn from the current policy: and the complementary slackness λ⋆ (x)F π̄ (x) = 0 hold. K Finally, we establish the optimality condition for π̄. 1 X F̂ψ (x) = max Qcψi (x, ak ), where ak ∼ πθ (·|x). At any iteration kj , we have L(π kj , λkj −1 , ρkj −1 ) ≤ K i∈{1,2} k=1 L(π, λkj −1 , ρkj −1 ) for all π ∈ Π. Because −J(π) and F π The loss functions for the critic networks are the same as are convex and strongly continuous, L(π, λ, ρ) is convex and strongly continuous, and thus l.s.c. with respect to π in the those in SAC: LQ (ϕi ) = E(x,a,r,x′ )∼B (yϕ − Qϕi (x, a))2 , weak topology [20, Corollary 3.9]. Since the inner product operator is continuous under weak convergence, together with π kj π̄ k ⋆ ⋆ ′ ′ ′ ′ F → F and λ ⇀ λ ∈ Ω , L(π, λ, ρ) is l.s.c. with yϕ = r(x, a) + γ min Qϕ̄i (x , a ) − α log πθ (a |x ) , i∈{1,2} respect to the joint sequence. Therefore, LQc (ψi ) = E(x,a,c,x′ )∼B (yψ − Qcψi (x, a))2 , L(π̄, λ⋆ , ρ̄) ≤ lim inf L(π kj , λkj −1 , ρkj −1 ) j→∞
≤ lim L(π, λkj −1 , ρkj −1 ) = L(π, λ⋆ , ρ̄). j→∞
yψ = c(x) + γ max Qcψ̄i (x′ , a′ ), i∈{1,2}
8
Algorithm 1 SAC-ALaM 1: Input θ, ϕ1,2 , ψ1,2 , ϕ̄1,2 , ψ̄1,2 , w. 2: for each iteration do 3: for each environment step do
Sample at ∼ πθ (·|xt ), xt+1 = f (xt , at ) B ← B ∪ {(xt , at , xt+1 , rt , ct )} end for for each gradient step do ϕi ← ϕi − βϕ ∇LQ (ϕi ), i = 1, 2 ψi ← ψi − βψ ∇LQc (ψi ), i = 1, 2 10: θ ← θ − βθ ∇Lπ (θ) 11: α ← α − βα ∇L(α) 12: if step mod mλ == 0 then 13: for each dual gradient step do 14: w ← w − βw ∇Lλ (w) 15: end for 16: if v > 1/ρ then 17: ρ ← min{σρ, ρmax } 18: end if 19: end if 20: ϕ̄i ← τ ϕi + (1 − τ )ϕ̄i , i = 1, 2 21: ψ̄i ← τ ψi + (1 − τ )ψ̄i , i = 1, 2 22: end for 23: end for 24: Output θ, ϕ1,2 , ψ1,2 , ϕ̄1,2 , ψ̄1,2 , w. 4: 5: 6: 7: 8: 9:
where the target network parameters (ϕ̄i , ψ̄i ) are updated via exponential moving averages. The temperature α is automatically tuned in a similar manner to SAC: L(α) = Ex∼B,a∼πθ [α(− log πθ (a|x) − H)], where H is the target entropy. The objective for updating the multiplier network aligns with Eq. (9), which can be approximated by Lλ (w) = Ex∼B [(λw (x) − sg(max{λw (x) + ρF̂ψ (x), 0}))2 ], where sg(·) denotes the stop-gradient operator. To ensure a smooth and precise approximation, we perform multiple smallstep gradient updates on this objective per cycle. Given that the multiplier updates rely on the policy being near convergence, we employ a two-timescale learning scheme. By introducing a delay interval mλ , the multiplier network is updated only every mλ policy steps. Following each dual update, we evaluate the average constraint violation metric v " u 2 # u λ (x) w v = tEx∼B max F̂ψ (x), − . ρ If v > ρ1 , we increase the penalty factor via ρ ← min{σρ, ρmax }; otherwise, ρ remains unchanged. The pseudocode for SAC-ALaM is summarized in Algorithm 1. VI. E XPERIMENTS In this section, we empirically evaluate SAC-ALaM to demonstrate its efficacy and competitive advantage in safe RL. Our evaluation highlights three core strengths:
SAC-ALaM achieves high returns while satisfying safety constraints on standard safe RL benchmark, outperforming state-of-the-art baselines. • SAC-ALaM effectively mitigates training oscillations, enabling stable convergence to the optimal feasible policies. • The multiplier network learns a well-calibrated risk representation, which provides interpretable safety feedback and demonstrates robust generalization in novel situations.
•
As illustrated in Figure 1, we evaluate ALaM across a diverse suite of environments from the Safety-Gymnasium benchmark [27]. We test multiple agents, ranging from Point and Car featuring low-dimensional state spaces for 2D kinematics and steering, to Swimmer, HalfCheetah, and Ant that introduce high-dimensional continuous control, demanding the agent to simultaneously achieve complex joint coordination, dynamic balance, and hazard avoidance. These agents are assigned to perform various tasks: moving rapidly without exceeding speed limits (Velocity), navigating to the green target while avoiding blue obstacles (Goal), moving as fast as possible along the green circular boundary without leaving the yellow bounded regions (Circle), and reaching and pressing orange buttons while evading hazards (Button). Furthermore, we examine two levels of safety difficulty (Level 1 and Level 2) for some scenarios. By combining these different agents, tasks, and constraint difficulties, we conduct a comprehensive evaluation of SAC-ALaM. TABLE I. Hyperparameters Hyperparameter Optimizer Discount factor (γ) Hidden layers Hidden neurons per layer Activation function Learning rate Learning rate for multiplier Total environment steps Replay buffer size Batch size Target entropy (H) Multiplier update interval (mλ ) Monte Carlo samples (K) Dual gradient steps Maximum penalty (ρmax ) Penalty scaling factor (σ)
Value RAD [28] 0.99 2 256 ReLU 1 × 10−4 1 × 10−5 4 × 106 2 × 106 256 − dim(A) 200 5 5 5.0 1.01
We compare SAC-ALaM against a comprehensive set of existing safe RL algorithms. The primary baselines are Lagrangian-based methods, including the scalar multiplier approach (SAC-Lag) [4] and the parameterized multiplier network (FAC [12], denoted as SAC-LagNet for notational consistency). To evaluate stabilization mechanisms, we include the PID Lagrangian method (SAC-PID) [13] and the augmented Lagrangian method (ASAC), adapted from APPO [16] to use SAC. By employing SAC as the shared backbone across all Lagrangian methods, we standardize the return optimization framework, isolating the performance impact of their respective safety mechanisms. Furthermore, we evaluate SAC-FPI [8], a
9
(a) Velocity
(b) Goal
(c) Circle
(d) Button
Fig. 1. Snapshots of four experimental task scenarios.
P3O CRPO SAC-Lag ASAC SAC-FPI SAC-PID SAC-LagNet SAC-ALaM
1.0
Normalized Return
0.8
0.6
0.4
0.2
0.0 0.8
0.6
0.4 Normalized Cost
0.2
0.0
Fig. 2. Comparison of normalized performance across 8 tasks. The data points and error bars denote the mean and 95% confidence intervals for episode returns and costs.
dynamic programming-based approach applying region-wise policy updates, together with two leading on-policy safe RL algorithms P3O [29] and CRPO [30]. To prevent overly conservative policies, we relax the strict safety constraint to F π (x) ≤ d, where d is a tunable tolerance threshold with a default value of 0.1. All hyperparameter settings are detailed in Table I. Figure 3 illustrates the learning curves for episode return and episode cost across the 8 evaluated tasks. SAC-ALaM demonstrates a distinct advantage in balancing reward maximization and constraint satisfaction. In contrast, SAC-Lag and SAC-LagNet suffer from pronounced training oscillations. Although ASAC and SAC-PID partially mitigate these fluctuations, this stability often comes at the expense of final returns. Furthermore, the on-policy baselines struggle with constraint satisfaction, even in relatively simple tasks such as PointGoal1 and CarGoal1. As environmental complexity increases (e.g., in Button environments), the baselines either suffer from severe constraint violations (SAC-Lag, P3O, CRPO) or exhibit excessive conservatism, thereby compromising final performance (SAC-FPI, SAC-PID, SAC-LagNet). SAC-ALaM, however, consistently learns safe and high-performing policies regardless of task difficulty. Figure 2 compares the normalized performance across all environments. To evaluate asymptotic
performance, we average the final 10% of training steps, apply per-task Min-Max normalization, and compute the crossenvironment means with 95% confidence intervals. With the x-axis inverted (where further right indicates lower costs), SAC-ALaM firmly occupies the optimal region in the topright corner with significantly narrower confidence intervals. This confirms that SAC-ALaM not only achieves the highest overall normalized performance but also demonstrates strong consistency across diverse tasks. To highlight SAC-ALaM’s ability to mitigate the training oscillations inherent to SAC-LagNet, Figure 4 visualizes their individual training dynamics. Specifically, SAC-LagNet relies on standard Lagrangian formulation and updates its multiplier network via conventional dual gradient ascent (3). In contrast, SAC-ALaM uses augmented Lagrangian with a supervised regression-based multiplier update. As observed, while SACLagNet suffers from severe fluctuations that ultimately degrade the final policy return, SAC-ALaM maintains a highly stable optimization process. It incurs fewer constraint violations during training and converges to a safe and effective policy. To demonstrate the advantage of dual supervised regression, Figure 5 presents an ablation study comparing SAC-ALaM against SAC-ALaM-GA, a variant utilizing traditional dual gradient ascent. As illustrated, SAC-ALaM-GA suffers from more training oscillations. This discrepancy arises because standard gradient ascent merely adjusts parameters based on local constraint violations, failing to approximate the theoretical dual target. This violates the theoretical convergence requirements of augmented Lagrangian methods. Furthermore, owing to the neural network’s inherent generalization, improper local gradient updates inevitably corrupt the multiplier’s estimations in neighboring regions, ultimately degrading the agent’s overall performance. Conversely, by explicitly fitting the theoretical target, SAC-ALaM consistently outperforms its gradient ascent counterpart in both task efficacy and safety adherence. Figure 6 visualizes the multiplier functions to illustrate risk identification capabilities. In the Goal task, the agent must reach the green target while avoiding blue obstacles. SACALaM effectively distinguishes hazards from safe zones. As velocity increases, multipliers inside the obstacles decrease due to shorter departure time, while these values in the surrounding areas increase for extended braking distance. In contrast, SACLagNet struggles to clearly separate safe from dangerous regions, collapsing into a uniform high-risk estimation at high speeds. Notably, despite extreme velocities being rarely
10
0.4
100
0.2
80
PointCircle2
4.0 1e6
40
3.8
4.0 1e6
40
20
5
0
0
1.0
1.5
2.0
2.5
3.0
3.5
Environment step
4.0 1e6
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Environment step
SwimmerVelocity
4.0 1e6
24
CarGoal1 0.4 0.2 0.0
3.8
4.0 1e6
3.8
4.0 1e6
12
0 0.5
1.0
1.5
2.0
2.5
3.0
3.5
Environment step
4.0 1e6
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Environment step
PointGoal1
30
45
18
6
0.0
PointCircle2
45
0.5
10
0 0.5
30
15
20
0.0
1.0
0.0
60
Episode cost
0 3.8
Episode cost
Episode cost
20
2
0.0
60
PointGoal1
25
4
Episode cost
SwimmerVelocity 80
4.0 1e6
CarGoal1
24
0
30 43
15
42
18
Episode return
Episode return
Episode return
15
Episode return
30 30
12 6
15 36 0
41 40 3.8 1.0
1.5
2.0
2.5
3.0
3.5
Environment step
4.0 1e6
0.0
1.0
1.5
2.0
4.0 1e6
0.0
3.8
4.0 1e6
16
8
0 1.0
1.5
2.0
2.5
3.0
3.5
Environment step
4.0 1e6
15
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Environment step
3.8 1.0
1.5
2.0
2.5
4.0 1e6
3.0
3.5
8
Environment step
P3O
120
1.0
1.5
2.0
2.5
3.0
3.5
1.5
2.0
2.5
3.0
3.5
Environment step
CRPO
SAC-Lag
ASAC
4.0 1e6
2
3.8
45
4.0 1e6
30
4.0 1e6
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0 1e6
3.0
3.5
4.0 1e6
Environment step PointButton2 24
10 5
18
0
4.0 1e6
4.0 1e6 3.5
4
CarCircle2
17
0.0
0.5
1.0
1.5
2.0
2.5
3.0
8 0
4.0 1e6 3.5
Environment step
SAC-FPI
16
−8
16
−10 1.0
3.0
0 0.5
3.8 −8
2.5
15
−5
0.5
2.0
0 4.0 1e6
15
0.0
1.5
60 3.8
20
16
4.0 1e6
1.0
PointButton2
Environment step
0
0.5
0.5
Environment step
2.5
0.0
2900 −3000
3.8 0.0
75
180
4.0 1e6
Episode return
Episode return
3000
4.0 1e6
0 0.0
−1500
3.5
60
24
3100
3.0
0.0
PointButton1
1500
2.5
CarCircle2
240
3.8
4.0 1e6
3000
0.0
2.0
5.0
30
HalfCheetahVelocity
0
1.5
300
0 0.5
1.0
0
24
0.0
0.5
Environment step
2
45
Episode cost
Episode cost
3.5
4
0.05 0.00
Episode return
3.0
PointButton1
60
0.10
32
2.5
Environment step
HalfCheetahVelocity
40
0.5
Episode return
0.5
32 −15
Episode cost
0.0
34
0
4.0 1e6
Episode cost
0
−15
SAC-PID
4.0 1e6
0.0
0.5
1.0
1.5
2.0
2.5
Environment step
SAC-LagNet
SAC-ALaM
Fig. 3. Learning curves of SAC-ALaM and baselines across 8 environments. The solid lines represent the average performance over 5 seeds, and the shaded areas indicate the 95% confidence intervals. The first row shows the average episode cost, and the second row shows the average episode return.
160
80
30
20
10
SwimmerVelocity
60
SAC-LagNet SAC-ALaM
45
320
Episode cost
240
SwimmerVelocity
400
40
Episode return
320
Episode cost
PointCircle2
50
SAC-LagNet SAC-ALaM
Episode return
PointCircle2
400
240
160
30 15 0
80 −15
0
0
0 −30
0.0
0.5
1.0
1.5
2.0
2.5
Environment step
3.0
3.5
4.0 1e6
0.0
0.5
1.0
1.5
2.0
2.5
Environment step
3.0
3.5
4.0 1e6
0.0
0.5
1.0
1.5
2.0
2.5
Environment step
3.0
3.5
4.0 1e6
0.0
0.5
1.0
Fig. 4. Training stability comparison in the PointCircle2 and SwimmerVelocity tasks.
1.5
2.0
2.5
Environment step
3.0
3.5
4.0 1e6
11
AntVelocity
30
PointButton1
40
SAC-ALaM SAC-ALaM-GA
24
PointGoal1
20
SAC-ALaM SAC-ALaM-GA
32
4
SAC-ALaM SAC-ALaM-GA
2
16
12
Episode cost
Episode cost
Episode cost
0 18
24
16
8
4
0
0
0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Environment step
4.0 1e6
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
Environment step
AntVelocity
4.0 1e6
1600
800
0.5
1.0
1.5
2.0
2.5
Environment step
3.0
3.5
4.0 1e6
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0 1e6
PointGoal1
30 24
8 0 −8
0
4.0 1e6
Environment step
Episode return
2400
16
Episode return
24
0.0
0.0
PointButton1
3200
3.8
8
6
0.0
Episode return
12
0.0
0.5
1.0
1.5
2.0
2.5
Environment step
3.0
3.5
4.0 1e6
18 12
26
6
24
0
22 3.8 0.0
0.5
1.0
1.5
2.0
2.5
Environment step
3.0
4.0 1e6 3.5
4.0 1e6
Fig. 5. Ablation of multiplier update. Solid lines and shaded regions denote the mean and 95% confidence intervals across 5 seeds. The first row shows the average episode cost, and the second row shows the average episode return.
explored during training, SAC-ALaM successfully provides accurate safety feedback. In the Circle task, the agent must navigate a green circular path within the grey boundary. SACALaM’s multiplier exhibits spatial symmetry when stationary, aligning with the underlying physical dynamics. Upon introducing eastward velocities, the network shifts high-risk estimations to the right-hand regions. Specifically, the high multiplier values are concentrated in the lower-right quadrant, where eastward momentum and counter-clockwise steering elevate collision probability; as the agent decelerates in the top-right area, the corresponding risk decreases. SAC-LagNet fails to capture these velocity-dependent risks. Overall, these visualizations confirm that the SAC-ALaM trains a physically consistent and robustly generalizable multiplier model for safety monitoring. VII. C ONCLUSION In this paper, we propose the ALaM framework for stable learning of state-wise multipliers in safe RL. By introducing an augmented Lagrangian with supervised regression to update the multiplier, ALaM stabilizes the training dynamics, with provable sequential convergence to an optimal policy. We further develop a practical safe RL algorithm, SAC-ALaM, by combining ALaM with soft actor-critic. Empirically, SACALaM yields high-performing, safety-constrained policies while providing a well-calibrated multiplier for risk assessment. R EFERENCES [1] S. E. Li, Reinforcement Learning for Sequential Decision and Optimal Control. Springer Verlag, Singapore, 2023.
[2] A. Gupta, S. Savarese, S. Ganguli, and F.-F. Li, “Embodied intelligence via learning and evolution,” Nature Communications, vol. 12, no. 1, p. 5721, 2021. [3] R. Rafailov, A. Sharma, E. Mitchell, C. D. Manning, S. Ermon, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in Advances in Neural Information Processing Systems, vol. 36, 2023, pp. 53 728–53 741. [4] A. Ray, J. Achiam, and D. Amodei, “Benchmarking safe exploration in deep reinforcement learning,” arXiv preprint arXiv:1910.01708, 2019. [5] H. Ma, J. Chen, S. E. Li, Z. Lin, Y. Guan, Y. Ren, and S. Zheng, “Modelbased constrained reinforcement learning using generalized control barrier function,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, 2021, pp. 4552–4559. [6] M. Ganai, Z. Gong, C. Yu, S. Herbert, and S. Gao, “Iterative reachability estimation for safe reinforcement learning,” in Advances in Neural Information Processing Systems, vol. 36, 2023, pp. 69 764–69 797. [7] D. Yu, W. Zou, Y. Yang, H. Ma, S. E. Li, Y. Yin, J. Chen, and J. Duan, “Safe model-based reinforcement learning with an uncertainty-aware reachability certificate,” IEEE Transactions on Automation Science and Engineering, vol. 21, no. 3, pp. 4129–4142, 2023. [8] Y. Yang, Z. Zheng, S. E. Li, W. Xu, J. Liu, X. Zhan, and Y.-Q. Zhang, “Feasible policy iteration for safe reinforcement learning,” arXiv preprint arXiv:2304.08845, 2025. [9] S. Paternain, L. Chamon, M. Calvo-Fullana, and A. Ribeiro, “Constrained reinforcement learning has zero duality gap,” in Advances in Neural Information Processing Systems, vol. 32, 2019, pp. 7555–7565. [10] C. Tessler, D. J. Mankowitz, and S. Mannor, “Reward constrained policy optimization,” in International Conference on Learning Representations, 2019, pp. 1–15. [11] Y. Chow, M. Ghavamzadeh, L. Janson, and M. Pavone, “Risk-constrained reinforcement learning with percentile risk criteria,” Journal of Machine Learning Research, vol. 18, no. 167, pp. 1–51, 2018. [12] H. Ma, Y. Guan, S. E. Li, X. Zhang, S. Zheng, and J. Chen, “Feasible actor-critic: Constrained reinforcement learning for ensuring statewise safety,” arXiv preprint arXiv:2105.10682, 2021. [13] A. Stooke, J. Achiam, and P. Abbeel, “Responsive safety in reinforcement learning by PID Lagrangian methods,” in Proceedings of the 37th International Conference on Machine Learning, vol. 119, 2020, pp. 9133–9143. [14] B. Peng, J. Duan, J. Chen, S. E. Li, G. Xie, C. Zhang, Y. Guan, Y. Mu,
12
CarCircle2
CarGoal1 SAC-ALaM (Vel: 0.0 m/s)
3
SAC-ALaM (Vel: 0.4 m/s, East)
SAC-ALaM (Vel: 1.5 m/s, East) 9.57
0
0.12 0.10
6.56
1
0.09
5.56
0
0.08 0.06
1
3.56
0.05
2
0.04
2.56
2
3
1.56 3
0.13
2
4.56 1
3
2
1
0
X
1
2
3
3
2
1
Goal
0
X
1
2
3
Hazard
3
2
1
0
X
1
2
3
4
0.56
0.02 4
3
2
1
0
X
1
2
3
4
4
3
2
SAC-LagNet (Vel: 0.0 m/s)
3
0
X
1
2
3
4
4
3
2
1
0
X
1
2
3
4
SAC-LagNet (Vel: 0.4 m/s, East)
SAC-LagNet (Vel: 1.5 m/s, East)
Boundary
0.33 0.31
2
0.30
6.94
1
0.29
6.16
0
0.28
5.38 1
SAC-LagNet (Vel: 0.25 m/s, East)
7.71
4.61
Y
0
SAC-LagNet (Vel: 0.05 m/s, East)
3
8.49
1
SAC-LagNet (Vel: 0.0 m/s)
4
9.26
0.27
1
0.26
2
0.25
3.83
2
3.06 3
2
1
0
X
1
2
3
3
Goal
2
1
0
X
1
Hazard
0.01
CarCircle2
2
Y
1
Target Circle
Vase
CarGoal1
3
SAC-ALaM (Vel: 0.25 m/s, East)
7.56
Y
Y
1
SAC-ALaM (Vel: 0.05 m/s, East)
3
8.57
2
SAC-ALaM (Vel: 0.0 m/s)
4
2
3
3
2
1
0
X
1
2
3
2.28
Vase
3 4
0.24 4
3
2
1
0
X
1
2
3
4
4
3
2
1
0
X
Target Circle
1
2
3
4
4
3
2
1
0
X
1
2
3
4
0.23
Boundary
Fig. 6. Heatmaps of the multiplier for SAC-ALaM and SAC-LagNet across varying agent velocities.
and E. Sun, “Model-based chance-constrained reinforcement learning via separated proportional-integral Lagrangian,” IEEE Transactions on Neural Networks and Learning Systems, vol. 35, no. 1, pp. 466–478, 2022. [15] L. Spoor, Á. Serra-Gómez, A. Plaat, and T. M. Moerland, “An empirical study of lagrangian methods in safe reinforcement learning,” in Differentiable Systems and Scientific Machine Learning Workshop, EurIPS, 2025, pp. 1–17. [16] J. Dai, J. Ji, L. Yang, Q. Zheng, and G. Pan, “Augmented proximal policy optimization for safe reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 6, 2023, pp. 7288–7295. [17] Y. Yang, Y. Zhang, W. Zou, J. Chen, Y. Yin, and S. E. Li, “Synthesizing control barrier functions with feasible region iteration for safe reinforcement learning,” IEEE Transactions on Automatic Control, vol. 69, no. 4, pp. 2713–2720, 2023. [18] Y. Yang, Z. Zheng, M. Tomizuka, C. Liu, and S. E. Li, “The feasibility of constrained reinforcement learning algorithms: A tutorial study,” Foundations and Trends in Systems and Control, vol. 13, no. 1, pp. 1–72, 2026. [19] D. P. Bertsekas, Constrained Optimization and Lagrange Multiplier Methods. Academic Press, 2014. [20] E. Zeidler, Nonlinear Functional Analysis and its Applications: II/B: Nonlinear Monotone Operators. Springer Science & Business Media, 2013. [21] R. Rockafellar, “On the maximal monotonicity of subdifferential mappings,” Pacific Journal of Mathematics, vol. 33, no. 1, pp. 209–216, 1970. [22] H. H. Bauschke and P. L. Combettes, Convex Analysis and Monotone Operator Theory in Hilbert Spaces, 2nd ed. Springer, 2017. [23] H. Brezis, Functional Analysis, Sobolev Spaces and Partial Differential Equations. Springer, 2011. [24] H. H. Bauschke and P. L. Combettes, “Correction to: Convex analysis and monotone operator theory in Hilbert spaces,” in Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer, 2020, pp. C1–C4. [25] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor,” in Proceedings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 1861–1870. [26] S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approximation error in actor-critic methods,” in Proceedings of the 35th International Conference on Machine Learning, vol. 80, 2018, pp. 1587–1596. [27] J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y. Geng, Y. Zhong, J. Dai, and Y. Yang, “Safety Gymnasium: A unified safe reinforcement learning benchmark,” in Advances in Neural Information Processing Systems, vol. 36, 2023, pp. 18 964–18 993.
[28] Y. Lyu, X. Zhang, S. E. Li, J. Duan, L. Tao, Q. Xu, L. He, and K. Li, “Conformal symplectic optimization for stable reinforcement learning,” IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 6, pp. 11 049–11 063, 2025. [29] L. Zhang, L. Shen, L. Yang, S. Chen, X. Wang, B. Yuan, and D. Tao, “Penalized proximal policy optimization for safe reinforcement learning,” in Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, 2022, pp. 3744–3750. [30] T. Xu, Y. Liang, and G. Lan, “CRPO: A new approach for safe reinforcement learning with convergence guarantee,” in Proceedings of the 38th International Conference on Machine Learning, vol. 139, 2021, pp. 11 480–11 491.
Jiaming Zhang Jiaming Zhang received her B.S. degree in mathematics and applied mathematics from the School of Mathematics, Shandong University, Jinan, China in 2024. She is currently pursuing her Ph.D. degree in the Department of Mathematical Sciences, Tsinghua University, Beijing, China. Her research interests include continuous optimization and safe reinforcement learning.
Yujie Yang Yujie Yang received his B.S. degree in automotive engineering from the School of Vehicle and Mobility, Tsinghua University, Beijing, China in 2021. He is currently pursuing his Ph.D. degree in the School of Vehicle and Mobility, Tsinghua University, Beijing, China. His research interests include safe reinforcement learning and decision and control of autonomous vehicles.
13
Yao Lyu Yao Lyu received his B.Eng. degree in 2019 and his Ph.D. degree in 2025 from Tsinghua University, where he currently serves as a Postdoctoral Researcher in the School of Vehicle and Mobility. His active research interests include end-to-end autonomous driving, embodied artificial intelligence, deep reinforcement learning, and neural network optimization. He has authored over 20 peerreviewed publications in top-tier venues and was awarded the CVCI 2023 Best Paper Award. Dr. Lyu actively contributes to the academic community, serving as a reviewer for IEEE TNNLS, IEEE Cyber, IEEE TITS, NeurIPS, ACC, CDC, etc.
Shengbo Eben Li Shengbo Eben Li (Senior Member, IEEE) received his M.S. and Ph.D. degrees from Tsinghua University in 2006 and 2009. He has worked at Stanford University, University of Michigan, and UC Berkeley. He is now a professor at Tsinghua University, working on intelligent vehicles and driver assistance, embodied intelligence for robotics, deep reinforcement learning, optimal control and estimation, etc. He is the author of over 250 peerreviewed journal/conference papers, and co-inventor of over 40 patents. Dr. Li has received over 20 prestigious awards, including Youth Sci. & Tech Award of Ministry of Education (annually 10 receivers in China), Natural Science Award of Chinese Association of Automation (First level), National Award for Progress in Sci & Tech of China, and best (student) paper awards or finalists of IEEE ITSC, IEEE IVS, IET ITS, ICCAS, IFAC MECC, CAA CVCI, IEEE ICUS, CCCC, IEEE ITSM, L4DC, Automotive Innovation, etc. He was a member of Board Governor of IEEE ITS Society. He serves as the director of Technical Committee on AI of SAE-China, deputy director of Technical Committee on Vehicle Control and Intelligence of CAA, and the leader of AI working group in China Industry Innovation Alliance for ICVs.He also serves as Senior AE of IEEE OJ ITS, AEs of IEEE ITSM, IEEE TITS, IEEE TNNLS, IEEE TCST, and area chairs of ICLR and ICML, etc.
Liping Zhang Liping Zhang is currently a tenured Professor in Department of Mathematical Sciences, Tsinghua University. She received her Ph.D. degree from the Academy of Mathematics and Systems Sciences, Chinese Academy of Sciences in 2001. Her research interests include continuous optimization, tensor analysis and computation, machine learning. She has published more than 70 research papers in international journals such as Mathematical Programming, SIAM Journal on Optimization, Mathematics of Computation, Mathematics of Operational Research, SIAM Journal on Matrix Analysis and Applications, Journal of Machine Learning Research, Expert Systems with Applications, etc.