arXiv CSopen access

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Ayoub Belouadah 1 Sylvain Kubler 1 Yves Le Traon 1

arXiv:2606.14415v1 [cs.AI] 12 Jun 2026

Abstract

this, typically formalizing the problem using the CMDP framework (Altman, 2021), where an agent must ensure that the expected cumulative cost remains below a prescribed limit, restricting the set of feasible policies.

Safe reinforcement learning (Safe RL) aims to maximize expected return while satisfying safety constraints, typically modeled as Constrained Markov Decision Processes (CMDPs). While primal-dual methods scale well to deep RL, they often suffer from delayed constraint correction, leading to oscillatory behavior and prolonged safety violations. In this paper, we propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates. CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety, compensating for delayed Lagrange multiplier updates, reducing oscillations near the boundary, and preserving the KKT solutions of the original constrained problem. Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns compared to state-of-the-art primal-dual and penalty-based methods.

A major line of work in safe RL addresses CMDP using second-order trust-region methods (Achiam et al., 2017; Yang et al., 2020; Milosevic et al., 2025), which approximate the constrained objective via sequential quadratic programs, while providing strong theoretical guarentees, they require costly Fisher matrix inversions. Empirical studies show that simpler first-order methods can perform comparably or even better on continuous control benchmarks (Ray et al., 2019). This has motivated primal–dual Lagrangian methods (Chow et al., 2018; Tessler et al., 2019), which jointly update policies and multipliers using first-order gradients, as well as penalty methods (Zhang et al., 2022; Liu et al., 2020) that augment the reward objective with a cost penalty. However, both exhibit limitations. First, penalty methods require carefully tuned (often large) penalty factors; otherwise, they yield either overly conservative policies or persistent constraint violations. Second, primal-dual methods suffer from a well-known dual-lag effect (Tessler et al., 2019; Paternain et al., 2019), leading to unsafe behavior when multipliers are small and excessive conservatism when they grow large, thus exhibiting oscillatory learning dynamics (Platt & Barr, 1987). Moreover, safety recovery requires accounting for the local sensitivity and steepness of the constraint function. Flat regions may require stronger corrective updates to return to feasibility, whereas steep regions call for more cautious steps to avoid overshooting. Yet, existing methods fail to exploit this structure, penalizing constraint violations uniformly regardless of local constraint steepness. As illustrated in Figure 1 (‘Primal-dual’) ignoring constraint sensitivity near steep boundaries can cause overshooting and oscillatory behavior, resulting in instability and inefficient safety recovery.

1. Introduction RL has shown remarkable success in high-dimensional decision-making problems, including robotic locomotion, autonomous driving, and complex game playing (Wang et al., 2022). A major limitation of standard RL is that it optimizes expected return without explicitly considering safety constraints, which is unacceptable in real-world applications where constraint violations can result in physical damage or unsafe behavior. Safe RL seeks to overcome

In this paper, we introduce Constraint-Sensitive Policy Optimization (CSPO), a first-order safe RL algorithm that retains the simplicity of primal–dual methods while explicitly exploiting local first-order constraint information. CSPO augments the primal update with a constraint correction that is applied only when violations occur. By scaling this correc-

1

SnT, University of Luxembourg. Correspondence to: Ayoub Belouadah <[email protected]>, Sylvain Kubler <[email protected]>, Yves Le Traon <[email protected]>. Proceedings of the 43 rd International Conference on Machine Learning, Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s).

1

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Steep constraint surface

tion using the norm of the constraint gradient, CSPO adapts the strength of constraint enforcement to the local constraint sensitivity, reducing delayed constraint correction and enabling faster, smoother returns to feasibility (see the CSPO dynamics in Figure 1). Importantly, CSPO remains equivalent to the original constrained problem in terms of both the feasible set and optimal solutions. In addition, we introduce three complementary metrics to evaluate safety recovery: Time-To-Safety (TTS), measuring how quickly feasibility is restored, Reward Preservation (RP), quantifying reward retention during recovery, and Violation frequency (VF) capturing how often safety constraints are violated throughout training. Our contributions are:

( largek∇g(θ)k)

π0

πsafe Primal-dual CSPO

• We propose CSPO, a first-order primal–dual safe RL algorithm that incorporates local constraint sensitivity to accelerate recovery from constraint violations.

Figure 1. Geometric intuition for CSPO on a toy constrained optimization problem. Dashed curves denote objective level sets f (θ), dotted curves denote constraint level sets g(θ), the purple curve is the boundary g(θ) = 0, and the shaded region is infeasible. Starting from an infeasible iterate π0 , standard Lagrangian updates may overshoot and oscillate in highly sensitive regions, whereas CSPO scales the constraint correction by |∇g(θ)|, enabling stable recovery to feasibility with reduced reward degradation.

• We demonstrate that CSPO preserves the feasible set and optimal solutions of the original constrained problem. • We introduce three safety metrics (TTS, RP, VF) to evaluate safety recovery dynamics.

transitions to instantaneous costs. For a stationary policy π, the discounted cost return for ci is:

• Empirically, CSPO demonstrates faster and more stable safety convergence with strong performance on continuous-control benchmarks.

Jci (π) = Eτ ∼π

"∞ X

# t

γ ci (st , at , st+1 ) .

(2)

t=0

2. Background

Given cost limits d1 , . . . , dm , the feasible policy set is Πsafe = {π ∈ Π : Jci (π) ≤ di ∀i}.

2.1. Markov Decision Processes We consider a discounted MDP M = (S, A, r, P, µ, γ), where S is the state space, A is the action space, r : S ×A× S → R is the reward function, P (s′ | s, a) is the transition kernel, µ is the initial state distribution, and γ ∈ [0, 1) is the discount factor. A stationary policy π is a distribution over actions given states, π(a | s), and induces a trajectory τ = (s0 , a0 , s1 , a1 , . . . ) with s0 ∼ µ, at ∼ π(· | st ), st+1 ∼ P (· | st , at ). The discounted return of π is " ∞ # X t J(π) = Eτ ∼π γ rt . (1)

(3)

Hence, safe RL aims to find an optimal feasible policy. π ⋆ = arg max J(π). π∈Πsafe

(4)

i.e., maximize return subject to the cost constraints. We similarly define Vcπi , Qπci , and Aπci as the value, state-action value, and advantage functions associated with P the cost func∞ tion ci defined respectively asP Vcπi (s) = Eτ ∼π [ t=0 γ t cit | ∞ π t s0 = s], Qci (s, a) = Eτ ∼π [ t=0 γ cit | s0 = s, a0 = a], and Aπci (s, a) = Qπci (s, a) − Vcπi (s).

We denote by V π(s) , Qπ (s, a), and Aπ (s, a) the value, state-action value, and advantage P functions of π, defined ∞ t respectively as V π (s) = E [ P∞ t τ ∼π t=0 γ rt | s0 = s], π Q (s, a) = Eτ ∼π [ t=0 γ rt | s0 = s, a0 = a], and Aπ (s, a) = Qπ (s, a) − V π (s).

2.3. Constrained Policy Optimization P∞ Let dπ (s) = (1 − γ) t=0 γ t Pr(st = s | π) denote the normalized discounted state-visitation distribution of policy π. For any bounded function f : S × A × S → R, let Jf (π) denote the corresponding discounted return, and Aπf its advantage function.

2.2. Constrained Markov Decision Processes (CMDP)

The performance difference lemma (Kakade & Langford, 2002) states that, for any two policies π and π ′ ,

t=0

A CMDP (Altman, 2021) augments a MDP with cost functions {ci }m i=1 , where each ci : S × A × S → R+ maps

Jf (π ′ ) − Jf (π) = 2

  1 Es∼dπ′ , a∼π′ Aπf (s, a) . 1−γ

(5)

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Applying (5) to the reward r and each cost ci allows rewriting the safe RL objective (4) as an iterative policy search. For parametrized stochastic policies πθ (a | s) with θ ∈ Rd and a current policy πθk , the updated policy πθk+1 is obtained by maximizing the reward advantage while satisfying the cost constraints:  πθ  πθk+1 = arg max Es∼dπθ , a∼πθ Ar k (s, a) (6)

However, exactly targeting g(θk+1 ) = 0 in one update can be overly aggressive under stochastic gradient estimates. Instead, we enforce a fractional decrease of the violation, g(θk+1 ) ≤ σ g(θk ), σ ∈ (0, 1], which corresponds to targeting the boundary of the scaled linearized constraint g(θk )+∇θ g(θk )⊤ ∆θ = σg(θk ). Solving the same minimal update problem with this target yields the update magnitude:

πθ

s.t.

Jci (πθk ) +

 πθ  1 Es∼dπθ , a∼πθ Aci k (s, a) ≤ di , 1−γ

∆θ⋆ = −α

3. Constraint-Sensitive Policy Optimization

The shortest distance formulation in (11) suggests that the magnitude required to reduce a given violation g(θk ) depends inversely on the local first-order geometry of the constraint, as captured by ∥∇θ g(θk )∥. In particular, large gradient norms correspond to steep constraint surfaces, where overly strong corrections may lead to overshooting and oscillations near the boundary. Conversely, small gradient norms correspond to flat surfaces, where stronger corrective updates are needed for efficient feasibility recovery. As illustrated in Figure 1, accounting for local constraint sensitivity prevents over-correction near steep boundaries while enabling stable recovery toward feasibility aligned with reward level sets. Rather than applying (11) as a direct one-step update, we encode this geometry into a smooth differentiable objective that minimizes the scaled distance to the safety boundary:

By definition of the advantage function, this constraint satisfies g(θk ) = Jc (πθk ) − d, which corresponds to the constraint violation of the current policy iterate πθk . We define the feasible parameter set as Γ = {θ ∈ Rd : g(θ) ≤ 0}. 3.1. First-Order Geometry of the Constraint Consider a policy update θk+1 = θk + ∆θ. For sufficiently small, trust-region bounded updates and assuming that g is continuously differentiable with locally Lipschitz gradient, the constraint admits the second-order expansion (8)

Equivalently, if ∇g is Lg -Lipschitz in a neighborhood of θk , L then g(θk + ∆θ) − g(θk ) − ∇θ g(θk )⊤ ∆θ ≤ 2g ∥∆θ∥2 . Under trust-region bounded updates ∥∆θ∥ ≤ ε for small ε > 0, the approximation error is therefore O(ε2 ), while the first-order change scales as O(ε). Consequently, for small steps, O(∥∆θ∥2 ) is negligible.

qk (θ) =

1 ∥∆θ∥2 2

s.t.

g(θk ) + ∇θ g(θk )⊤ ∆θ = 0.

(9)

Solving (9) yields: g(θk ) ∥∇θ g(θk )∥ (10) ∇θ g(θk ) denotes the gradient of the surrogate constraint with respect to the policy parameters, evaluated at the current iterate θk . The length of this update (∥∆θ⋆ ∥) gives us the shortest signed distance from θk to the feasible set Γ. ∆θ⋆ = −

g(θk ) ∇θ g(θk ), ∥∇θ g(θk )∥2

α wk [g(θ)]2+ , 2

wk =

1 (12) ∥∇θ g(θk )∥2 + ϵ

where α is the fractional violation reduction factor from (11), and ϵ > 0 is added for numerical stability. [x]+ = max(x, 0) activates the correction only when g(θ) > 0. By construction, the gradient of qk at θk recovers exactly the parameter update prescribed by (11), turning the geometric recovery step into an objective optimizable with standard gradient-based methods, while vanishing identically at feasible points to preserve equivalence with theoriginal con πθ strained problem. Let LR (θ) = Es∼dπθ , a∼πθ Ar k (s, a) , CSPO aims to solve the new equivalent constrained problem:

During constraint violation, we seek the minimal update to return to the safety boundary. Formulated as: ∆θ

g(θk ) ∥∇θ g(θk )∥ (11)

3.2. Constraint-Sensitive Objective

i h π 1 θ g(θ) = Jc (πθk ) + Es∼dπθ , a∼πθ Ac k (s, a) − d. (7) 1−γ

min

∥∆θ⋆ ∥ = α

With α = 1 − σ ∈ [0, 1] defined as the fractional reduction factor. See Appendix A for the detailed derivations.

We consider a differentiable parametric policy class πθ : θ ∈ Rd with current iterate θk . For clarity, we consider a single-constraint case and denote the surrogate cost constraint for policy πθ as:

g(θk+1 ) = g(θk ) + ∇θ g(θk )⊤ ∆θ + O(∥∆θ∥2 ).

g(θk ) ∇θ g(θk ), ∥∇θ g(θk )∥2

∥∆θ⋆ ∥ =

πθk+1 = arg min − LR (θ) + qk (θ) πθ

s.t.

g(θ) ≤ 0. (13)

Assumption 3.1. (Strict feasibility). There exists a constant ξ > 0 and a policy π̄ such that g(θ̄) ≤ −ξ, i.e., the constraint is satisfied with nonzero margin. 3

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Under Assumption 3.1, we introduce a nonnegative multiplier λ ≥ 0, forming the corresponding Lagrangian of (13): Lk (θ, λ) = − LR (θ) + qk (θ) + λ g(θ),

This corresponds to an approximate first-order KKT point of the surrogate constrained problem (13), satisfying: (i) ∥∇θ Lk (θ, λ)∥ ≤ ε, (ii) [g(θ)]+ ≤ ε, (iii) λ ≥ 0, (iv) λ[g(θ)]+ ≤ ε, where ε is controlled by the step-size schedule and problem regularity constants. Appendix C provides the detailed derivation and verifies the required assumptions for the underlying nonconvex-concave minimax formulation.

λ ≥ 0. (14)

We restrict the dual variable to a compact domain Λ = [0, λmax ] and consider the associated min-max problem: (θk+1 , λk+1 ) ∈ arg min arg max Lk (θ, λ). θ

λ∈Λ

(15)

3.4. Extension to Multiple Constraints CSPO naturally extends to the multi-constraint CMDP setting with m cost constraints. Let gi (θ) denote the surrogate constraint corresponding to cost ci . For each constraint, we define:

We solve (15) iteratively using a primal-dual scheme, performing gradient descent on the policy parameters θ and projected gradient ascent on the multiplier λ: ( θk+1 = θk − ηθ ∇θ Lk (θk , λk )   (16) λk+1 = λk + ηλ g(θk ) +

(i)

qk (θ) =

αi (i) w [gi (θ)]2+ , 2 k

(i)

wk = ∥∇gi (θk )∥−2 .

And the Lagrangian becomes

With ηθ , ηλ as the primal and dual step sizes. The gradient of the Lagrangian Lk (θ, λ) takes the form:

Lk (θ, λ) = −LR (θ)+

∇θ Lk (θ, λ) = − ∇θ LR (θ) + ∇θ qk (θ) + λ ∇θ g(θ)   = − ∇θ LR (θ) + λ + α wk [g(θ)]+ ∇θ g(θ) (17)

m  X

 (i) λi gi (θ) + qk (θ) ,

λi ≥ 0.

i=1

Allowing each constraint to be corrected independently according to its local sensitivity.

Equation (17) shows that CSPO uses an effective multiplier λeff = λ + αwk [g(θ)]+ , which strengthens constraint correction whenever g(θ) ≥ 0. The additional term enables smarter feasibility recovery while allowing λ to adapt more smoothly for long-term constraint satisfaction.

4. Algorithm We now describe the practical implementation of CSPO in a deep RL setting. We consider a class of parameterized stochastic policies Πθ = {πθ (a | s) : θ ∈ Rd }, represented by neural networks with fixed architecture. Directly minimizing (14) is intractable due to unknown future state distributions and poor sampling efficiency. Following Proximal Policy Optimization (PPO) (Schulman et al., 2017), CSPO uses an analogous clipped surrogate objective for the cost as for the reward. Thus, the practical optimization objective is derived from (14) as: 
 R L̂R (θ) = Et min rt (θ)AR t , clip(rt (θ), 1 − ϵ, 1 + ϵ)At (18) 
 C L̂C (θ) = Et max rt (θ)AC t , clip(rt (θ), 1 − ϵ, 1 + ϵ)At (19) 1 ĝ(θ) = Jc (πθk ) − d + L̂C (θ) (20) (1 − γ) α LCSPO (θ) = − L̂R (θ) + w [ĝ(θ)]2+ + λ ĝ(θ) (21) 2 where the Generalized Advantage Estimator (GAE) (Schulman et al., 2015b) is used to compute the reward and cost advantages AR and AC from trajectories t t c ∞ {st , at , rt , ct , Vt , Vt }t=0 collected under πθold . The imt) portance sampling ratio is rt (θ) = ππθθ (a(at |s . w = t |st )

Proposition 3.2. Assuming the reward objective LR (θ) and the constraint function g(θ) are continuously differentiable, the original constrained problem (6) and CSPO’s augmented problem (13) share the same set of KKT solutions. See Appendix B for proof. Assumption 3.3. LR and g are continuously differentiable with LR being LR -smooth and g being Lg -smooth. The gradients and constraint values are uniformly bounded: ∥∇g(θ)∥ ≤ Gg , ∥∇LR (θ)∥ ≤ GR , |g(θ)| ≤ Bg . The sensitivity weights satisfy wk ≤ wmax , and the dual domain is Λ = [0, λmax ]. 3.3. Convergence analysis Under Assumption 3.3 and appropriate step-size conditions, CSPO’s iterates approach an ε-stationary point of (14) at a rate of  3 2 2  L G λmax O , ε6 where L = LR + αwmax G2g + (λmax + αwmax Bg )Lg ,

old

1 ∥∇θ ĝ(θk )∥2 +ϵ is recomputed at the beginning of each policy

G = GR + (λmax + αwmax Bg )Gg . 4

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Algorithm 1 Constraint-Sensitive Policy Optimization

one CSPO primal update satisfies   g(θt+1 ) ≤ g(θt ) − η αw g(θt )∥∇g(θt )∥2 − δ + O(η 2 ) (23) δ In particular, if g(θt ) > αw∥∇g(θ , then g(θ ) < g(θ 2 t+1 t) t )∥ holds for sufficiently small η.

1: Initialize policy network πθ0 , value network Vψ , and

cost value network Vϕc

2: Initialize Lagrange multiplier λ0 ≥ 0 3: for episodic iteration k = 0, 1, 2, . . . do 4: Collect trajectories τ ∼ πθk 5: Estimate advantages AR , AC using GAE 6: Update value networks Vψ and Vϕc

Remark 4.3. Under PPO-style clipped surrogates used in (18)-(19) and sufficiently small step size η, the sufficient condition above simplifies to g(θt ) ≳ αδ . See Appendix E for proof.

Compute and detach wk = ∥∇θ ĝ(θ1k )∥2 +ϵ for t = 0, 1, . . . , T − 1 do Compute L̂R (θ) in Eq.(18) Compute L̂C (θ) in Eq.(19) Update policy parameters: θ ← θ − η∇θ L(θ, λk ) in Eq.(17) c θ ∥ πθ ) > δKL then 12: if KL(π k 13: break 14: end if 15: end for 16: Set θk+1 ← θ   17: Update λ: λk+1 = λk + ηλ (Jc (πθk ) − d) + 18: end for 7: 8: 9: 10: 11:

Proposition 4.2 shows that CSPO guarentees a decrease in constraint violation whenever g(θt ) exceeds a threshold proportional to α1 . Thus, larger α values activate feasibility-oriented updates at smaller violations, whereas smaller α postpone this activation and permit a wider reward–constraint trade-off before corrective behavior dominates. Computational Complexity. CSPO adds minimal overhead over first-order Lagrangian baselines, requiring one extra surrogate constraint-gradient evaluation per episodic iteration to compute wk , which is detached and kept constant during the inner-loop policy updates.

update step k using the current policy parameters and is treated as a fixed, detached constant during the subsequent policy updates. For numerical stability, we also incorporate clipping and Exponential Moving Average (EMA) smoothing to wk (See Appendix G for implementation details). Since qk (θ) is continuously differentiable, the surrogate objective (21) can be optimized using standard first-order optimizers (e.g., Adam (Kingma & Ba, 2017)). We present the pseudo-code of CSPO in Algorithm 1. This practical optimization scheme yields the following theoretical properties:

5. Experiments In this section, we empirically evaluate CSPO. Our experiments are designed to assess the constrained performance, safety recovery behavior, and robustness of CSPO in comparison with state-of-the-art safe RL methods. Specifically, we aim to answer the following questions: • Does CSPO achieve strong final performance while satisfying safety constraints? • Does CSPO enable faster recovery to feasibility and mitigating oscillations without sacrificing reward?

Proposition 4.1 (Inner-loop stationarity of CSPO). For a fixed episodic iteration k in Algorithm 1, with wk and λk −1 held constant during the inner-loop updates. Let {θt }Tt=0 denote the iterates generated by successive policy updates in Algorithm 1 (line 11) applied to the CSPO-augmented objective L(θ, λk ). Assuming L(θ, λk ) is L-smooth in θ, then the updates satisfy the stationarity rate min

0≤t≤T −1

∇θ L(θt , λk )

2

  1 =O T

• How sensitive is CSPO to different safety parameters? 5.1. Experimental Setup We evaluate CSPO on 9 continuous-control safety tasks (5 locomotion and 4 navigation) from the Safety Gymnasium (Ji et al., 2023) benchmark1 . The locomotion tasks incentivize forward progress while penalizing excessive velocity, whereas the navigation tasks reward reaching designated goals and impose penalties for entering unsafe regions.

(22)

See Appendix D for proof details. Proposition 4.2 (Local constraint decrease under CSPO updates). For a fixed episodic iteration k in Algorithm 1, let θt be an inner-loop iterate such that g(θt ) > 0. Assuming g is continuously differentiable with locally Lipschitz gradient and that ∥∇LR (θ)∥ ≤ GR and ∥∇g(θ)∥ ≤ Gg locally. Define δ := GR Gg . Then for sufficiently small step size η,

Baselines. We compare CSPO against representative baselines from the safe RL literature, including classic Lagrangians (i.e., PPO-Lag (Ray et al., 2019), CPPO-PID (Stooke et al., 2020)), and first-order methods (i.e., FOCOPS 1

5

https://safety-gymnasium.readthedocs.io/en/latest/

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

(Zhang et al., 2020), CUP (Yang et al., 2022)), penaltybased methods (i.e., P3O (Zhang et al., 2022), IPO (Liu et al., 2020), and EPO (Gao et al., 2024)) and augmented lagrangians (i.e., APPO (Dai et al., 2023)), and second-order quadratic methods (i.e., CPO (Achiam et al., 2017), PCPO (Yang et al., 2020), C-TRPO (Milosevic et al., 2025)). All methods and experiments have been implemented in the Omnisafe (Ji et al., 2024) library2 as well as our implementation for CSPO3 . Detailed hyperparameter settings and training curves are provided in Appendix F.

and RP across environments. Across all tasks, CSPO consistently achieves low violation frequency (VF) and fast recovery (TTS), while maintaining competitive reward preservation (RP). The full aggregated results are reported in Table 5. These aggregate trends are consistent with the training dynamics in Figure 3, where CSPO exhibits reduced cost oscillations leading to faster safety recovery compared to Lagrangian baselines. To better interpret cases where CSPO does not attain the lowest mean TTS, we analyze recovery time versus local constraint sensitivity. We compute st = ∥∇θ g(θt )∥ at violation states and classify them into flat/steep groups using a quantile split (flat: bottom 30%, steep: top 30%). Table 2 shows that TTS increases in high-sensitivity regions, where CSPO is more conservative to avoid overshooting and preserve reward, while recovery is faster in low-sensitivity regions. This trade-off between fast feasibility recovery in flat regions and reward-preserving recovery in steep regions aligns well with CSPO’s final constrained returns in Table 1.

5.2. Overall Performance We first evaluate the final constrained performance of CSPO. Table 1 reports the Interquartile Mean (IQM) return and cost over the last 100 epochs for all algorithms. Constraintviolating runs (average cost above the threshold) are considered poor performance from the perspective of safe RL. CSPO achieves competitive or superior constrained returns while respecting the cost limits. The gains are most pronounced in navigation tasks (CSPO outperforming all baselines), where precise cost control is required to avoid unsafe regions under sparse rewards. In locomotion environments with velocity-based constraints, CSPO remains competitive with state-of-the-art methods, consistently achieving the best returns while staying close to the safety threshold. Notably, without sensitivity scaling (wk = 1), CSPO reduces to a primal-dual update with a quadratic penalty term (similar in style to APPO) and degenerates to PPO-Lag when α = 0. Its consistent improvements over these two, in particular, isolate the benefit of the proposed constraint-sensitive correction, enabling faster feasibility recovery without compromising reward performance across diverse tasks.

5.4. Sensitivity to Safety Parameters We analyze the sensitivity of CSPO to key safety-related parameters, including the fractional reduction factor α and the cost threshold d. Figure 4 ablates α on PointGoal, showing it acts as an interpretable safety-recovery aggressiveness knob: smaller α yields slower, less intrusive feasibility recovery; while larger α enforces faster constraint satisfaction with temporary reward suppression. For the results reported in Table 1 we used fixed values of α per task family: α = 0.85 for locomotion tasks and α = 0.3 for navigation tasks. We provide further discussion and practical guidance for choosing α in Section 5.5 We further evaluate the robustness of CSPO to different safety budgets by training under multiple cost thresholds. As shown in Figure 5, CSPO maintains stable behavior across a wide range of cost limits, effectively adapting its performance to the available safety budget while preserving rapid recovery toward feasibility.

5.3. Safety metrics evaluation: TTS, RP, VF To characterize safety behavior beyond average constraint satisfaction, we analyze how frequently algorithms violate constraints and how quickly they recover when violations occur. Let Ck and Jk denote the average episode cost and return at epoch k, and let d be the cost limit. We define the empirical constraint violation as gk = Ck − d, where an epoch is safe if gk ≤ 0. Three metrics are defined: Violation frequency (VF): fraction of training epochs with gk > 0 within a fixed evaluation window (i.e., after 30% of the training budget). Time to safety (TTS): measures recovery speed: if a violation begins at epoch s and feasibility is recovered at epoch e > s, then TTS = e − s. Reward preservation (RP): quantifies performance retention during recovery and is defined as RP = Je /Js . All metrics are averaged over violation episodes and seeds.

Finally, since CSPO is a primal-dual method, we also evaluate its sensitivity to the initialization and learning rate of the Lagrange multiplier. As shown in Appendix I, CSPO remains comparatively robust to these choices, exhibiting reduced sensitivity to the dual hyperparameters. 5.5. Practical Choice of α Proposition 4.2 shows that the constraint decrease threshold scales as O(1/α), directly linking α to the aggressiveness of feasibility recovery. In practice, α can be selected based on the expected severity of constraint violations in the task: when violations are expected to be sudden and large (as in locomotion tasks), larger α promotes faster recovery; when

Figure 2 visualizes the joint relationship between VF, TTS, 2 3

https://github.com/PKU-Alignment/omnisafe https://github.com/serval-uni-lu/CSPO

6

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning Table 1. IQM performance and bootstrap 95% CI over 5 seeds on safe RL benchmarks. Cost thresholds C are shown in brackets. We bold the best return among methods that satisfy the constraints. CSPO delivers competitive or superior constrained performance while respecting cost limits, with pronounced gains in navigation tasks. Environment Ant C (25) Humanoid C (25)

CSPO

APPO

P3O

PPO-Lag

3231.6 ± 84.8 24.5 ± 0.3

2961.36 ± 92.37 24.49 ± 1.2

2514.81 ± 81.01 23.1 ± 0.5

3134.17 ± 91.88 24.3 ± 1.7

CUP

FOCOPS

PCPO

CPPOPID

C-TRPO

3055.80 ± 81.74 2839.65 ± 113.76 19.6 ± 1.1 26.6 ± 1.8

CPO

2860.88 ± 85.57 26.7 ± 2.4

1728.74 ± 227.2 18.5 ± 1.4

3190.0 ± 69.3 25.3 ± 0.4

3042.64 ± 57.1 18.0 ± 0.7

3123.60 ± 98.79 2754.55 ± 113.40 26.6 ± 0.9 13.9 ± 1.1

6496.85 ± 74.30 6092.95 ± 241.32 18.0 ± 3.1 25.3 ± 0.4

5684.22 ± 87.91 22.0 ± 1.1

6414.56 ± 54.75 6213.15 ± 126.53 2427.39 ± 504.08 5295.88 ± 231.07 4706.07 ± 267.57 25.49 ± 0.7 18.3 ± 2.9 18.9 ± 6.1 26.7 ± 2.6 14.7 ± 1.4

6417.05 ± 54.65 5450.52 ± 166.68 26.4 ± 0.6 15.9 ± 0.8

6270.25 ± 289.6 5745.65 ± 110.47 30.4 ± 4.2 13.1 ± 0.5

HalfCheetah 2007.90 ± 415.95 1771.32 ± 417.02 1745.84 ± 317.66 1841.38 ± 447.36 2464.80 ± 366.32 1718.19 ± 302.14 C (25) 15.5 ± 4.1 23.1 ± 0.5 23.4 ± 0.4 24.6 ± 3.6 19.7 ± 1.4 21.3 ± 3.1 607.20 ± 475.25 1639.22 ± 101.83 22.59 ± 7.1 28.1 ± 3.5

2870.70 ± 73.78 20.3 ± 3.5

1333.23 ± 39.90 18.3 ± 1.4

IPO

EPO

2797.52 ± 380.4 2748.80 ± 313.62 1825.65 ± 326.91 2102.11 ± 117.80 24.2 ± 0.4 15.3 ± 5.2 23.8 ± 0.8 10.9 ± 2.9

Hopper C (25)

1691.92 ± 70.0 1229.86 ± 306.10 1255.34 ± 245.60 7.1 ± 6.1 23.6 ± 3.0 24.4 ± 1.7

1661.60 ± 11.93 1613.64 ± 106.88 1112.55 ± 519.68 1579.19 ± 351.51 1597.48 ± 266.82 20.5 ± 9.3 19.6 ± 6.2 22.5 ± 1.8 21.4 ± 2.3 23.2 ± 1.6

Swimmer C (25)

91.40 ± 56.30 4.8 ± 5.1

74.72 ± 55.90 2.6 ± 1.2

40.94 ± 4.13 22.5 ± 3.1

64.26 ± 19.79 30.5 ± 4.6

44.32 ± 38.58 25.0 ± 0.8

42.39 ± 2.37 22.1 ± 2.7

49.48 ± 17.32 30.2 ± 3.7

32.53 ± 53.41 37.0 ± 19.0

67.12 ± 46.60 24.7 ± 1.0

68.20 ± 54.70 16.5 ± 6.9

45.32 ± 7.40 25.6 ± 1.4

41.85 ± 4.40 19.2 ± 0.9

CarButton C (25)

0.70 ± 0.32 23.4 ± 0.8

0.36 ± 0.11 24.8 ± 1.8

-0.14 ± 0.20 32.9 ± 2.3

0.52 ± 0.40 25.79 ± 2.2

0.85 ± 0.35 33.3 ± 2.8

-0.33 ± 1.30 36.2 ± 8.8

0.63 ± 0.73 37.6 ± 5.3

0.09 ± 0.20 35.6 ± 3.6

-1.36 ± 0.23 32.7 ± 4.9

0.65 ± 0.56 34.8 ± 1.4

1.13 ± 0.76 34.7 ± 1.0

0.14 ± 0.60 24.2 ± 1.3

CarGoal C (25)

27.11 ± 0.92 24.0 ± 0.4

26.00 ± 1.00 24.97 ± 0.8

21.71 ± 0.92 25.7 ± 0.4

27.70 ± 1.65 26.8 ± 2.0

26.67 ± 0.76 27.4 ± 0.6

20.46 ± 4.02 28.2 ± 4.1

20.78 ± 2.13 31.0 ± 2.9

21.43 ± 0.89 29.4 ± 1.5

6.23 ± 2.16 24.6 ± 2.7

23.07 ± 2.54 28.3 ± 0.5

27.41 ± 0.80 27.1 ± 0.6

25.05 ± 1.16 25.2 ± 0.5

PointButton C (25)

10.15 ± 0.20 24.4 ± 0.3

9.01 ± 1.04 25.2 ± 0.5

4.16 ± 1.02 26.3 ± 1.7

9.05 ± 1.65 24.1 ± 4.7

7.02 ± 0.61 25.0 ± 0.4

3.56 ± 1.62 30.1 ± 6.0

5.55 ± 1.52 26.8 ± 4.4

2.61 ± 1.49 33.5 ± 3.6

1.39 ± 1.31 24.0 ± 1.7

6.50 ± 1.55 28.9 ± 0.7

7.92 ± 1.30 28.6 ± 0.9

6.38 ± 1.40 24.3 ± 3.8

PointGoal C (25)

23.79 ± 0.75 23.7 ± 0.4

23.12 ± 0.35 24.3 ± 0.5

22.28 ± 0.81 24.1 ± 0.6

21.78 ± 2.38 22.0 ± 1.6

22.58 ± 0.43 26.9 ± 0.6

20.40 ± 3.82 25.3 ± 5.5

15.56 ± 4.36 30.8 ± 2.7

19.82 ± 1.27 27.1 ± 0.9

6.36 ± 3.16 25.1 ± 3.7

19.34 ± 1.03 24.8 ± 0.2

23.35 ± 0.77 27.5 ± 0.5

20.19 ± 1.00 24.6 ± 1.4

Table 2. CSPO recovery metrics conditioned on local constraint sensitivity (Flat: low ∥∇g∥, Steep: high ∥∇g∥). Env

Geometry

TTS

RP

Ant Ant Humanoid Humanoid HalfCheetah HalfCheetah

Flat Steep Flat Steep Flat Steep

3.63 5.25 2.33 6.17 4.19 7.45

1.02 0.99 1.00 0.99 1.04 1.00

185.98 ± 9.41 1340.49 ± 131.78 27.1 ± 2.8 12.1 ± 5.1

CSPO preserves equivalence with the original constrained problem. Limitations: While CSPO improves safety recovery dynamics, it relies on accurate cost-gradient estimates, which may become noisy in sparse or discontinuous settings, reducing normalization effectiveness. CSPO also introduces one additional parameter α, although bounded and interpretable, its optimal choice remains task-dependent. Designing more robust adaptive or learned sensitivity scaling mechanisms is left for future work.

violations are gradual and bounded, smaller α suffices.

6. Related Work

When the violation severity is difficult to characterize a priori, we propose an adaptive schedule motivated by the O(1/α) scaling of Proposition 4.2, setting α proportionally to the current violation magnitude:   C [J (πθk ) − d]+ αk = clip , 0, 1 , (24) d+ε

In this section, we discuss the relevant prior works in the safe RL literature. Trust-region methods. Trust-region approaches such as Constrained Policy Optimization (CPO) (Achiam et al., 2017) extend Trust Region Policy Optimization (TRPO) (Schulman et al., 2015a) by enforcing safety constraints within a local trust region, providing theoretical guarantees under idealized assumptions. More recently, C-TRPO (Milosevic et al., 2025) augments the trust-region formulation with a barrier-based divergence term to ensure that the trust region contains only safe policies. Projection-based variants, including PCPO (Yang et al., 2020), decompose the optimization into a reward maximization step followed by a projection back onto the feasible policy set. In practice, trust-region and projection-based methods require solving quadratic programs using conjugate gradient methods and backtracking line search, resulting in high computational cost and sensitivity to approximation errors.

so that αk is small near feasibility and grows under larger violations, providing a practical default that removes the need for task-specific tuning. Normalization by d renders the schedule budget-invariant across different cost thresholds. We compare both the fixed and adaptive schedules empirically in Appendix H. 5.6. Discussion Originality: CSPO demonstrates the importance of constraint sensitivity in safe policy optimization. Existing primal-dual methods typically apply uniform corrections, ignoring how sensitive the constraint is to parameter updates. This can lead to overshooting and oscillations in high-sensitivity regions, or slow recovery in low-sensitivity regions. In contrast, CSPO scales the constraint correction using the norm of the constraint gradient, enabling stable feasibility recovery using only first-order information. Since the correction activates only under violation,

Lagrangian and primal-dual methods. A widely adopted class of Safe RL methods reformulates CMDPs using Lagrangian duality, converting constraints into weighted penalties on expected costs. Representative approaches in7

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

TTS (Mean Time To Safety)

Ant 22.5 20.0 17.5 15.0 12.5 10.0 7.5 5.0

Humanoid

Hopper

9

20

8

15

7

5 0.0

0.2

0.4

0.8

0.0

PointGoal 20.0 17.5 15.0 12.5 10.0 7.5 5.0 2.5

0.2

0.4

0.6

0.8

PointButton

40

10

30

0.2

0.6

0.8

0.6

0.2

5 0.4

0.6

0.8

10 6 4 0.0

0.2

0.2

0.4

0.4

0.6

0.8

CSPO

0.0

0.2

0.4

0.6

0.8

CarButton

CSPO

0.0

8

CSPO

12 11 10 9 8 7 6 5

10 0.0

12

CarGoal

15

CSPO

14

20

0.8

20

4 0.4

0.4

25

6

0.0

0.2

35

8

CSPO

0.0

12

16

25

5

CSPO

4 0.6

30

10

5

CSPO

Swimmer 18

15

10

6 CSPO

HalfCheetah 35

0.6

0.8

APPO CPPOPID CUP FOCOPS CSPO PPO-Lag

CSPO

0.0

0.2

VF (Violation Frequency)

0.4

0.6

0.8

0.8

0.9

Reward Preservation (RP)

1.0

Figure 2. Safety recovery behavior across environments. Each point corresponds to one algorithm in a given environment, plotted by violation frequency (VF) and mean time to safety (TTS). Marker color encodes reward preservation (RP), with greener colors indicating better reward retention. Lower VF and lower TTS indicate safer and more stable behavior. Across all tasks, CSPO consistently achieves low VF and low TTS, while maintaining high RP.

clude PPO-Lag and TRPO-Lag variants (Ray et al., 2019), as well as general primal–dual optimization methods (Chow et al., 2018; Paternain et al., 2019; Tessler et al., 2019). While simple and scalable, these methods often suffer from oscillations and overshoot near the constraint boundary due to mismatched update timescales between the policy and the dual variable. CPPO-PID (Stooke et al., 2020) reduces oscillations via PID-controlled multiplier updates, and recent work (Chen et al., 2024) further couple this feedback with multiplier-dependent adaptive primal learning rates to scale the policy update, at the cost of introducing more hyperparameters to tune. We also consider FOCOPS (Zhang et al., 2020) and CUP (Yang et al., 2022), which rely on solving primal–dual subproblems through projection-based updates. Although theoretically motivated, such projections primarily enforce feasibility and do not preserve equivalence with the original constrained optimum, which can degrade empirical performance (Zhang et al., 2023).

oscillations. However, these methods typically rely on fixed or globally tuned penalty factors, without explicitly accounting for the local sensitivity of the constraint surface. For an extended discussion on the related works, see Appendix J

7. Conclusion and future works We proposed CSPO, a first-order, primal-dual algorithm for safe policy optimization that incorporates local constraint sensitivity to stabilize feasibility recovery. By scaling the constraint correction using the norm of the constraint gradient, CSPO mitigates overshooting and oscillations near the safety boundary while preserving reward performance, without relying on projections, large penalty coefficients, or second-order optimization. Experiments across navigation and locomotion tasks show that CSPO consistently improves recovery stability and constrained returns compared to strong safe RL baselines. Future work includes extending CSPO to multiple constraints and delayed or noisy cost signals, and generalizing the sensitivity scaling beyond the Euclidean norm using alternative local metrics (e.g Fisher), to incorporate curvature information of the constraint, yielding a geometry-aware extension of CSPO.

Penalty and augmented penalty methods. Penaltybased approaches replace the constrained objective with an unconstrained surrogate. IPO (Liu et al., 2020) employs logarithmic barrier functions, requiring feasible initialization and yielding suboptimal solutions in practice. P3O (Zhang et al., 2022) uses an exact ℓ1 penalty formulation but relies on sufficiently large penalty coefficients, which can amplify gradient estimation errors. EPO (Gao et al., 2024) employs an adaptive Exterior point penalty generated by a seperate Penalty Metric Network. APPO (Dai et al., 2023) introduces a quadratic penalty term to complement the Lagrangian multiplier, improving stability by dampening cost

Impact Statement This paper presents work whose goal is to advance the field of Safe Reinforcement Learning. While improved safety guarantees can benefit real-world decision-making systems, the methods developed here are general-purpose and we do not anticipate any immediate negative societal impacts that 8

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

PPO-Lag

CUP

APPO

Constraint

CSPO

CPPOPID

0 100

2

50

1

100 50

Cost

2000 0 0

0

100 200 300 400 500 Epoch

λ

10

λ

20 0

Return

FOCOPS

6 4 2 0

Cost

Return

Return

0

100 200 300 400 500 Epoch

0

0

Multiplier

100 200 300 400 500 Epoch

Figure 3. Episodic returns and costs for Lagrangian-based algorithms on PointGoal (top) and Ant (bottom) tasks. CSPO’s constraint sensitivity helps mitigating and damping oscillations arising from delayed dual updates.

Cost

Return

20 10 0

0

200

Epoch

400

d = 10

α = 0.5

α = 0.3 Constraint

α = 0.7

Return

d = 15

20

40 20 0

200

Epoch

10 0

400

Figure 4. Episodic returns and costs of CSPO with different α values on PointGoal. Larger α induces more aggressive sensitivityaware corrections, accelerating feasibility recovery at the expense of reward; smaller values yield conservative safety recovery.

Cost

Return

Return

α = 1.0

0

200

Epoch

400

d = 25

d = 35

Constraint

40 20 0

200

Epoch

400

Figure 5. Cost-threshold ablation on P OINT G OAL. CSPO adapts to increasingly strict cost limits while maintaining stable reward growth and rapid return to feasibility.

Dai, J., Ji, J., Yang, L., Zheng, Q., and Pan, G. Augmented proximal policy optimization for safe reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 7288–7295, 2023.

require specific discussion.

References Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International conference on machine learning, pp. 22–31. PMLR, 2017.

Gao, S., Ding, J., Fu, L., Wang, X., and Zhou, C. Exterior penalty policy optimization with penalty metric network under constraints. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24. International Joint Conferences on Artificial Intelligence Organization, 2024. URL https://doi. org/10.24963/ijcai.2024/443.

Altman, E. Constrained Markov decision processes. Routledge, 2021. Chen, W., Onyejizu, J., Vu, L., Hoang, L., Subramanian, D., Kar, K., Mishra, S., and Paternain, S. Adaptive primal-dual method for safe reinforcement learning. In Proceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2024, 2024. URL https://dl.acm.org/doi/10. 5555/3635637.3662881.

Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., Geng, Y., Zhong, Y., Dai, J., and Yang, Y. Safety gymnasium: A unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36: 18964–18993, 2023. Ji, J., Zhou, J., Zhang, B., Dai, J., Pan, X., Sun, R., Huang, W., Geng, Y., Liu, M., and Yang, Y. Omnisafe: An infrastructure for accelerating safe reinforcement learning research. Journal of Machine Learning Research, 25 (285):1–6, 2024.

Chow, Y., Ghavamzadeh, M., Janson, L., and Pavone, M. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18 (167):1–51, 2018. 9

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Kakade, S. and Langford, J. Approximately optimal approximate reinforcement learning. In Proceedings of the nineteenth international conference on machine learning, pp. 267–274, 2002.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Stooke, A., Achiam, J., and Abbeel, P. Responsive safety in reinforcement learning by pid lagrangian methods. In International Conference on Machine Learning, pp. 9133– 9143. PMLR, 2020.

Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization, 2017. URL https://arxiv.org/abs/ 1412.6980. Lin, T., Jin, C., and Jordan, M. On gradient descent ascent for nonconvex-concave minimax problems. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 6083–6093. PMLR, 13–18 Jul 2020. URL https://proceedings.mlr.press/ v119/lin20a.html.

Tessler, C., Mankowitz, D. J., and Mannor, S. Reward constrained policy optimization. In 7th International Conference on Learning Representations, ICLR 2019. OpenReview.net, 2019. URL https://openreview. net/forum?id=SkfrvsA9FX. Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., and Miao, Q. Deep reinforcement learning: A survey. IEEE Transactions on Neural Networks and Learning Systems, 35(4):5064–5078, 2022.

Liu, Y., Ding, J., and Liu, X. Ipo: Interior-point policy optimization under constraints. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 4940–4947, 2020.

Yang, L., Ji, J., Dai, J., Zhang, L., Zhou, B., Li, P., Yang, Y., and Pan, G. Constrained update projection approach to safe policy optimization. Advances in Neural Information Processing Systems, 35:9111–9124, 2022.

Milosevic, N., Müller, J., and Scherf, N. Embedding safety into RL: A new take on trust region methods. In Fortysecond International Conference on Machine Learning, 2025. URL https://openreview.net/forum? id=4zRb89SbzG.

Yang, T.-Y., Rosca, J., Narasimhan, K., and Ramadge, P. J. Projection-based constrained policy optimization. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum? id=rke3TJrtPS.

Nesterov, Y. et al. Lectures on convex optimization, volume 137. Springer, 2018.

Zhang, L., Shen, L., Yang, L., Chen, S., Yuan, B., Wang, X., and Tao, D. Penalized proximal policy optimization for safe reinforcement learning. In International Joint Conference on Artificial Intelligence, 2022. URL https://api.semanticscholar. org/CorpusID:249017615.

Paternain, S., Chamon, L., Calvo-Fullana, M., and Ribeiro, A. Constrained reinforcement learning has zero duality gap. Advances in Neural Information Processing Systems, 32, 2019. Platt, J. and Barr, A. Constrained differential optimization. In Neural information processing systems, 1987.

Zhang, L., Zhang, Q., Shen, L., Yuan, B., Wang, X., and Tao, D. Evaluating model-free reinforcement learning toward safety-critical tasks. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 15313– 15321, 2023.

Ray, A., Achiam, J., and Amodei, D. Benchmarking safe exploration in deep reinforcement learning, 2019. URL https://cdn.openai.com/ safexp-short.pdf. Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In Bach, F. and Blei, D. (eds.), Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 1889–1897, Lille, France, 07–09 Jul 2015a. PMLR. URL https://proceedings.mlr.press/v37/ schulman15.html.

Zhang, Y., Vuong, Q., and Ross, K. First order constrained optimization in policy space. Advances in Neural Information Processing Systems, 33:15338–15349, 2020.

Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. CoRR, abs/1506.02438, 2015b. URL https://api. semanticscholar.org/CorpusID:3075448. 10

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

A. Derivations for Section 3.1 A.1. Minimum update to the linearized safety boundary We derive the closed form of the minimal parameter update (in the Euclidean norm) that reaches the linearized constraint boundary. Consider the optimization problem: min 12 ∥∆θ∥2

∇θ g(θk )⊤ ∆θ + g(θk ) = 0

s.t.

∆θ

(25)

The objective is strictly convex and the constraint is affine, therefore strong duality holds and the KKT conditions are necessary and sufficient for optimality. From the Lagrangian
J (∆θ, ν) = 12 ∥∆θ∥2 + ν ∇θ g(θk )⊤ ∆θ + g(θk ) , stationarity w.r.t. ∆θ gives ∇∆θ J = ∆θ + ν∇θ g(θk ) = 0

⇒

∆θ⋆ = −ν ∇θ g(θk ).

Imposing the constraint yields ∇θ g(θk )⊤ ∆θ⋆ = −ν ∥∇θ g(θk )∥2 = −g(θk ), so ν=

g(θk ) . ∥∇θ g(θk )∥2

Therefore, ∆θ⋆ = −

g(θk ) ∇θ g(θk ). ∥∇θ g(θk )∥2

Finally, the optimal step length is ⋆ 2



∥∆θ ∥ =

g(θk ) ∥∇θ g(θk )∥2

2

∥∇θ g(θk )∥2 =

g(θk )2 , ∥∇θ g(θk )∥2

(26)

which gives ∥∆θ⋆ ∥ =

|g(θk )| . ∥∇θ g(θk )∥

(27)

In the safety-recovery setting, when g(θk ) > 0, (27) reduces to ∥∆θ⋆ ∥ =

g(θk ) . ∥∇θ g(θk )∥

A.2. Minimum update for the fractional violation reduction We derive the minimal update that achieves a fractional reduction of the linearized constraint violation. Let σ ∈ [0, 1) be the desired contraction factor, i.e., g(θk+1 ) ≤ σ g(θk ). (28) Using the first-order approximation g(θk + ∆θ) ≈ g(θk ) + ∇θ g(θk )⊤ ∆θ, Then consider the optimization problem min 21 ∥∆θ∥2 ∆θ

s.t.

∇θ g(θk )⊤ ∆θ + g(θk ) = σ g(θk ).

The Lagrangian is   J (∆θ, ν) = 21 ∥∆θ∥2 + ν ∇θ g(θk )⊤ ∆θ + (1 − σ) g(θk ) . 11

(29)

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Stationarity gives ∆θ + ν∇θ g(θk ) = 0, hence ∆θ⋆ = −ν ∇θ g(θk ). Enforcing the constraint yields −ν ∥∇θ g(θk )∥2 = −(1 − σ) g(θk ) Therefore ∆θ⋆ = −

⇒

ν=

(1 − σ) g(θk ) . ∥∇θ g(θk )∥2

(1 − σ) g(θk ) ∇θ g(θk ). ∥∇θ g(θk )∥2

Finally, the optimal step length is ∥∆θ⋆ ∥2 =



(1 − σ) g(θk ) ∥∇θ g(θk )∥2

2

∥∇θ g(θk )∥2 =

(1 − σ)2 g(θk )2 . ∥∇θ g(θk )∥2

(30)

This yields ∥∆θ⋆ ∥ = (1 − σ)

|g(θk )| . ∥∇θ g(θk )∥

(31)

For the infeasible case g(θk ) > 0, and setting α = 1 − σ ∈ [0, 1], this simplifies to ∥∆θ⋆ ∥ = α

g(θk ) , ∥∇θ g(θk )∥

which matches Eq. (11).

B. Proof of Proposition 3.2. Assume that the reward objective L(θ) and the constraint function g(θ) are continuously differentiable. Consider the original constrained problem min −LR (θ) s.t. g(θ) ≤ 0, (32) θ

and the CSPO-augmented constrained problem min −LR (θ) + qk (θ) θ

s.t.

g(θ) ≤ 0,

(33)

where

α wk [g(θ)]2+ . 2 Consider the same constraint qualification for both problems. Then the two problems share the same set of KKT solutions. qk (θ) =

Proof. Let (θ⋆ , λ⋆ ) be a KKT solution of the original problem (32). By primal feasibility, g(θ⋆ ) ≤ 0, which implies [g(θ⋆ )]+ = 0 and hence qk (θ⋆ ) = 0. Moreover, since the hinge term [g(θ)]2+ vanishes identically for all θ such that g(θ) ≤ 0, the sensitivity correction term does not contribute to the objective or its first-order variation at any feasible point. As a result, the stationarity condition of the CSPO-augmented problem reduces to −∇θ LR (θ⋆ ) + λ⋆ ∇θ g(θ⋆ ) = 0, which coincides exactly with the KKT stationarity condition of the original problem. The remaining KKT conditions—primal feasibility, dual feasibility, and complementary slackness—are identical for both problems. Therefore, (θ⋆ , λ⋆ ) is also a KKT solution of (33). Conversely, let (θ⋆ , λ⋆ ) be a KKT solution of the CSPO-augmented problem (33). By primal feasibility, g(θ⋆ ) ≤ 0, implying qk (θ⋆ ) = 0 and ∇qk (θ⋆ ) = 0. Thus, the stationarity condition of the CSPO-augmented problem reduces to that of the original constrained problem, and the remaining KKT conditions coincide. Hence (θ⋆ , λ⋆ ) is a KKT solution of (32). 12

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

C. Convergence analysis We establish convergence of CSPO’s idealized single-step iterates (16) by verifying the conditions of Lin et al. (2020) (Theorem 4.9) and deriving the CSPO-specific rate constants. The iterates (16) perform two-timescale Gradient Descent Ascent (GDA) on minθ maxλ∈Λ Lk (θ, λ), matching Algorithm 1 of Lin et al. (2020) under the correspondence x ↔ θ,

y ↔ λ,

f (x, y) ↔ Lk (θ, λ),

ηx ↔ ηθ ,

ηy ↔ ηλ ,

with wk evaluated at the current iterate θk and treated as a constant during the optimization of θ. Condition 1: Nonconvex-concave structure. Lk (θ, λ) = −LR (θ) + qk (θ) + λg(θ) is nonconvex in θ since LR (θ) is generally nonconvex, and linear (hence concave) in λ. The dual domain Λ = [0, λmax ] is convex and bounded with diameter D = λmax . This satisfies Assumption 4.7 of Lin et al. (2020). Condition 2: L-smoothness in θ. For any fixed λ ∈ Λ, we bound the gradient difference ∥∇θ Lk (θ, λ) − ∇θ Lk (θ′ , λ)∥ by decomposing into three terms using (17):

∇θ Lk (θ, λ) − ∇θ Lk (θ′ , λ) = − ∇LR (θ) − ∇LR (θ′ )
+ λ + αwk [g(θ)]+ ∇g(θ)
− λ + αwk [g(θ′ )]+ ∇g(θ′ ).
Adding and subtracting λ + αwk [g(θ′ )]+ ∇g(θ) and regrouping yields:
∇θ Lk (θ, λ) − ∇θ Lk (θ′ , λ) = − ∇LR (θ) − ∇LR (θ′ ) {z } | Term 1

+ αwk [g(θ)]+ − [g(θ′ )]+ ∇g(θ) | {z } Term 2

+ λ + αwk [g(θ′ )]+ ∇g(θ) − ∇g(θ′ ) . | {z } Term 3

Applying the triangle inequality and bounding each term: • ∥∇LR (θ) − ∇LR (θ′ )∥ ≤ LR ∥θ − θ′ ∥ by LR -smoothness of LR . • αwk |[g(θ)]+ − [g(θ′ )]+ |∥∇g(θ)∥ ≤ αwmax G2g ∥θ − θ′ ∥ since the hinge is 1-Lipschitz, ∥∇g∥ ≤ Gg , and wk ≤ wmax . • (λ + αwk [g(θ′ )]+ )∥∇g(θ) − ∇g(θ′ )∥ ≤ (λmax + αwmax Bg )Lg ∥θ − θ′ ∥ by Lg -smoothness of g and [g]+ ≤ Bg . Summing yields L-smoothness with L = LR + αwmax G2g + (λmax + αwmax Bg )Lg , satisfying the ℓ-smoothness requirement of Assumption 4.7. Condition 3: G-Lipschitz in θ.

From (17), uniformly over λ ∈ Λ: ∥∇θ Lk (θ, λ)∥ ≤ GR + (λmax + αwmax Bg )Gg =: G, |{z} | {z } ∥∇LR ∥

(λ+αwk [g]+ )∥∇g∥

so Φk (θ) = maxλ∈Λ Lk (θ, λ) is G-Lipschitz, satisfying the L-Lipschitz requirement of Assumption 4.7. 13

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Conclusion. All conditions of Theorem 4.9 of Lin et al. (2020) are satisfied with ℓ ↔ L, L-Lip ↔ G, D ↔ λmax . Applying Theorem 4.9 with step sizes     1 ε4 , ηλ = Θ ηθ = Θ L3 G2 λ2max L yields the complexity L3 λ2max ∆0 L3 G2 λ2max ∆Φ + O ε6 ε4 

 .

Absorbing the initialization-dependent quantities ∆Φ and ∆0 into constants and retaining the dominant term in ε gives the simplified rate  3 2 2  L G λmax O . ε6 to an ε-stationary point of (16)

D. Proof of Proposition 4.1 For a fixed episodic iteration k in Algorithm 1, let L(θ, λk ) denote CSPO’s objective in (21), with wk and λk held constant −1 throughout the inner loop updates, and assuming that L(·, λk ) is L-smooth in θ. Let {θt }Tt=0 be generated by T gradient steps: θt+1 = θt − η ∇θ L(θt , λk ), t = 0, . . . , T − 1, with step size η ∈ (0, 1/L]. By the descent lemma for L-smooth functions (Nesterov et al., 2018), for any θ and ∆, L(θ + ∆, λk ) ≤ L(θ, λk ) + ⟨∇θ L(θ, λk ), ∆⟩ +

L ∥∆∥2 . 2

Applying this inequality at θ = θt with ∆ = −η ∇θ L(θt , λk ) yields Lη 2 2 2 L(θt+1 , λk ) ≤ L(θt , λk ) − η ∥∇θ L(θt , λk )∥ + ∥∇θ L(θt , λk )∥ 2   Lη 2 = L(θt , λk ) − η 1 − ∥∇θ L(θt , λk )∥ . 2 1 Since η ≤ 1/L, we have 1 − Lη 2 ≥ 2 , and therefore

L(θt+1 , λk ) ≤ L(θt , λk ) −

η 2 ∥∇θ L(θt , λk )∥ . 2

Summing over t = 0 to T − 1 gives T −1

η X 2 ∥∇θ L(θt , λk )∥ ≤ L(θ0 , λk ) − Lk (θT , λk ) ≤ L(θ0 , λk ) − L⋆ (λk ), 2 t=0 where L⋆ (λk ) := inf θ L(θ, λk ). Dividing both sides by ηT /2 yields
T −1 2 L(θ0 , λk ) − L⋆ (λk ) 1 X 2 ∥∇θ L(θt , λk )∥ ≤ . T t=0 ηT Finally, since mint at ≤ T1

P

t at for nonnegative at , we obtain

  2 L(θ0 , λk ) − L⋆ (λk ) 1 min ∥∇θ L(θt , λk )∥ ≤ =O , 0≤t≤T −1 ηT T 2

which proves the stationarity rate in Proposition 4.1. 14

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

E. Proof of Proposition 4.2 let L(θ, λk ) denote CSPO’s objective in (21) and consider one inner-loop gradient update θt+1 = θt − η ∇θ L(θt , λk ),

(34)

with step size η > 0. First-order expansion of the constraint. Assuming g is continuously differentiable with locally Lipschitz gradient. Then, for sufficiently small η, g(θt+1 ) = g(θt ) + ∇g(θt )⊤ (θt+1 − θt ) + O(∥θt+1 − θt ∥2 ) = g(θt ) + ∇g(θt )⊤ (θt+1 − θt ) + O(η 2 ).

(35)

Additionally, assuming the gradients are locally bounded as ∥∇LR (θ)∥ ≤ GR and ∥∇g(θ)∥ ≤ Gg . Constraint decrease bound.

Let θt be infeasible, i.e., g(θt ) > 0. Then [g(θt )]+ = g(θt ) and ∇θ α2 w[g(θ)]2+

θ=θt

= αw g(θt ) ∇g(θt ).

Differentiating the CSPO’s objective at θt yields   ∇θ L(θt , λk ) = − ∇LR (θt ) + λ + αwg(θt ) ∇g(θt ).

(36)

g(θt+1 ) = g(θt ) − η ∇g(θt ), ∇θ L(θt , λk ) + O(η 2 ).

(37)

g(θt+1 ) = g(θt ) + η ∇g(θt ), ∇LR (θt ) − η λ ∥∇g(θt )∥2 − η αw g(θt )∥∇g(θt )∥2 + O(η 2 ).

(38)

Substituting (34) into (35) gives

Using (36) in (37) yields

Since λ ≥ 0, the term −η λ ∥∇g(θt )∥2 is non-positive and can be dropped, giving g(θt+1 ) ≤ g(θt ) + η ∇g(θt ), ∇LR (θt ) − η αw g(θt )∥∇g(θt )∥2 + O(η 2 ).

(39)

By Cauchy–Schwarz and the bounded-gradient assumption, ∇g(θt ), ∇LR (θt ) ≤ ∥∇g(θt )∥ ∥∇LR (θt )∥ ≤ Gg GR =: δ. Substituting this bound into (39) yields   g(θt+1 ) ≤ g(θt ) − η αw g(θt )∥∇g(θt )∥2 − δ + O(η 2 ), which proves Eq. (23). Finally, a one-step decrease in constraint violation (g(θt+1 ) < g(θt )) is ensured whenever the leading-order term is strictly negative, i.e., αw g(θt )∥∇g(θt )∥2 > δ, and η is sufficiently small so that the O(η 2 ) term does not dominate the first-order decrease. Rearranging gives the sufficient condition δ g(θt ) > , αw∥∇g(θt )∥2 completing the proof. 15

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning Table 3. Training parameters. Parameter

CSPO

APPO

P3O

IPO

PPO-Lag

CUP

FOCOPS

TRPOPID

CPPOPID

CPO

PCPO

C-TRPO

Number of hidden layers Number of hidden units Activation function Discount factor γ GAE parameter λGAE Actor learning rate ηπ Critic learning rate ηVR Training steps Steps per epoch Update iterations Batch size Penalty factor PPO Clip ratio ϵ Trust region [δ − , δ + ] cg iterations Damping coefficient Backtrack iterations Lagrange multiplier init. Lagrange multiplier learning rate

2 64 tanh 0.99 0.95 3e-4 3e-4 107 2e4 10 512 N/A 0.2 [0,2e-2] N/A N/A N/A 0.001 0.035

2 64 tanh 0.99 0.95 3e-4 3e-4 107 2e4 10 512 0.2 0.2 [0,2e-2] N/A N/A N/A 0.001 0.035

2 64 tanh 0.99 0.95 3e-4 3e-4 107 2e4 10 512 20 0.2 [0,2e-2] N/A N/A N/A N/A N/A

2 64 tanh 0.99 0.95 3e-4 3e-4 107 2e4 10 512 10 0.2 [0,2e-2] N/A N/A N/A N/A N/A

2 64 tanh 0.99 0.95 3e-4 3e-4 107 2e4 10 512 N/A 0.2 [0,2e-2] N/A N/A N/A 0.001 0.035

2 64 tanh 0.99 0.95 3e-4 3e-4 107 2e4 10 512 N/A 0.2 [0,2e-2] N/A N/A N/A 0.001 0.01

2 64 tanh 0.99 0.95 3e-4 3e-4 107 2e4 10 512 N/A 0.2 [0,2e-2] N/A N/A N/A 1.0 0.01

2 64 tanh 0.99 0.95 3e-4 3e-4 107 2e4 10 128 N/A N/A [0,1e-2] 15 0.1 15 N/A N/A

2 64 tanh 0.99 0.95 3e-4 3e-4 107 2e4 10 512 N/A 0.2 [0,2e-2] N/A N/A N/A 0.001 0.035

2 64 tanh 0.99 0.95 N/A 1e-3 107 2e4 10 128 N/A N/A [0,1e-2] 15 0.1 15 N/A N/A

2 64 tanh 0.99 0.95 N/A 1e-3 107 2e4 10 128 N/A N/A [0,1e-2] 15 0.1 15 N/A N/A

2 64 tanh 0.99 0.95 N/A 1e-3 107 2e4 10 256 N/A N/A [0,1e-2] 10 0.1 10 N/A N/A

Table 4. Choice of fractional reduction factor α in each task.

α

Point Goal

Point Button

Car Goal

Car Button

Ant

Humanoid

HalfCheetah

Hopper

Swimmer

0.3

0.3

0.3

0.3

0.85

0.85

0.85

0.85

0.85

Justification of Remark 4.3. Assuming the inner-loop iterates remain within a trust region around θk , i.e., for a sufficiently small neighborhood radius ϵ: ∥θt − θk ∥ ≤ ε, and that ∇g is Lg -Lipschitz on this region. Then ∥∇g(θt ) − ∇g(θk )∥ ≤ Lg ε, which implies ∥∇g(θk )∥ − Lg ε ≤ ∥∇g(θt )∥ ≤ ∥∇g(θk )∥ + Lg ε. 2

With wk = 1/∥∇g(θk )∥ , it follows that (∥∇g(θk )∥ + Lg ε)2 (∥∇g(θk )∥ − Lg ε)2 ≤ wk ∥∇g(θt )∥2 ≤ . 2 ∥∇g(θk )∥ ∥∇g(θk )∥2 Hence, for sufficiently small ε, wk ∥∇g(θt )∥2 remains close to 1 throughout the inner loop, so the sufficient decrease condition in Proposition 4.2 is well-approximated by the rule of thumb g(θt ) ≳ δ/α.

F. Training curves and parameters In this section we report the training parameters used in the experiments for CSPO and the other baselines, as well as the full training curves. Table 3 shows the hyperparameters used for all baselines, and Table 4 shows the choice of α for CSPO in each task. Training curves are shown in Figure 6 for navigations tasks and Figure 7 for locomotion.

G. Additional implementation details Stabilizing the sensitivity weight w. CSPO uses the sensitivity weight wk ∝ 1/∥∇θ ĝ(θk )∥2 computed from automatic differentiation of the surrogate constraint ĝ. In practice, raw gradient-norm estimates can be noisy under minibatch sampling and PPO-style clipping. To improve numerical stability, we apply three standard safeguards when computing w. (i) ϵ-regularization.

We add a small constant ϵ > 0 to avoid division by zero: wraw =

1 , ∥∇θ ĝ(θk )∥2 + ϵ 16

Cost

Return

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

SafetyPointGoal1-v0

25 20 15 10 5 0

10 0 80

60

60

40

40

20

20 0

CSPO

SafetyCarButton1-v0

10.0 7.5 5.0 2.5 0.0 2.5

20

80

0

SafetyCarGoal1-v0

30

SafetyPointButton1-v0 15 10 5 0 100 80 60 40 20

150 100 50

200 Epoch

400

0

200 Epoch

400

APPO

P3O

PPO-Lag

CPO

CUP

0

0

200 Epoch

FOCOPS

PCPO

400

0 EPO

200 Epoch

CPPOPID

400

C-TRPO

IPO

Figure 6. Training curves of CSPO and baselines over 5 seeds on 4 Safety Gymnasium navigation tasks. The x-axis shows training epochs, the y-axis return or cost; solid lines denote the mean, shaded areas the standard deviation, and the dashed line the cost threshold (25).

Return

SafetyAntVelocity-v1

SafetyHalfCheetahVelocity-v1

3000

3000

2000

2000

1000

1000

Cost

1500 1000 500

0

0

SafetyHopperVelocity-v1 2000

0

70 60 50 40 30 20 10 0

75

100

50

75

25

50

0

25

25

0

CSPO

APPO

400

0

P3O

200 Epoch

400

PPO-Lag

CPO

0

200 Epoch

CUP

SafetySwimmerVelocity-v1 150 100 50 0 250

60 50 40 30 20 10 0

100

125

200 Epoch

SafetyHumanoidVelocity-v1

125

150

0

7000 6000 5000 4000 3000 2000 1000 0

200 150 100 50 0

400

0

200 Epoch

FOCOPS

PCPO

EPO

400

0

CPPOPID

200 Epoch

C-TRPO

400

IPO

Figure 7. Training curves of CSPO and baselines over 5 seeds on 5 Safety Gymnasium locomotion tasks. The x-axis shows training epochs; y-axis return or cost; solid lines denote the mean, shaded areas the standard deviation, and the dashed line the cost threshold (25).

Algorithm 2 Stabilized sensitivity weight computation for CSPO 1: Input: rollout batch τ ∼ πθk , cost advantages ÂC , old log-probabilities log πθk (a|s), EMA state w̄k−1 , smoothing

coefficient β ∈ [0, 1), ϵ > 0, clipping bounds wmin , wmax .

2: Compute PPO ratio: rθ (s, a) = exp log πθ (a|s) − log πθk (a|s)

h

i

3: Define surrogate constraint-gradient signal : ĝ(θ) ≜ E(s,a)∼τ rθ (s, a) ÂC (s, a) 4: Compute constraint gradient via automatic differentiation: i h

gθ ← ∇θ ĝ(θ) θ=θ

k

= E(s,a)∼τ ÂC (s, a) ∇θ log πθk (a|s)

5: Compute raw weight: wraw ← ∥g ∥12 +ϵw θ

6: Clip weight: wclip ← clip(wraw , wmin , wmax ) 7: EMA smoothing: w̄k ← β w̄k−1 + (1 − β) wclip 8: Detach and return stabilized weight: wk ← stopgrad(w̄k ) 9: Output wk

17

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

(ii) Clipping.

We clamp wraw to a bounded interval to prevent extreme values in very flat or very steep regions: wclip = clip(wraw , wmin , wmax ) .

(iii) Exponential moving average (EMA).

We further smooth the clipped weight using an EMA to reduce the variance:

w̄k = β w̄k−1 + (1 − β) wclip ,

β ∈ [0, 1).

The smoothed weight w̄k is then used in the policy update, and we detach it from the computation graph (i.e., no gradients are backpropagated through w), so that the policy update treats w as a constant within the inner loop: wk := stopgrad(w̄k ). This ensures that the sensitivity scaling acts as a stable, violation-dependent normalization factor rather than introducing additional higher-order gradient terms. Algorithm 2 summarizes this process. Gradient of the surrogate constraint. The true constraint residual is JC (πθk ) − d, which is estimated from rollouts but is not differentiable with respect to θ since it depends on environment transitions. Therefore, we compute ∇θ ĝ(θk ) from a differentiable PPO-style surrogate that depends on θ only through the policy ratio rθ (s, a). In particular, at θ = θk we have rθk (s, a) = 1 while ∇θ rθ |θk = ∇θ log πθk (a|s), yielding a standard policy-gradient form for the surrogate constraint gradient.

H. Fixed vs adaptive fractional reduction factor α In addition to the fixed-α ablation in Section 5.4, we evaluate a simple violation-adaptive heuristic for the sensitivity coefficient. Motivated by Proposition 4.2, which indicates that the feasibility activation threshold scales as O(1/α), we increase α proportionally to the current constraint violation. Specifically, at epoch k we define ϕk = JC (πθk ) − d and set αk = clip([ϕk ]+ /(d + ε), 0, 1). Figure 8 compares fixed α = 0.3 against the adaptive schedule on 4 navigation tasks and Figure 9 compares fixed α = 0.85 against the adaptive schedule on 4 locomotion tasks. Overall, adaptive α yields comparable feasibility behavior, but does not consistently improve reward performance over a well-tuned fixed α.

I. Sensitivity to dual hyperparameters We analyze the sensitivity of CSPO to the initialization and learning rate of the Lagrange multiplier and compare it against PPO-Lag on representative navigation and locomotion tasks. Learning curves are shown in Figures 10 - 11. Initialization of the multiplier. Varying the initial multiplier value λ0 (Figure 10), we observe that for CSPO, large initial values (e.g., λ0 = 0.1) induce conservative early behavior, resulting in lower returns and persistent under-utilization of the safety budget. In contrast, smaller values (λ0 ∈ [0.001, 0.05]) consistently converge to similar final returns and constraint satisfaction levels. PPO-Lag exhibits markedly higher sensitivity: large λ0 often leads to overly conservative policies, while small λ0 can cause prolonged constraint violations or oscillatory cost behavior. These results indicate that CSPO recovers more reliably from unfavorable multiplier initialization, reflecting improved robustness rather than complete insensitivity. Learning rate of the multiplier. We further vary the multiplier learning rate ηλ (Figure 11). CSPO remains stable across a broad range of values, with changes in ηλ primarily affecting transient dynamics rather than final performance. In contrast, PPO-Lag shows stronger coupling between ηλ and training stability, manifesting as larger cost oscillations and increased sensitivity to tuning. This behavior is consistent with the design of CSPO: sensitivity-aware correction absorbs part of the feasibility restoration that would otherwise rely solely on the dual update, reducing the burden on the multiplier dynamics and improving robustness to its step size.

J. Extended Related works J.1. Preliminary We briefly summarize the CMDP formulation and the constrained RL problem discussed in Section 2. A CMDP augments m an MDP with m cost functions {ci }m i=1 and corresponding limits {di }i=1 . For a stationary policy π, the discounted cost 18

Return

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

SafetyPointGoal1-v0

25 20 15 10 5 0

SafetyCarGoal1-v0

30

SafetyCarButton1-v0 1

20

0

10

1

Cost

20 0

0

100

50

80 40

40

60 40

30

20

60

40

5

3

80 60

10

2

0

SafetyPointButton1-v0

20

20

100 200 300 400 500 Epoch

0

100 200 300 400 500 Epoch = 0.3

0

100 200 300 400 500 Epoch

0

100 200 300 400 500 Epoch

= adaptive

Figure 8. Fixed vs. adaptive α in CSPO on four navigation tasks across 5 seeds. We compare CSPO with a fixed (α = 0.3) against a simple violation-adaptive schedule (Appendix H). Top: episodic return. Bottom: episodic cost with the cost limit shown as a dashed line. Curves show mean across seeds and shaded regions indicate variability.

SafetyAntVelocity-v1

Return

3000 2000

SafetyHalfCheetahVelocity-v1 1500

6000

1000

1000

4000

500

2000

0 50

30

0

40

Cost

30

10 0

0

200 Epoch

400

20

30

20

20

10

10

0

0

40

30

20

SafetyHumanoidVelocity-v1

2000

1000 0

SafetyHopperVelocity-v1

10

0 0

200 Epoch

400

0

= 0.85

= adaptive

200 Epoch

400

0

0

200 Epoch

400

Figure 9. Fixed vs. adaptive α in CSPO on four locomotion tasks across 5 seeds. We compare CSPO with a fixed (α = 0.3) against a simple violation-adaptive schedule (Appendix H). Top: episodic return. Bottom: episodic cost with the cost limit shown as a dashed line. Curves show mean across seeds and shaded regions indicate variability.

19

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

PointGoal

25

Ant 3000

15

Return

Return

20 10

2000 1000

5 0

0 0

100

200

300

400

500

100

200

0

100

200

300

400

500

300

400

500

40

80

30

Cost

60

Cost

0

40

20 10

20 0

100

200

Epoch

300

0 = 0.1

400 0 = 0.01

0

500 0 = 0.05

0 = 0.001

CSPO

Epoch

PPO-Lag

Figure 10. Training curves of CSPO vs PPO-Lag across four different initializations of the Lagrange multiplier λ.

PointGoal

25

Ant 3000

15

Return

Return

20 10

2000 1000

5

0

0 0

100

200

300

400

500

0

100

200

0

100

200

300

400

500

300

400

500

40 60

30

Cost

Cost

50 40 30

20 10

20 0

100

200

Epoch = 0.2

300

400 = 0.12

0

500 = 0.07

= 0.035

CSPO

Epoch

PPO-Lag

Figure 11. Training curves of CSPO vs PPO-Lag across four different learning rates η for the Lagrange multiplier λ.

20

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning Table 5. Recovery metrics across algorithms and environments. TTS: time to safety (epochs). RP: reward preservation (Jafter /Jbefore ). #V: Number of violations. Values are mean ± std across seeds. Environment

PointGoal

PointButton

CarGoal

CarButton

Ant

Humanoid

HalfCheetah

Hopper

Swimmer

CSPO

PPO-Lag

APPO

FOCOPS

CUP

CPPOPID

TTS

3.098 ± 0.421

7.110 ± 3.081

4.089 ± 0.383

21.452 ± 18.541

9.336 ± 8.728

6.477 ± 1.515

RP

1.000 ± 0.003

1.002 ± 0.014

0.996 ± 0.003

0.9889 ± 0.0074

0.9792 ± 0.0361

0.9928 ± 0.0166

#V

66.4 ± 23.29

111.6 ± 30.63

116.33 ± 26.83

214.5 ± 36.37

98.2 ± 109.33

143.6 ± 21.89

TTS

3.703 ± 0.411

5.144 ± 2.14

5.155 ± 0.695

9.479 ± 5.699

12.208 ± 5.236

5.041 ± 2.158

RP

0.961 ± 0.014

0.980 ± 0.016

0.942 ± 0.007

0.9984 ± 0.0073

0.9732 ± 0.0238

1.0132 ± 0.0934 124.6 ± 14.81

#V

104.0 ± 20.02

82.6 ± 28.27

145.0 ± 11.37

130.2 ± 17.61

126.0 ± 16.05

TTS

4.204 ± 0.611

7.240 ± 1.850

4.382 ± 0.651

39.515 ± 42.679

9.872 ± 7.085

6.131 ± 0.862

RP

1.000 ± 0.006

1.004 ± 0.006

0.994 ± 0.004

1.0318 ± 0.0684

1.0098 ± 0.0229

0.9932 ± 0.0171

#V

116.8 ± 13.718

141.2 ± 16.05

134.6 ± 19.32

200.8 ± 17.61

117.6 ± 15.65

148.0 ± 26.07

TTS

4.525 ± 0.270

11.834 ± 3.46

4.977 ± 0.626

11.372 ± 1.951

10.367 ± 5.576

12.260 ± 2.338

RP

1.005 ± 0.347

0.850 ± 0.259

1.051 ± 0.3247

0.800 ± 0.08

0.8464 ± 0.4363

0.8264 ± 0.0519

#V

116.33 ± 11.67

214.2 ± 27.97

141.8 ± 19.12

209.5 ± 36.24

160.5 ± 10.14

107.6 ± 13.50

TTS

5.151 ± 0.954

14.045 ± 3.210

4.591 ± 0.304

23.988 ± 4.726

21.388 ± 5.019

8.870 ± 0.398

RP

1.001 ± 0.001

0.9979 ± 0.0038

0.9975 ± 0.0010

1.0330 ± 0.0130

1.0229 ± 0.0202

1.0046 ± 0.0041 134.8 ± 14.72

#V

122.6 ± 22.04

122.0 ± 17.79

138.6 ± 8.20

150.4 ± 41.53

124.8 ± 13.10

TTS

4.867 ± 0.524

9.690 ± 1.706

4.188 ± 0.750

8.139 ± 2.425

6.267 ± 1.525

6.479 ± 0.318

RP

1.006 ± 0.005

1.0065 ± 0.0074

1.0001 ± 0.0009

1.0282 ± 0.0197

1.0575 ± 0.0590

1.0046 ± 0.0051 124.0 ± 10.93

#V

101.00 ± 23.56

140.4 ± 25.30

135.4 ± 11.39

79.00 ± 11.23

36.00 ± 5.71

TTS

8.229 ± 3.949

26.290 ± 15.492

3.206 ± 0.821

35.733 ± 19.674

30.533 ± 3.620

9.323 ± 2.197

RP

1.003 ± 0.002

1.0075 ± 0.0045

0.9986 ± 0.0025

1.0249 ± 0.0286

1.0202 ± 0.0199

1.0018 ± 0.0015

#V

40.25 ± 22.11

110.2 ± 19.17

83.8 ± 33.40

116.88 ± 12.53

90.20 ± 17.13

118.4 ± 4.77

TTS

2.333 ± 1.040

20.656 ± 14.980

2.961 ± 1.373

14.111 ± 10.238

23.683 ± 8.318

9.479 ± 1.506

RP

1.002 ± 0.002

0.8188 ± 0.1797

0.9966 ± 0.0104

1.0041 ± 0.0093

1.0678 ± 0.1872

0.9337 ± 0.1296

#V

13.333 ± 5.77

83.2 ± 43.25

129.2 ± 39.25

110.8 ± 17.28

90.8 ± 8.31

114.22 ± 13.34

TTS

4.387 ± 2.030

8.656 ± 2.002

5.214 ± 2.120

77.500 ± 76.062

38.450 ± 20.613

12.796 ± 2.642

RP

0.9106 ± 0.9853

0.811 ± 0.179

1.001 ± 0.002

1.0135 ± 0.1580

1.0065 ± 0.0082

0.9915 ± 0.0235

#V

20.10 ± 5.35

28.0 ± 2.175

25.45 ± 4.45

50.33 ± 16.35

93.60 ± 11.38

116.45 ± 17.37

return is Jci (π) = Eτ ∼π

"∞ X

# t

γ ci (st , at , st+1 ) ,

(40)

t=0

and the feasible policy set is Πsafe = {π ∈ Π : Jci (π) ≤ di , ∀i}.

(41)

Safe reinforcement learning aims to maximize return subject to feasibility, π ⋆ = arg maxπ∈Πsafe J(π). Let dπ denote the normalized discounted state-visitation distribution of π. For any bounded function f , the performance difference lemma (Kakade & Langford, 2002) gives Jf (π ′ ) − Jf (π) =

  1 Es∼dπ′ , a∼π′ Aπf (s, a) , 1−γ

(42)

Applying (42) to the reward r and each cost ci allows rewriting the safe RL objective as an iterative policy search. For parametrized stochastic policies πθ (a | s) with θ ∈ Rd and a current policy πθk , the updated policy πθk+1 is obtained by maximizing the reward advantage while satisfying the cost constraints  πθ  πθk+1 = arg max Es∼dπθ , a∼πθ Ar k (s, a) (43) πθ

s.t.

Jci (πθk ) +

 πθ  1 Es∼dπθ , a∼πθ Aci k (s, a) ≤ di , 1−γ 21

∀i.

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

This formulation provides the common basis for primal–dual and trust-region methods for CMDPs, and will be used to analyze constraint violation dynamics and safety recovery behavior in later sections. J.2. Classical Primal-Dual Methods for CMDPs We recall the standard primal–dual approach for constrained reinforcement learning. We consider parameterized stochastic policies πθ and a single cost constraint Jc(θ) ≤ d. Defining the constraint function g(θ) := Jc (θ) − d, and the reward  πθ objective J(θ) = Es∼dπθ , a∼πθ Ar k (s, a) the constrained optimization problem can be written as min −J(θ) θ

s.t.

g(θ) ≤ 0.

The associated Lagrangian is L(θ, λ) = − J(θ) + λ g(θ),

λ ≥ 0.

(44)

Classical primal–dual methods seek a saddle point of (44) by alternating updates of the primal variable θ and the dual variable λ. Using first-order methods, the updates take the form   θk+1 = θk − ηθ ∇θ J(θk ) − λk ∇θ g(θk ) , (45)   λk+1 = λk + ηλ g(θk ) + , (46) where ηθ , ηλ > 0 are step sizes and [·]+ denotes projection onto the nonnegative orthant. In this formulation, the dual variable λ accumulates constraint violation over iterations: when g(θk ) > 0, λ increases and strengthens the influence of the constraint gradient in subsequent parameter updates. Constraint satisfaction is therefore enforced indirectly through the growth of λ, which balances reward minimization and feasibility. This parameter-space primal–dual scheme constitutes the canonical first-order approach for CMDPs and underlies many safe reinforcement learning algorithms, including PPO-Lag (Ray et al., 2019) and related methods. J.3. Augmented Lagrangian Methods for CMDPs Augmented Lagrangian methods extends primal-dual by adding a quadratic deviation penalty to the objective. A standard approach is to convert the inequality constraint into equality by introducing a nonnegative slack variable h(θ) = g(θ)+s, s ≥ 0 Consider a CMDP with a single expected cost constraint The augmented Lagrangian associated with this equalityconstrained problem is ρ Lρ (θ, s, λ) = −J(θ) + λh(θ) + h(θ)2 2 (47)
ρ
2 = −J(θ) + λ g(θ) + s + g(θ) + s 2 where λ ∈ R is the Lagrange multiplier and ρ > 0 is a penalty parameter. Minimizing (47) with respect to the slack variable admits a closed-form solution,   s⋆ (θ, λ) = − g(θ) − λρ + . Substituting s⋆ back into (47) yields the Powell–Hestenes–Rockafellar (PHR) form of the augmented Lagrangian, Lρ (θ, λ) = −J(θ) +

 2 1  λ + ρg(θ) + − λ2 , 2ρ

(48)

which penalizes constraint violations through a squared hinge term. The corresponding gradient with respect to the policy parameters θ takes the form n o
∇θ Lρ (θ, λ) = ∇θ J(θ) − 1 g(θ) ≥ − λρ λ + ρg(θ) ∇θ g(θ), 22

(49)

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

showing that augmented Lagrangian methods effectively activate the quadratic penalty early before violation happens, when the constraint g(θ) exceeds the − λρ threshold. Yielding a dynamic safety threshold that changes whenever the dual variable λ and the penalty factor ρ are updated. The dual variable is typically updated via,   λk+1 = λk + ρg(θk ) + , (50) so that the same penalty parameter ρ governs both the magnitude of the primal correction and the growth of the multiplier. Additionally, the penalty factor is also iteratively increased whenever the constraint violation does not decrease Augmented Lagrangian methods improve upon pure penalty approaches by preserving the original constrained optimum and, under suitable conditions, admitting exact solutions with finite ρ. However, in stochastic and nonconvex settings such as deep reinforcement learning, coupling the primal correction and the dual update through a shared penalty parameter can lead to aggressive multiplier growth and oscillatory behavior, often requiring careful task-dependent tuning of ρ. J.4. Penalty Methods for CMDPs Penalty methods enforce the constraint by incorporating it directly into the objective through a penalty term, yielding the unconstrained problem
Pρ (θ) = −J(θ) + ρ ϕ g(θ)
(51) = −J(θ) + ρ ϕ Jc (θ) − d , where ρ > 0 is a penalty coefficient and ϕ : R → R+ is a nonnegative penalty function satisfying ϕ(g) = 0 for g ≤ 0. Theoretical guarantees of penalty methods depend on the choice of ϕ. For smooth penalties such as quadratic functions, exact equivalence with the original constrained problem typically requires ρ → ∞. Finite exactness can be achieved only for specific nonsmooth penalties under additional regularity conditions. As a result, penalty methods often require large or problem-dependent penalty coefficients to enforce feasibility, which can amplify gradient variance and lead to overly conservative behavior in stochastic and nonconvex settings such as deep reinforcement learning. J.5. Related Safe RL Algorithms In this section we detail the optimization process and objective of the related safe RL algorithms discussed in Section 6, being categorized into primal-dual, augmented lagrangian, penalty and second-order trust region methods. J.5.1. CPO (ACHIAM ET AL ., 2017) Constrained Policy Optimization (CPO) formulates policy updates as a trust-region constrained optimization problem. Given the current policy πθk , CPO computes πθk+1 by solving h π i θ max Es∼dπθ , a∼πθ Ar k (s, a) k πθ ∈Πθ i h π 1 θ (52) s.t. Jc (πθk ) + Es∼dπθ , a∼πθ Ac k (s, a) ≤ d, k 1−γ D̄KL (πθ ∥ πθk ) ≤ δ. Since (52) is intractable, CPO applies local approximations. Using first-order Taylor expansions of the reward and cost objectives and a second-order approximation of the KL constraint, the update is approximated as the quadratic program max (θ − θk )⊤ g θ

s.t. (θ − θk )⊤ a ≤ −c,

(53)

⊤

(θ − θk ) H(θ − θk ) ≤ δ, where g = ∇θ J(θk ), a = ∇θ Jc (θk ), c = Jc (πθk ) − d, and H is the Fisher information matrix associated with the KL divergence. Problem (53) admits a closed-form solution via its dual. Let (λ⋆ , ν ⋆ ) denote the optimal dual variables associated with the trust-region and cost constraints, respectively. If the linearized problem is feasible, the update direction is θk+1 = θk +

1 −1 H (g − ν ⋆ a) . λ⋆ 23

(54)

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

Otherwise, the update follows the boundary of the cost constraint, r 2δ θk+1 = θk − H −1 a. ⊤ a H −1 a

(55)

The required matrix-vector products with H −1 are computed using conjugate gradient methods, yielding a natural-gradient step that enforces feasibility at each iteration. J.5.2. PCPO (YANG ET AL ., 2020) Projection-Based Constrained Policy Optimization (PCPO) reformulates constrained policy optimization as a two-stage procedure consisting of a reward improvement step followed by a projection onto the feasible set. Given the current policy πθk , PCPO first computes an intermediate policy by solving the unconstrained trust-region problem h π i θ πθk+ 1 = arg max Es∼dπθ , a∼πθ Ar k (s, a) k πθ ∈Πθ 2 (56) s.t. D̄KL (πθ ∥ πθk ) ≤ δ. If the resulting policy violates the cost constraint, PCPO performs a projection step to restore feasibility: πθk+1 = arg min D(πθ , πθk+ 1 ) πθ ∈Πθ

2

s.t. Jc (πθk ) +

i h π 1 θ Es∼dπθ , a∼πθ Ac k (s, a) ≤ d, k 1−γ

(57)

where D(·, ·) is a chosen divergence measure. Applying the same local approximations as in CPO—first-order expansions of the reward and cost objectives and a quadratic approximation of the trust-region constraint—yields the update s   2δ g ⊤ H −1 a + c −1 θk+1 = θk − H g − max 0, L−1 a, (58) g ⊤ H −1 g a⊤ L−1 a where g = ∇θ J(θk ), a = ∇θ Jc (θk ), c = Jc (πθk ) − d, H is the Fisher information matrix, and L = H when D is the KL divergence (or L = I for an ℓ2 metric). Unlike CPO, which solves a joint saddle-point problem, PCPO decouples reward optimization and constraint satisfaction via explicit projection, simplifying the update at the cost of enforcing feasibility only after the reward step. J.5.3. C-TRPO (M ILOSEVIC ET AL ., 2025) Constrained Trust Region Policy Optimization (C-TRPO) extends trust-region policy optimization by embedding safety directly into the trust region itself. Rather than enforcing constraints through penalties or dual variables, C-TRPO controls constraint satisfaction via a constraint-aware divergence. Given a safe policy πk ∈ Πsafe , C-TRPO computes the next policy by solving πk+1 = arg max Aπr k (π) π∈Π

s.t. D̄C (π∥πk ) ≤ δ,

(59)

where Aπr k (π) denotes the expected reward advantage and D̄C is a surrogate constraint divergence. The surrogate divergence is defined as D̄C (π∥πk ) = D̄KL (π∥πk ) + β D̄ϕ (π∥πk ),

(60)

where D̄KL is the standard discounted KL divergence and D̄ϕ is a Bregman-type divergence derived from the cost advantage Aπc k (π), approximating changes in expected cost to first order. The parameter β > 0 controls the influence of the constraint on the trust region geometry. 24

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

The optimization in (59) is solved using standard TRPO-style approximations: a linearization of the objective and a quadratic approximation of the divergence, leading to a second-order update computed via conjugate gradients. If the current policy is unsafe, C-TRPO switches to a recovery step that minimizes the cost under a standard KL trust region. Overall, C-TRPO can be interpreted as a trust-region method that incorporates constraint information directly into the policy-space geometry, yielding conservative but stable updates near the constraint boundary. J.5.4. FOCOPS (Z HANG ET AL ., 2020) First-Order Constrained Optimization in Policy Space (FOCOPS) formulates safe policy optimization directly in the non-parameterized policy space, avoiding second-order approximations while enforcing constraints via a primal–dual structure. Given the current policy πθk , FOCOPS first solves the constrained optimization problem in policy space h π i θ π ⋆ = arg max Es∼dπθ , a∼π Ar k (s, a) k π∈Π h π i 1 θ s.t. Jc (πθk ) + Es∼dπθ , a∼π Ac k (s, a) ≤ d, k 1−γ D̄KL (π ∥ πθk ) ≤ δ. When πθk is feasible, the solution in policy space admits a closed-form expression:  
1 πθk πθk (a|s) πθk ⋆ exp Ar (s, a) − νAc (s, a) , π (a|s) = Zλ,ν (s) λ

(61)

(62)

where Zλ,ν (s) is the normalizing partition function, and λ, ν ≥ 0 are dual variables obtained by solving min λν + ν b̃ + λEs∼dπθ , a∼π⋆ [log Zλ,ν (s)] ,

λ,ν≥0

k

b̃ = (1 − γ)(d − Jc (πθk )).

(63)

Since π ⋆ generally lies outside the parameterized policy class Πθ , FOCOPS performs a projection step back to the parametric space: 
 θk+1 = arg min Es∼dπθ KL πθ (·|s) ∥ π ⋆ (·|s) . (64) θ

k

FOCOPS thus implements a first-order primal–dual update entirely in policy space, replacing trust-region constraints with KL-regularized exponentiated updates and avoiding second-order optimization. J.5.5. CUP (YANG ET AL ., 2022) Constrained Update Projection (CUP) follows a two-step optimization scheme that alternates between a regularized policy improvement step and a constraint-aware projection step. Unlike second-order trust-region methods, CUP relies only on first-order information while explicitly enforcing feasibility. Given the current policy πθk , CUP first computes an intermediate policy πθk+ 1 by solving 2 ( ) h π i θk πθk+ 1 = arg max Es∼dλπ , a∼πθ AGAE (s, a) − αq Es∼dλπ [KL(πθk (·|s) ∥ πθ (·|s))] ,

Policy improvement.

2

πθ ∈Πθ

θk

(65)

θk

πθ

k where AGAE denotes the generalized advantage estimate, dλπθ is the λ-discounted state visitation distribution, and αq > 0 k controls the KL regularization strength.

Projection.

The intermediate policy is then projected back onto the constraint set by solving   πθk+1 = arg min D πθ , πθk+ 1 πθ ∈Πθ 2 h π i 1 θk s.t. Jc (πθk ) + Es∼dλπ , a∼πθ AGAE,c (s, a) θk 1 − γ̃ + βq Es∼dλπ [KL(πθk (·|s) ∥ πθ (·|s))] ≤ d, θk

25

(66)

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning πθ

k where AGAE,c is the cost advantage estimate, γ̃ is an effective discount factor, and βq > 0 controls the constraint regularization.

CUP can thus be interpreted as a first-order projection method that enforces constraint satisfaction by explicitly projecting policy updates onto a linearized feasible set, while maintaining stability through KL regularization. J.5.6. P3O (Z HANG ET AL ., 2022) Penalized Proximal Policy Optimization (P3O) formulates constrained policy optimization as an unconstrained problem by introducing an exact penalty constructed via hinge (ReLU) operators. Given a CMDP with constraints Jci (π) ≤ di , P3O optimizes the following penalized objective: θk+1 = arg min Es∼dπk , a∼πθ [−AπRk (s, a)] θ n X  
o +κ max 0, Es∼dπk , a∼πθ AπCki (s, a) + (1 − γ) JCi (πk ) − di ,

(67)

i

where κ > 0 is a penalty factor, AπRk and AπCki denote reward and cost advantages, and the hinge operator activates only when the corresponding constraint is violated. The penalty term vanishes when all constraints are satisfied, reducing (67) to standard policy optimization in the feasible region. Crucially, P3O interprets the hinge penalty as an exact penalty function. Specifically, it is shown that if λ̄ denotes the optimal Lagrange multiplier vector of the original constrained problem, then for any κ ≥ ∥λ̄∥∞ , the penalized problem (67) shares the same set of optimal solutions as the original constrained optimization problem. J.5.7. IPO (L IU ET AL ., 2020) Interior-point Policy Optimization (IPO) addresses constrained policy optimization by incorporating barrier functions directly into the objective, following the classical interior-point theory. Given constraints JCi (π) ≤ di , IPO introduces the shifted constraint ḡi (π) := JCi (π) − di , and replaces the hard feasibility requirement ḡi (π) ≤ 0 with a logarithmic barrier. The resulting unconstrained objective takes the form X
max LIPO (θ) = LCLIP (θ) + ϕ ḡi (πθ ) , (68) θ

i

where LCLIP denotes the PPO clipped surrogate objective and ϕ(x) =

1 log(−x), t

x < 0,

(69)

is a logarithmic barrier function with temperature parameter t > 0. The barrier term diverges to −∞ as ḡi (πθ ) ↑ 0, preventing iterates from crossing the constraint boundary. Increasing t improves the approximation to the hard feasibility indicator but simultaneously induces steeper gradients near the boundary. J.5.8. EPO (G AO ET AL ., 2024) Exterior Penalty Policy Optimization (EPO) formulates constrained policy optimization as an unconstrained problem using an exterior penalty function. Starting from the surrogate constrained objective, EPO defines the reward and constraint surrogates FR (π) = Es∼dπk , a∼π [AπRk (s, a)] ,

FC (π) = JC (πk ) +

Constraint violations are penalized through a learned penalty metric
2 Φ(FC+ (π)) = αFC+ (π) + (1 − α) FC+ (π) , 26

1 Es∼dπk , a∼π [AπCk (s, a)] − d. 1−γ

FC+ (π) = max{FC (π), 0},

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

where α ∈ [0, 1] balances linear and quadratic penalties. The resulting exterior penalty objective is max FR (π) −

π∈Πθ

1 Φ FC+ (π) , µ

(70)

with penalty factor µ > 0. EPO employs a Penalty Metric Network to adapt Φ(·) across near- and far-boundary regions of the constraint space, enabling stronger penalties for large violations and smoother corrections near feasibility. By solving a sequence of penalty problems with decreasing µt → 0, EPO guarantees monotonic reduction of constraint violations and convergence to a feasible optimum under suitable conditions. J.5.9. APPO (DAI ET AL ., 2023) Augmented Proximal Policy Optimization (APPO) follows exactly theAugmented Lagrangian methods described in  J.3. Using the local surrogate reward LπRk (π) = Es∼dπk , a∼π AπRk (s, a) and constraint surrogate ϕπcik (π) = Jci (πk ) +  π  1 k 1−γ Es∼dπk , a∼π Aci (s, a) − bi , APPO considers the equality-constrained reformulation ϕπcik (π) + pi = 0,

pi ≥ 0,

and constructs the augmented Lagrangian # "  2  2 X λ λ σ i max + ϕπcik (π), 0 − i2 , L(π, λ, σ) = −LπRk (π) + 2 σ σ i

(71)

where λi ≥ 0 are Lagrange multipliers and σ > 0 is a penalty factor. The resulting primal gradient takes the form ∇π L = −∇π LπRk (π) +

X  1 ϕπcik (π) ≥ − λσi

λi + σϕπcik (π) ∇π ϕπcik (π),

(72)

i

reducing oscillations and dual-lag with the early activation of the penalty when the constraint violation exceeds the threshold − σλ . APPO updates the multipliers via   λi,k+1 = λi,k + σ ϕπcik (πk ) + . (73) APPO establishes that, for sufficiently large but finite σ, the augmented problem is exact, i.e., it shares the same optimal solution set as the original constrained problem. This avoids the infinite or very large penalty factors required by classical penalty methods, while stabilizing constraint satisfaction through the quadratic deviation term. J.5.10. CPPOPID (S TOOKE ET AL ., 2020) Constraint-Controlled PPO with PID Lagrangian (CPPOPID) extends the classical primal–dual approach by modifying the dual update dynamics rather than the primal objective. Starting from the standard Lagrangian formulation with constraint g(π) = JC (π) − d ≤ 0, CPPOPID retains the Lagrangian objective L(π, λ) = J(π) − λg(π), but replaces the integral-only dual ascent update with a proportional–integral–derivative (PID) controller. To stabilize learning when λ becomes large, CPPOPID introduces a rescaled primal objective θ∗ (λ) = arg max θ

1 J(πθ ) − λJC (πθ ) , 1+λ

(74)

which preserves the direction of the Lagrangian gradient while normalizing its magnitude. Equivalently, defining u = λ 1+λ ∈ [0, 1], the policy gradient can be written as ∇θ L = (1 − u) ∇θ J(πθ ) − u ∇θ JC (πθ ). The Lagrange multiplier is updated using a PID control rule, X
λk = KP ∆k + KI ∆t + KD (∆k − ∆k−1 )+ + , t≤k

27

∆k = JC (πk ) − d,

(75)

(76)

CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning

where KP , KI , KD ≥ 0 are proportional, integral, and derivative gains. CPPOPID can thus be viewed as a primal–dual method with controlled dual dynamics: the primal update remains Lagrangian, while proportional and derivative terms in the multiplier update aim to reduce oscillations and overshoot caused by integralonly dual ascent but in return introducing more hyperparameters to tune.

28