Provably Safe, Yet Scalable Reinforcement Learning
arXiv:2606.14536v1 [cs.LG] 12 Jun 2026
Kai S. Yun MIT [email protected]
Zeyang Li MIT [email protected]
Navid Azizan MIT [email protected]
Abstract Safe reinforcement learning (RL) aims to learn policies that optimize rewards while satisfying constraints. Predominant approaches rely on soft-constrained policy optimization, which has achieved empirical success but does not provide formal safety guarantees for the learned policy. In contrast, methods with strict guarantees typically rely on explicit certificate functions, whose construction requires the direct synthesis and verification of control-invariant sets, a process that scales poorly with state dimension and often yields overly conservative behavior. In this paper, we present the Provably Safe, yet Scalable RL (PS2-RL) framework, a novel two-phase architecture for learning provably safe policies in a scalable manner, designed to overcome the key bottlenecks of prior methods. Rather than explicitly computing invariant sets, PS2-RL leverages a learned backup policy to forward-integrate the system dynamics, generating an implicit control-invariant set online. In the first phase, the backup policy is trained with our proposed safe-arrival value function, which characterizes the optimal backup policy for invariant-set construction. In the second phase, an RL policy is trained end-to-end through a differentiable projection layer that strictly enforces the safety guarantees induced by the learned backup policy. By maximizing the volume of the implicit controlinvariant set in the first phase, the resulting PS2 policy from the second phase is performant and scalable, while maintaining provable safety. Crucially, PS2-RL imposes no restrictions on the underlying RL algorithm and can be plugged into any existing training pipeline. We establish theoretical guarantees for the proposed framework and evaluate it on robotic control tasks with state dimensions up to 10, a regime in which prior provably safe RL methods struggle or become impractical.
1
Introduction
Reinforcement learning (RL) has achieved remarkable success in controlling complex robotic systems [24, 44, 27, 38]. However, the lack of safety guarantees still hinders the deployment of RL on real-world systems. To mitigate this issue, safe RL aims to learn policies that achieve high rewards while satisfying safety constraints. Predominant approaches use constrained policy optimization, such as Lagrangian-based methods, where constraints are typically imposed through either cost value functions or neural certificate functions [3, 1, 15, 48, 22, 39, 56, 33]. A key limitation of this line of work is that, although the learned policies may perform well empirically in some cases, they provide no formal guarantee of constraint satisfaction; as a result, catastrophic failures can still occur during deployment. Another line of work in safe RL seeks to learn provably safe policies [14, 54, 49, 2, 59, 11]. These methods typically require verified explicit certificate functions representing control-invariant sets, such as control barrier function (CBF) [4–6] or safety index (SI) [34, 51], whose synthesis and verification scale poorly with state dimension [10, 42, 36, 17]. Therefore, these certificate-based methods often yield overly conservative behavior [35, 13]. Correspondence to Zeyang Li ([email protected]).
Figure 1: Overview of the PS2-RL framework.
The backup control barrier function (BCBF) framework [13, 21] offers a promising way to avoid the fundamental difficulty of directly synthesizing a valid explicit control-invariant set. Its key idea is to enlarge a small, known safe invariant set by forward-integrating a deterministic backup policy. The implicit set induced by this rollout is control-invariant by construction, respects bounded actuator limits, and reduces the safety condition to relative degree one regardless of the underlying dynamics. Thus, safety can be specified in its original form without high-order-style augmentations [53, 46]. Crucially, this framework also preserves the convexity of the resulting safety-filter constraints, allowing them to be solved efficiently. However, BCBF shifts rather than eliminates the difficulty: it replaces the direct synthesis of an explicit control-invariant set with the design of a backup policy, but does not provide a principled way to obtain such a policy. The quality of the backup policy critically affects the size of the resulting invariant set. Hand-designed analytic policies, such as linear quadratic regulators (LQRs), are easy to certify but typically recover only a small portion of the full invariant set [13, 29]. In this paper, we use BCBF as a building block and address this limitation by training the safe-arrival component of the backup policy via RL. The learned policy steers the system into a small certified region around an equilibrium, where a closed-form linear controller takes over. The BCBF construction then automatically extends safety from the certified base set to the entire implicit invariant set, without requiring certification of the learned safe-arrival policy itself. We introduce PS2-RL (Provably Safe, yet Scalable RL), a two-phase framework for training provably safe control policies without excessive conservatism or performance sacrifice. A visual overview of our method is in Fig. 1. Phase I trains a safe-arrival policy with RL through a novel indicator-based safe-arrival value function, whose unique optimum induces the time-optimal policy that drives the system safely to a target set. Phase II trains a PS2 policy end-to-end through a controlinvariant layer (CIL), constructed from the learned backup policy in Phase I. On unicycle lane-keeping and powerloop tracking for a 10-dimensional quadrotor, PS2-RL achieves 100% safety across both training and deployment while exceeding the performance of all baselines. Our contributions are: • PS2-RL framework: A two-phase RL framework with formal safety guarantees and scalability to high-dimensional, input-constrained systems, without synthesizing an explicit invariant set. • Safe-arrival value function: A novel objective that encodes the time-optimal behavior of safely driving the system to a target set. We show that it admits a self-consistency condition and a Bellman equation, enabling the training of a neural backup policy with standard RL pipelines. The learned backup policy enlarges the implicit control-invariant set, empirically yielding substantially larger safe regions than analytic alternatives. • Control-invariant layer: A differentiable projection layer that enforces the BCBF constraints, enabling efficient end-to-end training of provably safe RL policies. Crucially, we show that the differentiable projection does not compromise the expressiveness of the policy network, preserving universal approximation. • Theoretical justifications: Comprehensive theoretical results for the proposed framework, including guarantees for each individual phase and their connection in the full PS2-RL pipeline. Related work. We group related studies into five categories: (i) safe RL via constrained policy optimization, (ii) safe RL with verified certificate, (iii) backup control barrier functions, (iv) RL for reach-avoid specifications, and (v) differentiable optimization layers. See App. B for details. 2
2
Preliminaries
2.1
Backup Control Barrier Functions
Throughout this work, we consider the following control-affine, continuous-time system: ẋ = f (x) + g(x)u, x ∈ X ⊆ Rn , u ∈ U ⊆ Rm (1) where x is the state, X is the state space, u is the control input, and U is a compact set of admissible inputs encoding actuator limits. The functions f and g are locally Lipschitz continuous. Given a state-feedback policy π : X → U, let Φπ : Rn × R → Rn denote the closed-loop flow map under fπ (x) := f (x) + g(x)π(x), where Φπ (x0 , t) is the state at time t starting from x0 := x(t0 ). Safety of the system (1) is specified with a safe set S := {x ∈ X : hS (x) ≥ 0} for a continuously differentiable hS : Rn → R, and enforced in the sense of control invariance [43, 6, 34]. Control barrier functions (CBFs) provide a differentiable certificate of control invariance via an inequality condition ∇h(x)⊤ (f (x) + g(x)u) + α(h(x)) ≥ 0 enforced online within a safety filter [4, 5]. The key challenge lies in the synthesis of a CBF whose zero-superlevel set is invariant. As discussed in Sec. 1 and App. B, classical tools for computing the explicit invariant set have various limitations. The backup control barrier function (BCBF) framework [13, 21] circumvents this by leveraging a deterministic backup policy πb : X → U and a small, known control-invariant base set B := {x : hB (x) ≥ 0} ⊆ S. Then, the T -time constrained backup-induced control-invariant set under πb is: Definition 2.1 (Backup-induced Control-Invariant Set). CT (πb ) is the set of all states from which πb recovers to B within the backup horizon T while keeping the trajectory in S: CT (πb ) = R (B, S, fπb , T ) := {x ∈ X : Φπb (x, T ) ∈ B ∧ Φπb (x, t′ ) ∈ S ∀t′ ∈ [0, T ]} . (2) CT (πb ) is control-invariant under πb with B ⊆ CT ⊆ S, and yields the BCBF, an implicit CBF: ′ hCT (x) = min ′ min {hS (Φπb (x, t ))} , hB (Φπb (x, T )) (3) t ∈[0,T ]
The non-smooth min in (3) is handled by enforcing the CBF inequality condition pointwise along the backup flow and at the horizon. The resulting BCBF constraints are sufficient for the original CBF condition on hCT [13, Prop. 2] and are always feasible by construction under control limits [13, Thm. 2], without ever requiring an explicit invariant set computation, making BCBF an ideal backbone for PS2-RL. We defer the explicit forms of CBF and BCBF constraints to Sec. 4.1.1 and App. C. 2.2
Reinforcement Learning
We model the control task as a Markov decision process (MDP) (X , U, F, r, γ) with deterministic transitions F : X × U → X , reward r : X × U → R, and discount factor γ ∈ (0, 1). A neural policy πϕ : X → U, parameterized by ϕ, maps states to control P∞inputs. The objective is to find a policy that maximizes expected discounted return J(πϕ ) = E[ k=0 γ k r(xk , uk )] for an initial-state distribution ρ0 , under closed loop dynamics xk+1 = F (xk , πϕ (xk )) with x0 ∼ ρ0 . The stateπ πϕ P∞ k action value function Qγ ϕ (x, u) := Ex,u k=0 γ r(xk , uk ) satisfies the self-consistency condition πϕ πϕ Qγ (x, u) = r(x, u) + γQγ (F (x, u), πϕ (F (x, u))) for policy πϕ . 2.3
Hard-Constrained Neural Networks
A core requirement of PS2-RL is to enforce hard constraints on a neural network’s output, while preserving differentiability for end-to-end training. We build on HardNet-CVX [41], which projects a neural network’s output onto a state-dependent convex set via a differentiable layer. In our notation, we refer to this construction as the constraint projection layer: Definition 2.2 (Constraint Projection Layer). Given an RL policy πϕ : X → U and a state-dependent convex set K(x) ⊆ U, the constraint projection layer P is defined as: 2
P(πϕ )(x) = arg min ∥u − πϕ (x)∥2 u
s.t.
u ∈ K(x).
(4)
This projection is computed by a differentiable convex optimization layer, allowing policy gradients to propagate through P during end-to-end training. Crucially, P inherits the universal approximation property of HardNet-CVX [41, Thm 12]. In Sec. 4.4, we instantiate K(x) with affine constraints derived from the BCBF framework to define PS2-RL’s control-invariant layer. 3
3
Problem Formulation
We now formalize our main problem: synthesize a reward-maximizing RL policy that is provably safe under control limits for every initial state of interest. First, we make the following connection: Remark 3.1. The system (1) is continuous time, while the RL policy is evaluated at sample times tk = k ∆t and its output is held by zero-order hold (ZOH). Accordingly, the transition map xk+1 = F (xk , uk ) introduced in Sec. 2.2 represents the exact ZOH dynamics of (1) over one sampling interval. Our theoretical guarantees are for this sampled-data discrete dynamics. Under standard Lipschitz regularity and a sufficiently fast control rate, the sampled-data set-invariance conditions guarantee that BCBF conditions enforced at sample instants imply continuous-time safety of the ZOH trajectory [45, 20, 13]. With this connection between continuous- and discrete-time established, we present our main problem: Problem 1 (PS2-RL Formulation). Given the control-affine system (1) and its ZOH discretization F , a safe set S ⊆ X , an admissible control input set U ⊆ Rm , an initial-state distribution ρ0 supported on X0 ⊆ S, a reward function r : X × U → R for task performance, and a discount factor γ ∈ (0, 1), find the policy π that solves "∞ # X (5) max J(π) = Ex0 ∼ρ0 , uk =π(xk ) γ k r(xk , uk ) π
s.t.
xk+1 =F (xk ,uk )
xk ∈ S,
uk ∈ U,
k=0
∀k ≥ 0, ∀x0 ∈ X0 .
We use Πsafe to denote the feasible policy class of Problem 1, which constrains each realized control input uk and the resulting state xk , i.e., the requirements xk ∈ S and uk ∈ U must hold pointwise in k, for every trajectory induced by π, and for every initial state in X0 . We require the learned policy for Problem 1 to be provably safe, i.e., to provably belong to Πsafe . Note that predominant safe RL algorithms based on constrained policy optimization [3, 1, 15, 48, 22, 39, 56, 33, 31] may adopt similar formulations, such as state-wise safe RL [58]. However, the policies learned by these methods generally do not come with formal guarantees of membership in Πsafe .
4
Provably Safe, yet Scalable RL
Problem 1 requires maximizing task return while guaranteeing safety under bounded control. PS2-RL decomposes this into two subproblems (Sec. 4.1). Then, we first train a safe-arrival policy using the safe-arrival value function to quickly drive the system to a certified base set while avoiding unsafe states (Secs. 4.2–4.3). Second, the resulting composed backup policy instantiates the control-invariant layer, a differentiable projection layer mapping inputs to the BCBF-admissible set. This enables end-to-end training of an RL policy through the safety constraints (Sec. 4.4). By enlarging the implicit invariant set before optimizing within it, PS2-RL achieves both high returns and guaranteed safety. All proofs are deferred to the Appendix. 4.1
Decomposing Safety and Performance from Problem 1
We decompose Problem 1 into two subproblems: a BCBF-constrained policy optimization for a fixed backup policy πb and horizon T (Sec. 4.1.1) and a safe-arrival set maximization problem for the backup policy itself (Sec. 4.1.2), and show their composition certifies an inner approximation of Problem 1 (Sec. 4.1.3). 4.1.1
BCBF-constrained policies under a fixed backup policy ∂Φ
(x,t)
b BCBF-admissible input set. Let Ψπb (x, t) = π∂x denote the sensitivity of the backup flow starting at x. The BCBF derivative terms at the current state under candidate input u are as follows:
⊤
DtS′ (x, u) := ∇hS (Φπb (x, t′ )) [Ψπb (x, t′ ) (f (x) + g(x)u) − fπb (Φπb (x, t′ ))] , ⊤
DTB (x, u) := ∇hB (Φπb (x, T )) Ψπb (x, T ) (f (x) + g(x)u) . 4
Given extended class-K∞ functions αS and αB , the BCBF-admissible input set is KBCBF (x; πb ) := u ∈ U : DtS′ (x, u) + αS (hS (Φπb (x, t′ ))) ≥ 0 t′ ∈ [0, T ],
(6) DTB (x, u) + αB (hB (Φπb (x, T ))) ≥ 0 . Given x, these constraints are affine in u since the backup rollout and sensitivity are computed under a fixed πb . Hence KBCBF (x; πb ) is convex whenever U is convex. Moreover, πb (x) is feasible for (6) for all x ∈ CT (πb ) [13, Thm. 2]. With the BCBF-admissible set, we present our first subproblem: Subproblem 1 (BCBF-Constrained Policy Optimization). Fix a backup policy πb and backup horizon T , and assume X0 ⊆ CT (πb ). Find a ϕ-parameterized policy πϕ that maximizes the RL objective while satisfying the BCBF-constraints: max J(πϕ ) s.t. πϕ (x) ∈ KBCBF (x; πb ) ∀x ∈ CT (πb ). (7) πϕ
Remark 4.1. In practice, we enforce the constraints on a finite relative-time mesh 0 = t′0 < t′1 < · · · < t′N = T . Under standard Lipschitz regularity [13] and Remark 3.1, sampled BCBF enforcement at instants tk implies continuous-time safety of the ZOH trajectory. 4.1.2
Learning the safe-arrival policy to enlarge the implicit invariant set
The conservatism in Subproblem 1 is governed by the backup-induced set CT (πb ): a small CT (πb ) gives the policy a narrow region in which to optimize. To enlarge this region without losing the BCBF guarantee, we anchor the construction of the backup policy with a certified base set and use a learnable safe-arrival policy only to steer states to that base set, as follows: πB (x), x ∈ B πb (x) = (8) πSA (x), x ∈ / B.
Here, πB : B → U is the base controller, which renders the base set B forward invariant under fπB , and πSA : X \ B → U is the safe-arrival policy, which is tasked with steering the system from a state outside B into B within the backup horizon T while staying in the safe set S. While the terminal pair (B, πB ) needs to be certified for forward invariance, the safe-arrival policy πSA does not.
Certified base set and base controller. Constructing a valid (B, πB ) is the prerequisite for a BCBF, and thus also for our PS2-RL framework. Here, we show that this construction is always possible and tractable for nonlinear dynamical systems under a mild condition. Crucially, this does not require computationally expensive synthesis or global search. Intuitively, constructing a small local invariant set around an equilibrium is much easier than synthesizing a globally valid explicit invariant set, which is often intractable for high-dimensional systems. Assumption 4.1 (Local Stabilizability). There exists x⋆ ∈ int(S) and u⋆ ∈ int(U ) such that: (a) (x⋆ , u⋆ ) is an equilibrium of (1), i.e., f (x⋆ ) + g(x⋆ )u⋆ = 0; and (b) the linearization (A, B) of (1) ⋆ ) about (x⋆ , u⋆ ), with A = ∂(f +gu and B = g(x⋆ ), is locally stabilizable. ∂x x=x⋆ Assumption 4.1 is standard in nonlinear control [28] and is satisfied by any continuously differentiable system in a neighborhood of an equilibrium (x⋆ , u⋆ ) that admits first-order-controllable linearization. Theorem 4.1 (Existence of a Certified Local Base Set). Under Assumption 4.1, there exists a linear feedback gain K ∈ Rm×n , a symmetric positive-definite matrix P ≻ 0, and a constant c̄ > 0 such that the base controller πB (x) = u⋆ − K(x − x⋆ ) and the sublevel set Bc = x ∈ X : (x − x⋆ )⊤ P (x − x⋆ ) ≤ c , c ∈ (0, c̄] together satisfy the following properties: (i) Bc ⊆ S; (ii) πB (x) ∈ U for every x ∈ Bc ; (iii) Bc is control-invariant under πB ; and (iv) every trajectory with x(0) ∈ Bc satisfies x(t) → x⋆ as t → ∞ and x(t) ∈ Bc for all t ≥ 0. Thm. 4.1 shows that any stabilizing linear feedback K (e.g., pole placement, LQR, etc.) followed by a sublevel-set check for input feasibility and S-containment (i.e., B ⊆ S) produces a valid (B, πB ). Hereafter, we fix one valid choice (B, πB ) for the base set and base controller produced by Thm. 4.1. With (B, πB ) fixed, CT (πb ) is governed by the safe-arrival policy πSA and the backup horizon T . While CT (πb ) monotonically increases with T in the set inclusion sense [13, Lemma 1], it also increases the number of constraints in KBCBF and could render the optimization problem inefficient. A more straightforward solution is to come up with a better πSA that can bring more states in S into B given a fixed horizon. To this end, we introduce our second subproblem, learning a safe-arrival policy πSA to maximize the safe-arrival set, and thereby the BCBF-induced invariant set CT . 5
Subproblem 2 (Safe-Arrival Set Maximization). Given a certified base set B and base controller πB from Thm. 4.1, a backup horizon T > 0, a compact design region Ω ⊆ S, and a finite reference θ measure µ on Ω \ B, find a θ-parameterized safe-arrival policy πSA : X \ B → U that maximizes the measure of states in Ω \ B from which it can safely arrive at B within T : πB (x), x ∈ B, θ max µSA πSA ; Ω, B := µ CT (πbθ ) ∩ (Ω \ B) s.t. πbθ (x) = (9) θ θ πSA (x), x ∈ / B. πSA 4.1.3
Certified decomposition guarantee
Subproblem 2 enlarges the backup-induced invariant set CT (πbθ ) over which safety is guaranteed, and motivates the safe-arrival value formulation (Sec. 4.2). Subproblem 1 then optimizes the task return inside the corresponding BCBF-admissible input set (6). Accordingly, for a fixed backup policy πb , we define the BCBF-feasible policy class as ΠBCBF (πb ) := {π : X → U : π(x) ∈ KBCBF (x; πb ) ∀x ∈ CT (πb )}. The following result formalizes this decomposition.
Theorem 4.2 (Certified Decomposition and Exactness of PS2-RL). Suppose (B, πB ) satisfies Thm. 4.1. Let πb⋆ be a solution of Subproblem 2, and let πϕ⋆ be a solution of Subproblem 1 with the fixed backup policy πb⋆ . Assume the exact BCBF constraints in (6) are enforced. Then: (i) ΠBCBF (πb⋆ ) ⊆ Πsafe and hence πϕ⋆ is feasible for Problem 1; (ii) πϕ⋆ is optimal among all policies certified by πb⋆ , i.e., J(πϕ⋆ ) = supπ∈ΠBCBF (πb⋆ ) J(π) ≤ supπ∈Πsafe J(π); and (iii) if there exists a ⋆ ⋆ globally optimal solution πsafe of Problem 1 such that πsafe ∈ ΠBCBF (πb⋆ ), then πϕ⋆ is also globally optimal for Problem 1. Thus, the two subproblems solve a certified inner approximation of Problem 1. They solve Problem 1 exactly whenever the learned backup policy induces a BCBF-feasible class containing an optimal safe policy. 4.2
Safe-Arrival Value Function
Subproblem 2 optimizes the backup policy to maximize the size of the induced implicit invariant set, thereby imposing fewer restrictions on the RL policy in Subproblem 1 and enabling potentially higher rewards. To this end, we introduce the safe-arrival value function, a novel indicator-style objective that quantifies a policy’s ability to safely return the system to the target set. Optimizing the safe-arrival value function encourages the policy to reach B quickly while avoiding the failure set F := X \ S, thereby enlarging CT (πb ). We define b(x) := 1B (x), f (x) := 1F (x), and c(x) := 1 − b(x) − f (x), so that c(x) = 1 precisely on the continuation set X \ (B ∪ F ). We refer to the policy optimized for this purpose as the safe-arrival policy πSA . Definition 4.1 (Safe-Arrival Value Function). For a deterministic safe-arrival policy πSA and discount factor β ∈ (0, 1), the discounted safe-arrival Q-function is: "∞ ! # k−1 X Y πSA πSA k QSA,β (x, u) := Ex,u β c(xτ ) b(xk ) , (10) k=0
τ =0
where x0 = x, u0 = u, xk+1 = F (xk , uk ), and uk = πSA (xk ) for k ≥ 1, with the empty product πSA SA interpreted as 1. The state-value function is VSA,β (x) := QπSA,β (x, πSA (x)). Qk−1 The factor τ =0 c(xτ ) is a “survival” gate: it is 1 only while the rollout remains in the continuation set, and becomes 0 immediately after the trajectory enters either B or F. Hence (10) rewards exactly the first safe arrival to B, discounted by how many steps it takes. Under deterministic dynamics, (10) equals β N if the rollout reaches B safely in exactly N steps, and 0 otherwise. Consequently, maximizing the safe-arrival value function trains a policy that first prefers safe arrival over failure, and among safe-arrival policies prefers those that reach B sooner. Importantly, safe-arrival value satisfies the following self-consistency condition: SA SA QπSA,β (x, u) = b(x) + β c(x) QπSA,β (F (x, u), πSA (F (x, u))) ,
(11)
as well as a corresponding Bellman equation. This structure enables the safe-arrival policy to be trained using standard RL pipelines. The full details and additional properties of the safe-arrival value function are deferred to App. D. 6
4.3
Phase I: Training the Safe-Arrival Policy
Phase I addresses Subproblem 2 (Fig. 1, left panel) by training a parameterized safe-arrival policy θ πSA with the discounted Bellman recursion (11). Letting ρarr be a design distribution supported on Ω \ B, we optimize the following objective: h θ i πSA θ max JSA,β (θ) := Ex∼ρarr QSA,β (x, πSA (x)) . (12) θ
The general Phase I training procedure and algorithm are in App. E.6. Note that the only ingredients specific for the safe-arrival policy training are the indicators b, f , c, first-hit termination at B and F, and (12). Everything else can be supplied by a chosen RL backbone. Hence, the safe-arrival policy training can be instantiated with actor-critic, Q-learning, and related methods. 4.4
Phase II: Training with the Control-Invariant Layer
⋆ Phase II solves Subproblem 1 (Fig. 1, right panel). With πSA , the safe-arrival policy trained from Phase I, we define the fixed composed backup policy by setting πb⋆ (x) = πB (x) for x ∈ B and ⋆ πb⋆ (x) = πSA (x) otherwise. With πb⋆ fixed, the backup-induced invariant set CT (πb⋆ ) and the BCBFadmissible set KBCBF (x; πb⋆ ) are fixed as well. Assuming U is convex, KBCBF (x; πb⋆ ) is convex for each x ∈ CT (πb⋆ ) because the BCBF constraints are affine in u under a fixed backup flow. Extending from the constraint projection layer (Def. 2.2), we define the control-invariant layer as follows. Definition 4.2 (Control-Invariant Layer). Given the fixed composed backup policy πb⋆ and backup horizon T , the control-invariant layer is the state-dependent projection operator PCIL (πϕ )(x) := arg min ∥u − πϕ (x)∥22 s.t. u ∈ KBCBF (x; πb⋆ ). (13) u
(13) is essentially the BCBF-QP written as a differentiable projection layer. Note that for a relativetime mesh of N + 1 nodes (Remark 4.1), this QP has at least N + 2 + 2m affine constraints in u. The constraint count thus grows linearly with the backup horizon, which precludes a closed-form solution and motivates the usage of a differentiable QP solver (App. E.7). We now define our PS2 policy: Definition 4.3 (PS2 Policy). For πϕ : X → U, the PS2 policy is ϕ πPS2 (x; πb⋆ ) := PCIL (πϕ )(x).
(14)
This reparameterization turns Subproblem 1 into unconstrained optimization over policy parameters: ϕ maxϕ J(πPS2 (·; πb⋆ )). Because the projection layer is differentiable, gradients propagate end-to-end through (13). Hence Phase II is agnostic to the RL backbone with the only architectural modification being the control-invariant layer appended to the policy output. Regardless of the stochasticity of the policy output, the projection is applied deterministically. Therefore, PS2-RL guarantees safety in both training and deployment, and is expressive as shown in the following theoretical results. Theorem 4.3 (Safety Guarantee of PS2-RL). Suppose (B, πB ) satisfies Thm. 4.1, let πb⋆ be the fixed composed backup policy, and assume X0 ⊆ CT (πb⋆ ). Then for any policy πϕ , the corresponding ϕ ϕ PS2 policy satisfies πPS2 ∈ ΠBCBF (πb⋆ ). By Thm. 4.2, πPS2 ∈ Πsafe and is therefore feasible for Problem 1. Every realized input satisfies uk ∈ U and the sampled closed-loop trajectory satisfies xk ∈ S for all k ≥ 0 and all x0 ∈ X0 . Under the sampled-data conditions, the continuous-time ZOH trajectory remains in S as well. Theorem 4.4 (Universal Approximation of PS2-RL). Fix πb⋆ , assume U is convex, and let Ω ⊆ CT (πb⋆ ) be compact. Let C(Ω, U ) denote the space of continuous maps Ω → U with norm ∥π∥∞ := supx∈Ω ∥π(x)∥∞ . For function classes GNN , G ⊆ C(Ω, U ), assume GNN universally approximates G, i.e., for every π ∈ G and ϵ > 0, there exists πϕ ∈ GNN such that ∥π − πϕ ∥∞ < ϵ. Define the class of PS2 policies obtained by composing (13) with GNN -class policies as ⋆ ΠGPNN S2 (πb ; Ω) := {PCIL (πϕ ) : πϕ ∈ GNN } , and the BCBF-feasible policy class on Ω as ΠBCBF (πb⋆ ; Ω) := {π ∈ G : π(x) ∈ KBCBF (x; πb⋆ ) ∀x ∈ Ω} .
(15)
(16)
⋆ ⋆ 1 Then ΠGPNN S2 (πb ; Ω) universally approximates ΠBCBF (πb ; Ω) under ∥ · ∥∞ .
1 The same statement holds in Lp (Ω, U ) by the corresponding Lp version of [41, Thm. 12], where Lp (Ω, U ) denotes R p-integrable maps with norm ∥π∥p = ( Ω ∥π(x)∥pp dx)1/p .
7
Together, Thms. 4.3 and 4.4 show that the CIL enforces safety without sacrificing expressivity. Thm. 4.3 guarantees that every projected policy is BCBF-feasible and hence safe, while Thm. 4.4 shows that the projection preserves the approximation power of the chosen policy architecture over the BCBF-feasible class. Thus, PS2-RL combines hard safety guarantees with the policy expressivity needed for high-performance RL. 4.5
Scalability of PS2-RL
The scalability of PS2-RL comes from reducing provably safe RL to rollout-based operations and a control-space projection, rather than a global invariant set computation. From a practical standpoint, PS2-RL is simple to instantiate: hS can be a distance-to-obstacle margin, and the only component that must be explicitly certified is the local base pair (B, πB ), which is easily obtained via Thm. 4.1. The learned safe-arrival policy need not be certified separately, as once it is fixed, our framework extends the base set to the rollout-induced invariant set CT (πb⋆ ) by construction, avoiding direct synthesis or verification of a global invariant set. Moreover, the CIL remains small in decision dimension even when the state is high-dimensional. With N backup nodes, nS safety constraints, and nB terminal constraints, the CIL has m control variables and O(N nS + nB + 2m) affine rows.2 Thus, the number of constraints grows only linearly with the backup mesh size or the system dimension. In contrast, computing the global invariant set by discretizing the Hamilton-Jacobi reachability PDE incurs complexity that grows exponentially with the state dimension [10]. Furthermore, PS2-RL is agnostic to the RL backbone and the policy architecture: the CIL is appended after the output of πϕ , and Thm. 4.4 shows that this projection preserves universal approximation over feasible policies. Thus, unlike post-hoc DNN verification methods that are architecture-dependent [52, 12], PS2-RL is not limited to a specific network family. Finally, although our presentation uses an analytic control-affine model, PS2-RL does not fundamentally rely on a hand-derived symbolic decomposition f (x) + g(x)u. More generally, given a differentiable simulator, backup rollouts and the sensitivities needed by the CIL can be computed by automatic differentiation for only the safety-relevant outputs. The differentiable-simulator variant is further discussed in App. E.5.
5
Experiments
We evaluate PS2-RL on two tracking tasks whose reference trajectories intentionally leave the safe set. The first is a low-dimensional unicycle lane-keeping task, where dense grid evaluation lets us visualize how the learned safe-arrival policy enlarges the backup-induced invariant set. The second is an agile 10-dimensional quadrotor powerloop task with body rate and thrust as control, which tests scalability under tight actuation and a high-relative-degree altitude constraint. In both cases, the reward is a negative weighted tracking cost. Full details and extended analyses are provided in App. F.1 and App. G.1 for the unicycle environment, and App. F.2 and App. G.2 for the quadrotor. θ Baselines and protocol. We compare PS2SA (PS2+πSA ) against seven baselines: (Pen) low/highpenalty SAC; (Con) CPO and SAC-Lag. [1, 22, 55]; and (PSRL) CBF-RL [54], MPS [11], and PS2ABP , where PS2ABP replaces the learned safe-arrival policy with an analytic one. CBF-RL uses a sum-of-squares CBF for unicycle and an HOCBF for quadrotor, and the filter is used for training but removed at evaluation, following [54]. We give MPS a longer recovery horizon than PS2-RL. Tables 1–2 report sampled-data metrics under each method’s controller. All methods are trained with 10 seeds and evaluated on 1,000 episodes per seed, and we report the IQM with 95% CIs and the aggregated safety. See App. F.3 for PS2-RL implementation details and App. F.4 for the baselines.
5.1
Unicycle: Lane-Keeping
The unicycle state is x = [y, v, ψ]⊤ , and inputs are acmd ∈ [−5, 5] m/s2 and rcmd ∈ [−1, 1] rad/s. The reference is a 20 sec sinusoid with amplitude 2.5 m and desired speed vdes = 5 m/s, while the safe set is S = {x : |y| ≤ 1.8, |ψ| ≤ π/3}. The base set is an LQR-induced ellipsoid about 2 For the explicit QP construction, see App. E.7.1.
8
θ x⋆ = [0, vdes , 0]⊤ . Dense safe-arrival evaluation on a 201×121×201 grid shows that the learned πSA enlarges the safe-arrival measure µSA from 0.23 for the analytic policy to 0.33. Figure 2 visualizes this larger induced set and the resulting less-conservative tracking behavior.
Figure 2: Unicycle lane-keeping. The learned safe-arrival policy enlarges the backup-induced invariant set compared to the one induced by the analytic policy, enabling PS2SA to track closer to the intentionally unsafe reference while remaining inside the lane constraints. Table 1 shows that PS2SA achieves 100% safety with the best lateral and heading tracking among methods that are also 100% safe. Compared with PS2ABP , the learned policy reduces lateral RMSE from 0.97 m to 0.52 m, showing that Phase I directly reduces downstream conservatism. The higher velocity RMSE of PS2SA is consistent with the task objective, which prioritizes lateral tracking over speed tracking through a larger reward weight. PS2SA therefore trades speed accuracy for much lower lateral and heading errors. SAC-Penhigh is also safe, but is substantially more conservative, while the low-penalty and CMDP baselines achieve tracking only by allowing frequent violations. CBF-RL and MPS have formal safety mechanisms, but under the evaluated sampled-data deployment they are not uniformly safe. Table 1: Unicycle lane-keeping experiment results across models trained with 10 different seeds. Safe RL
ψ (rad) (↓) IQM [95% CI]
Total Safety (↑) Safe Ep/Total Ep
Per-seed Safety (↑) IQM [95% CI]
Worst Viol. (↓) y (m)
Pen
SAC-Penlow SAC-Penhigh
0.53 [0.34, 0.96] 1.11 [0.99, 1.25]
2.14 [1.20, 3.11] 1.85 [1.26, 2.84]
0.19 [0.09, 0.38] 0.15 [0.13, 0.19]
0.0% [0.0%, 3.0%] 100% [100%, 100%]
8.12 0.00
SAC-Lag. CPO
0.52 [0.40, 0.66] 2.51 [1.66, 7.29]
1.22 [0.99, 1.54] 4.44 [3.05, 5.36]
0.12 [0.09, 0.14] 0.27 [0.20, 0.57]
1.8% ± 5.4% 100% ± 0.0%
Con
Safety Performance
v (m/s) (↓) IQM [95% CI]
16.7% [0.0%, 66.7%] 73.7% [32.5%, 99.6%]
1.03 76.94
PSRL
Tracking Performance (RMSE) y (m) (↓) IQM [95% CI]
CBF-RL MPS PS2ABP PS2SA
1.78 [1.75, 1.80] 0.68 [0.55, 0.78] 0.97 [0.97, 0.98] 0.52 [0.51, 0.54]
4.72 [3.69, 4.91] 0.48 [0.43, 0.53] 0.43 [0.35, 0.55] 0.83 [0.76, 0.88]
0.27 [0.24, 0.30] 0.17 [0.14, 0.19] 0.15 [0.14, 0.15] 0.11 [0.10, 0.11]
100% [99.6%, 100%] 97.8% [97.5%, 98.2%] 100% [100%, 100%] 100% [100%, 100%]
0.00 0.67 0.00 0.00
5.2
30.0% ± 45.8% 64.2% ± 40.2%
99.8% ± 0.003% 97.8% ± 0.005% 100% ± 0.0% 100% ± 0.0%
Quadrotor: Powerloop Maneuver
The quadrotor state is x = [p⊤ , v⊤ , q⊤ ]⊤ ∈ R10 , and the input is thrust acmd ∈ [0, 4g] and body rates ωcmd = [ωx , ωy , ωz ]⊤ ∈ [−18, 18]3 rad/s. The reference is a powerloop inspired by [26]: starting at 4.5 m/s, the quadrotor tracks a vertical loop of radius 1.5 m centered at [0, 0, 2]⊤ m while executing a 360◦ flip. The safe set is S = {x : pz ≤ 3}. Although the safety specification is a single altitude bound, the constraint is enforced through the full 10-dimensional dynamics and all actuations. Table 2 shows that PS2SA is the best-performing method among all policies with 100% safety. Relative to PS2ABP , it reduces all RMSEs by approximately 40–57%. Compared to MPS, it reduces p-RMSE by about 70% despite MPS being given a longer recovery horizon. The penalty and CBF-RL baselines still exhibit violations, and the CMDP baselines show the limitation of enforcing safety θ only in expectation. The learned πSA is especially important in this agile setting. Fig. 3 shows that θ πSA recovers from task-relevant near-ceiling states much better compared to the analytic policy. Quantitatively, the learned safe-arrival policy increases task-relevant recoverability from 60.9% to 85.9% over the analytic backup, with the largest gains near the ceiling (35.5% → 69.3%). This larger near-ceiling safe-arrival set explains the downstream tracking performance gap in Table 2. 9
θ Figure 3: Safe-arrival policies in action: (left) πSA learned policy and (middle) πSA analytic policy. The base set trajectory segments are controlled by the base controller πB . (Right) PS2 policies θ tracking the powerloop reference (dashed line): PS2SA with πSA and PS2ABP with πSA .
Table 2: Quadrotor powerloop experiment results across models trained with 10 different seeds. Safe RL
θq (rad) (↓) IQM [95% CI]
Total Safety (↑) Safe Ep/Total Ep
Per-seed Safety (↑) IQM [95% CI]
Worst Viol. (↓) z (m)
Pen
SAC-Penlow SAC-Penhigh
1.00 [0.72, 1.36] 1.14 [1.02, 1.48]
1.88 [1.51, 2.36] 2.24 [1.84, 2.63]
1.15 [0.74, 1.59] 1.54 [1.30, 1.79]
45.5% [7.2%, 89.7%] 100% [99.9%, 100%]
0.62 0.004
SAC-Lag. CPO
1.94 [1.70, 2.53] 10.62 [7.08, 13.17]
4.30 [3.97, 4.58] 11.55 [9.28, 13.94]
2.12 [1.77, 2.26] 2.17 [2.04, 2.36]
47.3% ± 44.6% 99.9% ± 6e-4%
Con
Safety Performance
v (m/s) (↓) IQM [95% CI]
64.3% [19.0%, 100%] 83.3% [33.3%, 100%]
1.79 2.41
PSRL
Tracking Performance (RMSE) p (m) (↓) IQM [95% CI]
CBF-RL MPS PS2ABP PS2SA
2.29 [1.71, 2.97] 1.99 [1.95, 2.05] 1.39 [1.30, 1.53] 0.60 [0.53, 0.70]
4.36 [4.00, 4.64] 4.07 [3.89, 4.21] 1.74 [1.62, 1.92] 0.96 [0.75, 1.18]
2.06 [1.80, 2.29] 1.38 [1.31, 1.51] 0.75 [0.71, 0.90] 0.45 [0.33, 0.53]
100% [99.7%, 100%] 100% [100%, 100%] 100% [100%, 100%] 100% [100%, 100%]
0.21 0.00 0.00 0.00
58.6% ± 43.9% 70.0% ± 45.8%
99.8% ± 0.005% 100% ± 0.0% 100% ± 0.0% 100% ± 0.0%
Ablations and computation. App. G.3 isolates the control-invariant layer’s role on the quadrotor task. Removing it after PS2-RL training drops safety from 100% to 0%, while adding it only as a post-hoc filter remains safe but yields 43–53% worse tracking than end-to-end PS2-RL training. App. G.4 shows that the computational overhead is modest. Safe-arrival policy training takes 14 minutes for unicycle and 40 minutes for quadrotor. PS2SA training takes 4.6 hours for unicycle and 13.6 hours for quadrotor, and inference takes 0.35 ms for unicycle and 0.80 ms for quadrotor. All training and deployment were run as single-GPU jobs on a node with one NVIDIA V100 GPU.
6
Conclusion and Limitations
In this work, we present PS2-RL, a two-phase framework for safe RL that learns a backup policy with the safe-arrival value function and then trains the task policy end-to-end through a differentiable control-invariant layer. By replacing explicit synthesis of control-invariant sets with learned backup rollouts and a safe control-space projection, PS2-RL is a scalable RL framework that attains hard safety guarantees while retaining strong task performance. Experiments on lane keeping and a 10-dimensional quadrotor powerloop show 100% safety for PS2-RL during deployment and substantially lower tracking error than the baselines, demonstrating its scalability, performance, and safety guarantees. Limitations. The formal guarantees of PS2-RL require a control-affine analytic dynamics model or a differentiable simulator. These assumptions are reasonable for provable safety: without prior knowledge of the system dynamics, formal guarantees for a neural-network policy are generally impossible. Extending PS2-RL to uncertain dynamics and perception-based constraints, as well as evaluating it on hardware, are important directions for future work.
10
References [1] Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 22–31. JMLR.org, 2017. [2] Mohammed Alshiekh, Roderick Bloem, Rüdiger Ehlers, Bettina Könighofer, Scott Niekum, and Ufuk Topcu. Safe reinforcement learning via shielding. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018. [3] Eitan Altman, Said Boularouk, and Didier Josselin. Constrained markov decision processes with total expected cost criteria. In VALUETOOLS 2019, page 191–192, New York, NY, USA, 2019. Association for Computing Machinery. [4] Aaron D Ames, Jessy W Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs with application to adaptive cruise control. In Decision and Control (CDC), 2014 IEEE 53rd Annual Conference on, pages 6271–6278. IEEE, 2014. [5] Aaron D. Ames, Xiangru Xu, Jessy W. Grizzle, and Paulo Tabuada. Control barrier function based quadratic programs for safety critical systems. IEEE Transactions on Automatic Control, 62(8):3861–3876, 2017. [6] Aaron D. Ames, Samuel Coogan, Magnus Egerstedt, Gennaro Notomista, Koushil Sreenath, and Paulo Tabuada. Control barrier functions: Theory and applications. In 18th European Control Conference, ECC 2019, pages 3420–3431. IEEE, 2019. [7] Brandon Amos and J. Zico Kolter. Optnet: differentiable optimization as a layer in neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 136–145. JMLR.org, 2017. [8] Brian D. O. Anderson and John B. Moore. Optimal control: linear quadratic methods. Prentice-Hall, Inc., USA, 1990. [9] MOSEK ApS. The MOSEK MATLAB API manual. Version 11.1.11, 2026. URL https://docs.mosek. com/latest/matlabapi/index.html. [10] Somil Bansal, Mo Chen, Sylvia Herbert, and Claire J. Tomlin. Hamilton-jacobi reachability: A brief overview and recent advances. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pages 2242–2253, 2017. [11] Osbert Bastani. Safe reinforcement learning with nonlinear dynamics via model predictive shielding. In 2021 American Control Conference (ACC), pages 3488–3494, 2021. [12] Robin A Brown, Edward Schmerling, Navid Azizan, and Marco Pavone. A unified view of SDP-based neural network verification through completely positive programming. In International conference on artificial intelligence and statistics, pages 9334–9355. PMLR, 2022. [13] Yuxiao Chen, Mrdjan Jankovic, Mario Santillo, and Aaron D. Ames. Backup control barrier functions: Formulation and comparative study. In 2021 60th IEEE Conference on Decision and Control (CDC), pages 6835–6841, 2021. [14] Richard Cheng, Gábor Orosz, Richard M. Murray, and Joel W. Burdick. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the ThirtyThird AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019. [15] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone. Risk-constrained reinforcement learning with percentile risk criteria. Journal of Machine Learning Research, 18(1):6070–6120, 2017. [16] Andrew Clark. A semialgebraic framework for verification and synthesis of control barrier functions. IEEE Transactions on Automatic Control, 70(5):3101–3116, 2025. [17] Hongkai Dai and Frank Permenter. Convex synthesis and verification of control-lyapunov and barrier functions with input constraints. In 2023 American Control Conference (ACC), pages 4116–4123. IEEE, 2023.
11
[18] Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1587–1596. PMLR, 2018. [19] Andrea Goertzen, Kaveh Alim, Youngjae Min, and Navid Azizan. HardNet++: Nonlinear constraint enforcement in neural networks. arXiv preprint arXiv:2604.19669, 2026. [20] Thomas Gurriet, Petter Nilsson, Andrew Singletary, and Aaron D. Ames. Realizable set invariance conditions for cyber-physical systems. In 2019 American Control Conference (ACC), pages 3642–3649, 2019. [21] Thomas Gurriet, Mark Mote, Andrew Singletary, Petter Nilsson, Eric Feron, and Aaron D. Ames. A scalable safety critical control framework for nonlinear systems. IEEE Access, 8:187249–187275, 2020. [22] Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort. In Proceedings of the 2020 Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Research, pages 1110–1120. PMLR, 2021. [23] Kai-Chieh Hsu, Vicenç Rubies-Royo, Claire Tomlin, and Jaime Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning. In Robotics: Science and Systems (RSS), 2021. [24] Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. Science Robotics, 4(26): eaau5872, 2019. [25] Zachary Jarvis-Wloszek, Ryan Feeley, Weehong Tan, Kunpeng Sun, and Andrew Packard. Some controls applications of sum of squares programming. In 42nd IEEE International Conference on Decision and Control, volume 5, pages 4676–4681 Vol.5. IEEE, 2003. [26] Elia Kaufmann, Antonio Loquercio, René Ranftl, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Deep drone acrobatics. RSS: Robotics, Science, and Systems, 2020. [27] Elia Kaufmann, Leonard Bauersfeld, Antonio Loquercio, Matthias Müller, Vladlen Koltun, and Davide Scaramuzza. Champion-level drone racing using deep reinforcement learning. Nature, 620:982–987, 2023. [28] Hassan K. Khalil. Nonlinear Systems. Prentice Hall, 3rd edition, 2002. [29] Taekyung Kim, Aswin D. Menon, Akshunn Trivedi, and Dimitra Panagou. Backup-based safety filters: A comparative review of backup cbf, model predictive shielding, and gatekeeper. arXiv: 2604.02401, 2026. [30] Milan Korda, Didier Henrion, and Colin N Jones. Convex computation of the maximum controlled invariant set for polynomial control systems. SIAM Journal on Control and Optimization, 52(5):2944–2969, 2014. [31] Zeyang Li and Navid Azizan. Safe multi-agent reinforcement learning with convergence to generalized nash equilibrium. arXiv preprint arXiv:2411.15036, 2024. [32] Zeyang Li, Chuxiong Hu, Weiye Zhao, and Changliu Liu. Learning predictive safety filter via decomposition of robust invariant set. arXiv preprint arXiv:2311.06769, 2023. [33] Zeyang Li, Chuxiong Hu, Yunan Wang, Yujie Yang, and Shengbo Eben Li. Safe reinforcement learning with dual robustness. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(12):10876–10890, 2024. [34] Changliu Liu and Masayoshi Tomizuka. Control in a safe set: Addressing safety in human-robot interactions. In Proceedings of the ASME 2014 Dynamic Systems and Control Conference (DSCC), volume 3, page V003T42A003. ASME, 10 2014. [35] Simin Liu, Changliu Liu, and John Dolan. Safe control under input limits with neural control barrier functions. In Proceedings of The 6th Conference on Robot Learning, volume 205 of Proceedings of Machine Learning Research, pages 1970–1980. PMLR, 2023. [36] Simin Liu, Kai S. Yun, John M. Dolan, and Changliu Liu. Synthesis and verification of robust-adaptive safe controllers. In 2024 European Control Conference (ECC), pages 2265–2272, 2024. [37] Johan Löfberg. Yalmip : A toolbox for modeling and optimization in matlab. In 2004 IEEE International Symposium on Computer Aided Control Systems Design, 2004. [38] Renzhi Lu, Jie Wang, Zonghe Shao, Ruijuan Chen, Lijun Zhu, Yuzhi Jiang, Yunyi Pang, Dongfang Liang, Yang Shi, and Han Ding. Deep reinforcement learning for real-world humanoid robot locomotion control with automatic reward learning. Research, 9:1123, 2026.
12
[39] Haitong Ma, Changliu Liu, Shengbo Eben Li, Sifa Zheng, and Jianyu Chen. Joint synthesis of safety certificate and safe control policy using constrained reinforcement learning. In Learning for Dynamics and Control Conference, pages 97–109. PMLR, 2022. [40] Kostas Margellos and John Lygeros. Hamilton–jacobi formulation for reach–avoid differential games. IEEE Transactions on Automatic Control, 56(8):1849–1861, 2011. [41] Youngjae Min and Navid Azizan. HardNet: Hard-constrained neural networks with universal approximation guarantees. arXiv preprint arXiv: 2410.10807, 2025. [42] Ian M. Mitchell, Alexandre M. Bayen, and Claire J. Tomlin. A time-dependent hamilton-jacobi formulation of reachable sets for continuous dynamic games. IEEE Transactions on Automatic Control, 50(7):947–957, 2005. [43] Mitio Nagumo. Über die lage der integralkurven gewöhnlicher differentialgleichungen. Proceedings of the Physico-Mathematical Society of Japan. 3rd Series, 24:551–559, 1942. [44] Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 91–100. PMLR, 2022. [45] Andrew Singletary, Yuxiao Chen, and Aaron D. Ames. Control barrier functions for sampled-data systems with input delays. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 804–809, 2020. [46] Xiao Tan, Wenceslao Shaw Cortez, and Dimos V. Dimarogonas. High-order barrier functions: Robustness, safety, and performance-critical control. IEEE Transactions on Automatic Control, 67(6):3021–3028, 2022. [47] Sunbochen Tang, Andrea Goertzen, and Navid Azizan. LMI-Net: Linear matrix inequality–constrained neural networks via differentiable projection layers. arXiv preprint arXiv:2604.05374, 2026. [48] Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2019. [49] Sander Tonkens and Sylvia Herbert. Refining control barrier functions through hamilton-jacobi reachability. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 13355–13362, 2022. [50] Kevin Tracy and Zachary Manchester. On the differentiability of the primal-dual interior-point method. arXiv preprint arXiv:2406.11749, 2024. [51] Tianhao Wei and Changliu Liu. Safe control algorithms using energy functions: A unified framework, benchmark, and new directions. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 238–243. IEEE, 2019. [52] Tianhao Wei, Hanjiang Hu, Luca Marzari, Kai S. Yun, Peizhi Niu, Xusheng Luo, and Changliu Liu. Modelverification.jl: A comprehensive toolbox for formally verifying deep neural networks. In Computer Aided Verification, pages 395–408. Springer Nature Switzerland, 2025. [53] Wei Xiao and Calin Belta. High-order control barrier functions. IEEE Transactions on Automatic Control, 67(7):3655–3662, 2022. [54] Lizhi Yang, Blake Werner, Massimiliano de Sa, and Aaron D. Ames. Cbf-rl: Safety filtering reinforcement learning in training with control barrier functions. arXiv preprint arXiv: 2510.14959, 2026. [55] Qisong Yang, Thiago D. Simão, Simon H. Tindemans, and Matthijs T. J. Spaan. Wcsac: Worst-case soft actor critic for safety-constrained reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, 35(12):10639–10646, 2021. [56] Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference on Machine Learning, pages 25636–25655. PMLR, 2022. [57] Kai S. Yun, Rui Chen, Chase Dunaway, John M. Dolan, and Changliu Liu. Safe control of quadruped in varying dynamics via safety index adaptation. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pages 7771–7777, 2025. [58] Weiye Zhao, Tairan He, Rui Chen, Tianhao Wei, and Changliu Liu. State-wise safe reinforcement learning: a survey. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI ’23, 2023. [59] Weiye Zhao, Feihan Li, Tairan He, and Changliu Liu. Implicit safe set algorithm for provably safe reinforcement learning. Journal of Artificial Intelligence Research, 84, 2025.
13
Appendix A Notations and Symbols
15
B Related Work
17
C Background on Safe Control Theory
18
C.1 Control Invariance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
18
C.2 Control Barrier Function . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
C.3 Backup Control Barrier Function . . . . . . . . . . . . . . . . . . . . . . . . . . .
19
D Safe-Arrival Value Function
21
D.1 Undiscounted Safe-Arrival Value . . . . . . . . . . . . . . . . . . . . . . . . . . .
21
D.2 Discounted Safe-Arrival Value . . . . . . . . . . . . . . . . . . . . . . . . . . . .
24
D.3 Behavior of Optimal Safe-Arrival Actions . . . . . . . . . . . . . . . . . . . . . .
26
E PS2-RL Details and Proofs
28
E.1 Base Set Certification Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28
E.2 PS2-RL Decomposition Proof . . . . . . . . . . . . . . . . . . . . . . . . . . . .
28
E.3 PS2-RL Safety Guarantee Proof . . . . . . . . . . . . . . . . . . . . . . . . . . .
29
E.4 PS2-RL Universal Approximation Proof . . . . . . . . . . . . . . . . . . . . . . .
29
E.5 Constructing the CIL from Differentiable Simulators . . . . . . . . . . . . . . . .
30
E.6 Phase I: Safe-Arrival Policy Training . . . . . . . . . . . . . . . . . . . . . . . . .
30
E.7 Phase II: Control-Invariant Layer and PS2 Policy . . . . . . . . . . . . . . . . . .
31
F Experiment Details
34
F.1
Unicycle Environment Details . . . . . . . . . . . . . . . . . . . . . . . . . . . .
34
F.2
Quadrotor Environment Details . . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
F.3
PS2-RL Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . .
35
F.4
Baseline Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . . .
39
G Additional Results, Ablation Studies, and Computation
42
G.1 Unicycle: Extended Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
42
G.2 Quadrotor: Extended Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . .
44
G.3 Ablation Studies: Role of the Control-Invariant Layer . . . . . . . . . . . . . . . .
45
G.4 Computation Details and Time . . . . . . . . . . . . . . . . . . . . . . . . . . . .
47
14
A
Notations and Symbols
For convenience, we summarize the notation used in the paper’s main text. Appendix-only proof variables, algorithmic implementation variables, and experimental-detail symbols that do not appear in the main text are omitted. Tables 3–6 group the remaining symbols by category, with the right column indicating where each symbol is first introduced. Table 3: Sets, spaces, and policy classes used in the main text. Symbol
Description
First used
R, Rn , Rm n, m X ⊆ Rn U ⊆ Rm S B CT (πb ) R(B, S, fπb , T )
Real numbers; n-dim. real space; m-dim. real space State and control-input dimensions State space of interest Compact admissible input set encoding actuator limits Safe set, S := {x ∈ X : hS (x) ≥ 0} Certified base set, B := {x : hB (x) ≥ 0} ⊆ S T -time backup-induced control-invariant set under policy πb States that reach B within horizon T under πb while remaining in S Failure set, F := X \ S Compact design region and finite reference measure for safearrival set maximization Initial-state support and initial-state distribution Space of continuous maps Ω → U endowed with the sup norm ∥π∥∞ := supx∈Ω ∥π(x)∥∞ Chosen pre-CIL policy architecture class and target policy class, with GNN assumed to universally approximate G Class of PS2 policies obtained by composing the CIL with policies in GNN Feasible policy class for Problem 1, enforcing xk ∈ S and uk ∈ U pointwise BCBF-feasible policy class certified by a fixed backup policy πb BCBF-feasible target policy subclass restricted to compact Ω
Sec. 2.1 Eq. (1) Eq. (1) Eq. (1) Sec. 2.1 Sec. 2.1 Def. 2.1 Def. 2.1
F Ω, µ X0 , ρ 0 C(Ω, U), ∥ · ∥∞ GNN , G ⋆ ΠGPNN S2 (πb ; Ω)
Πsafe ΠBCBF (πb ) ΠBCBF (πb⋆ ; Ω)
Sec. 4.2 Subproblem 2 Problem 1 Thm. 4.4 Thm. 4.4 Thm. 4.4 Problem 1 Sec. 4.1.3 Thm. 4.4
Table 4: Dynamics, time, and local-base-set quantities used in the main text. Symbol
Description
First used
x, u
Continuous-time state and control input; also state/action in the sampled MDP Time derivative of the state Drift of the control-affine system Control vector field of the control-affine system Closed-loop vector field under policy π, fπ (x) := f (x) + g(x)π(x) Closed-loop flow map under policy π from initial state x0 Backup flow map under πb Deterministic sampled/ZOH transition map, xk+1 = F (xk , uk ) Continuous time and relative backup-rollout time Discrete timestep and dummy product/summation index Sampling instant tk = k∆t for sampling period ∆t Backup horizon and backup-mesh size; for a mesh 0 = t′0 < · · · < t′N = T , there are N + 1 rollout nodes and T = N ∆t Sensitivity of the backup flow, ∂Φπb (x, t)/∂x Equilibrium state and input satisfying f (x⋆ ) + g(x⋆ )u⋆ = 0 Linearization matrices about (x⋆ , u⋆ ); distinct from the set B Linear feedback gain for the base controller πB (x) = u⋆ − K(x − x⋆ ) Positive-definite matrix defining the local ellipsoidal base set Base set level, base set upper-level, and local base set Bc = {x : (x − x⋆ )⊤ P (x − x⋆ ) ≤ c}
Eq. (1)
ẋ f : Rn → Rn g : Rn → Rn×m fπ (x) Φπ (x0 , t) Φπb (x, t) F t, t′ k, τ tk , ∆t T, N Ψπb (x, t) x⋆ , u⋆ A, B K P ≻0 c, c̄, Bc
15
Eq. (1) Eq. (1) Eq. (1) Sec. 2.1 Sec. 2.1 Def. 2.1 Sec. 2.2 Sec. 2.1, 4.1.1 Sec. 2.2, Eq. (10) Rem. 3.1 Def. 2.1, Sec. 4.3 Sec. 4.1.1 Assump. 4.1 Assump. 4.1 Thm. 4.1 Thm. 4.1 Thm. 4.1
Table 5: Policies, objectives, and value functions used in the main text. Symbol
Description
First used
π:X →U πϕ πb πB πSA θ πSA πbθ ⋆ πSA , πb⋆
Generic state-feedback policy Neural RL policy parameterized by ϕ Backup policy used to induce CT (πb ) Base controller that renders B forward invariant Safe-arrival policy used outside B in the composed backup policy Learned safe-arrival policy with parameters θ θ Composed learned backup policy induced by πB and πSA Trained safe-arrival policy and fixed composed backup policy used in Phase II PS2 policy obtained by projecting πϕ through the controlinvariant layer MDP reward function for task performance Expected discounted return objective
Sec. 2.1 Sec. 2.2 Sec. 2.1 Sec. 4.1.2 Eq. (8) Subproblem 2 Subproblem 2 Sec. 4.4
ϕ πPS2 (x; πb⋆ )
r(x, u) J(π), J(πϕ ) γ ∈ (0, 1) π Qγ ϕ (x, u) β ∈ (0, 1) πSA SA QπSA,β , VSA,β JSA,β (θ) ρarr θ µSA (πSA ; Ω, B)
Discount factor for the task RL objective State-action value function for the task RL objective under πϕ Discount factor for the safe-arrival value objective Safe-arrival Q- and state-value functions under πSA Safe-arrival objective optimized in Phase I Design distribution for Phase I safe-arrival training Measure objective for safe-arrival set maximization
Def. 4.3 Sec. 2.2 Sec. 2.2, Problem 1 Sec. 2.2 Sec. 2.2 Def. 4.1 Eq. (10) Eq. (12) Sec. 4.3 Eq. (9)
Table 6: Barrier, indicator, and projection quantities used in the main text. Symbol
Description
First used
hS , hB ∇h(x) α, αS , αB hCT DtS′ (x, u), DTB (x, u)
Differentiable set functions defining S and B Gradient of a differentiable barrier/certificate function Extended class-K∞ functions used in CBF/BCBF inequalities Implicit BCBF defining the backup-induced set CT BCBF derivative terms for the safe-set rollout and terminal baseset condition Generic state-dependent convex constraint set in a projection layer BCBF-admissible input set for a fixed backup policy πb Constraint projection layer based on HardNet-CVX Control-invariant layer, i.e., the BCBF-QP projection onto KBCBF (x; πb⋆ ) used by PS2-RL Euclidean norm used in projection objectives Indicators of the base set B and failure set F Indicators of B, F, and the continuation set, with c = 1 − b − f Numbers of scalar safe-set rollout constraints and terminal baseset constraints used in the finite-mesh CIL construction
Sec. 2.1 Sec. 2.1 Sec. 2.1, Eq. (6) Eq. (3) Sec. 4.1.1
K(x) KBCBF (x; πb ) P PCIL ∥ · ∥2 1B , 1F b(x), f (x), c(x) nS , nB
16
Def. 2.2 Eq. (6) Def. 2.2 Def. 4.2 Def. 2.2 Sec. 4.2 Sec. 4.2 Sec. 4.5
B
Related Work
In this section, we review prior work related to our proposed framework. We organize existing methods into five categories. Safe RL via Constrained Policy Optimization. This line of work imposes constraints from learned value functions through constrained policy optimization, such as Lagrangian methods. A widely adopted formulation is the constrained Markov decision process (CMDP) framework [3], where an auxiliary cost function is introduced and the expected trajectory cost is constrained to remain below a predefined threshold [1, 15, 48, 22]. More recently, state-wise safe RL methods [58] have gained attention: instead of enforcing safety in expectation through a cost value function, they learn a neural certificate function and impose it as a constraint to encourage state-wise safety [39, 56, 33]. However, neither class of methods can guarantee the safety of the learned policy. First, the constraints are provided by learned neural-network values, which are not provably reliable in general. Second, constrained policy optimization itself does not guarantee that the outputs of the policy network will satisfy the imposed constraints. Although these methods scale to expressive policies and high-dimensional systems, and can achieve good empirical performance in some cases, catastrophic failures may still occur during deployment due to the lack of formal safety guarantees. This gap motivates the study of provably safe RL. Safe RL via Verified Certificate. A second line of work attaches a formal safety certificate to the policy. Unlike neural certificate functions learned purely from loss functions, formal safety guarantees require the synthesis and verification of a valid certificate. Common certificate functions include control barrier function (CBF) [4–6], safety index (SI) [34, 51], and reachability value function [10, 42]. Although they take different forms, these certificates can be viewed as functional representations of control-invariant sets, characterizing states from which persistent safety can be maintained under an admissible controller. In safe RL, such certificates are typically used as safety filters or shields that modify unsafe actions proposed by a learned policy [14, 54, 49, 2, 59, 11]. The main challenge is that obtaining a valid certificate amounts to synthesizing or verifying a controlinvariant set, which poses significant scalability challenges. Hamilton–Jacobi (HJ) reachability analysis [10, 42] provides a formal mathematical framework for computing maximal control-invariant sets for nonlinear systems. However, it requires solving a nonlinear partial differential equation (PDE) over a discretized state space, whose computational cost scales exponentially with the state dimension. Sum-of-squares (SOS) optimization [25, 30, 57, 36, 17, 16] synthesizes polynomial safety certificates by relaxing polynomial nonnegativity conditions into SOS constraints, which can be solved via semidefinite programming. While effective for polynomial dynamics and semialgebraic safe sets, SOS methods scale poorly with state dimension and polynomial degree, and often produce conservative invariant sets that are much smaller than the maximal sets computed by HJ reachability when the latter is tractable. Compared to the above approaches, PS2-RL avoids the fundamental bottleneck of explicit invariantset synthesis by learning a backup policy that induces an implicit control-invariant set online. This implicit set is then used to provide formal safety guarantees while enabling scalability to higherdimensional systems. Backup Control Barrier Functions. Our framework builds on the backup control barrier function (BCBF) framework [13, 21, 45]. During online deployment, a BCBF certifies the set of states from which a fixed deterministic backup policy can return the system to a small base set within a finite horizon while remaining inside the safe set. This construction yields an implicit control-invariant set, whose associated safety-filter constraint is of relative degree one by construction and can explicitly account for input limits [6]. However, the backup policy is typically hand-designed, such as an LQR around an equilibrium [8, 28], and the resulting implicit invariant set is therefore often limited in size [29]. PS2-RL departs from this line of work in two key aspects. First, we learn the backup policy using the proposed safe-arrival value function, which characterizes time-optimal safe recovery to the base set. Second, the BCBF constraints are embedded into a control-invariant layer via differentiable projection [41, 50], enabling the RL policy to be trained end-to-end while preserving formal safety guarantees, rather than using the safety filter only as a post-hoc correction during deployment. 17
RL for Reach-Avoid Specifications. Optimal control for reach-avoid specifications [40] jointly captures the objective of reaching a target set while avoiding a failure set, making it a natural formulation for designing backup policies. However, classical optimal-control approaches for reachavoid tasks are computationally expensive and quickly become intractable for high-dimensional systems. More recently, RL-based methods have been proposed to solve reach-avoid problems with improved scalability, making them more tractable for complex dynamical systems [23, 32]. Hsu et al. [23] propose a time-discounted reach-avoid Bellman equation built from dense target and safety-margin functions, typically instantiated with signed-distance-like geometric quantities inside a nested max–min backup. Although this formulation captures reach-avoid semantics, the learned critic depends not only on the logical event of reaching the target before failure, but also on the choice, scale, and geometry of these task-specific shaping functions. This makes the optimization sensitive and difficult in practice: stable learning requires warm-starting with a standard RL objective and progressively annealing the discount factor. In contrast, our method learns an indicator-style probabilistic critic that directly represents the safe-arrival event. The critic is aligned with the underlying reach-avoid specification rather than an auxiliary dense geometric surrogate, leading to cleaner credit assignment and easier optimization. As a result, our RL implementation follows a standard training pipeline and learns effective policies without task-specific heuristics. Moreover, although our main theoretical development focuses on deterministic systems and deterministic backup policies, the safe-arrival value function is probabilistic by construction and naturally extends to stochastic dynamics and stochastic policies. Differentiable Optimization Layers. This line of work embeds optimization problems as differentiable layers inside neural networks. OptNet [7] differentiates through quadratic programs via implicit differentiation of the Karush–Kuhn–Tucker (KKT) conditions, enabling constrained optimization modules to be trained end-to-end. HardNet [41] provides a unified framework for efficiently training neural networks under general affine and convex constraints, with universal-approximation guarantees for networks equipped with such constrained layers. Recent variants include LMI-Net [47], which specializes to linear matrix inequalities, and HardNet++ [19], which handles more general nonlinear constraints. The control-invariant layer proposed in PS2-RL inherits the universal-approximation guarantee of HardNet-CVX. Unlike prior differentiable layers, however, its constraints are not specified directly in closed form; instead, they are induced by the rollout of a learned backup policy. This allows PS2-RL to enforce rollout-based control invariance while retaining the expressiveness needed for high-performance policies on high-dimensional systems.
C
Background on Safe Control Theory
This appendix provides the safe control background used in Sec. 2.1 and Sec. 4.1.1. We focus on the deterministic, control-affine system (1) in Sec. 2.1. For convenience, the system is restated below: ẋ = f (x) + g(x)u, x ∈ X ⊆ R n , u ∈ U ⊆ Rm , (17) where U encodes actuator limits and X is the state space of interest. The functions f : Rn → Rn and g : Rn → Rn×m are assumed locally Lipschitz, so that the closed-loop trajectories are well defined under the feedback policies considered in this work. C.1
Control Invariance
As mentioned in Sec. 2.1, safety of the system (17) is specified through a safe set S := {x ∈ X : hS (x) ≥ 0} (18) S n where h : R → R is continuously differentiable. Importantly, not every state in S is necessarily safe in the dynamical sense. Under bounded inputs, there may be states inside S from which all admissible control inputs eventually lead to a safety violation. Thus, safe control typically seeks an invariant subset of S, rather than enforcing membership in S alone [6, 34, 51]. Definition C.1 (Control Invariance). A set I ⊆ S is control-invariant if there exists a state-feedback policy π : X → U such that, for every initial condition x(0) ∈ I, the corresponding closed-loop trajectory satisfies x(t) ∈ I forall t ≥ 0. This definition is the controlled analogue of Nagumo’s set-invariance condition [43]. In the ideal case, one would compute the maximal control-invariant subset of S and restrict the controller to remain inside it. However, computing such a set is difficult for nonlinear systems with input limits. 18
C.2
Control Barrier Function
Control barrier functions provide a differential certificate for such invariant subsets. Let C := {x ∈ X : hC (x) ≥ 0} ⊆ S
(19)
be a candidate control-invariant set. A control barrier function certifies that, at each point in C, there exists an admissible input that prevents hC from decreasing too quickly [4]. Definition C.2 (Control Barrier Function). A continuously differentiable function hC : Rn → R is a control barrier function (CBF) [4] for the zero-superlevel set C := {x : hC (x) ≥ 0} ⊆ S if there exists an extended class-K∞ function α : R → R such that, for all x ∈ C, sup ∇hC (x)⊤ (f (x) + g(x)u) + α(hC (x)) ≥ 0. (20) u∈U
The condition (20) guarantees that the following state-dependent feasible input is nonempty: KC (x) := u ∈ U : ∇hC (x)⊤ (f (x) + g(x)u) + α(hC (x)) ≥ 0 .
(21)
A standard CBF safety filter minimally modifies a nominal input unom by solving 2
usafe (x) = arg min ∥u − unom ∥2 s.t.
u∈U ⊤
∇hC (x) (f (x) + g(x)u) + κ (hC (x)) ≥ 0.
(22) (23)
When U is convex, this is a convex quadratic program since the CBF constraint (20) is affine in u [5]. Proposition C.1 (CBF Invariance Guarantee). Suppose hC is a CBF for C and a locally Lipschitz feedback policy π satisfies π(x) ∈ KC (x) for all x ∈ C. Then C is forward invariant under π. Consequently, if x(0) ∈ C, then x(t) ∈ C ⊆ S for all t ≥ 0. Proof. Proof in [5, Cor. 2]. The main challenge is not using a CBF once it is known, but synthesizing a valid hC whose superlevel set is control-invariant under the bounded input set U. The backup-CBF construction addresses this by avoiding the explicit synthesis of a global invariant set. C.3
Backup Control Barrier Function
Backup control barrier functions [13, 21] start from a small certified base set and enlarge it implicitly using the rollout of a deterministic backup policy. Let B := {x ∈ X : hB (x) ≥ 0} ⊆ S
(24)
be a known control-invariant base set, and let πB : B → U be a certified base controller. In PS2-RL, this base pair is constructed locally around an equilibrium in Thm. 4.1. Let πb : X → U be a deterministic backup policy that agrees with the certified base controller on B, i.e., πb (x) = πB (x) for x ∈ B. Define the closed-loop backup trajectory as fπb (x) := f (x) + g(x)πb (x),
(25)
and let Φπb (x, t) denote the corresponding flow map. Definition C.3 (Backup-induced Control-Invariant Set). Given a backup horizon T > 0, the T -time backup-induced set is CT (πb ) := R(B, S, fπb , T ) := {x ∈ X : Φπb (x, T ) ∈ B ∧ Φπb (x, t′ ) ∈ S
∀t′ ∈ [0, T ]} . (26)
Thus, CT (πb ) contains exactly the states from which the backup policy reaches the certified base set within time T while remaining in the safe set throughout the backup rollout. Proposition C.2 (Backup-Induced Invariance). If B is control-invariant under πB and πb = πB on B, then B ⊆ CT (πb ) ⊆ S, and CT (πb ) is control-invariant. Proof. Proof in [13, Thm. 1]. 19
The set CT (πb ) can be represented as the zero-superlevel set of the implicit CBF ′ hCT (x) = min ′ min hS (Φπb (x, t )) , hB (Φπb (x, T )) . t ∈[0,T ]
(27)
The nonsmooth minimum in (27) makes a direct CBF-QP inconvenient. Instead, BCBF enforces sufficient CBF inequalities pointwise along the backup rollout and at the terminal base-set condition. Let ∂Φπb (x, t′ ) ∂x
Ψπb (x, t′ ) :=
(28)
denote the sensitivity of the backup flow with respect to its initial condition. When fπb is differentiable, Ψπb satisfies the variational equation d ∂fπb Ψπ (x, t′ ) = (Φπb (x, t′ )) Ψπb (x, t′ ), dt′ b ∂x
Ψπb (x, 0) = I.
(29)
For a candidate current input u, define the rollout and terminal derivative terms Ψπb (x, t′ ) f (x) + g(x)u − fπb (Φπb (x, t′ )) , ⊤ DTB (x, u) := ∇hB (Φπb (x, T )) Ψπb (x, T ) f (x) + g(x)u . DtS′ (x, u) := ∇hS (Φπb (x, t′ ))
⊤
(30) (31)
The subtraction term in DtS′ appears because the safety constraints along the backup rollout are indexed by future time. As real time advances, the same future point on the backup rollout moves closer by the backup dynamics fπb . The terminal condition uses a fixed terminal horizon T , so it has no corresponding subtraction term. Given extended class-K∞ functions αS and αB , the BCBF-admissible input set is KBCBF (x; πb ) := u ∈ U : DtS′ (x, u) + αS (hS (Φπb (x, t′ ))) ≥ 0 ∀t′ ∈ [0, T ], DTB (x, u) + αB (hB (Φπb (x, T ))) ≥ 0 .
(32)
For fixed x and fixed πb , the rollout Φπb (x, t′ ) and sensitivity Ψπb (x, t′ ) are constants with respect to the optimization variable u. Therefore, all constraints in (32) are affine in u, and KBCBF (x; πb ) is convex whenever U is convex. The corresponding BCBF-QP is usafe (x) = arg min u
s.t.
∥u − unom (x)∥22
(33)
u ∈ KBCBF (x; πb ).
(34)
Proposition C.3 (BCBF Feasibility and Safety). For every x ∈ CT (πb ), the set KBCBF (x; πb ) is nonempty. In particular, πb (x) ∈ KBCBF (x; πb ). Moreover, any locally Lipschitz policy π that satisfies π(x) ∈ KBCBF (x; πb ) for all x ∈ CT (πb ) renders CT (πb ) forward invariant, and therefore keeps the trajectory inside S. Proof. Proof in [13, Thm. 2]. Finally, BCBFs also address the relative-degree issue that arises when the original safety function hS depends on states that are not directly actuated. The composed functions hS ◦ Φπb (·, t′ ) depend on the current input through the flow sensitivity Ψπb (x, t′ )g(x). Under mild controllability and nondegeneracy assumptions, these composed constraints have relative degree one for positive rollout times [13, Thm. 3]. This is why PS2-RL can specify safety directly in physically meaningful coordinates, such as altitude or lane position, without constructing high-order CBFs for the full system. 20
D
Safe-Arrival Value Function
The safe-arrival value function is a central ingredient of Phase I (Sec. 4.3). This section provides a formal characterization of the proposed value function and its key properties. Although we introduce it for training the backup policy in our PS2-RL framework, the construction may be of independent interest and is not limited to the deterministic setting: the same methodology extends naturally to stochastic dynamics and randomized policies. For this reason, we present the definitions in terms of trajectory laws, so that the notation applies directly to stochastic settings while reducing to pathwise statements in the deterministic case considered here. We consider a deterministic control system xk ∈ X ,
xk+1 = F (xk , uk ),
uk ∈ U,
(35)
together with deterministic policies π : X → U. Let B ⊆ X denote the base set, let S ⊆ X denote the safe set, and define the failure set by F := X \ S.
(36)
H := X \ (B ∪ F).
(37)
The continuation set is
Throughout, we assume B ∩ F = ∅, so that B, F, and H form a disjoint partition of X . We use the indicators b(x) = 1B (x),
c(x) = 1 − b(x) − f (x) = 1H (x).
f (x) = 1F (x),
For an initial state-action pair (x, u) ∈ X × U and a policy π, let π ωx,u = (x0 , u0 , x1 , u1 , . . .)
(38)
(39)
denote the rollout generated by x0 = x,
u0 = u,
xk+1 = F (xk , uk ) for k ≥ 0,
uk = π(xk ) for k ≥ 1.
(40)
π Because the dynamics and policy are deterministic, the rollout ωx,u is unique. π π We write Px,u for the induced law on trajectory space and Ex,u for expectation with respect to this
law. In the deterministic setting,
π , Pπx,u = δωx,u
π Eπx,u [Z] = Z(ωx,u )
(41)
for every trajectory functional Z. Thus any event depending only on the controlled rollout has probability either 0 or 1. We keep the law-and-expectation notation because it remains unchanged when randomness is introduced into the policy or the dynamics. The hitting times of the base and failure sets are the trajectory functionals τB (ω) = inf{k ≥ 0 : xk ∈ B},
τF (ω) = inf{k ≥ 0 : xk ∈ F},
(42)
with the convention inf ∅ = ∞. When the rollout is clear, we simply write τB and τF . The safe-arrival event is ESA := {ω : τB (ω) < τF (ω), τB (ω) < ∞}.
(43)
This event occurs exactly when the trajectory arrives at the base set in finite time before entering the failure set. D.1
Undiscounted Safe-Arrival Value
We first define the undiscounted safe-arrival value. This value is the feasibility identifier associated with the safe-arrival specification: it identifies whether a policy succeeds in arriving at the base set before failure. Definition D.1 (Undiscounted Safe-Arrival Value). For a deterministic policy π, define QπSA (x, u) := Pπx,u (ESA ) = Pπx,u (τB < τF , τB < ∞).
(44)
π VSA (x) := QπSA (x, π(x)).
(45)
The associated state-value function is
21
The event-based definition admits an equivalent pathwise representation in terms of the indicator functions in (38). Proposition D.1 (Indicator Representation of Undiscounted Safe-Arrival Value). For every deterministic policy π and every (x, u) ∈ X × U, ! # " ∞ k−1 X Y QπSA (x, u) = Eπx,u c(xi ) b(xk ) , (46) i=0
k=0
where the empty product is interpreted as 1. Proof. Define the pathwise quantity
∞ X
Y :=
k=0
We show that
k−1 Y
!
c(xi ) b(xk ).
i=0
(47)
Y = 1{τB <τF , τB <∞} .
(48)
Suppose first that τB < τF and τB < ∞. Then xi ∈ H for all i < τB , so c(xi ) = 1 for i = 0, . . . , τB − 1. Moreover, xτB ∈ B, so b(xτB ) = 1. Hence the term in Y with k = τB equals Qk−1 1. For k < τB , we have xk ∈ / B, so b(xk ) = 0. For k > τB , the product i=0 c(xi ) contains the factor c(xτB ) = 0. Therefore all terms except the term with k = τB vanish, and Y = 1.
Conversely, suppose that the event {τB < τF , τB < ∞} does not occur. If the trajectory enters F before entering B, then b(xk ) = 0 for all k ≤ τF , and for all k > τF the product contains the factor c(xτF ) = 0. Hence every term in Y is zero. If the trajectory never enters B, then b(xk ) = 0 for all k, so again Y = 0.
Thus (48) holds pathwise. Taking expectation with respect to Pπx,u gives (46). The indicator representation yields the following one-step self-consistency structure. Proposition D.2 (Self-Consistency Condition of Undiscounted Safe-Arrival Value). For every deterministic policy π and every (x, u) ∈ X × U, QπSA (x, u) = b(x) + c(x) QπSA (F (x, u), π(F (x, u))) .
Proof. Starting from (46), QπSA (x, u) = Eπx,u
"∞ X
k=0
"
k−1 Y
!
c(xi ) b(xk )
i=0
= Eπx,u b(x0 ) +
#
(49)
∞ X
k=1
= b(x) + c(x) Eπx,u
k−1 Y
!
c(xi ) b(xk )
i=0
"∞ X
(50)
k−1 Y
!
#
(51) #
c(xi ) b(xk ) .
(52)
! j ∞ X Y QπSA (x, u) = b(x) + c(x) Eπx,u c(xi ) b(xj+1 ) .
(53)
k=1
Re-indexing with j = k − 1 gives
j=0
i=1
i=1
Let x+ = F (x, u) = x1 . From step 1 onward, the shifted rollout is the rollout initialized at (x+ , π(x+ )) under the same policy π. Therefore, ! j ∞ X Y Eπx,u c(xi ) b(xj+1 ) = QπSA (x+ , π(x+ )). (54) j=0
i=1
Substituting x+ = F (x, u) proves the result.
22
The optimal undiscounted values are Q⋆SA (x, u) := sup QπSA (x, u),
⋆ VSA (x) := max Q⋆SA (x, u). u∈U
π
(55)
Since the first action u is fixed in Q⋆SA (x, u) and only the continuation policy remains to be optimized after the successor state, the optimal values satisfy Q⋆SA (x, u) = b(x) + c(x) max Q⋆SA F (x, u), u+ . (56) u+ ∈U
Equivalently,
⋆ ⋆ VSA (x) = max [b(x) + c(x)VSA (F (x, u))] .
(57)
u∈U
The undiscounted value defines the states from which safe arrival is feasible. Definition D.2 (Safe-arrival region). The deterministic safe-arrival region is ⋆ RSA := {x ∈ X : VSA (x) = 1}.
(58)
Equivalently, RSA is the set of states from which there exists a deterministic policy that arrives at B before entering F. Definition D.3 (Minimum safe arrival time). Define n d(x) := inf k ∈ {0, 1, 2, . . .} : ∃(u0 , . . . , uk−1 ) ∈ U k such that xi+1 = F (xi , ui ) for i = 0, . . . , k − 1, o xi ∈ H for all i < k, xk ∈ B , x0 = x,
(59)
with the convention d(x) = ∞ if the set above is empty. For k = 0, the safety condition over i < k is vacuous, so d(x) = 0 exactly when x ∈ B. The following finite-step construction gives an equivalent characterization of the safe-arrival region. Proposition D.3 (Finite-step characterization of the safe-arrival region). Define R0 := B, Rn+1 := Rn ∪ {x ∈ H : ∃u ∈ U such that F (x, u) ∈ Rn } ,
n ≥ 0.
(60) (61)
Then RSA = Moreover, for every x ∈ RSA ,
∞ [
n=0
Rn = {x ∈ X : d(x) < ∞}.
d(x) = min{n ≥ 0 : x ∈ Rn }.
(62)
(63)
Proof. We prove by induction that x ∈ Rn if and only if safe arrival from x can be achieved in at most n steps. For n = 0, this holds because R0 = B. Suppose the claim holds for some n ≥ 0. If x ∈ Rn+1 , then either x ∈ Rn , in which case the induction hypothesis applies, or x ∈ H and there exists u ∈ U such that F (x, u) ∈ Rn . In the latter case, applying u for one step and then using the induction hypothesis from F (x, u) gives safe arrival in at most n + 1 steps. Conversely, suppose safe arrival from x can be achieved in at most n + 1 steps. If it can be achieved in at most n steps, then x ∈ Rn ⊆ Rn+1 . Otherwise, the first step is taken from a state in H, and some first action u leads to a successor from which safe arrival can be achieved in at most n steps. By the induction hypothesis, this successor lies in Rn , so x ∈ Rn+1 . S∞ Thus n=0 Rn is exactly the set of states with finite minimum safe arrival time. This set is also RSA by the definition of the optimal undiscounted value. The identity (63) follows from the same induction argument and the definition of d(x). 23
D.2
Discounted Safe-Arrival Value
We now define the discounted Safe-Arrival Value. The discounted value retains the same first-arrival structure as the undiscounted one, while assigning larger value to earlier safe arrival at the base set. Definition D.4 (Discounted Safe-Arrival Value). Fix a discount factor β ∈ (0, 1). For a deterministic policy π, define "∞ ! # k−1 X Y π π k QSA,β (x, u) := Ex,u β c(xi ) b(xk ) . (64) i=0
k=0
The associated state-value function is π VSA,β (x) := QπSA,β (x, π(x)).
(65)
The discounted sum pays only at the first safe arrival time. This gives both a hitting-time interpretation and a random-horizon interpretation. Proposition D.4 (Interpretations of Discounted Safe-Arrival Value). For every deterministic policy π and every (x, u) ∈ X × U, QπSA,β (x, u) = Eπx,u β τB 1{τB <τF , τB <∞} . (66)
Moreover, let N be independent of the controlled rollout and satisfy PN (N = k) = (1 − β)β k ,
k = 0, 1, 2, . . . .
(67)
Then
Proof. Define
QπSA,β (x, u) = Pπx,u ⊗ PN (τB < τF , τB ≤ N ) . Yβ :=
∞ X
β
k
k=0
k−1 Y i=0
!
c(xi ) b(xk ).
(68)
(69)
The same pathwise argument used in the proof of Proposition D.1 shows that, on the event {τB < τF , τB < ∞}, the only nonzero term in Yβ occurs at k = τB , and its value is β τB . Outside this event, all terms vanish. Hence Yβ = β τB 1{τB <τF , τB <∞} .
(70)
Taking expectation proves (66). For the random-horizon identity, note that PN (N ≥ k) =
∞ X
n=k
(1 − β)β n = β k .
By independence of N and the controlled rollout, Pπx,u ⊗ PN (τB < τF , τB ≤ N ) ∞ X = Pπx,u (τB = k < τF ) PN (N ≥ k) =
k=0 ∞ X
β k Pπx,u (τB = k < τF )
k=0 = Eπx,u β τB 1{τB <τF , τB <∞} .
Combining this identity with (66) proves (68).
The discounted value satisfies the following one-step self-consistency relation. 24
(71)
(72) (73) (74)
Proposition D.5 (Self-Consistency Condition of Discounted Safe-Arrival Value). For every deterministic policy π and every (x, u) ∈ X × U, QπSA,β (x, u) = b(x) + β c(x) QπSA,β (F (x, u), π(F (x, u))) .
(75)
Proof. Starting from (64), "
QπSA,β (x, u) = Eπx,u b(x0 ) +
∞ X
k=1
= b(x) + β c(x) Eπx,u
βk
k−1 Y
!
c(xi ) b(xk )
i=0
"∞ X
β
k−1
k=1
k−1 Y
∞ X QπSA,β (x, u) = b(x) + β c(x) Eπx,u βj
!
j Y
(76) #
c(xi ) b(xk ) .
i=1
Re-indexing with j = k − 1 gives
#
!
(77)
c(xi ) b(xj+1 ) .
(78)
⋆ VSA,β (x) := max Q⋆SA,β (x, u).
(80)
j=0
i=1
Let x+ = F (x, u) = x1 . The shifted rollout from step 1 onward is the rollout initialized at (x+ , π(x+ )) under the same policy π. Therefore, ! j ∞ X Y Eπx,u βj c(xi ) b(xj+1 ) = QπSA,β (x+ , π(x+ )). (79) j=0
i=1
Substituting x+ = F (x, u) proves the result. Define the optimal discounted values by
Q⋆SA,β (x, u) := sup QπSA,β (x, u),
u∈U
π
The corresponding Bellman equation is Q⋆SA,β (x, u) = b(x) + β c(x) max Q⋆SA,β F (x, u), u+ . + u ∈U
Equivalently,
⋆ ⋆ VSA,β (x) = max b(x) + β c(x)VSA,β (F (x, u)) . u∈U
(81)
(82)
The discounted Bellman operator (Tβ Q)(x, u) := b(x) + β c(x) max Q(F (x, u), u+ ) + u ∈U
(83)
is a β-contraction in the sup norm. Indeed, for any two bounded action-value functions Q1 , Q2 , |(Tβ Q1 )(x, u) − (Tβ Q2 )(x, u)| ≤ β c(x) max Q1 (F (x, u), u+ ) − Q2 (F (x, u), u+ ) +
(84)
≤ β∥Q1 − Q2 ∥∞ .
(85)
u ∈U
Consequently, the discounted Bellman equation has a unique bounded fixed point, namely Q⋆SA,β . The discounted and undiscounted fixed-policy values have the same success support. Proposition D.6 (Discounting Preserves Safe-Arrival Feasibility). For every deterministic policy π, every (x, u) ∈ X × U, and every β ∈ (0, 1), QπSA (x, u) = 1
⇐⇒
QπSA,β (x, u) > 0.
(86)
Moreover, lim QπSA,β (x, u) = QπSA (x, u). β↑1
25
(87)
Proof. Since the rollout is deterministic, either the safe-arrival event occurs or it does not. If it occurs, then τB = N for some finite N , and QπSA (x, u) = 1,
QπSA,β (x, u) = β N > 0.
(88)
If it does not occur, then both values are zero. This proves (86). The limit (87) follows from β N → 1 as β ↑ 1 for every finite N . The next proposition gives the precise time-optimal implication of discounting. Proposition D.7 (Arrival-Time Optimality of the Discounted Value). For every x ∈ X , d(x) β , d(x) < ∞, ⋆ VSA,β (x) = 0, d(x) = ∞. Moreover, for every (x, u) ∈ X × U, 1, ⋆ QSA,β (x, u) = β 1+d(F (x,u)) , 0,
x ∈ B, x ∈ H and d(F (x, u)) < ∞, otherwise.
(89)
(90)
Consequently, for every β ∈ (0, 1),
⋆ ⋆ RSA = {x ∈ X : VSA (x) = 1} = {x ∈ X : d(x) < ∞} = {x ∈ X : VSA,β (x) > 0},
(91)
and ⋆ ⋆ lim VSA,β (x) = VSA (x). β↑1
(92)
Proof. Fix x ∈ X . Under deterministic dynamics and a deterministic policy, either the trajectory arrives at B before F in exactly N steps for some finite N , or the safe-arrival event does not occur. In the first case, the discounted value is β N ; in the second case, it is zero. Since β ∈ (0, 1), maximizing the discounted value among successful policies is equivalent to minimizing the number of steps required for safe arrival. The minimum such number is d(x). Hence (89) holds. The action-value expression (90) follows by separating the first action. If x ∈ B, safe arrival has already occurred at time 0, so the value is 1. If x ∈ F, failure has already occurred at time 0, so the value is 0. If x ∈ H, taking action u first moves the system to F (x, u). Safe arrival remains possible exactly when d(F (x, u)) < ∞, in which case the shortest safe arrival time after taking u is 1 + d(F (x, u)). The equivalence (91) follows from (89) and the definition of RSA . Finally, if d(x) < ∞, then ⋆ β d(x) → 1 as β ↑ 1; if d(x) = ∞, then VSA,β (x) = 0 for every β ∈ (0, 1). This proves (92). D.3
Behavior of Optimal Safe-Arrival Actions
The preceding results imply a complete qualitative description of optimal safe-arrival actions across the state space. The relevant state-space partition is X = B ∪˙ F ∪˙ (RSA ∩ H) ∪˙ (H \ RSA ).
(93)
Proposition D.8 (Qualitative Structure of Optimal Safe-Arrival Actions). The optimal safe-arrival values and actions satisfy the following four cases. 1. If x ∈ B, then
⋆ VSA (x) = 1,
⋆ VSA,β (x) = 1.
(94)
Every action has the same value, because safe arrival has already occurred at time 0. 2. If x ∈ F, then
⋆ VSA (x) = 0,
⋆ VSA,β (x) = 0.
Every action has the same value, because failure has already occurred at time 0. 26
(95)
3. If x ∈ RSA ∩ H, then an action u ∈ U is optimal for the undiscounted optimal action-value if and only if Q⋆SA (x, u) = 1.
(96)
F (x, u) ∈ RSA .
(97)
Equivalently,
Thus an undiscounted optimal first action preserves safe-arrival feasibility. A complete policy must still select future actions so that B is reached in finite time; the undiscounted criterion does not distinguish among successful policies with different arrival times. For the discounted criterion, Q⋆SA,β (x, u) =
1+d(F (x,u)) β , 0,
F (x, u) ∈ RSA , F (x, u) ∈ / RSA .
(98)
Hence u is discounted-optimal at x if and only if F (x, u) ∈ RSA
and
d(F (x, u)) = d(x) − 1.
(99)
Therefore a discounted optimal policy decreases the minimum safe arrival time by one at every pre-arrival step and reaches B in exactly d(x) steps.
4. If x ∈ H \ RSA , then
⋆ VSA (x) = 0,
⋆ VSA,β (x) = 0.
(100)
No policy can safely arrive at B from such a state. Hence every action has value zero under the safe-arrival objectives unless an additional secondary criterion is imposed. Proof. If x ∈ B, then τB = 0 < τF , so both the undiscounted and discounted optimal values equal 1, independently of the chosen action. If x ∈ F, then τF = 0, so the safe-arrival event cannot occur, and both values equal 0. ⋆ Now let x ∈ RSA ∩ H. Since x ∈ RSA , we have VSA (x) = 1. Therefore an action u is optimal for ⋆ the undiscounted objective exactly when QSA (x, u) = 1. By (56), and since b(x) = 0 and c(x) = 1 for x ∈ H, ⋆ Q⋆SA (x, u) = VSA (F (x, u)).
(101)
Thus Q⋆SA (x, u) = 1 if and only if F (x, u) ∈ RSA .
The discounted action-value expression (98) follows from (90). Since β ∈ (0, 1), maximizing β 1+d(F (x,u)) over feasible successors is equivalent to minimizing d(F (x, u)). By the definition of d(x), the minimum feasible successor distance is d(x) − 1. Therefore u is discounted-optimal if and only if (99) holds. Finally, if x ∈ H \ RSA , then d(x) = ∞. Hence no policy can safely arrive at B from x, so both the undiscounted and discounted optimal values are zero. This proves the final case. Remark D.1 (Roles of Undiscounted and Discounted Safe-Arrival Value). The undiscounted safearrival value defines the safe-arrival specification and the feasible safe-arrival region ⋆ RSA = {x ∈ X : VSA (x) = 1}.
(102)
The discounted safe-arrival value preserves this region, ⋆ RSA = {x ∈ X : VSA,β (x) > 0},
β ∈ (0, 1),
(103)
and further ranks successful policies according to their arrival time. Thus the undiscounted value provides the feasibility semantics, while the discounted value provides a contraction-based, timesensitive objective for RL training. 27
E
PS2-RL Details and Proofs
E.1
Base Set Certification Proof
Theorem 4.1 (Existence of a Certified Local Base Set). Under Assumption 4.1, there exists a linear feedback gain K ∈ Rm×n , a symmetric positive-definite matrix P ≻ 0, and a constant c̄ > 0 such that the base controller πB (x) = u⋆ − K(x − x⋆ ) and the sublevel set Bc = x ∈ X : (x − x⋆ )⊤ P (x − x⋆ ) ≤ c , c ∈ (0, c̄] together satisfy the following properties: (i) Bc ⊆ S; (ii) πB (x) ∈ U for every x ∈ Bc ; (iii) Bc is control-invariant under πB ; and (iv) every trajectory with x(0) ∈ Bc satisfies x(t) → x⋆ as t → ∞ and x(t) ∈ Bc for all t ≥ 0. Proof. By Assumption 4.1(b), there exists K ∈ Rm×n such that A − BK is Hurwitz. Fix any such K and any Q ≻ 0. By the Lyapunov equation for stable linear systems [28, Thm 4.6], there exists a unique P ≻ 0 with (A − BK)⊤ P + P (A − BK) = −Q.
(104)
Define the error coordinate e := x − x⋆ and the candidate Lyapunov function as V (e) = e⊤ P e. The closed-loop nonlinear dynamics under πB is ẋ = f (x) + g(x)(u⋆ − K(x − x⋆ )). Setting F (e) := f (x⋆ + e) + g(x⋆ + e)(u⋆ − Ke), a Taylor expansion of F at e = 0 combined with F (0) = f (x⋆ ) + g(x⋆ )u⋆ = 0 (Assumption 4.1(a)) gives ė = F (e) = (A − BK) e + ρ(e),
∥ρ(e)∥ = o(∥e∥) as e → 0,
(105)
where the remainder ρ inherits the smoothness of f and g. Using (104) and (105), V̇ (e) = 2 e⊤ P ė = − e⊤ Q e + 2 e⊤ P ρ(e), | {z } =: R(e)
with |R(e)| ≤ 2∥P ∥ ∥e∥ ∥ρ(e)∥ = o(∥e∥2 ). Hence there exists r1 > 0 such that for every e with 0 < ∥e∥ ≤ r1 , V̇ (e) ≤ − 21 λmin (Q) ∥e∥2 < 0. Let c1 := λmin (P ) r12 . Then {e : V (e) ≤ c1 } ⊆ {e : ∥e∥ ≤ r1 } and V̇ (e) < 0 on {0 < V (e) ≤ c1 }, establishing Thm. 4.1(c) and (d) for any c ≤ c1 by the standard Lyapunov invariance theorem [28, Thm. 4.1]. It remains to shrink c1 so that Thm. 4.1(a) and (b) also hold. Since x⋆ ∈ int(S) and S = {x : hS (x) ≥ 0} is the zero-superlevel set of the continuous function hS , there exists c2 > 0 such that Bc2 ⊆ S. Analogously, since u⋆ ∈ int(U), U is compact, and the map πB (x) = u⋆ − K(x − x⋆ ) is continuous, there exists c3 > 0 such that πB (Bc3 ) ⊆ U. Taking c̄ := min{c1 , c2 , c3 } > 0 yields all four properties simultaneously for every c ∈ (0, c̄]. E.2
PS2-RL Decomposition Proof
Theorem 4.2 (Certified Decomposition and Exactness of PS2-RL). Suppose (B, πB ) satisfies Thm. 4.1. Let πb⋆ be a solution of Subproblem 2, and let πϕ⋆ be a solution of Subproblem 1 with the fixed backup policy πb⋆ . Assume the exact BCBF constraints in (6) are enforced. Then: (i) ΠBCBF (πb⋆ ) ⊆ Πsafe and hence πϕ⋆ is feasible for Problem 1; (ii) πϕ⋆ is optimal among all policies certified by πb⋆ , i.e., J(πϕ⋆ ) = supπ∈ΠBCBF (πb⋆ ) J(π) ≤ supπ∈Πsafe J(π); and (iii) if there exists a ⋆ ⋆ globally optimal solution πsafe of Problem 1 such that πsafe ∈ ΠBCBF (πb⋆ ), then πϕ⋆ is also globally optimal for Problem 1. Thus, the two subproblems solve a certified inner approximation of Problem 1. They solve Problem 1 exactly whenever the learned backup policy induces a BCBF-feasible class containing an optimal safe policy. 28
Proof. We first prove (i). Let π ∈ ΠBCBF (πb⋆ ) and let x0 ∈ X0 . By Subproblem 2, X0 ⊆ CT (πb⋆ ), so x0 ∈ CT (πb⋆ ). The set CT (πb⋆ ) is the zero-superlevel set of the implicit BCBF hCT (x) = min ′ min hS (Φπb⋆ (x, t′ )), hB (Φπb⋆ (x, T )) . t ∈[0,T ]
By construction of KBCBF (x; πb⋆ ), every action u = π(x) with x ∈ CT (πb⋆ ) satisfies the pointwise BCBF inequalities on the backup rollout and the terminal base-set condition. These pointwise inequalities are a sufficient condition for the CBF condition on hCT [13, Prop. 2]. Therefore CT (πb⋆ ) is forward invariant under any policy in ΠBCBF (πb⋆ ). Since CT (πb⋆ ) ⊆ S, the resulting trajectory satisfies xπ (t; x0 ) ∈ S for all t ≥ 0. Moreover, KBCBF (x; πb⋆ ) ⊆ U by definition, so the executed input also satisfies the actuator constraint. Hence π ∈ Πsafe , proving ΠBCBF (πb⋆ ) ⊆ Πsafe . Statement (ii) follows directly from the definition of πϕ⋆ as a solution of Subproblem 1. Since Subproblem 1 optimizes J over ΠBCBF (πb⋆ ), J(πϕ⋆ ) = sup J(π). π∈ΠBCBF (πb⋆ )
The inequality sup
≤
J(π)
π∈ΠBCBF (πb⋆ )
sup J(π) π∈Πsafe
then follows from part (i). ⋆ Finally, suppose there exists a globally optimal safe policy πsafe ∈ ΠBCBF (πb⋆ ). Since πϕ⋆ maximizes ⋆ J over ΠBCBF (πb ), ⋆ J(πϕ⋆ ) ≥ J(πsafe ). ⋆ ⋆ But by part (i), πϕ ∈ Πsafe , and πsafe is globally optimal over Πsafe , so ⋆ J(πϕ⋆ ) ≤ J(πsafe ).
⋆ Thus J(πϕ⋆ ) = J(πsafe ), proving that πϕ⋆ is globally optimal for Problem 1.
E.3
PS2-RL Safety Guarantee Proof
Theorem 4.3 (Safety Guarantee of PS2-RL). Suppose (B, πB ) satisfies Thm. 4.1, let πb⋆ be the fixed composed backup policy, and assume X0 ⊆ CT (πb⋆ ). Then for any policy πϕ , the corresponding ϕ ϕ PS2 policy satisfies πPS2 ∈ ΠBCBF (πb⋆ ). By Thm. 4.2, πPS2 ∈ Πsafe and is therefore feasible for Problem 1. Every realized input satisfies uk ∈ U and the sampled closed-loop trajectory satisfies xk ∈ S for all k ≥ 0 and all x0 ∈ X0 . Under the sampled-data conditions, the continuous-time ZOH trajectory remains in S as well. Proof. For every x ∈ CT (πb⋆ ), the control-invariant layer returns the projection of πϕ (x) onto KBCBF (x; πb⋆ ). Hence ϕ πPS2 (x; πb⋆ ) ∈ KBCBF (x; πb⋆ ) ∀x ∈ CT (πb⋆ ), which is exactly the defining property of ΠBCBF (πb⋆ ). Thus, the following holds: ϕ πPS2 ∈ ΠBCBF (πb⋆ ). The conclusion then follows immediately from Thm. 4.2.
E.4
PS2-RL Universal Approximation Proof
Theorem 4.4 (Universal Approximation of PS2-RL). Fix πb⋆ , assume U is convex, and let Ω ⊆ CT (πb⋆ ) be compact. Let C(Ω, U ) denote the space of continuous maps Ω → U with norm ∥π∥∞ := supx∈Ω ∥π(x)∥∞ . For function classes GNN , G ⊆ C(Ω, U ), assume GNN universally approximates G, i.e., for every π ∈ G and ϵ > 0, there exists πϕ ∈ GNN such that ∥π − πϕ ∥∞ < ϵ. Define the class of PS2 policies obtained by composing (13) with GNN -class policies as ⋆ ΠGPNN S2 (πb ; Ω) := {PCIL (πϕ ) : πϕ ∈ GNN } , and the BCBF-feasible policy class on Ω as ΠBCBF (πb⋆ ; Ω) := {π ∈ G : π(x) ∈ KBCBF (x; πb⋆ ) ∀x ∈ Ω} .
⋆ ⋆ Then ΠGPNN S2 (πb ; Ω) universally approximates ΠBCBF (πb ; Ω) under ∥ · ∥∞ .
29
(15)
(16)
Proof. For every x ∈ Ω ⊆ CT (πb⋆ ), the set KBCBF (x; πb⋆ ) is nonempty by BCBF feasibility and convex because U is convex and the BCBF constraints are affine in u under the fixed backup flow. Thus (13) is precisely a HardNet-CVX projection with state-dependent convex feasible set KBCBF (x; πb⋆ ). The result follows directly from the universal-approximation theorem for HardNet-CVX [41, Thm. 12]. E.5
Constructing the CIL from Differentiable Simulators
As briefly discussed in Sec. 4.5, PS2-RL does not fundamentally require a hand-derived symbolic decomposition f (x) + g(x)u. Suppose instead that a differentiable simulator F∆t is available. For a fixed backup policy, define the candidate-control-input rollout z0 (u) = F∆t (x, u), zi+1 (u) = F∆t zi (u), πb⋆ (zi (u)) . For each backup rollout condition ℓi (zi (u)) ≥ 0, corresponding to either a sampled safe-set condition or the terminal base-set condition, automatic differentiation gives ai (x) = ∇u ℓi (zi (u)) u=u , b
ub = πb⋆ (x),
and hence the local affine row ℓi (zi (ub )) + ai (x)⊤ (u − ub ) ≥ 0.
These simulator-generated rows can be inserted into the same control-invariant layer. For highdimensional simulators, one need not materialize the full state sensitivity matrix, as vector–Jacobian products or reduced sensitivities for only the constraint-relevant outputs are sufficient. This simulator-based variant should be interpreted as a discrete-time, locally affine realization of PS2RL. With an exact control-affine model, the rows in Eq. (6) are exact and the guarantees in Thm. 4.3 apply directly. With a general differentiable simulator, the local rows are first-order approximations, and in that case one can add conservative margins or impose a trust region around ub . Thus, the main scaling bottleneck is not closed-form dynamics or explicit invariant-set synthesis, but access to a differentiable rollout model whose accuracy is sufficient over the backup horizon. Employing PS2-RL with a differentiable simulator, without the analytical dynamics, is left as future work. E.6
Phase I: Safe-Arrival Policy Training
Here, we expand on Phase I beyond the high-level description in Sec. 4.3. Phase I addresses θ Subproblem 2 by learning a parameterized safe-arrival policy πSA that enlarges the backup-induced control-invariant set CT (πbθ ), given the certified base pair (B, πB ) from Thm. 4.1. Fig. 4 (top) summarizes the resulting training loop: the policy interacts with the environment, stepwise indicators of B, F, and the continuation set are recorded along each rollout, and the parameters are updated with the discounted safe-arrival value Bellman recursion (11). Algorithm 1 Safe-Arrival Policy Training Require: sampler ρarr on Ω \ B, dynamics F , sets B and S, discount β 1: Initialize safe-arrival policy parameters θ and value/Q-function parameters 2: while not converged do 3: Sample initial states x0 ∼ ρarr θ 4: Collect rollouts under πSA until xk ∈ B or xk ∈ F = X \ S, 5: For each xk , assign indicators b(xk ), f (xk ), and c(xk ) = 1 − b(xk ) − f (xk ) 6: Update the chosen value/Q estimator using the safe-arrival value Bellman recursion (11) 7: Improve θ with the chosen RL backbone 8: end while θ 9: return πSA and freeze the composed πbθ for Phase II The Phase I objective can be written as
h θ i πSA θ max JSA,β (θ) := Ex∼ρarr QSA,β (x, πSA (x)) . θ
(106)
Algorithm 1 summarizes Phase I at the framework level. In practice, one can take Ω = S, or a smaller design region of interest inside S, and choose ρarr to focus training on the portion of state space from 30
Phase I: Safe-Arrival Policy Training Action uk
Safe-Arrival Policy θ (xk ) πSA
Environment xk+1 = F (xk , uk )
Indicators b(xk ), f (xk ), c(xk ) (Time-Optimal Arrival at B)
Transfer & Freeze Backup Policy
Phase II: PS2 Policy Training Learned(Backup Policy πB (x), x ∈ B, ⋆ πSA (x), x ∈ /B
πb∗ (x) =
BCBF-Admissible Set Construction Forward integrate Φπb (x, t) Compute constraints: A(x)u ≤ b(x) State x
End-to-End Backpropagation RL Policy (Unconstrained Policy) πϕ (·)
πϕ (x)
A(x), b(x)
Control-Invariant Layer (Differentiable QP Solver) min ∥u − πϕ (x)∥22
u
u
Environment xk+1 = F (xk , uk )
s.t. A(x)u ≤ b(x)
State x, Reward r
Figure 4: Block-diagram view of the two-phase PS2-RL framework, complementing Fig. 1. θ Phase I (top): the safe-arrival policy πSA is trained against the environment using the indicators b(xk ), f (xk ), c(xk ) of B, F, and the continuation set, optimizing the discounted safe-arrival value Bellman recursion. Phase II (bottom): the composed backup policy πb⋆ is frozen and used to construct the BCBF-admissible set online for each timestep. The control value from the RL policy πϕ is projected onto this set by the control-invariant layer (a differentiable QP), and task-reward gradients propagate end-to-end through the projection back into πϕ . which safe arrival is desired. Details on ρarr for each experiment are provided in App. F.3. The only safe-arrival policy-specific ingredients are the indicators b, f , c, first-hit termination at B or F, and the objective function with the safe-arrival value. Everything else can be supplied by a chosen RL backbone, such as actor-critic, Q-learning, fitted value iteration, and related methods. E.7
Phase II: Control-Invariant Layer and PS2 Policy
This appendix expands the control-invariant layer construction and PS2-RL policy training in Sec. 4.4, which corresponds to the bottom panel of Fig. 4. The main text defines the control-invariant layer as the projection of a nominal policy output onto the BCBF-admissible set KBCBF (x; πb⋆ ). Here, we write the finite-mesh QP explicitly in the standard affine form and describe the differentiable solver used in implementation. E.7.1
Control-Invariant Layer Construction and Solver
Construction of the BCBF-admissible input set. For this subsection, we allow the safe set and base set to be defined by multiple differentiable inequalities, S = {x ∈ X : hS,j (x) ≥ 0, j = 1, . . . , nS }, B = {x ∈ X : hB,ℓ (x) ≥ 0, ℓ = 1, . . . , nB }.
(107) (108)
The scalar case used in the main text is recovered by setting nS = nB = 1. Let the finite backup mesh be 0 = t′0 < t′1 < · · · < t′N = T . For a fixed state x and the frozen backup policy πb⋆ , define xbi (x) := Φπb⋆ (x, t′i ),
Ψi (x) := Ψπb⋆ (x, t′i ) = 31
∂Φπb⋆ (x, t′i ) , ∂x
fib (x) := fπb⋆ xbi (x) .
(109)
The current-state control-affine terms are denoted by f0 := f (x) and g0 := g(x). For every safety constraint component hS,j and backup node t′i , the BCBF rollout condition is h i ∇hS,j (xbi )⊤ Ψi (x) f0 + g0 u − fib (x) + αS hS,j (xbi ) ≥ 0. (110)
Equivalently, this is an affine inequality in the current input u: aSi,j (x)⊤ u ≤ bSi,j (x),
(111)
aSi,j (x)⊤ := −∇hS,j (xbi )⊤ Ψi (x)g0 , h i bSi,j (x) := αS hS,j (xbi ) + ∇hS,j (xbi )⊤ Ψi (x)f0 − fib (x) .
(112)
with
At the terminal node T = t′N , each base-set constraint component gives ∇hB,ℓ (xbN )⊤ ΨN (x) f0 + g0 u + αB hB,ℓ (xbN ) ≥ 0,
(113)
(114)
or equivalently
⊤ B aB ℓ (x) u ≤ bℓ (x),
(115)
⊤ b ⊤ aB ℓ (x) := −∇hB,ℓ (xN ) ΨN (x)g0 , b b ⊤ bB ℓ (x) := αB hB,ℓ (xN ) + ∇hB,ℓ (xN ) ΨN (x)f0 .
(116)
where
Stacking all safety rows and terminal rows gives AS (x) ABCBF (x) := ∈ Rnc ×m , AB (x)
bS (x) bBCBF (x) := ∈ Rnc , bB (x)
(117)
(118)
where nc = (N + 1)nS + nB . Thus the finite-mesh BCBF constraints can be written compactly as ABCBF (x)u ≤ bBCBF (x).
(119)
U = {u ∈ Rm : umin ≤ u ≤ umax },
(120)
For a boxed input limit set
the actuator constraints are AU u ≤ b U ,
AU :=
Im , −Im
bU :=
umax . −umin
(121)
Then, the control-invariant layer is a quadratic program: PCIL (πϕ )(x) = arg min ∥u − πϕ (x)∥22 u
s.t.
(122)
ABCBF (x)u ≤ bBCBF (x), AU u ≤ bU .
Slack-regularized QP. In implementation, we solve a slack-regularized version of (122). Specifically, the QP decision variable is z = [u⊤ , δ]⊤ ∈ Rm+1 , where δ ≥ 0 is a scalar slack shared across all BCBF rows: (u⋆ , δ ⋆ ) = arg min ∥u − πϕ (x)∥22 + λδ δ 2 u,δ
s.t.
ABCBF (x)u − 1nc δ ≤ bBCBF (x),
(123) AU u ≤ bU ,
δ ≥ 0,
where λδ ≫ 1. In our experiments, λδ,unicycle = 105 and λδ,quadrotor = 106 . Note that the control-invariant layer returns only the control component, λδ PCIL (πϕ )(x) := u⋆ .
32
(124)
The slack variable is included for numerical robustness. The exact BCBF theory is continuous-time, while the implemented layer uses a finite backup mesh, numerical integration of the backup flow and sensitivity, and a sampled-data controller whose action is held by ZOH. Near the boundary of CT (πb⋆ ), these approximations can create small artificial infeasibilities even when the ideal continuous-time BCBF condition is feasible. A heavily penalized slack prevents such numerical infeasibilities from causing solver failure or unstable gradients. The slack relaxes only the BCBF rows, not the control limits AU u ≤ bU . Thus every executed control still satisfies u⋆ ∈ U. The zero-slack case δ ⋆ = 0 recovers the hard BCBF constraints in (119), and the formal safety guarantees in Sec. 4.4 correspond to this exact hard-constraint setting. Table 7 confirms that the slack-regularized control-invariant layer is effectively operating in the hard-constraint regime throughout evaluation. Across both environments and both PS2-RL variants, the mean and median slack values are on the order of 10−5 or smaller, and even the 99th percentile remains at most 5.10 × 10−5 . Moreover, steps with δ > 10−4 never occur in the unicycle evaluations and occur only rarely in the quadrotor evaluations, even for the more aggressive learned variant. Thus, the slack term mainly absorbs small numerical discrepancies from finite-mesh backup integration and sampled-data implementation, rather than serving as a meaningful relaxation of the BCBF constraints. Consistent with this interpretation, the PS2-RL variants retain the 100% safety and strong tracking performance reported in the main experimental Tables 1 and 2. Table 7: Control-invariant layer’s slack value during evaluation. For each PS2-RL variant and task, we aggregate the scalar QP slack δ over all control-invariant layer solves from 10 seeds and 1,000 evaluation episodes per seed. We report the mean, median, 99th percentile, and fraction of steps with δ > 10−4 . Across both tasks, the slack remains near 10−5 and large slack values are extremely rare, indicating that the relaxation acts primarily as a numerical safeguard rather than an active softening of the BCBF constraints. Unicycle (Total Steps Each: 1.06 × 106 )
Quadrotor (Total Steps Each: 4.0 × 106 )
Safe RL
δ-mean
δ-med.
δ-p99
Frac. > 10−4
δ-mean
δ-med.
δ-p99
Frac. > 10−4
PS2ABP PS2SA
9.92e-06 8.97e-06
9.60e-06 9.04e-06
1.61e-05 1.56e-05
0% 0%
5.41e-06 2.49e-05
4.80e-06 4.41e-06
1.47e-05 5.10e-05
3.77 × 10−4 % 0.088%
Differentiable QP solver. The optimization problems (122) and (123) are convex QP by construction. Our implementation solves them with qpax from [50]. All quantities used to assemble ABCBF (x) and bBCBF (x) are computed with differentiable JAX operations, including the backup rollout, sensitivity propagation, row construction, and the QP solve. Consequently, the policy update backpropagates directly through the network without an estimator or a separate surrogate loss. The differentiable solver provides the corresponding gradient used by the RL optimizer. E.7.2
PS2-RL Policy Training through the Control-Invariant Layer
Phase II freezes the composed backup policy πB (x), x ∈ B, ⋆ πb (x) = ⋆ πSA (x), x ∈ / B, and trains only the nominal task policy πϕ . For a deterministic actor, the executed PS2 action is ϕ uk = πPS2 (xk ; πb⋆ ) := PCIL (πϕ )(xk ).
For a stochastic actor, the nominal action is first sampled from the policy distribution and the control-invariant layer is then applied deterministically: unomk ∼ πϕ (· | xk ),
uk = PCIL (unomk )(xk ).
The environment and replay buffer therefore only see projected controls uk . In our implementation, the actor loss evaluates the critic at the projected action u⋆ , and target-policy actions in Bellman backups are projected through the same control-invariant layer. Thus the actor is optimized for task return inside the BCBF-admissible action set. 33
Because the control-invariant layer is differentiable, the actor-gradient path contains the QP solution: ∇ϕ Lactor =
∂Lactor ∂u⋆ ∂unom ∂u⋆ ∂unom ∂ϕ
This is the key distinction between PS2-RL and a post-hoc non-differentiable shield: the projection layer is present during both data collection and gradient-based policy improvement. The resulting policy is therefore trained to produce high-return nominal actions whose projections remain close to the nominal command while satisfying the BCBF-induced constraints.
F
Experiment Details
F.1
Unicycle Environment Details
System dynamics. We consider the following unicycle dynamics for the lane keeping experiment: " # " # ẏ v sin(ψ) 0 0 acmd v̇ = 0 + 1 0 (125) rcmd 0 0 1 ψ̇
where the state x comprises the lateral position y, velocity v, and the heading angle ψ, with control inputs consisting of acceleration acmd ∈ [−5, 5] m/s2 and yaw rate rcmd ∈ [−1, 1] rad/s. We implement the continuous-time dynamics in sampled-data form with ∆t = 0.05 sec. Moreover, all controllers employ a control frequency of 20 Hz, matching the sampling period. Safe set. The safe set is specified as S = {x : |y| ≤ ymax , |ψ| ≤ ψmax }, where ymax = 1.8 m and ψmax = π/3. Thus, we have four safety constraints: hS,2 (x) = ymax − y, hS,4 (x) = ψmax − ψ.
hS,1 (x) = ymax + y, hS,3 (x) = ψmax + ψ,
Note that the safety specification constrains only the lateral position and heading, since these are the variables directly associated with leaving the lane. Reference trajectory, reward, episode setup. The unicycle task is to track a sinusoidal lane reference 2πt yref (t) = 2.5 sin , vref (t) = 5, 10 with heading reference ẏref (t) ψref (t) = arcsin . vref (t) The reference lasts 20 sec and deliberately violates the lane constraint, since its lateral amplitude 2.5 m exceeds ymax = 1.8 m. The trajectory-following reward uses the normalized tracking error for step k yk − yref,k vk − vref,k wrap(ψk − ψref,k ) acmd,k rcmd,k , , , , , ξk = ymax 5 ψmax amax rmax and the step-wise reward is rk = −ξk⊤ Lξk with L = diag(50, 20, 10, 0.05, 0.05), amax = 5, and rmax = 1. The reward weights are selected by fine-tuning a vanilla tracking policy trained to follow the reference trajectory without any safety mechanism or safety-aware objective. That is, these weights were chosen purely for nominal sinewave tracking performance, without safety in mind. Since the environment uses ∆t = 0.05 and the reference is 20 seconds, the episode horizon is 400 steps. 34
F.2
Quadrotor Environment Details
System dynamics. experiment:
We consider the following quadrotor dynamics for the powerloop tracking ṗ = v,
v̇ = −ge3 + R(q)(acmd e3 ), −qx −qy −qz −qz qy q Ξ(q) = w qz qw −qx −qy qx qw
q̇ =
1 Ξ(q)ωcmd , 2
where Ξ(q) is the quaternion kinematic matrix. The state x = [p⊤ , v⊤ , q⊤ ]⊤ ∈ R10 consists of the inertial position p = [px , py , pz ]⊤ , the inertial linear velocity v = [vx , vy , vz ]⊤ , and the unit quaternion q⊤ = [qw , qx , qy , qz ]⊤ is the global, inertial orientation. The control input u = [acmd , ωcmd ⊤ ]⊤ consists of the mass-normalized thrust, acmd ∈ [0, 4g], for gravitational acceleration g = 9.81m/s2 , and the body rate, ωcmd = [ωx , ωy , ωz ]⊤ , where ωi ∈ [−18, 18]rad/s, i ∈ {x, y, z}, is defined with respect to the quadrotor’s body frame. Here, ∆t = 0.02 sec, and the control frequency is 50 Hz for all controllers. Safe set. The safe set for the quadrotor is defined as S = {x : pz ≤ zceil }, where zceil = 3m represents a hard ceiling. There is no safety constraint for the ground. Note that hS (x) = zceil − z has relative degree 2 for thrust, but the relative degree for bodyrate inputs is higher. Reference trajectory, reward, episode setup. The quadrotor task tracks a powerloop reference inspired by [26], shown in Fig. 5. The reference follows a vertical circular loop of radius 1.5m centered at [0, 0, 2]⊤ m, starting at the bottom of the loop with tangential speed 4.5m/s. The resulting reference completes one full loop in approximately 2.1 sec and is sampled at ∆t = 0.02 sec. It is intentionally unsafe, as the loop √ apex exceeds the ceiling zceil = 3m, and the tangential speed exceeds the free-fall threshold ε 1.5g (ε = 1.1) required for dynamic feasibility at the apex [26]. The attitude reference simultaneously commands an aggressive 360◦ flip, forcing the controller to trade off near-ceiling translational tracking, agile attitude-rate tracking, and hard safety enforcement at the most dynamically constrained portion of the maneuver. At step k, the reward is the negative weighted tracking cost rk = − wp,xy ∥px,y − pref,x,y ∥22 + wp,z (pz − pref,z )2 + wv ∥v − vref ∥22 + watt ∥eatt ∥22 + ∥ωcmd − ωref ∥2Wω + wa a2cmd + wΩ ∥ωcmd ∥22 ,
where eatt = sgn(qe,w ) qe,xyz and qe = qref ⊗ q⋆ . The weights are wp,xy = 2.5, Wω = diag(0.10, 0.20, 0.05),
wp,z = 2.0, wa = 0.01,
wv = 4.0, wΩ = 0.01.
watt = 16.0,
(126)
As in the unicycle experiment, these weights were selected by fine-tuning a vanilla tracking policy with no safety mechanism, and purely for nominal powerloop tracking of the aggressive position and attitude references shown in Fig. 5. The environment uses ∆t = 0.02 sec and an episode horizon of 106 steps, identical to the length of the powerloop reference. F.3 F.3.1
PS2-RL Implementation Details Unicycle: Phase I
Base set and controller. Let x⋆ = [0, vdes , 0]⊤ be the cruising equilibrium, with vdes = 5 and u⋆ = 02 , and define the local error state e(x) = [ y, v − vdes , ψ ]⊤ . We linearize (125) about (x⋆ , u⋆ ) and discretize it with the same ∆t = 0.05 sec as the environment timestep. We then solve the discrete-time LQR problem for (Ad , Bd ), i.e., the discretized linearization, with Qd = diag(1, 1, 1) and Rd = diag(0.01, 0.5). Let P ≻ 0 be the positive-definite solution of the discrete algebraic Riccati equation and K = (Rd + Bd⊤ P Bd )−1 Bd⊤ P Ad . 35
(127)
Figure 5: Quadrotor powerloop reference trajectory. Note that the powerloop includes an aggressive attitude reference that commands a full 360◦ flip at the apex. Then, the base controller is the LQR feedback πB (x) = −Ke(x),
(128)
where we clip the resulting control input to be within U. The base set is the ellipsoid B = {x ∈ X : e(x)⊤ P e(x) ≤ cB },
hB (x) = cB − e(x)⊤ P e(x),
(129)
with cB = 0.3. The actuator-admissible upper bound for the LQR ellipsoid is 2 rmax a2max c̄ = min , ≈ 1.05, Ka P −1 Ka⊤ Kr P −1 Kr⊤ where Ka and Kr are the acceleration and yaw-rate rows of K, respectively. Hence cB = 0.3 keeps the unclipped LQR action strictly within the input limits over B. The corresponding coordinate radii are approximately |y| ≤ 0.209, |v − vdes | ≤ 0.342, and |ψ| ≤ 0.110, well inside the safe bounds |y| ≤ 1.8 and |ψ| ≤ π/3. Design region and reference measure. We instantiate the design region as the following box n πo Ω = x ∈ X : |y| ≤ 1.8, v ∈ [0, 12], |ψ| ≤ 3 so that Ω ⊆ S covers the entire safety-relevant portion of the state space. The reference measure underlying µSA is taken to be the uniform measure on Ω \ B, which we approximate at evaluation time by a uniform 201 × 121 × 201 grid over (y, v, ψ). We count the grid points that safely arrive at θ B within horizon T = 1.0 sec under πSA , and divide by the total number of grid points in Ω \ B. Safe-arrival policy backbone. Phase I instantiates Alg. 1 with a TD3-style off-policy backθ bone [18]. We maintain a deterministic actor πSA and twin safe-arrival critics Q1SA,β and Q2SA,β , each with their corresponding target networks updated via Polyak averaging. Both critics are trained with the discounted safe-arrival Bellman recursion (11) using a Huber loss. Episodes terminate on first-hit entry into the base set B (success) or the failure set F (failure). Curriculum and initial-state sampling. Initial states are drawn from a curriculum-controlled distribution ρarr that progressively expands during training. Concretely, we maintain a scalar curriculum scale s ∈ [0, 1], initialized at s0 = 0.2, and sample initial states uniformly from y ∼ U (−ys , ys ),
v ∼ vdes + U (−vs , vs ),
ψ ∼ U(−ψs , ψs ),
where each radius ys , vs , and ψs interpolates linearly with s. We reject any sample that lies inside the base set B, so that all training initial states belong to Ω \ B. The scale s is incremented by 0.005 whenever the rolling success rate over a window of 50 episodes exceeds 0.9, with at least 50 episodes 36
between increments. This mild curriculum starts the policy from states close to B, where short safe arrivals are easy to discover, and progressively pushes the initial-state distribution outward to cover most of Ω \ B once the policy has learned to recover from such states. Note that curriculum learning is an optimization heuristic we employ, not a required component of safe-arrival policy training. The full hyperparameters are summarized in Table 8. F.3.2
Unicycle: Phase II
PS2 policy backbone. Phase II is instantiated with a SAC backbone [22, 55] for the policy πϕ , and the control-invariant layer PCIL is appended to its output. The composed backup policy πb⋆ from Phase I is fixed, and PCIL is constructed from the BCBF rollout of πb⋆ as described in App. E.7. During both data collection and policy updates, the nominal SAC action is projected through the control-invariant layer before being applied to the environment. Thus, the replay buffer stores projected controls, the actor loss evaluates the critic at projected actions, and target actions in Bellman updates are also projected. The SAC actor is a tanh-squashed Gaussian policy, and evaluation uses the deterministic mean action followed by the same projection. Initial states are sampled uniformly from CT (πb⋆ ). Episodes have a fixed horizon of 400 steps, matching the 20-second sinusoidal reference with ∆t = 0.05. Control-invariant layer parameters. The unicycle safety set has four scalar inequalities, and the terminal base condition is the single LQR ellipsoid hB (x) ≥ 0. With T = 1.0 second and ∆t = 0.05, we have N = 20 backup steps. Thus, the finite-mesh BCBF construction contributes 4(N + 1) + 1 = 85 BCBF rows. Together with the input box and the numerical slack nonnegativity row, the implemented QP has 90 inequalities. The full hyperparameters are summarized in Table 9. Table 8: PS2-RL Phase I (safe-arrival policy) hyperparameters. Quantity
Unicycle
Total environment steps Update schedule Replay buffer / minibatch size Actor / critic architecture Actor / critic LR (Adam) Polyak averaging coefficient τ Policy update delay Twin critic target Critic loss Exploration noise / clip Safe-arrival discount β ∆t Backup horizon T Curriculum (s0 , ∆s) Curriculum success window / threshold Curriculum min episodes between updates Curriculum mechanism
F.3.3
6
3 × 10 1 grad. step / 8 env. steps 4 × 105 / 128 MLP, hidden [128, 128] 3 × 10−4 / 3 × 10−4 0.0025 every 2 critic updates min(Q1SA,β , Q2SA,β ) Huber, δ = 1.0 0.08 / 0.10 0.92 0.05 1.0 sec (0.2, 0.005) 50 ep. / 0.9 50 isotropic radius scaling
Quadrotor 5 × 106 1 grad. step / 8 env. steps 4 × 105 / 128 MLP, hidden [128, 128] 10−4 / 3 × 10−4 0.0025 every 2 critic updates min(Q1SA,β , Q2SA,β ) Huber, δ = 1.0 0.10 / 0.10 0.99 0.02 2.0 sec (0.0, 0.10) 50 ep. / 0.8 100 region-mixture and per-region radius
Quadrotor: Phase I
Base set and controller. For the base set and controller construction for the quadrotor experiment, we use the reduced hover-error state xe = [ pz − zdes , vx , vy , vz , 2qerr,x , 2qerr,y , 2qerr,z ]⊤ ∈ R7 ,
where 2qerr,{x,y,z} are first-order rotation-angle error coordinates obtained from the sign-corrected quaternion error. The hover equilibrium is x⋆e = 07 with zdes = 2 and u⋆ = [g, 0, 0, 0]⊤ . In these coordinates, we compute the reduced-state discrete-time linearization about hover to retrieve (Ad , Bd ). We use the same ∆t = 0.02 sec as the environment timestep to construct (Ad , Bd ), and solve the discrete-time LQR problem with Qd = diag(1.0, 0.16, 0.16, 0.4, 0.8, 0.8, 0.16) and Rd = diag(0.02, 0.012, 0.012, 0.004). Let P ≻ 0 be the discrete Riccati solution from (127) and the base controller is the hover LQR feedback πB (x) = u⋆ − Kxe 37
Table 9: PS2-RL Phase II (PS2 policy) hyperparameters. Quantity
Unicycle
Quadrotor
6
10 1 grad. step / 8 env. steps 3 × 105 / 64 MLP, hidden [128, 128] 10−4 / 3 × 10−4 / 10−4 0.99 / 0.005 α0 = 0.2, αmin = 0.01, H̄ = −2 5.0 / 5 × 106 yes yes (4.0, 2.0) 105 qpax [50] 400 steps (20 sec) none
Total environment steps Update schedule Replay buffer / minibatch size Actor / critic architecture Actor LR / critic LR / temp. LR (Adam) Discount γ / Polyak coefficient τ Entropy temperature Gradient norm clip / Q clipping Project actor actions in actor loss Project target-actor actions in critic backup Class-K∞ gains (αS , αB ) QP slack penalty λδ Differentiable QP solver Episode horizon Warm start
1.5 × 106 1 grad. step / 8 env. steps 3 × 105 / 64 MLP, hidden [256, 256] 5 × 10−5 / 10−4 / 5 × 10−5 0.99 / 0.005 α0 = 0.2, αmin = 0.01, H̄ = −4 5.0 / 5 × 106 yes yes (4.0, 2.0) 106 qpax [50] 106 steps (≈ 2.1 sec) vanilla powerloop-tracking SAC
where we clip the input to be within U. The base set is the ellipsoid B = {x ∈ X : x⊤ e P xe ≤ cB },
hB (x) = cB − x⊤ e P xe ,
with cB = 8.0. This is the value of both the codebase’s LQR terminal set and LQR capture set in the reported quadrotor safe-arrival and PS2-RL runs. The actuator-admissible upper bound associated with the unclipped LQR action is c̄ =
ū2i ≈ 12.38, i∈{1,...,4} Ki P −1 Ki⊤ min
where ūi denotes the one-sided actuator margin around u⋆ for control channel i. Hence cB = 8.0 keeps the unclipped base-controller action inside the input limits over B. At this level, the largest possible deviations in the reduced coordinates are approximately |pz − zdes | ≤ 0.490, |2qerr,x | ≤ 1.177,
|vx | ≤ 1.986, |2qerr,y | ≤ 1.177,
|vy | ≤ 1.986, |2qerr,z | ≤ 2.437.
|vz | ≤ 1.282,
In particular, the altitude radius gives pz ≤ zdes + 0.490 < zceil = 3, so the LQR base ellipsoid lies strictly below the ceiling safety boundary. Design region and reference measure. The quadrotor state space is 10-dimensional, so a uniform grid over S is computationally intractable. We therefore anchor the design region to the powerloop reference trajectory itself. Concretely, we collect 20 rollout traces from a vanilla SAC powerloop tracker trained without any safety mechanism, and define the design region Ω as the union of perturbation balls around the states visited by these traces, with per-axis maximum perturbations of 0.4 m in position, 1.5 m/s in linear velocity, 30◦ in body tilt, and 12◦ in yaw. This concentrates Ω on a tube around task-relevant powerloop states while still containing aggressive deviations from the reference, including states close to the ceiling. Note that we enforce Ω ⊆ S by rejection sampling, accepting only perturbed states for which hS (x) ≥ 0.
To further stratify Ω along the structure of the powerloop task, we partition the trace into four sub-regions: a general-trace region of safe trace states away from both the ceiling and the base set; a near-ceiling region of safe trace states within 0.25 m below zceil ; a bridge region of synthetic states right below the ceiling, obtained by linearly interpolating between the unsafe endpoints of the reference; and a capture-shell region of safe trace states close to B. The reference measure underlying µSA is taken to be the region-weighted uniform measure on the four sub-regions, with greater weight assigned to the near-ceiling and bridge regions since these are the states that limit performance during powerloop tracking. We approximate µSA at evaluation time by drawing 1024, 1024, 1024, and 512 perturbed initial states from the four sub-regions, simulating the candidate safe-arrival policy for T = 2.0 sec, and counting the fraction that safely reach B while remaining in S. Safe-arrival policy backbone. backbone [18] as the unicycle.
Phase I for the quadrotor uses the same TD3-style off-policy
38
Curriculum and initial-state sampling. As in the unicycle, training initial states are drawn from a curriculum-controlled ρarr on the train split, but the scale s ∈ [0, 1] now controls the relative emphasis across the four sub-regions rather than a single perturbation radius: at s = 0, initial states are concentrated on the general-trace and capture-shell regions where short safe arrivals are easy, and as s → 1 it shifts toward the harder near-ceiling and bridge regions. The bridge and near-ceiling states are perturbed most aggressively. Samples that fall outside S or inside B are rejected. The full hyperparameters are summarized in Table 8, alongside the unicycle settings. F.3.4
Quadrotor: Phase II
PS2 policy backbone. Phase II for the quadrotor uses the same SAC + control-invariant layer backbone as the unicycle, with two task-specific differences. First, initial states are drawn with ±0.1 m position perturbations around the powerloop start state and episodes have a fixed horizon of 106 steps matching the powerloop reference at ∆t = 0.02 sec. Second, we warm-start training from a vanilla powerloop-tracking SAC checkpoint. This is based on the warm-start scheme suggested in HardNet-CVX [41, App. C.1], where the nominal network is first trained without the projection layer to ease optimization through it. Crucially, the control-invariant layer projects every action throughout training and deployment, so the formal safety guarantee holds regardless of how πϕ is initialized. Control-invariant layer parameters. The quadrotor safety set is the single ceiling inequality hS (x) = zceil − pz ≥ 0, and the terminal base condition is again a single LQR ellipsoid. With T = 2.0 seconds and ∆t = 0.02, we have N = 100 backup steps. Thus, the finite-mesh BCBF construction contributes 1 · (N + 1) + 1 = 102 BCBF rows, and together with the input box and the slack nonnegativity row, the implemented QP has 111 inequalities. The full hyperparameters are summarized in Table 9, alongside the unicycle settings. F.4
Baseline Implementation Details
Common training and evaluation protocol. All baselines are trained with 10 random seeds per task on the same trajectory-tracking environments, task rewards, initial-state sampler, and reference trajectories as PS2-RL (App. F.1, F.2). The unicycle environment uses ∆t = 0.05 sec, and the episode length is 400 steps. For the quadrotor, ∆t = 0.02 sec, and the episode length is 106 steps. The hyperparameters used for the baselines intentionally match the PS2-RL Phase II training settings whenever the same quantity applies for each environment, including the hidden layer width and depth, batch size, total training steps, learning rates, entropy, neural network initialization, etc. F.4.1
RL with Violation Penalty
We implement the penalty baseline with the same SAC backbones as Phase II in PS2-RL, but with the control-invariant layer components disabled. All other hyperparameters and configurations remain the same as PS2-RL. Safety enters only through an additive reward penalty rkpen = rktask − λpen 1{xk+1 ∈ / S}, where rktask = −ξk⊤ Lξk is the task reward described in App. F.1 and App. F.2. We report two penalty strengths, SAC-Penlow with λpen = 1.0 and SAC-Penhigh with λpen = 1000.0. For the quadrotor penalty baseline, training directly from a random policy often spent most of the early training horizon far from useful powerloop tracking behavior. We therefore warm-started the final quadrotor penalty runs from a vanilla SAC tracker trained only to follow the powerloop trajectory, with no safety objective. This gave the penalty method a strong tracking initialization and made the comparison more favorable to the baseline. F.4.2
Safe RL via Constrained Policy Optimization
We implement Constrained Policy Optimization (CPO) [1] and SAC-Lagrangian following the standard constrained-policy and Lagrangian safe-RL formulations [22, 55]. These methods use the same trajectory-tracking reward as above and introduce an auxiliary cost ck = 1{xk+1 ∈ / S}. The cost limit is set to 0.0 in all runs. This is the strictest binary-cost setting, as any unsafe transition contributes positive cost, while a perfectly safe rollout has zero cost. As usual for CMDP methods, 39
this constraint is optimized as an expected cost constraint during training, rather than as a pointwise per-trajectory guarantee. Constrained Policy Optimization (CPO). Our CPO implementation uses a Gaussian policy with two hidden layers and a separate reward-value and cost-value network. Each policy update collects a batched rollout, computes generalized advantage estimates for reward and cost, and solves the local trust-region constrained update using conjugate gradient and backtracking line search, following [1]. SAC-Lagrangian. The SAC-Lagrangian baseline uses the same off-policy SAC backbone as the penalty baseline, but learns separate double critics for reward and cost. The actor minimizes the Lagrangian SAC objective, E[αlr log πϕ (u | x) − Qr (x, u) + λQc (x, u)], where λ ≥ 0 is updated by projected gradient ascent on the empirical cost violation. During tuning, we tried warm-starting the quadrotor CMDP baselines from the same vanilla powerloop-tracking SAC checkpoint used by the quadrotor penalty baseline. In those trials, the post-training CPO and SAC-Lagrangian updates did not move the policy far enough away from the unsafe vanilla tracker, i.e., tracking remained close to the warm-start behavior, and thereby, ceiling violations remained frequent. Hence, the final reported CMDP baselines use non-warm-start training. F.4.3
Safe RL via Verified Certificate
CBF-RL. We implement the CBF-RL method from [54], where CBF is used for action filtering and reward shaping during training, but the resulting policy is deployed without a runtime filter. The training-time, closed-form safety filter in [54] is as follows: ( unom , a(x)⊤ unom ≥ b, ⊤ (130) ucbf−rl (x) = unom )a(x) , otherwise, unom − (b(x)−a(x) ∥a(x)∥2 where a(x) = ∇hC (x) and b(x) = −αhC (x) for a CBF hC , and unom is the output from a nominal policy. Note that the original CBF-RL does not consider explicit control limits. In our implementation, we choose to clip the filtered control input ucbf−rl : usafe = clip(ucbf−rl , U). Moreover, during training, [54] penalizes unsafe behavior through reward shaping with a penalty term: ∥unom − usafe ∥2 − 1 , (131) rcbf−rl (x, u) = min a(x)⊤ unom − b(x), 0 + exp − σ2
where σ is a scaling scalar. Note that we use usafe , the clipped control input, in (131), whereas the original CBF-RL in [54] uses the unclipped input ucbf−rl . First, valid CBFs need to be synthesized/designed to be used within the CBF-RL framework. For the unicycle task, we synthesize a CBF using sum-of-squares programming (SOSP), based on the implementation in [13], where the details for the SOS formulation can be found. The SOSP was modeled in MATLAB using the YALMIP toolbox [37] and solved utilizing the MOSEK optimization suite [9]. For the quadrotor task, computational synthesis of a formal control-invariant set is not practical, as the system is 10-dimensional, nonlinear, quaternion-valued, and subject to tight actuator bounds. HJ reachability scales very poorly with dimension, making it an infeasible option for our 10-dimensional system. SOSP is restricted to polynomial dynamics and a fixed polynomial degree, as shown in the unicycle CBF-RL case. However, the quadrotor system (126) is quaternion-based, which is non-polynomial, and even after converting to Euler-angle based and applying polynomial relaxation, the resulting semidefinite program scales poorly, making it difficult at this state dimension. We therefore choose the high-order CBF (HOCBF) formulation [53], which sidesteps formal synthesis by recursively building barrier conditions through Lie derivatives. For a relative-degree-γ safety function hS (x), HOCBF defines h0 (x) := hS (x),
hi (x) := ḣi−1 (x) + αi (hi−1 (x)),
i = 1, . . . , γ,
and enforces hγ (x, u) ≥ 0 with u ∈ U. For our ceiling constraint, hS (x) = zceil − pz . Using (126), ḣS (x) = −vz ,
ḧS (x, u) = g − acmd R33 (q). 40
With extended class-K∞ functions αi (s) = ki s, this gives the implementable second-order HOCBF condition: ḣC (x) = g − acmd R33 (q) − (k1 + k2 )vz + k1 k2 (zceil − pz ) ≥ 0. Although this is a valid HOCBF condition, it exposes a key limitation of HOCBFs for this system. The constraint is affine only in the thrust command and contains no direct dependence on the body-rate commands. The body rates influence ceiling safety only through future attitude evolution, so the filter cannot fully exploit the available control authority at the current step. In other words, the thrust acmd has relative degree 2, while the body rate ωcmd has a higher relative degree. This is precisely the issue PS2-RL avoids, as the induced BCBF constraints are relative-degree-one for all actuators by construction. The CBF-RL policy optimizer uses the same SAC implementation as the penalty baseline. During training, for each nominal action unom , we apply the closed-form CBF-RL filter in (130), clip the result to the control limits, and step the environment with the filtered action usafe . The CBF-based penalty term (131) is added to the task reward with weight wcbf pen . Post-training evaluation uses the deterministic actor mean without the CBF filter, matching the CBF-RL protocol of learning a policy that is deployed without runtime shielding. During tuning, we swept various hyperparameter combinations, and report the combination that resulted in the best safety and tracking performances. Namely, we tried α ∈ {0.01, 0.03, 0.05, 0.1, 1.0, 2.0}, wcbf pen ∈ {10, 100}, σ = 0.5, and warm-start/non-warm-start variants for the quadrotor. Model Predictive Shielding (MPS). We implement Model Predictive Shielding [11], a runtime switching shield rather than a projection layer or a reward-penalty method. It uses three policies: a learned nominal task policy π̂, a learned recovery policy πrec , and a fixed equilibrium controller πeq . At deployment time, the shield first checks whether applying π̂ for one step leaves the system in a state that can be recovered to a stable invariant set within N steps under πrec . If so, it applies the nominal action; otherwise it switches to the recovery policy or, once inside the invariant set, the equilibrium LQR controller. The shielded action is π̂(ok ), RecN (f (xk , π̂(ok )), k + 1) = 1, πeq (xk ), xk ∈ Xinv , πMPS (xk , k) = π (o ), RecN (xk , k) = 1, rec k πeq (xk ), otherwise.
The final branch is a best-effort fallback, and logged separately. Both learned policies are trained with the differentiable model-based backpropagation-through-time (BPTT) procedure from [11]. For a deterministic actor πθ , we roll out the known dynamics for a fixed horizon of N -timesteps and update θ with Adam on the negative discounted return. The nominal policy π̂ is trained only on the task reward, with no shield and no safety penalty. The recovery policy πrec is trained independently with the shaped recovery reward from [11].
MPS shares parts of the high-level structure of PS2-RL: both methods use a learned recovery/arrival policy with an LQR-stabilized equilibrium, and both forward-integrate the dynamics under the policy to verify safe arrival to a small invariant set. However, MPS fundamentally differs from PS2-RL in that it uses a switching mechanism with a non-differentiable post-hoc shield. Specifically, MPS runs either π̂(x) or the recovery policy in full, depending on whether the flow under π̂(x) is recoverable. Furthermore, as the recovery policy is a non-differentiable post-hoc wrapper, π̂ is trained without exposure to the safety constraint, i.e., the task policy learns to maximize performance, and the shield simply catches violations at deployment. On the other hand, PS2-RL projects πϕ (x) onto the BCBF-admissible set via the control-invariant layer, returning the closest safe action to the nominal. This exploits the available safety margin smoothly rather than triggering extensive overrides at the boundary of the certified set, as shown in Fig. 6. Moreover, the control-invariant layer is differentiable, and gradients propagate through the projection back into πϕ during training. Thus, the PS2 policy learns to optimize task performance within the certified set rather than against it. Note that we grant MPS twice the recovery horizon (40 steps) used by PS2-RL’s backup policy (20 steps) in the unicycle experiment. Similarly, MPS is allowed 106 recovery steps in the quadrotor experiment, higher than PS2-RL’s 100 steps. This is so that MPS can expand its certified recoverable 41
set and give the baseline its best-case performance. Despite this advantage, PS2-RL achieves higher tracking performance in both experiments. PS2ABP : PS2-RL with an analytic backup policy. PS2ABP is implemented as the same Phase II PS2-RL algorithm as PS2SA , but with Phase I safe-arrival policy learning disabled. That is, instead θ of using the learned safe-arrival policy πSA , PS2ABP uses an analytic safe-arrival controller. The certified base set and LQR base controller remain the same as PS2SA . Thus, the comparison between PS2ABP and PS2SA isolates the effect of the learned safe-arrival policy on the downstream RL task performance. For the unicycle task, the analytic safe-arrival controller is the same LQR controller as the base controller (128). For the quadrotor task, we employ an aggressive cascaded PID-style recovery controller. The outer loop generates a virtual acceleration that damps translational motion, regulates altitude toward the hover region, and becomes more conservative near the ceiling. This virtual acceleration is converted into a thrust command and desired attitude, while an inner-loop quaternion attitude controller tracks that attitude until the trajectory enters the base set. After fixing πSA , we train the PS2ABP policy with the same Phase II settings as PS2SA .
G
Additional Results, Ablation Studies, and Computation
Checkpoint selection and evaluation. All methods (PS2SA and the seven baselines) are trained with identical 10 seeds. During training, we periodically evaluate each method on 10 episodes and select the best-performing checkpoint per seed. Each selected checkpoint is then evaluated on 1,000 episodes, and all reported metrics in Tables 1, 2, 10, 11 are aggregated across the resulting 10 × 1, 000 = 10, 000 episodes per method. Metrics. For each selected checkpoint, the evaluator reduces every rollout to episode-level RMSEs (root mean square errors) for the reported tracking coordinates, a binary safe-episode indicator, the episode’s maximum constraint violation, and its cumulative exceedance. An episode is safe only if no sampled state violates any safety constraint. Tables 1 and 2 report the interquartile mean (IQM) and 95% confidence interval across the 10 seed-level summaries for the RMSE columns and per-seed safety. “Total Safety” is the safe-episode fraction over all 10,000 evaluation episodes, and “Worst Viol.” is the largest episode-wise violation over all seeds. Tables 10 and 11 report the complementary aggregations of the same evaluation episodes used in Tables 1 and 2: mean ± standard deviation across the 10 seed-level summaries, and “Worst-seed Safe%” as the minimum seed safety rate. Thus, the tables in this Appendix are not new experimental data but rather emphasize variability, outlier seeds, and the severity of violations. G.1
Unicycle: Extended Analysis
Phase I analysis. The unicycle task permits dense evaluation of the safe-arrival set size over the full design region. The learned safe-arrival policy increases the safe-arrival fraction from 0.227 for the analytic policy to 0.326 over Ω. Moreover, the learned set covers 99.08% of the analytic set, so Phase I strictly enlarges the safe-arrival set rather than relocating it. The expansion is especially relevant around the task speed. At vdes = 5 m/s, the learned safe-arrival slice area is 1.43× larger. This result is visually shown in Fig. 2. Phase II analysis. The Phase II results in Tables 1 and 10 are consistent with this enlargement of the safe-arrival set. Among methods that are uniformly safe across all seeds, PS2SA obtains the lowest lateral and heading errors, with y-RMSE 0.53 ± 0.02 m and ψ-RMSE 0.11 ± 0.01 rad. Compared with PS2ABP , this reduces lateral error by roughly 46% and heading error by roughly 27%, while preserving 100% safety and zero measured violation. The velocity error of PS2SA is higher than that of PS2ABP and MPS, but this reflects a different safety-performance tradeoff: PS2ABP and MPS stay closer to a conservative recovery behavior, whereas PS2SA uses the larger admissible set induced by the learned backup policy to track the unsafe lateral reference more closely. After accounting for both the diagonal weights (50 for y vs. 20 for v) and the per-component normalizers (1.8 vs. 5), the reward penalizes squared lateral error roughly 19× more heavily than squared velocity error, making this tradeoff a clear net gain under the task objective. SAC-Penhigh also achieves 100% safety, but 42
Figure 6: Representative lateral trajectories on the unicycle lane-keeping task. Each panel plots the lateral position y(t) of one method over a 20-second rollout. The dotted black curve denotes the unsafe sinusoidal reference, and the dashed red lines denote the lane safety boundary |y| = ymax = 1.8 m. PS2SA closely tracks the reference while remaining inside the safe lane, whereas PS2ABP is safe but more conservative due to the smaller analytic-backup-induced admissible set. CBF-RL and SACPenhigh also remain safe but sacrifice substantial tracking performance, MPS exhibits interventioninduced nonsmooth behavior near the boundary, and the penalty/CMDP methods with weaker effective safety enforcement violate the lateral constraint.
43
Table 10: Unicycle experiment results, across models trained with 10 different seeds, each evaluated for 1,000 episodes. The mean and standard deviations across all 10,000 episodes are shown. “Worstseed Safe %” is the safety rate of the unsafest model. Tracking Performance (RMSE)
Pen
SAC-Penlow SAC-Penhigh
Con
SAC-Lag. CPO
PSRL
Safe RL
CBF-RL MPS PS2ABP PS2SA
Safety Performance
y (m)
v (m/s)
ψ (rad)
Worst-seed Safe %
Mean Max. Viol. (m)
Mean Cumul. Viol. (m)
0.68 ± 0.48 1.13 ± 0.23
2.18 ± 1.13 2.08 ± 1.08
0.26 ± 0.27 0.16 ± 0.05
0.0% 100%
0.77 ± 0.73 0.00 ± 0.00
54.12 ± 46.30 0.00 ± 0.00
1.77 ± 0.09 0.67 ± 0.16 0.98 ± 0.00 0.53 ± 0.02
4.25 ± 1.18 0.48 ± 0.08 0.45 ± 0.15 0.82 ± 0.08
0.27 ± 0.05 0.17 ± 0.03 0.15 ± 0.00 0.11 ± 0.01
99.1% 96.8% 100% 100%
0.54 ± 0.18 4.83 ± 6.25
1.28 ± 0.41 4.25 ± 1.63
0.12 ± 0.04 0.41 ± 0.37
0.0% 0.0%
0.40 ± 0.38 7.01 ± 13.66
0.00 ± 0.00 0.004 ± 0.001 0.00 ± 0.00 0.00 ± 0.00
26.41 ± 31.06 1005.29 ± 2000.82 0.00 ± 0.00 0.03 ± 0.008 0.00 ± 0.00 0.00 ± 0.00
its much larger lateral and velocity errors show the conservatism of enforcing safety only through a large reward penalty. Table 10 also exposes the instability of methods whose safety is not enforced pointwise. SAC-Penlow and SAC-Lagrangian have competitive average tracking on some episodes, but their worst-seed safety is 0%, with nonzero mean maximum and cumulative violations. CPO is particularly heavy-tailed, with large tracking variance and very large cumulative violations. CBF-RL and MPS are closer to the safe-RL goal, but neither is uniformly safe in the unicycle evaluation. For CBF-RL, this is expected because the runtime filter is removed at deployment. For MPS, the rare violations appear to arise from chattering in the switching shield. Near the recoverability boundary, the controller can rapidly alternate between the policies, making the sampled-data rollout sensitive to integration, thresholding, and finite-precision errors. Thus, the rare MPS violations in the unicycle task are consistent with a sampled-data artifact under the coarser ∆t = 0.05. In contrast, no MPS violations are observed in the quadrotor task, where the controller is evaluated at the finer period ∆t = 0.02. The MPS rollout shown in Fig. 6 illustrates this behavior: the shielded trajectory repeatedly rides the safety boundary and then undergoes abrupt intervention-induced switches. G.2
Quadrotor: Extended Analysis
Phase I analysis. Since a dense grid evaluation over the full safe set is not possible for this 10dimensional system, we evaluate Phase I on the task-relevant distribution introduced in App. F.3.3. Over the design region Ω, the learned safe-arrival policy increases the recoverability rate from 60.9% for the analytic policy to 85.9%. The gains are largest where the analytic safe-arrival policy limits downstream tracking performance. On near-ceiling states, recoverability improves from 35.5% to 69.3%; on bridge states below the unsafe loop apex, it improves from 47.3% to 78.6%; on general trace states, it improves from 77.7% to 96.1%; and on capture-shell states, it reaches 100%. Fig. 7 visualizes this difference. The two rows show safe-arrival rollouts from task-relevant quadrotor states, with side and front views. The translucent red plane denotes the hard ceiling, and the gold segments indicate the portion after the trajectory enters the certified base set, where the base controller πB takes over. While the learned safe-arrival policy bends the trajectories away from the ceiling and into the base set, the analytic policy is less effective from aggressive near-ceiling and bridge states. Phase II analysis. The Phase II results in Tables 2 and 11 show that the larger learned backupinduced set directly improves the task performance. Among the methods with 100% safety in every seed, PS2SA has the best tracking performance: 0.63 ± 0.13 m position RMSE, 0.97 ± 0.24 m/s velocity RMSE, and 0.43 ± 0.12 rad attitude RMSE. While PS2ABP and MPS baselines achieve 100% safety, they do so by inducing a smaller or less task-aligned admissible set, which forces the task policy to cut the loop and deviate from the reference. SAC-Penlow tracks parts of the loop well but has 0% worst-seed safety. SAC-Penhigh is almost safe, but still has a worst-seed safety rate of 99.8% and is substantially more conservative than PS2SA . The CMDP methods optimize safety only through expected costs and therefore do not enforce pertrajectory ceiling satisfaction; both CPO and SAC-Lagrangian have 0% worst-seed safety. CBF-RL is much closer to safe behavior, but its worst-seed safety rate is 98.4%, and its tracking errors remain large. The decrease in performance is mainly because the quadrotor HOCBF constraint acts directly only on thrust, with no dependence on the body rate commands that are crucial for agile attitude 44
θ Figure 7: Safe-arrival policies in action: (top) πSA learned policy and (bottom) πSA analytic policy. The base set trajectory segments are controlled by the base controller πB .
recovery. MPS, by contrast, remains safe in this task, but is much more conservative due to switching into recovery/equilibrium behavior far away from the ceiling. Fig. 8 illustrates these distinctions. Table 11: Quadrotor experiment results, across models trained with 10 different seeds, each evaluated for 1,000 episodes. The mean and standard deviations across all 10,000 episodes are shown. “Worstseed Safe %” is the safety rate of the unsafest model. Tracking Performance (RMSE)
Pen
SAC-Penlow SAC-Penhigh
Con
SAC-Lag. CPO
PSRL
Safe RL
CBF-RL MPS PS2ABP PS2SA
G.3
Safety Performance
p (m)
v (m/s)
θq (rad)
Worst-seed Safe %
Mean Max. Viol. (m)
Mean Cumul. Viol. (m)
1.04 ± 0.45 1.25 ± 0.33
1.95 ± 0.65 2.24 ± 0.51
1.17 ± 0.52 1.54 ± 0.39
0.0% 99.8%
0.15 ± 0.18 0.00 ± 1.2e-7
2.71 ± 3.68 8.0e-7 ± 2.5e-6
4.33 ± 0.38 4.07 ± 0.25 1.76 ± 0.21 0.97 ± 0.24
2.05 ± 0.28 1.41 ± 0.12 0.82 ± 0.23 0.43 ± 0.12
98.4% 100% 100% 100%
8.7e-5 ± 2.6e-4 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00
7.0e-4 ± 2.1e-3 0.00 ± 0.00 0.00 ± 0.00 0.00 ± 0.00
2.14 ± 0.70 10.28 ± 4.36 2.36 ± 0.72 2.00 ± 0.07 1.42 ± 0.15 0.63 ± 0.13
4.29 ± 0.50 11.64 ± 3.35
2.01 ± 0.35 2.19 ± 0.19
0.0% 0.0%
0.28 ± 0.48 0.42 ± 0.75
9.54 ± 17.98 16.02 ± 30.67
Ablation Studies: Role of the Control-Invariant Layer
We perform paired ablations on the quadrotor powerloop task to isolate the two roles of the controlinvariant layer (CIL): providing safety guarantees at deployment and end-to-end constrained policy optimization during training. For this diagnostic study, we select the best-tracking PS2SA checkpoint among the ten quadrotor seeds and evaluate three policies tied to the same training run. PS2SA is the full policy. PS2SA w/o CIL uses the same trained policy network but disables the projection layer at deployment. Vanilla+CIL uses the unconstrained tracking policy that warm-started this PS2SA run, but applies the same CIL only at evaluation. Thus, the first ablation removes the certified projection after training, while the second tests whether a post-hoc filter alone is sufficient without training through it. The results are presented in Table 12 and the pz -trajectories are plotted in Fig. 9. 45
Figure 8: Quadrotor powerloop trajectories for all evaluated methods. Each method is shown with side and front views of a representative rollout. The dashed black line denotes the unsafe powerloop reference, and the semi-transparent red plane denotes the hard ceiling at zceil = 3m. Note that while we show the ground as a grid, there is not safety constraint for it. PS2SA tracks the aggressive reference most closely while remaining below the ceiling. PS2ABP and the other certified/shielded baselines are more conservative, while penalty- and CMDP-based baselines either deviate substantially from the maneuver or exhibit ceiling violations.
Table 12: Paired ablation on the role of the control-invariant layer (CIL) in the quadrotor powerloop task. All rows are evaluated for 1,000 episodes using components from the same selected PS2SA run: Vanilla+CIL uses the unconstrained tracking warm-start model with the CIL added only at evaluation, PS2SA w/o CIL disables the CIL after PS2-RL training, and PS2SA is the full model. Whenever active, the CIL uses the same learned safe-arrival backup policy. All metrics are averaged over 1,000 episodes. Violation metrics are ceiling violations in pz . Tracking Performance
Safety Performance
Safe RL
p (m)
v (m/s)
θq (rad)
Total Safety %
Mean Max. Viol. (m)
Mean Cumul. Viol. (m)
Vanilla + CIL PS2SA w/o CIL
0.9283 0.7395
1.1336 3.5522
0.4314 1.0292
100% 0.0%
0.00 0.1975
0.00 2.6964
PS2SA
0.5275
0.6267
0.2043
100%
0.00
0.00
46
Figure 9: Altitude traces for the control-invariant layer ablation. For each policy, we plot the besttracking episode, measured by tracking error, among its 1,000 evaluation episodes. Aggregated metrics over all episodes are reported in Table 12. The dashed black curve is the unsafe powerloop reference and the dashed red line is the ceiling zceil = 3 m. Policies evaluated with the controlinvariant layer remain below the ceiling, while PS2SA without the control-invariant layer exhibits a small overshoot near the loop apex.
Deployment Ablation: removing the control-invariant layer. The safety guarantee of PS2-RL is a guarantee on the composed projected policy PCIL (πϕ ), not on the raw nominal network πϕ alone. Table 12 confirms that this distinction is essential in deployment. When the CIL is disabled after training, the resulting PS2SA w/o CIL violates the ceiling in every evaluation episode, dropping from 100% safety to 0.0% safety. Figure 9 visualizes this failure mode. Note that the 0.0% safety rate is based on the hard-constraint metric, where any ceiling crossing marks the episode unsafe. The unfiltered policy’s failures are small altitude overshoots, with a mean peak violation of 0.1975 m above the 3 m ceiling. The larger cumulative violation reflects these small exceedances accumulated across violating timesteps, rather than large off-grid excursions. This shows that the policy network has not simply internalized the hard safety constraint, but has learned to optimize task performance through the projection layer. The CIL is therefore the safety-guaranteeing component that must remain active at deployment. Training Ablation: post-hoc projection without CIL in training. The Vanilla+CIL row evaluates the opposite ablation: an unconstrained vanilla tracker is shielded by the CIL only at deployment. This policy is safe in all 1,000 episodes, as expected from the projection onto the BCBF-admissible set. However, its tracking performance is substantially worse than the full PS2SA policy. Training end-to-end through the CIL improves the position, velocity, and attitude errors by approximately 43%, 45%, and 53%, respectively, compared to the post-hoc shielding. Thus, while the CIL alone can enforce safety at evaluation time, exposing the RL policy to the same projection during training is what allows the policy to learn high-performance actions within the certified admissible set. Together, these ablations show that PS2-RL is not merely a test-time filter nor a nominal policy that can be deployed without its filter: the control-invariant layer is required for the deploymenttime guarantee, and differentiable training through the control-invariant layer is required to recover performance under that guarantee. This behavior is consistent with observations in [41]. G.4
Computation Details and Time
Computation resources. All experiments were run as single-GPU jobs on a node with one NVIDIA V100 GPU and an Intel Xeon Gold 6248 CPU allocation with 20 CPU cores per job. We used JAX with CUDA for batched environment rollouts, policy updates, backup-flow integration, and controlinvariant layer evaluation. No multi-GPU training was used. The only computation outside this JAX pipeline was the CBF-RL baseline for the unicycle, where MATLAB was used to synthesize a polynomial CBF via sum-of-squares programming with YALMIP and MOSEK [37, 9]. Timing protocol. We report two types of timing. First, Phase I training time is the wall-clock time required to train the safe-arrival policy before freezing the composed backup policy πb⋆ . This cost is paid only for PS2SA and only once per safe-arrival policy design. PS2ABP uses an analytic backup 47
policy and therefore has no Phase I learning cost. Second, Phase II training time is the wall-clock time of the PS2 policy training job with the control-invariant layer. The Phase II timing includes the repeated construction of the BCBF-admissible set, the control-invariant layer QP solves, and policy/critic updates. Inference time is measured as the per-control-step state-to-action latency of the deployed PS2 policy after JAX compilation and warm-up. It includes backup rollout evaluation, BCBF constraint assembly, and the QP solve, but excludes environment stepping and logging. All Phase II training and inference statistics in Table 13 are averaged over the same 10 random seeds used in the main experiments. Phase I safe-arrival training cost. For the final learned backup policies used in PS2SA , Phase I safe-arrival policy training required 842.24 seconds for the unicycle task and 2408.89 seconds for the quadrotor task. Phase II training and deployment cost. Table 13 reports the computational cost of the two PS2-RL variants. The Phase II training times are on the order of 4.6 hours for the unicycle task and 13–14 hours for the quadrotor task. The increase for the quadrotor is expected because the system has a higher-dimensional state, a four-dimensional control input, and a longer backup rollout, which increases the cost of backup flow and sensitivity integration. Nevertheless, the online control-invariant layer remains small in decision dimension. Thus, the deployment-time cost is dominated by evaluating the backup rollout and solving a small QP, rather than by explicit invariant-set computation. The measured per-step inference latency is well below the environment control period in both tasks. For PS2SA , the unicycle latency is 0.35 ms, compared with a 50 ms control period, and the quadrotor latency is 0.80 ms, compared with a 20 ms control period. Table 13: Computation time for PS2-RL, averaged across 10 Phase II random seeds. “Training time” denotes the wall-clock time for Phase II task-RL training with the control-invariant layer. “Inference time” denotes the per-control-step latency of the deployed PS2-RL policy after compilation. Unicycle: Lane-Keeping
Quadrotor: Powerloop
Safe RL
Training Time (s)
Inference Time (µs)
Training Time (s)
Inference Time (µs)
PS2ABP PS2SA
16731.87 ± 1558.27 16482.99 ± 1068.22
360.48 ± 24.87 347.74 ± 12.62
46859.70 ± 1719.70 48938.12 ± 2932.48
758.82 ± 85.89 804.35 ± 170.91
48