Dyna-Style Safety Augmented Reinforcement Learning: Staying Safe in the Face of Uncertainty
Artur Eisele * 1 Bernd Frauenknecht * 1 Friedrich Solowjow 1 Sebastian Trimpe 1
arXiv:2604.25508v1 [cs.LG] 28 Apr 2026
Abstract
Safety Filter Learning
Model Learning
Safety remains an open problem in reinforcement learning (RL), especially during training. While safety filters are promising to address safe exploration, they are generally poorly suited for high-dimensional systems with unknown dynamics. We propose Dyna-style Safety Augmented Reinforcement Learning (Dyna-SAuR), a novel algorithm that learns both a scalable safety filter and a control policy using a learned uncertaintyaware dynamics model, while requiring minimal domain knowledge. The filter avoids failures and high uncertainty regions. Thus, better models expand the set of safe and certain states, reducing filter conservatism. We present the effectiveness of Dyna-SAuR on goal-reaching CartPole as well as MuJoCo Walker, reducing failures compared to state-of-the-art methods by 2 orders of magnitude.
Synthetic
A
Data
Safety
Filter
Safe Control Learning and Data Collection Environment Data
Figure 1. Dyna-SAuR mechanism. An uncertainty-aware dynamics model is used to train a safety filter that avoids both failures and uncertain regions of the model. The filter is used to safely learn a control policy in the environment. The collected data is used to improve the dynamics model, which expands the certain area and reduces conservatism in the next iteration.
1. Introduction
model. Dyna-SAuR enables safe exploration, solely relying on a notion of failure and some initial environment data.
Exploration safety during training is a substantial challenge in reinforcement learning (RL), making direct learning on hardware impractical for many applications. Safe RL methods typically treat safety as a soft constraint (Achiam et al., 2017; As et al., 2025) or use safety filters from control theory (Bansal et al., 2017; Ames et al., 2019; Wabersich et al., 2023). The former ensures safety only in expectation, while the latter requires extensive domain knowledge and struggles with high-dimensional problems. While results towards learning safety filters with RL were recently presented (Lavanakul et al., 2024), practical limitations such as access to a simulator and scaling of the method remain.
Main Idea Evaluating safety involves identifying if subsequent failures occur, which can be assessed using a learned dynamics model in synthetic rollouts without risk. However, the model is reliable only for state-action pairs in which it can accurately represent the dynamics, as illustrated in Figure 1. We train a safety filter to avoid both unsafe and uncertain regions. Deploying the safety filter in the environment enables safe control policy learning within the model’s certain area. As training progresses, environment data improves the model, reducing uncertain regions. This gives more room for safe exploration to find better policies.
We introduce Dyna-style Safety Augmented Reinforcement Learning (Dyna-SAuR), a model-based reinforcement learning (MBRL) method that concurrently learns a control policy and a safety filter using an uncertainty-aware dynamics
To address the technical challenges of building this architecture, we define safety within an uncertain model, present a compact RL formulation of the safety filter learning problem, and provide practical guidelines to scale up. In empirical evaluation on goal-reaching CartPole and MuJoCo Walker, Dyna-SAuR matches or exceeds the performance of state-of-the-art safe RL, while reducing failures during training by at least two orders of magnitude.
*
Equal contribution 1 Institute for Data Science in Mechanical Engineering, RWTH Aachen University, Aachen, Germany. Correspondence to: Artur Eisele <[email protected]>. Preprint. April 29, 2026.
1
Safety Augmented Model-Based Reinforcement Learning
2. Background
defined as entering a set of failure states SF ⊆ S, such that the complement SS := S \ SF defines the set of safe states1 . However, some safe states inevitably lead to failure. Therefore, we aim to stay within a finite horizon2 viability kernel SV ⊆ SS that we define inspired by probabilistic viability theory (Massiani et al., 2026; Aubin et al., 2011). Definition 2.1. (Finite-Horizon Viability) A state s is viable if it is in the finite horizon viability kernel SV := s ∈ S | ∃π ∈ Π : (4) Pπ [∀t ≤ T, St ∈ SS | S0 = s] ≥ 1 − δ .
Introducing the concepts of MBRL and safe learning allows us to clearly specify the challenges of safety filter learning. 2.1. Model-based Reinforcement Learning We consider sequential decision-making in an environment modeled as a discrete-time Markov decision process (MDP) M = {S, A, p, r, ρ0 , γ}, with states St ∈ S ⊆ RnS and actions At ∈ A = [−1, 1]nA . Rewards Rt ∈ R are generated by the reward function r : S × A × S → R, while the environment dynamics p : S × A → ∆(S) propagate the system state, with ∆ indicating a stochastic mapping. Starting from an initial state S0 ∼ ρ0 , the goal of RL is to find the optimal policy π ∗ : S → A within the set of policies Π that maximizes the sum of rewards discounted by 0 ≤ γ < 1, referred to as return, in expectation "∞ # X ∗ t π = arg max Eπ γ Rt+1 . (1) π∈Π
That is, there exists a policy such that the system remains within the set of safe states with high probability 1 − δ for a considered finite time horizon T ∈ N when starting from s. States outside SV form the unviability kernel, from which failure is unavoidable with high probability for horizon T . Definition 2.2. (Viable Action) For a viable state st ∈ SV , a viable action aV t preserves viability of the next state. Definition 2.3. (Viable Policy) The set of viable policies ΠV consists of all policies that select only viable actions. ΠV := π ∈ Π | ∀s ∈ SV : (5) Pπ [∀t ≤ T, St ∈ SS | S0 = s] ≥ 1 − δ .
t=0
Where needed, we use St , At , and Rt to refer to random variables and distinguish them from their realizations st , at , and rt . Further, the state-action occupancy of a policy π T −1
1 X Pπ [St = s, At = a] T →∞ T t=0
ρπ (s, a) := lim
A safety filter κ : Π → ΠV avoids failures during training by projecting the set of policies into the set of viable policies. This is typically achieved by mapping actions of the policy in Π to the closest viable action. For control-affine dynamics p, the discriminator between viable and unviable actions is a state-dependent hyperplane wt⊤ a ≥ bt with a normal vector wt ∈ RnA and an offset bt ∈ R (Lavanakul et al., 2024). Thus, the safety filter problem is described by
(2)
describes the probability of being in a state-action pair for a policy π interacting with the environment dynamics p. To reduce costly and dangerous data collection in the environment, model-based RL trains dynamics models p̂(·|St , At ) to approximate the environment dynamics (Deisenroth & Rasmussen, 2011; Hafner et al., 2023). In Dyna-style architectures (Sutton, 1991; Janner et al., 2019), p̂ is used to simulate interactions for training model-free RL algorithms. However, p̂ is typically unreliable outside the training data distribution. Therefore, models such as the probabilistic ensemble (PE) (Lakshminarayanan et al., 2017) capture epistemic uncertainty, where ensemble disagreement indicates prediction inaccuracy. This uncertainty is used to define a sufficiently certain set E ⊆ (S ×A) where the environment is modeled accurately (Frauenknecht et al., 2024). We follow Frauenknecht et al. (2025) and use the set E := {(st , at ) ∈ S × A | H(Ŝt+1 ) ≤ λ1 },
π aV t = κ (st ) := arg min ∥a − π(st )∥2 a∈A s.t. wt⊤ a ≥ bt .
(6)
However, obtaining these state-dependent hyperplanes using classical methods (Ames et al., 2019; Bansal et al., 2017) requires substantial domain knowledge and scales poorly with the system dimension. Thus, Lavanakul et al. (2024) formulate the idea of training a filter policy µ(st ) that parametrizes the hyperplane via a function h, resulting in the filter at =κπµ (st ) := arg min ∥a − π(st )∥2
a∈A ⊤ s.t. wt a ≥ bt with (wt , bt ) = h(µ(st )).
(3)
(7)
This learned filter κπµ : Π → Img(κπµ ) with image Img(κπµ ) ⊆ Π does not necessarily project into ΠV for all µ in the set of filter policies µ ∈ M, but a suitable µ yielding a valid safety filter needs to be learned.
where the entropy of the model’s predictive distribution H(Ŝt+1 ), indicating uncertainty, is upper bounded by λ1 . 2.2. Safe Learning
1
Discussion of terminology in Appendix E.2 Since we consider unbounded process noise, we cannot make infinite horizon statements (Gao et al., 2021).
We adopt the notion of training time safety (Brunke et al., 2022), minimizing failures during training. Failures are
2
2
Safety Augmented Model-Based Reinforcement Learning
3. Problem Statement
Algorithm 1 Dyna-SAuR
Dyna-SAuR requires solving two RL problems: the filter policy learning problem µ and the control policy learning problem π. Both of them leverage a learned uncertaintyaware dynamics model p̂ with a sufficiently certain subset E defined in (3) and the set of viable policies ΠV (E) ⊆ ΠV that additionally remain within E. Before defining ΠV (E), we formulate the Filter Policy Learning Problem3
input D0p̂,p , randomly initialized p̂0 , π0 and µ0 for j ∈ {1, . . . , J} Dyna-SAuR iterations do p̂,p Train dynamics model p̂j using data from Dj−1 (Algorithm 3) while Filter evaluation not successful do Train filter policy µj using p̂j and πj−1 (Algorithm 4) Evaluate filter policy µj (Algorithm 5) end while Train control policy πj using p̂j and µj and collect environment data Djp̂,p (Algorithm 6) end for output p̂J , µJ , πJ
µ∗ = arg max{Img(κπµ ) ⊆ ΠV (E)}. µ∈M
(8)
That is, finding a filter policy µ∗ that projects into the largest possible subset of ΠV (E). Projecting into a subset of ΠV (E) makes κπµ a valid safety filter, while aiming for the largest subset results in the least restrictive valid filter. Given a filter policy µ, we aim to find the control policy π ∗ that maximizes the expected return of the filtered policy κπµ , resulting in the Control Policy Learning Problem "∞ # X π ∗ = arg max Eκπµ γ t Rt+1 . (9) π∈Π
subset E, the certain safe set SS (E) comprises state-action pairs where the model is certain and the state is safe SS (E) := (st , at ) ∈ S × A | H(Ŝt+1 ) ≤ λ1 , st ∈ SS . (10) Consequently, the finite-horizon certain viability kernel n SV (E) := s ∈ S | ∃π ∈ Π : h i o (11) Pπ ∀t ≤ T, (Ŝt , At ) ∈ SS (E) | S0 = s ≥ 1 − δ
t=0
To address the filter policy learning problem (8) via MBRL in a scalable fashion, we answer the following questions: (i) How does viability extend to uncertainty-aware dynamics models, i.e., how does ΠV (E) look like? (ii) What is a unique representation of the hyperplane parameters w and b for an efficient action space design? (iii) What is an informative data distribution to learn µ in the face of a finite sampling budget from the model?
comprises all states from which a policy exists that keeps the model in the certain safe set for a given time horizon with high probability. Finally, the set of certain viable policies n ΠV (E) := π ∈ Π | ∀s ∈ SV (E) : h i o (12) Pπ ∀t ≤ T, (Ŝt , At ) ∈ SS (E) | S0 = s ≥ 1 − δ .
Subsequently, we combine recent results in model-based data generation (Frauenknecht et al., 2025), actor-critic learning (Seo et al., 2025), and filtered policy learning (Bejarano et al., 2025; Markgraf et al., 2025) to address control policy learning (9).
allows to stay within the certain viability kernel with high probability. Thus, using the model p̂ to train µ restricts the filter learning objective (8) to the subspace ΠV (E) ⊆ ΠV , which becomes less restrictive as E grows.
We limit the input to the learning algorithm to some initial data D0p̂,p from the environment and the failure set SF .
5. Dyna-SAuR: Dyna-style Safety Augmented Reinforcement Learning
The remainder of this paper focuses on the key technical contributions for solving (8), in particular, Section 4 addresses question (i) and Section 5 questions (ii) and (iii). For a detailed description of the full algorithm, including the solution to (9), we refer to Appendix A.
In the following, we introduce the Dyna-SAuR architecture with a focus on solving the filter policy learning problem (8) using MBRL. In particular, we introduce the MDP for filter policy learning in Section 5.1, present an efficient action space representation in Section 5.2, formulate the learning problem in Section 5.3, and propose practical approximations to make the learning problem scalable in Section 5.4.
4. Viability in the Uncertainty-Aware Model To prevent failures in the environment, µ is trained exclusively using the dynamics model. Thus, we need to address question (i) and extend the viability concepts introduced in Section 2.2 to account not only for safety but also certainty under the model dynamics. Based on the sufficiently certain 3
Algorithm 1 outlines the iterative training process.4 First, the dynamics model p̂ is trained on initial environment data 4 We use iteration index j as a subscript for learnable objects, denoting parameters at convergence after the j th iteration.
See Appendix E.1 for a more detailed discussion of (8).
3
Safety Augmented Model-Based Reinforcement Learning
D0p̂,p . Second, the filter policy µ is trained exclusively on model-based rollouts from p̂ until passing model-based evaluation. Third, the control policy π is trained using both model-based data and environment interactions while being filtered through µ. Observed environment transitions are added to Djp̂,p . Retraining p̂ on this expanded data in iteration j + 1 enlarges E, enabling a less restrictive µ. At each Dyna-SAuR iteration j, both the filter policy µ and the control policy π are retrained from scratch, to avoid local optima (Nikishin et al., 2022).
ut µ wt , bt at π
pSF
h
κπµ
aV t
st
p st+1
Next, we discuss learning the filter policy µ and refer to Appendix A for a detailed description of the full algorithm. Figure 2. Safety Filter MDP Dynamics pSF . Given a state st the control policy π generates a control action at . The safety filter κ (7) is parametrized through a hyperplane action ut by the filter policy µ, via the bijective transform h, and yields a viable control V action aV t . The control MDP dynamics p transition based on at .
5.1. Safety Filter MDP Formulating the filter policy objective (8) as an RL probSF lem requires defining the = safety filter MDP M SF SF SF S, U, p , r , ρ0 , γSF , which shares only the state space S with the control MDP M from Section 2.1. We define the space of hyperplane actions U, such that µ : S → U with a subsequent bijective transform h that maps ut to wt and bt . The benefits of this design are discussed in Section 5.2. The dynamics pSF : S ×U → ∆(S) are depicted in Figure 2. They comprise elements of the standard RL problem, namely any control policy π ∈ Π and the environment dynamics p approximated by the model p̂, as well as the safety filter components comprising the bijective transformation h and the safety filter κ. Further, the safety filter reward function rSF : S × U × S → R needs to reflect the goal of finding a least restrictive projection Π → ΠV (E) formulated in (8) and is discussed in Section 5.3. Since µ trains purely on model-based rollouts, the start state distribution ρSF 0 ⊆S does not need to match ρ0 . We define a more informative distribution in Section 5.4. Finally, the discount factor γSF influences the time horizon T in (12) as it induces a random stopping time with expectation E[T ] = 1−γ1 SF .
h : U → RnA × R that enforces intersection with A. Theorem 5.1. The function h mapping each u ∈ U to a discriminating hyperplane intersecting A = [−1, 1]nA is bijective. It is defined as h(ut ) = (wt , bt ) with wt =
(14)
Proof. See Appendix B.1, Theorem B.2 Figure 3 illustrates the transformation. The normal vector wt points in the direction of ut , where actions at in the positive direction of wt are classified as viable. The offset bt is computed using the distance from the origin to the closest vertex of A along wt , given by ∥wt ∥1 as shown in Lemma B.1 in Appendix B.1. We scale this distance by (2∥ut ∥2 − 1) ∈ [−1, 1] by construction in (13). Thus, ∥ut ∥2 indicates restrictiveness: values near 0 classify most actions as viable, while values near 1 classify most as unviable.
5.2. Efficient Filter Action Space Formulation
This design yields a minimal search space for µ, improving the scalability of data-driven safety filter learning.
Addressing question (i), we present a novel, expressive design of U. Lavanakul et al. (2024) propose learning hyperplane parameters directly as ũt = [wt⊤ , bt ]⊤ , which causes several problems. First, this formulation is unbounded with Ũ = RnA +1 and overparametrized since c wt⊤ · a ≥ c bt yields identical hyperplanes for any c ∈ R. Second, Ũ does not enforce hyperplane intersection with the bounded control action space A. These issues create a large, ambiguous search space that unnecessarily complicates filter learning.
5.3. Filter Policy Learning as an RL Problem Next, we aim for a general reward function that enforces to stay within SS (E) under dynamics pSF . We define ( 1, if st+1 ∈ SS (E), rSF (st , ut , st+1 ) = (15) − 1−γ1 SF , otherwise
Instead, we define the space of hyperplane actions as the unit hyperball in RnA U := {u ∈ RnA | 0 < ∥u∥2 ≤ 1} ,
ut , and bt = (2∥ut ∥2 − 1)∥wt ∥1 . ∥ut ∥2
and terminate, whenever SS (E) is left. As a consequence, the safety filter action value function "∞ # X SF t SF Q (st , ut ) := Eu∼µ,s∼pSF γSF rt+1 st , ut (16)
(13)
resulting in a bounded search space. Subsequently, we obtain the hyperplane parameters via the bijective function
t=0
4
Safety Augmented Model-Based Reinforcement Learning A
1
U
generate large amounts of data, and actor-critic algorithms tailored for large datasets (Seo et al., 2025). Addressing question (iii), we next present heuristics for concentrating this sampling budget in meaningful areas of S × U .
1
u2 -1
u1
1
u3
h 1
-1
Generally, RL methods with function approximation minimize the mean squared value error (Sutton & Barto, 2018) Z Z 2 ξ := ρ(s, u) q SF (s, u) − QSF (s, u) du ds (19)
w·a≥b
-1
-1
S
U
SF
where q is the true filter action value function and ρ(s, u) is the state-action distribution used for training. Equation (19) shows that QSF is generally more accurate in highdensity regions of ρ(s, u). This distribution depends on ρSF 0 , µ, and pSF . We shape ρ(s, u) by oversampling informative SF states in ρSF 0 and finding a suitable representation of p .
Figure 3. Hyperplane Action Space U. Example mappings between hyperplane actions u ∈ U and discriminating hyperplanes in A. Larger u correspond to more restrictive hyperplanes.
is bounded with QSF (st , ut ) ∈ [− 1−γ1 SF , 1−γ1 SF ] for all state, filter action pairs st , ut . Here, a value of − 1−γ1 SF corresponds to failing in the next transition, and 1−γ1 SF corresponds to not failing at all. In particular, safety filter action values directly encode an expected time to failure 1 − QSF (s, µ(s))(1 − γSF ) Eµ [TF |s] = logγSF (17) 2
We use three sources for starting states ρSF 0 in model-based rollouts. First, ρSF 0 must naturally include the environment distribution ρ0 , approximated from D0p̂,p , to mimic rollouts in the environment. Second, failed rollouts τF that left SS (E) should be overrepresented, since µ does not yet provide a valid safety filter in these areas and requires more training. Third, rollouts τG yielding high returns concerning the control reward r should be overrepresented, since an overly conservative filter would be especially harmful in these areas. Thus, we want to make sure that the filter policy RL objective (18) converges concerning the regularization loss ∥ut ∥2 and finds a least restrictive projection. This yields
under µ as derived in Lemma B.3 in Appendix B.2. Thus, maximizing QSF results in enforcing to stay within SS (E) for as long as possible, which closely reflects (12). To obtain not only a valid but also a least restrictive projection Π → ΠV (E), we leverage the fact that ∥ut ∥2 indicates filter restrictiveness, with lower norms corresponding to more viable actions. Thus, maximizing µ(st ) ← arg max QSF (st , µ(st )) − c∥µ(st )∥2 µ∈M
ρSF 0 = ν1 ρ0 + ν2 U(TF ) + ν3 U(TG )
(20)
P3 with i=1 νi = 1 and νi ≥ 0, which is the mixture distribution over ρ0 and uniformly sampling states from the trajectory sets τF ∈ TF and τG ∈ TG .
(18)
with regularization term ∥µ(st )∥2 scaled by c, closely resembles the original filter policy learning problem (8).
Finally, we need to approximate pSF with respect to the stochasticity induced by π ∈ Π. However, Π is typically a continuous and unbounded search space. Thus, sampling policies at random is unlikely to yield satisfactory results in the face of a finite sampling budget. We, therefore, take another perspective on the problem. The projection Π → ΠV (E) into the set of viable policies is identical to projecting the state action occupancy of Π
5.4. Addressing Stochasticity via Data Shaping Solving the RL objective (18) in practice is challenging due to the randomness in the dynamics pSF depicted in Figure 2. While the filter components h and κ are deterministic, and the control MDP dynamics p typically introduce only mild stochasticity, allowing any π ∈ Π makes pSF highly stochastic. Thus, areas in S × U need to be heavily sampled to obtain reliable value estimates QSF (st , ut ) and train an effective filter policy in these areas. Learning an expressive QSF everywhere, akin to Hamilton-Jacobi reachability methods, is infeasible for systems of the considered complexity (Wabersich et al., 2023). Instead, we leverage two principles that helped scale dynamic programming to modern deep RL: large-scale, parallelized data generation, and concentrating data and representational capacity on relevant regions of S × U. We use accurate long-horizon rollouts using the model p̂ (Frauenknecht et al., 2025) that can cheaply
ρπ∈Π (s, a) := Eπ∈Π ρπ (s, a).
(21)
into the state action occupancy of viable policies ρπ∈ΠV (E) (s, a), that has no probability mass outside of SS (E). So, if the filter projects from one policy set into another, it also projects the corresponding occupancy κπµ : Π → ΠV (E) ⇐⇒ κπµ : ρπ∈Π → ρπ∈ΠV (E) .
(22)
Consequently, it suffices for filter policy learning to approximate ρπ∈Π sufficiently well even if the control action generating process is not an element of Π. 5
Safety Augmented Model-Based Reinforcement Learning
We approximate the state action occupancy of Π using a convex combination of two components: the occupancy of the latest control policy πj−1 with exploration noise ϵ scaled by σ2 , and the occupancy of pure noise ϵ scaled by σ3 : ρπ∈Π (s, a) ≈ ν4 ρπj−1 +σ2 ϵ (s, a) + ν5 ρσ3 ϵ (s, a),
(23)
where ν4 + ν5 = 1 and ν4 , ν5 ≥ 0. We use temporally correlated pink noise ϵ ∼ PN, which provides a favorable trade-off between local and global exploration (Eberhard et al., 2023). This yields a sufficiently diverse approximation of ρπ∈Π for learning a robust filter policy µ, while overrepresenting the current best solution πj−1 to the control policy learning problem (9) making sure µ performs well for greedy control policies.
(a) Goal-reaching CartPole with (b) Walker with constraints constraints (red) and goal (blue). (red). Figure 4. Safe Learning Environments with Constraints
Infoprop-Dyna, and pretrain PPO-Lagrangian and DH-RL for the same amount of transitions as stored in D0p̂,p before reporting performance and counting safety violations. Appendix D provides a detailed discussion of the experimental setup, including the generation of D0p̂,p .
6. Experiments and Discussion We empirically evaluate Dyna-SAuR on a low-dimensional (S ⊆ R4 , A ⊆ R) goal-reaching CartPole task for detailed analysis and the high-dimensional (S ⊆ R17 , A ⊆ R6 ) MuJoCo Walker task to illustrate the scalability of the method. The experiments will show that Dyna-SAuR matches or excels the control performance of state-of-the-art safe RL approaches, while reducing accumulated failures during training by at least two orders of magnitude. We introduce the experimental setup in Section 6.1, discuss control performance, safety violations, and expansion of the initial data set in Section 6.2, and ablate the design decisions introduced in Section 5 in Section 6.3.
6.2. Control Performance and Safety Figure 5 reports control returns (top row) and accumulated failures (bottom row) over environment interactions for both tasks. Since each Dyna-SAuR comprises retraining the policy from scratch, only the final evaluation return of each iteration is shown as a green dot to maintain readability. All failures incurred during training are included. Dyna-SAuR shows the strongest control performance of all safe RL approaches on goal-reaching CartPole and performs on par with PPO-Lagrangian on Walker. Infoprop-Dyna consistently outperforms the safe RL approaches concerning control performance, yet it is unaware of safety. Dyna-SAuR reduces the amount of accumulated failures by at least two orders of magnitude compared to the other methods.
6.1. Experimental Setup The evaluation tasks are depicted in Figure 4. First, goalreaching CartPole extends the standard task (Towers et al., 2025) by adding constraints to the pole angle θ and the cart position x, depicted in red, while the corresponding velocities ẋ and θ̇ remain unconstrained. The control objective is reaching a goal cart position, marked in blue, which requires the cart to drive to the right while obeying safety constraints. Second, we extend MuJoCo Walker (Todorov et al., 2012), with angle and position constraints on the torso, depicted in red. The objective remains walking as fast as possible.
Figure 6 investigates the exploration behavior of DynaSAuR, providing insight to the learning dynamics. The top row compares the initial data set D0p̂,p to trajectories under the final filtered policy κπµJJ , depicted in Figure 6 in blue and green, respectively. Similarly, the bottom row compares the initial certain set E0 in grey to the final set EJ in pink. For goal-reaching CartPole, we plot the subspace x × θ that indicates a large disconnect between initial data and the final policy. In particular, the initial data and accurate set are far from the goal state. Consequently, E needed to expand substantially, to learn higher performing policies, which explains the gradual increase in control performance in Figure 5. Since Dyna-SAuR accumulates at most one failure on goal-reaching CartPole, the gradual expansion of E yielding less restrictive µ appears to work as intended.
We compare Dyna-SAuR to (a) PPO-Lagrangian (Ray et al., 2019), a state-of-the-art model-free safe RL approach that tries to minimize failures in expectation; (b) DH-RL (Lavanakul et al., 2024), a safe RL method that learns a safety filter policy through interaction with a simulator; and (c) Infoprop-Dyna (Frauenknecht et al., 2025), a recent MBRL method that is not aware of safety as a baseline. Dyna-SAuR can be viewed as an extension to DH-RL, as it learns a filter policy with action space U (Section 5.2) from a model, and to Infoprop-Dyna as it extends MBRL to safe RL.
Due to the 17-dimensional state space, we perform a principal component analysis to compare the distributions in Walker. While a substantial difference between initial data and final policy distribution, as well as the initial and final
We provide the initial dataset D0p̂,p to both Dyna-SAuR and 6
Safety Augmented Model-Based Reinforcement Learning
Figure 5. Control Return and Accumulated Failures (log scale) over Environment Interactions. Experiments are run for 10 random seeds with solid lines representing the mean and shaded areas the 99% confidence interval. We plot performance after every Dyna-SAuR iteration as a dot, as the agent is retrained from scratch between iterations. All failures during retraining are reported. Dyna-SAuR meets or excels over the control performance of safe RL algorithms (PPO-Lagrangian, DH-RL) while substantially reducing failures.
6.3. Ablation of Filter Policy Design Decisions We present ablation results of the design decisions presented in Section 5 for goal-reaching CartPole in Figure 7 with the corresponding results for Walker in Appendix C.
Goal
First, the predominant importance of formulating the action filter space as presented in Section 5.2 is illustrated when compared to learning w, b directly as proposed in (Lavanakul et al., 2024), shown in pink. The method is incapable of learning a reliable safety filter that allows for high control returns. Second, Removing the regularization loss on u from the objective (18) introduced in Section 5.3, shown in purple, still yields a good filter but shows weaker control performance. This indicates a valid but restrictive projection into ΠV (E), as expected. Third, removing the heuristics for shaping ρSF 0 presented in Section 5.4 and starting from ρ0 , instead, yields the blue curve. In this case, we observe an increased amount of failures, indicating the importance of overrepresenting TF , and slightly weaker control performance as TG is not represented.
Figure 6. Exploration Throughout Training. Top row: Initial data D0p̂,p (blue) vs. Data of the final filtered policy κπµJJ (green); Bottom row: Initial certain set E0 (grey) vs. final certain set EJ (pink). Dyna-SAuR explores the environment beyond initial data in both tasks, while the effect appears more pronounced in CartPole.
6.4. Safety Filter Analysis Finally, we observe the safety filter κπµJJ after the final training iteration J for goal-reaching CartPole in Figure 8. The top row shows the Cartpole at the beginning, in the middle, and at the end of a trajectory in the environment. The bottom row shows µJ in the x × θ subspace, with velocities given below each frame specifying the current state indicated by the black dot. Filtering the 1-dimensional action, namely applying a force in the interval [−1, 1] to the cart, is achieved by reducing the interval. This is visualized via color-coding, where blue and orange indicate that only force to the right or
certain set, can be observed, the effect is less pronounced than in CartPole. Combined with the strong but stagnant control performance in Figure 5, this suggests that efficiently expanding E in high-dimensional spaces is a limitation of the current approach. 7
Safety Augmented Model-Based Reinforcement Learning
ments by formulating a cost budget per state (Zhao et al., 2023). Methods addressing state-wise safety of the deployed agent often require minimal prior knowledge (Cheng et al., 2019; Bharadhwaj et al., 2021; Zhang et al., 2023). Conversely, approaches addressing training-time safety typically rely on strong assumptions, such as known dynamics or prior safe controllers (Berkenkamp et al., 2017; Fisac et al., 2017; Wachi et al., 2018; Wei & Liu, 2022). Recovery RL (Thananjeyan et al., 2020) utilizes prior data to avoid failures but is restricted to stable systems. Viability-based approaches yield rigorous safety filters providing hard state-wise safety guarantees, but often face computational limits or demand substantial domain knowledge (Wabersich et al., 2023). Methods based on Hamilton–Jacobi reachability, control barrier functions, or Lyapunov functions (Bastani & Li, 2021; Shao et al., 2021; Emam et al., 2022; Choi et al., 2021; Luo & Ma, 2021) generally require nominal models or invariant sets.
Figure 7. Ablation of Dyna-SAuR Design Choices. Removing the action formulation of Section 5.2 substantially impedes performance and safety. Removing the regularization loss of Section 5.3 and start state distribution of Section 5.4 yields weaker results concerning performance and safety, respectively.
Data-driven approximations to viability-based safety filters reduce these requirements but face limitations, including the inability to handle discontinuous dynamics (Wang et al., 2023) and the need for unfiltered environment interaction or accurate sampling models (So et al., 2024; Ganai et al., 2023; Lavanakul et al., 2024). To our knowledge, Dyna-SAuR is the first scalable approach to approximate viability-based filters for diverse systems with minimal domain knowledge.
8. Conclusion
Figure 8. Final safety filter κπµJJ in goal-reaching CartPole. The filter is less restrictive around the upper equilibrium and for low velocities, indicated by the comparatively large white areas, and becomes more restrictive as deflections and velocities increase.
Safe exploration is a fundamental concern in RL, yet traditional safety filters require substantial domain knowledge and struggle with high-dimensional problems. Conversely, RL excels at learning control for high-dimensional systems that are hard to model from first principles. We propose Dyna-SAuR, a scalable, data-driven approach to filter synthesis that jointly learns control and filter policies using a learned uncertainty-aware dynamics model. Inspired by viability theory, we present a compact RL formulation that approximates a rigorous safety filter matching the predictive capabilities of the learned dynamics model. Specifically, we formalize viability within a learned model, introduce a novel filter parametrization, and provide scaling guidelines. Building on the learned model, we limit required domain knowledge to a notion of failure and an initial dataset. In empirical evaluations, Dyna-SAuR matches or exceeds state-of-the-art safe RL control performance while reducing failures by at least two orders of magnitude. While future work should focus on further improving exploration in highdimensional spaces, we believe Dyna-SAuR represents a significant step towards safer, more reliable RL agents.
left is considered viable, respectively, while white indicates that all actions are considered viable. We observe a clear correlation between the pole angle and the set of viable actions, which aligns with the task dynamics. Further, we see a relatively broad set of viable actions around the upright position at low velocities, which grows more restrictive as deflections and velocities increase.
7. Related Work Besides penalizing constraint violations in the reward function (Massiani et al., 2022), safety in RL is frequently addressed through constrained Markov decision processes (CMDPs) (Altman, 2021), where the expected cumulative safety cost is constrained by a budget (Wabersich et al., 2023). While widely adopted (Achiam et al., 2017; Tessler et al., 2019; Ray et al., 2019; Sootla et al., 2022; Bai et al., 2023; Zhang et al., 2020; 2022; As et al., 2025), CMDPs address safety in expectation rather than state-wise. State-wise constrained MDPs (SCMDPs) make stricter state8
Safety Augmented Model-Based Reinforcement Learning
Acknowledgement
via conservative natural policy gradient primal-dual algorithm. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023.
We thank Lukas Kesper, Devdutt Subhasish, and PierreFrançois Massiani for the valuable discussions on the work presented in this paper. The authors gratefully acknowledge the computing time provided to them at the NHR Center NHR4CES at RWTH Aachen University (project number p0022301). This is funded by the Federal Ministry of Education and Research, and the state governments participating on the basis of the resolutions of the GWK for national high performance computing at universities (www.nhr-verein.de/unsere-partner). Friedrich Solowjow is supported by the KI-Starter grant by the state of NRW. This work is funded in part under the Excellence Strategy of the Federal Government and the Lander (G:(DE-82)EXS-SFOPSF854).
Bansal, S., Chen, M., Herbert, S., and Tomlin, C. J. Hamilton-jacobi reachability: A brief overview and recent advances. In 2017 IEEE 56th Annual Conference on Decision and Control (CDC), pp. 2242–2253. IEEE, 2017. Bastani, O. and Li, S. Safe reinforcement learning via statistical model predictive shielding. Robotics: Science and Systems XVII, 2021.
Impact Statement
Bejarano, F. P., Brunke, L., and Schoellig, A. P. Safety filtering while training: Improving the performance and sample efficiency of reinforcement learning agents. IEEE Robotics and Automation Letters, 10(1):788–795, January 2025.
We present Dyna-SAuR, an MBRL algorithm that jointly learns a control policy and a safety filter from an uncertaintyaware model. Addressing exploration safety, Dyna-SAuR contributes to the general applicability of RL methods. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Berkenkamp, F., Turchetta, M., Schoellig, A. P., and Krause, A. Safe model-based reinforcement learning with stability guarantees. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pp. 908–919, 2017.
References Achiam, J., Held, D., Tamar, A., and Abbeel, P. Constrained policy optimization. In International conference on machine learning, pp. 22–31. PMLR, 2017.
Bharadhwaj, H., Kumar, A., Rhinehart, N., Levine, S., Shkurti, F., and Garg, A. Conservative safety critics for exploration. In International Conference on Learning Representations, 2021.
Altman, E. Constrained Markov Decision Processes: Stochastic Modeling. Routledge, Boca Raton, 1 edition, December 2021. ISBN 978-1-315-14022-3. doi: 10.1201/9781315140223.
Brunke, L., Greeff, M., Hall, A. W., Yuan, Z., Zhou, S., Panerati, J., and Schoellig, A. P. Safe learning in robotics: From learning-based control to safe reinforcement learning. Annual Review of Control, Robotics, and Autonomous Systems, 5(Volume 5, 2022):411–444, 2022.
Ames, A. D., Coogan, S., Egerstedt, M., Notomista, G., Sreenath, K., and Tabuada, P. Control barrier functions: Theory and applications. In 2019 18th European control conference (ECC), pp. 3420–3431. Ieee, 2019.
Cheng, R., Orosz, G., Murray, R. M., and Burdick, J. W. End-to-end safe reinforcement learning through barrier functions for safety-critical continuous control tasks. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019.
As, Y., Sukhija, B., Treven, L., Sferrazza, C., Coros, S., and Krause, A. Actsafe: Active exploration with safety constraints for reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025. Aubin, J.-P., Bayen, A. M., and Saint-Pierre, P. Viability and Capturability, pp. 43–104. Springer Berlin Heidelberg, Berlin, Heidelberg, 2011. ISBN 978-3-642-16684-6. doi: 10.1007/978-3-642-16684-6 2.
Choi, J., Castañeda, F., Tomlin, C. J., and Sreenath, K. Reinforcement learning for safety-critical control under model uncertainty, using control lyapunov functions and control barrier functions. Robotics: Science and Systems (RSS), 2021.
Bai, Q., Bedi, A. S., and Aggarwal, V. Achieving zero constraint violation for constrained reinforcement learning 9
Safety Augmented Model-Based Reinforcement Learning
Deisenroth, M. P. and Rasmussen, C. E. Pilco: a modelbased and data-efficient approach to policy search. In Proceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, pp. 465–472. Omnipress, 2011.
Lavanakul, W., Choi, J., Sreenath, K., and Tomlin, C. Safety filters for black-box dynamical systems by learning discriminating hyperplanes. In Abate, A., Cannon, M., Margellos, K., and Papachristodoulou, A. (eds.), Proceedings of the 6th Annual Learning for Dynamics; Control Conference, volume 242 of Proceedings of Machine Learning Research, pp. 1278–1291. PMLR, 2024.
Eberhard, O., Hollenstein, J., Pinneri, C., and Martius, G. Pink noise is all you need: Colored noise exploration in deep reinforcement learning. In Proceedings of the Eleventh International Conference on Learning Representations, 2023.
Luo, Y. and Ma, T. Learning barrier certificates: Towards safe reinforcement learning with zero training-time violations. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021.
Emam, Y., Notomista, G., Glotfelter, P., Kira, Z., and Egerstedt, M. Safe reinforcement learning using robust control barrier functions. IEEE Robotics and Automation Letters, PP:1–8, 01 2022.
Markgraf, H., Sawant, S., Krasowski, H., Schäfer, L., Gros, S., and Althoff, M. Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?, 2025. URL https://arxiv.org/abs/ 2509.12833. Version Number: 1.
Fisac, J. F., Akametalu, A. K., Zeilinger, M. N., Kaynama, S., Gillula, J. H., and Tomlin, C. J. A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control, 64: 2737–2752, 2017.
Massiani, P.-F., Heim, S., Solowjow, F., and Trimpe, S. Safe value functions. IEEE Transactions on Automatic Control, 68(5):2743–2757, 2022.
Frauenknecht, B., Eisele, A., Subhasish, D., Solowjow, F., and Trimpe, S. Trust the model where it trusts itself: model-based actor-critic with uncertainty-aware rollout adaption. In Proceedings of the 41st International Conference on Machine Learning, ICML’24, 2024.
Massiani, P.-F., von Rohr, A., Haverbeck, L., and Trimpe, S. Viability of future actions: Robust safety in reinforcement learning via entropy regularization. In Ribeiro, R. P., Pfahringer, B., Japkowicz, N., Larrañaga, P., Jorge, A. M., Soares, C., Abreu, P. H., and Gama, J. (eds.), Machine Learning and Knowledge Discovery in Databases. Research Track, pp. 129–145, Cham, 2026. Springer Nature Switzerland.
Frauenknecht, B., Subhasish, D., Solowjow, F., and Trimpe, S. On rollouts in model-based reinforcement learning. Int. Conf. on Learning Representations, 2025. Ganai, M., Gong, Z., Yu, C., Herbert, S. L., and Gao, S. Iterative reachability estimation for safe reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Nikishin, E., Schwarzer, M., D’Oro, P., Bacon, P.-L., and Courville, A. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp. 16828–16847. PMLR, 2022.
Gao, Y., Johansson, K. H., and Xie, L. Computing probabilistic controlled invariant sets. IEEE Transactions on Automatic Control, 66(7):3138–3151, 2021.
Ray, A., Achiam, J., and Amodei, D. Benchmarking safe exploration in deep reinforcement learning. Preprint, OpenAI, San Francisco, CA, 2019.
Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. Mastering Diverse Domains through World Models. Int. Conf. on Learning Representations, January 2023.
Seo, Y., Sferrazza, C., Geng, H., Nauman, M., Yin, Z.-H., and Abbeel, P. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control. arXiv:2505.22642, 2025.
Janner, M., Fu, J., Zhang, M., and Levine, S. When to Trust Your Model: Model-Based Policy Optimization. In Wallach, H., Larochelle, H., Beygelzimer, A., AlchéBuc, F. d., Fox, E., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Shao, Y., Chen, C., Kousik, S., and Vasudevan, R. Reachability-based trajectory safeguard (rts): A safe and fast reinforcement learning safety layer for continuous control. IEEE Robotics and Automation Letters, PP:1–1, 03 2021.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, pp. 6405–6416. Curran Associates Inc., 2017.
So, O., Serlin, Z., Mann, M., Gonzales, J., Rutledge, K., Roy, N., and Fan, C. How to train your neural control 10
Safety Augmented Model-Based Reinforcement Learning
barrier function: Learning safety filters for complex inputconstrained systems. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 11532– 11539, 2024.
Wang, Y., Zhan, S. S., Jiao, R., Wang, Z., Jin, W., Yang, Z., Wang, Z., Huang, C., and Zhu, Q. Enforcing hard constraints with soft barriers: safe reinforcement learning in unknown stochastic environments. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
Sootla, A., Cowen-Rivers, A. I., Jafferjee, T., Wang, Z., Mguni, D. H., Wang, J., and Ammar, H. Saute RL: Almost surely safe reinforcement learning using state augmentation. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 20423–20443. PMLR, 17–23 Jul 2022.
Wei, T. and Liu, C. Safe control with neural network dynamic models. In Firoozi, R., Mehr, N., Yel, E., Antonova, R., Bohg, J., Schwager, M., and Kochenderfer, M. (eds.), Proceedings of The 4th Annual Learning for Dynamics and Control Conference, volume 168 of Proceedings of Machine Learning Research, pp. 739–750. PMLR, 23–24 Jun 2022.
Sutton, R. S. Dyna, an integrated architecture for learning, planning, and reacting. SIGART Bull., 1991.
Zhang, L., Shen, L., Yang, L., Chen, S., Wang, X., Yuan, B., and Tao, D. Penalized Proximal Policy Optimization for Safe Reinforcement Learning. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, pp. 3744–3750, Vienna, Austria, July 2022. International Joint Conferences on Artificial Intelligence Organization.
Sutton, R. S. and Barto, A. G. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. Tessler, C., Mankowitz, D. J., and Mannor, S. Reward constrained policy optimization. In International Conference on Learning Representations, 2019.
Zhang, L., Zhang, Q., Shen, L., Yuan, B., Wang, X., and Tao, D. Evaluating model-free reinforcement learning toward safety-critical tasks. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023.
Thananjeyan, B., Balakrishna, A., Nair, S., Luo, M., Srinivasan, K. P., Hwang, M., Gonzalez, J. E., Ibarz, J., Finn, C., and Goldberg, K. Recovery rl: Safe reinforcement learning with learned recovery zones. IEEE Robotics and Automation Letters, 6:4915–4922, 2020. Todorov, E., Erez, T., and Tassa, Y. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026–5033, 2012.
Zhang, Y., Vuong, Q., and Ross, K. W. First order constrained optimization in policy space. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20. Curran Associates Inc., 2020.
Towers, M., Kwiatkowski, A., Balis, J. U., Cola, G. D., Deleu, T., Goulão, M., Andreas, K., Krimmel, M., KG, A., Perez-Vicente, R. D. L., Terry, J. K., Pierré, A., Schulhoff, S. V., Tai, J. J., Tan, H., and Younis, O. G. Gymnasium: A standard interface for reinforcement learning environments. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2025.
Zhao, W., He, T., Chen, R., Wei, T., and Liu, C. State-wise safe reinforcement learning: A survey. In International Joint Conference on Artificial Intelligence. IJCAI, 2023.
Wabersich, K. P., Taylor, A. J., Choi, J. J., Sreenath, K., Tomlin, C. J., Ames, A. D., and Zeilinger, M. N. Datadriven safety filters: Hamilton-jacobi reachability, control barrier functions, and predictive methods for uncertain systems. IEEE Control Systems Magazine, 43(5):137– 177, 2023. Wachi, A., Sui, Y., Yue, Y., and Ono, M. Safe exploration and optimization of constrained mdps using gaussian processes. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1), Apr. 2018. 11
Safety Augmented Model-Based Reinforcement Learning
A. Detailed Algorithm Description of Dyna-SAuR In this section, we provide a more detailed overview of how the three learning problems depicted in Figure 9 interact with each other. We begin with model learning in Section A.1, then continue with the filter learning problem in Section A.2, and conclude with the (safety-filtered) control learning problem in Section A.3. Algorithm 2 connects all learning problems into the Dyna-SAuR algorithm. Algorithm 2 Dyna-SAuR input D0p̂,p , randomly initialized p̂0 , π0 and µ0 for j ∈ {1, . . . , J} Dyna-SAuR iterations do p̂,p Train dynamics model p̂j using data from Dj−1 (Algorithm 3) while Filter evaluation not successful do Train filter policy µj using p̂j and πj−1 (Algorithm 4) Evaluate filter policy µj (Algorithm 5) end while Train control policy πj using p̂j and µj and collect environment data Djp̂,p (Algorithm 6) end for output p̂J , µJ , πJ
A.1. Probabilistic Ensemble Model Learning Algorithm 3 Dynamics Model Learning input Djp̂,p , p̂ω for model training steps do Sample mini-batch of N Model transitions from D p̂,p Sample ensemble member p̂ωe with e ∼ U({1, . . . , E}) P 1 ωe ← arg minωe − N log p̂ωe (st+1 |st , aV t ) end for output p̂ω
In the beginning of the Dyna-SAuR algorithm (j = 0) and after each control learning phase (j), we use the model p̂,p learning buffer Dj−1 to train a Probabilistic Ensemble Model (Lakshminarayanan et al., 2017) by minimizing the negative log-likelihood, as described in Algorithm 3. Probabilistic Ensemble Models separate aleatoric and epistemic uncertainty. Aleatoric uncertainty arises from the inherent stochasticity of the system, while epistemic uncertainty is caused by lack of data. Ensemble disagreement is used to approximate epistemic uncertainty and serves as an indicator of prediction inaccuracy. Recent work on probabilistic ensemble models (Frauenknecht et al., 2025) restricts model rollouts to a sufficiently certain set Ej ⊆ (S × A) in which the environment dynamics are modeled accurately. This set is defined as Ej := {(st , at ) ∈ S × A | H(Ŝt+1 ) ≤ λ1 },
(24)
where the entropy of the model’s predictive distribution, H(Ŝt+1 ), which indicates uncertainty, is upper bounded by λ1 . The p̂,p data used to train the model of iteration j is Dj−1 and corresponds to Ej−1 . In addition, the accumulated information loss is thresholded for each trajectory τ̂ τ̂ = (ŝ0 , a0 , ŝ1 , . . . , at−1 , ŝT ) by λ2 . This defines sufficiently accurate trajectories of length t′ ∈ 1, . . . , T as t′ X ′ ′ (ŝt , at )tt=0 ∈ E t H(Ŝt+1 ) ≤ λ2 ,
(25)
(26)
t=0
which we use throughout the paper when referring to model rollouts. The authors of (Frauenknecht et al., 2025) provide heuristics for determining suitable values of λ1 and λ2 based on the training data of the current model. We proceed with filter learning in the current Dyna-SAuR iteration. 12
Safety Augmented Model-Based Reinforcement Learning
Dπ,p
Dπ,p̂
st , at , st+1 , rt+1 − ∥aVt − at ∥2
Control Learning
ŝt , at , ŝt+1 , rt+1 − ∥aVt − at ∥2
Q(st , at )
st+1 p(· | st , at )
st
κ(at | wt , bt )
π(st )
r(st , at , st+1 )
aVt
at
p̂(· | st , at ) st
ut µ(st )
ŝt+1
wt , bt
rSF (st , ut , st+1 )
h(ut ) Dp̂,p
QSF (st , ut )
Dµ,p̂
st , aVt , st+1 , rt+1
SF ŝt , ut , ŝt+1 , rt+1 Model Learning Safety Filter Learning st, rt
st+1 , rt+1
Figure 9. Overview of the three learning problems in Dyna-SAuR and their interactions. During filter learning, the control policy π is SF fixed. The filter replay buffer Dµ,p̂ is populated using model rollouts ŝt , ut , ŝt+1 , rt+1 , where the filter policy µ modifies potentially unsafe actions at into safe actions aVt while exploring the action space. During control learning, the filter µ is fixed. The control policy π maintains two replay buffers: an environment control buffer Dπ,p and a model control buffer Dπ,p̂ . Model rollouts, in which actions proposed by π are filtered by µ, are used to populate Dπ,p̂ . In parallel, the control policy interacts with the environment, with its actions filtered by µ, to populate Dπ,p . In both cases, the environment reward is penalized by the magnitude of filtering, ∥aVt − at ∥2 . The unfiltered action at is stored in both control buffers. Transitions from the environment control buffer are additionally used to populate the model learning buffer Dp̂,p , without reward penalization and using the filtered action aVt , as this is the action applied to the true dynamics p.
13
Safety Augmented Model-Based Reinforcement Learning
Algorithm 4 Filter Policy Learning SF SF SF µ,p̂ input p̂j , πj−1 , µϕ , µϕ̄ , QSF =∅ θ , Qθ̄ , Qθ , Qθ̄ , D 1
1
2
2
Randomly initialize ϕ, θ1 , θ2 and ϕ̄ ← ϕ, θ̄1 ← θ1 , θ̄2 ← θ2 for ROSF model-based rollouts do σ1 ∼ U(σmin , σmax ) Sample exploration scale (Seo et al., 2025) β β β Choose between stochastic performance policy or pink noise process as behavior policy β(st ) is either πj−1 (st ) + σ2 · ϵβ t or σ3 · ϵt with ϵt ∼ PN ŝ0 ∼ ρSF Sample initial state from modified starting state distribution introduced in Section 5.4 0 P while t < T and H(Ŝt ) < λ1 and tt′ =0 H(Ŝt′ ) < λ2 do at ∼ β(ŝt ) Sample potentially unsafe action from behavior policy µ µ ut ∼ µϕ (ŝt ) + σ1 · ϵµ Sample filter action exploring with pink noise t with ϵt ∼ PN aV Potentially filter behavior policy according to (7) t = κ(at , wt , bt ) with (wt , bt ) = h(ut ) ŝt+1 ∼ p̂(·|ŝt , aV Sample next state via the Infoprop rollout (Frauenknecht et al., 2025) t ) SF rt+1 = r SF (ŝt , ut , ŝt+1 ) Get next reward according to (15) SF D µ,p̂ ← D µ,p̂ ∪ {ŝt , ut , ŝt+1 , rt+1 } Store transition in filter replay buffer end while end for for update steps do Sample mini-batch of N transitions from D µ,p̂ Pt+n−1 k−t SF y= γ rk+1 + γ n mini∈{1,2} QSF (ŝt+n , µϕ̄ (ŝt+n ) + σ4 · ϵ) with ϵ ∼ N (0, I) Compute the n-step TD target y k=t θ̄i P 2 1 (y − QSF (ŝ , u )) Update the Q-functions by minimizing the squared TD error θi ← arg minθi N t t θi if policy update frequency then P 1 (QSF ϕ ← arg maxϕ N Update policy by maximizing Q-value while penalizing excessive filtering (18) θ1 (ŝt , µϕ (ŝt ) − c∥µϕ (ŝt )∥2 ) θ̄i ← τ̄ θi + (1 − τ̄ )θ̄i Polyak averaging ϕ̄ ← τ̄ ϕ + (1 − τ̄ )ϕ̄ Polyak averaging end if end for output µϕ
A.2. Filter Learning Problem A.2.1. F ILTER P OLICY L EARNING To learn the filter µj in iteration j of Dyna-SAuR, we require the current model p̂j and the control policy πj−1 from the previous iteration. The learning procedure for the filter is described in Algorithm 4. We adapt the recent model-free RL algorithm FastTD3 (Seo et al., 2025), which has demonstrated strong performance by fully exploiting parallelization in TD3. FastTD3 employs massively parallelized environments, large batch sizes, n-step returns, multiple exploration noise scales, and a distributional critic. Since the distributional critic improves sample efficiency at the cost of increased wall-clock runtime, we adopt all components of FastTD3 except the distributional critic. This design choice is motivated by the fact that sample complexity is not critical in our setting, as we rely on parallel rollouts from copies of the learned model rather than parallel simulation of the real environment. SF Each filter learning phase is initialized from scratch, using newly initialized Q-functions QSF θ1 and Qθ2 , as well as a newly SF SF initialized policy µϕ , together with their corresponding target networks Qθ̄1 , Qθ̄2 , and µϕ̄ . The filter replay buffer Dµ,p̂ is also reset at the beginning of each learning phase.
In parallel, we perform ROSF parallel model rollouts. For each rollout, a noise scale is sampled uniformly as σ1 ∼ U(σmin , σmax ) (Seo et al., 2025). For each rollout, the behavior policy is chosen with equal probability to be either a pink-noise process or the control policy augmented with pink-noise exploration. Rollouts are terminated if ŝt+1 ∈ / SS (Ej−1 ). Rollouts are truncated if the maximum episode length is reached or if the accumulated information loss is over the threshold as in (26). Following the diagram in Figure 9 and Algorithm Algorithm 4, in each model rollout a potentially unsafe action at proposed by the behavior policy is sampled to evaluate the exploratory hyperplane action ut . This action is filtered by solving the quadratic program defined in (7). The resulting filtered action is then applied to the model via an Infoprop rollout (Frauenknecht et al., 2025) to sample the next state ŝt+1 . The reward function provides a positive signal when the filter agent keeps the state within the intersection of the safe and certain sets, and otherwise penalizes the agent according to (15). The resulting transition is stored in the filter replay buffer. After each step across all model rollouts, the Q-functions and policy are trained using FastTD3. A large mini-batch is sampled to compute the n-step TD target y, which is then used to update the Q-functions by minimizing the squared TD 14
Safety Augmented Model-Based Reinforcement Learning
error. Delayed policy updates are performed, where we introduce the action regularization term in (18) to penalize excessive filtering. As in FastTD3, Polyak averaging is used for updating the target networks. After a fixed number of steps we start the evaluation of the current filter µj . A.2.2. F ILTER E VALUATION Algorithm 5 Filter Evaluation input µj for number of evaluations do β β β β(st ) is either πj−1 (st ) + σ2 · ϵβ t or σ3 · ϵt with ϵt ∼ PN ŝ0 ∼ ρEVAL (·) 0 P while t < T and H(Ŝt ) < λ1 and tt′ =0 H(Ŝt′ ) < λ2 do at ∼ β(ŝt ) ut = µj (ŝ) aV t = κ(at , wt , bt ) with (wt , bt ) = h(ut ) ŝt+1 ∼ p̂(·|ŝt , aV t ) end while end for output TF , TG
Choose between stochastic performance policy or pink noise process as behavior policy Sample initial state Sample potentially unsafe action from behavior policy Sample deterministic filter action Potentially filter behavior policy according to (7) Sample next state via the Infoprop rollout (Frauenknecht et al., 2025)
Update starting states
The goal of the filter evaluation is to determine if the RL filter training is finished. To this end, we perform model rollouts using the greedy filter policy µ and evaluate how long the behavior policy, defined as in the previous section, can be kept safe. Each evaluation model rollout is initialized from a starting state sampled from ρEVAL (·), the evaluation starting-state 0 distribution, which is defined as a mixture of the control starting-state distribution ρ0 and states from TG . We compute the average episode length across evaluation rollouts and terminate filter training if this metric does not improve compared to previous evaluations. We adopt this heuristic to make multiple RL filter trainings feasible within a reasonable time frame. In addition to deciding when to end filter training and proceed to control learning, we update the filter starting-state ρSF 0 distribution after each evaluation. Parts of each terminated trajectory are added to the failure set TF , while parts of each truncated trajectory will be added to TG . We utilize the concept of maximum time to failure, which describes the maximum number of steps for which a state in the unviability kernel can be kept safe before it inevitably becomes unsafe. For most systems, this quantity is infeasible to compute exactly. We therefore design a heuristic that approximates the maximum time to failure using an interval defined by two hyperparameters, [MTFearly , MTFlate ]. This interval is intended to capture the range in which the true maximum time to failure is expected to lie. Given a terminated trajectory τF with terminal time T , we sample MTFnum states from (ŝT −MTFearly , . . . , ŝT −MTFlate )
(27)
and add them to TF for each failed rollout. The motivation behind this heuristic is to sample training starting states in the vicinity of failure regions of the filter that can still be corrected. By sampling in this manner, we heuristically select states from the viability kernel. From the truncated trajectories τG , we aim to overrepresent those that yield high returns with respect to the control reward r, as an overly conservative safety would be particularly harmful in these regions for the subsequent control policy learning problem. At the same time, to ensure that selected starting states are viable, we apply a heuristic that only selects high-return trajectories whose Q-values, used here as proxies for the expected time to failure, exceed a predefined threshold. After a variable number of training and evaluation rounds, the method proceeds to control policy learning. A.3. Control Learning Problem At the start of each control policy RL phase, we are given the filter µj from the current iteration and the model p̂j−1 from the previous iteration. The training procedure for the control policy is described in Algorithm 6. Essentially, this procedure follows the Infoprop algorithm, with FastTD3 serving as the model-free RL backbone. Additionally, we incorporate recent insights from (Bejarano et al., 2025; Markgraf et al., 2025) by penalizing the control agent for deviations from the safety-filtered action, as illustrated in Figure 9. The penalized reward is given by rt − ∥at − aVt ∥2 , where the Euclidean distance measures the deviation between the control agent’s proposed action and the safety-filtered action. 15
Safety Augmented Model-Based Reinforcement Learning
Algorithm 6 Control Policy Learning and Environment Data Collection p̂,p input p, p̂j , µj , πψ , πψ̄ , Qϑ1 , Qϑ̄1 , Qϑ2 , Qϑ̄2 , D π,p = ∅, D π,p̂ = ∅, Dj−1 Randomly initialize ψ, ϑ1 , ϑ2 and ψ̄ ← ψ, ϑ̄1 ← ϑ1 , ϑ̄2 ← ϑ2 s0 ∼ ρ0 for environment steps do β β at ∼ πψ (st ) + σ2 · ϵβ Sample explorative control action t with ϵt ∼ PN ut = µj−1 (st ) Sample filter policy action V at = κ(at , wt , bt ) with (wt , bt ) = h(ut ) Filter control action st+1 ∼ p(·|st , aV Sample next state of environment by executing filtered action t ) rt+1 = r(st , aV Determine control reward t , st+1 ) D p̂,p ← D p̂,p ∪ {st , aV Add filtered action to model learning buffer t , st+1 , rt+1 } D π,p ← D π,p ∪ {st , at , st+1 , rt+1 − ∥aV Add unfiltered action to environment control buffer t − at ∥2 } if model-based rollout frequency then for number of model-based rollouts do σ1 ∼ U(σmin , σmax ) Sample exploration scale (Seo et al., 2025) ŝ0 ∼ D π,p Pt while t < T and H(Ŝt ) < λ1 and t′ =0 H(Ŝt′ ) < λ2 do π π at ∼ πψ (ŝt ) + σ1 · ϵπ Sample explorative control action t with ϵt ∼ PN ut = µj (ŝt ) Sample deterministic filter action aV Potentially filter behavior policy according to (7) t = κ(at , wt , bt ) with (wt , bt ) = h(ut ) ŝt+1 ∼ p̂(·|ŝt , aV Sample next state via the Infoprop rollout (Frauenknecht et al., 2025) t ) rt+1 = r(ŝt , aV Determine control reward t , ŝt+1 ) D π,p̂ ← D π,p̂ ∪ {ŝt , at , ŝt+1 , rt+1 − ∥aV Add unfiltered action to model control buffer t − at ∥2 } end while end for end if for update steps do Sample mini-batch of N transitions from D π,p̂ ∪ D π,p y = rt+1 − ∥aV Compute the TD target y t − at ∥2 + γ mini∈{1,2} Qϑ̄i (ŝt+1 , πψ̄ (ŝt+1 ) + σ4 · ϵ) with ϵ ∼ N (0, I) P 1 (y − Qϑi (ŝt , at ))2 Update the Q-functions by minimizing the squared TD error ϑi ← arg minϑi N if policy update frequency Pthen 1 Qϑ1 (ŝt , πψ (ŝt )) Update policy by maximizing Q-value ψ ← arg maxψ N ϑ̄i ← τ̄ ϑi + (1 − τ̄ )ϑ̄i Polyak averaging ψ̄ ← τ̄ ψ + (1 − τ̄ )ψ̄ Polyak averaging end if end for end for output πψ , Djp̂,p
16
Safety Augmented Model-Based Reinforcement Learning
When applying this penalty, it is important to store the unfiltered action at in both the environment control buffer Dπ,p and the model control buffer Dπ,p̂ . The filtered action aVt , which is executed in the environment, is stored in the model learning buffer Dp̂,p . Training is terminated after a fixed number of environment interactions. The Dyna-SAuRtraining loop continues with the next iteration of model learning.
17
Safety Augmented Model-Based Reinforcement Learning
B. Proofs B.1. Bijection Between Parameter Space and Discriminating Hyperplanes Lemma B.1 (Maximum Offset of a Hypercube Intersecting Hyperplane). Let w ∈ RnA be a normalized normal vector, i.e., ∥w∥2 = 1. A hyperplane {a ∈ A | w⊤ a ≥ b} intersects the hypercube A = [−1, 1]nA only if |b| ≤ ∥w∥1 . Proof. A hyperplane intersects [−1, 1]nA if and only if there exists some a ∈ [−1, 1]nA such that w⊤ a = b. Hence, a necessary and sufficient condition is |b| ≤ max n |w⊤ a|. a∈[−1,1] A
Since the maximization of a linear function over a hypercube is attained at the vertices, we have |b| ≤
max n w⊤ a =
a∈[−1,1] A
nA X
|wi | = ∥w∥1 .
i=1
Theorem B.2 (Bijection). The function h mapping each u ∈ U to a discriminating hyperplane intersecting A = [−1, 1]nA is bijective. It is defined as h(ut ) = (wt , bt ) with wt =
ut , and bt = (2 · ∥ut ∥2 − 1)∥wt ∥1 . ∥ut ∥2
Proof. Injectivity: For any u ∈ U = {u ∈ RnA | ∥u∥2 ≤ 1}, the hyperplane {a ∈ A | w⊤ a = b}, with h(u) = (w, b) intersects the action space A = [−1, 1]nA . By construction, w is a normalized normal vector with ∥w∥2 = 1. Since ∥u∥2 ≤ 1, we have 2∥u∥2 − 1 ∈ [−1, 1], and therefore |b| = |2∥u∥2 − 1| ∥w∥1 ≤ ∥w∥1 . By Lemma B.1, these conditions are sufficient for the hyperplane to intersect [−1, 1]d . Given u, the construction uniquely determines both the normalized normal vector w and the offset b. Hence, h is injective. Surjectivity. Let (w, b) be a hyperplane intersecting [−1, 1]nA with normalized normal ∥w∥2 = 1. By Lemma B.1 the absolute offset must be smaller than the maximal offset |b| ≤ ∥w∥1 . Define 1 b l= +1 , u = w l. 2 ∥w∥1 Then l = ∥u∥2 ≤ 1 and u ∈ U . Substituting u into h recovers (w, b) by construction. Hence, h is surjective. Therefore, h is bijective. B.2. Expected Time to Failure Lemma B.3 (Expected Time to Failure). A filter action value function QSF (s, u) approximating the action values of a filter policy µ(s) for a reward function ( 1, if st+1 ∈ SS (E), SF r (st , ut , st+1 ) = − 1−γ1 SF , otherwise encodes the expected time to failure TF under the filter policy µ in the following form: 1 − QSF (s, µ(s))(1 − γSF ) Eµ [TF |s] = logγSF 2 18
Safety Augmented Model-Based Reinforcement Learning
Proof. The safety filter return GSF for a trajectory terminating after TF steps is given by SF
G
=
TX F −1
TF t γSF · 1 + γSF
t=0
1 − 1 − γSF
TF TF 1 − γSF γSF = − 1 − γSF 1 − γSF
=
(28)
TF 1 − 2γSF . 1 − γSF
The state value function V SF (s) = QSF (s, µ(s)) (Sutton & Barto, 2018) encodes the expected return under the policy µ QSF (s, µ(s)) = Eµ GSF | S = s, U = µ(s) = V SF (s) = Eµ GSF | s # " TF 1 − 2γSF s = Eµ 1 − γSF E [T |s]
1 − 2γSFµ F = 1 − γSF Rearranging (29) yields the desired result.
19
(29)
Safety Augmented Model-Based Reinforcement Learning
Figure 10. Ablation of Dyna-SAuR design choices for Walker. Removing the starting state distribution introduced in Section 5.4 or the action regularization introduced in Section 5.3 leads to reduced control performance and increased accumulated failures. In contrast, incorrect parameterization of the hyperplane defining action introduced in Section 5.2 prevents effective learning, indicating its central role in the method.
C. Ablations As visualized in Figure 10, the Walker ablation exhibits trends similar to those observed for CartPole with Dyna-SAuR depicted in Figure 7. When removing the efficient action-space parameterization of the safety filter, we observe the highest number of accumulated failures and the lowest return. This indicates that the proposed parameterization reduces the search space and improves the effectiveness of learning safety filters with RL. When the action regularization is removed, the safety filter RL agent more easily learns restrictive safety filters, which leads to a lower final return. When removing the rewards encouraging safe starting states as well as the failure-preventing starting states, i.e., without starting-state distribution shaping, the final return is also reduced.
20
Safety Augmented Model-Based Reinforcement Learning
D. Experimental Setup D.1. Prior Data Since safety without any prior knowledge is an ill-defined problem (As et al., 2025), we assume the availability of some prior data to train an initial dynamics model. In our experiments, this prior data is generated using a stabilizing controller. Each action produced by the prior controller is perturbed with pink noise and every second step replaced by a uniformly sampled action to improve action-space coverage and enable reliable uncertainty estimation. For CartPole, we design a standard equilibrium-stabilizing Linear Quadratic Regulator (LQR). For Walker, we train an RL agent to step forward and maintain balance. We are using a partially trained version of this agent to generate prior data. For CartPole, we use 30,000 environment interactions, while for Walker we use 4,000,000 interactions. Due to the high dimensionality of the Walker environment, this larger dataset is necessary to sufficiently cover a local region of the state–action space and to obtain an adequately accurate initial dynamics model. Nevertheless, as shown in Fig. 6, our method generalizes beyond the initial data distribution and learns a walking gait that is not present in the prior data. D.2. Hyperparameters All experiments were conducted over 10 random seeds, with hyperparameters chosen to ensure fair and comparable comparisons across methods. For all environments, all episodes are truncated after 500 steps. D.2.1. H YPERPARAMETERS DYNA -SAU R The hyperparameters for Dyna-SAuR are shown in Table 1. D.2.2. H YPERPARAMETERS BASELINE I NFOPROP As shown in Table 2, we use the standard Infoprop (Frauenknecht et al., 2025) hyperparameters for the baseline. The only exception is that we increase the retraining interval (measured in environment steps) at which the model is retrained. This change is made to keep model training computationally feasible due to the relatively large amount of prior data in the Walker environment, and to match the retraining interval used by the Dynasaur model. D.2.3. H YPERPARAMETERS B ENCHMARK PPO-L AGRANGIAN As shown in Table 3, we use standard PPO-Lagrangian (Ray et al., 2019) hyperparameters for the benchmark. D.2.4. H YPERPARAMETERS B ENCHMARK RL-DH As shown in Table 4, we use standard PPO-Lagrangian (Lavanakul et al., 2024) hyperparameters for the benchmark.
21
Safety Augmented Model-Based Reinforcement Learning
Table 1. Hyperparameters for Dyna-SAuR on Walker and CartPole environments.
Hyperparameter Model Learning Ensemble size E Number of hidden layers Number of hidden neurons Learning rate Weight decay Patience for early stopping Batch size N Model Safety Filter Learning Number of model rollouts ROSF Minimal noise scale σmin Maximum noise scale σmax Accurate quantile ζ1 Exceptionally accurate quantile ζ2 Learning rate Buffer size filter buffer Dµ,p̂ Batch size N Polyak averaging factor τ̄ Policy update frequency Action regularization factor c Pink noise process scale σ3 Policy exploration scale σ2 TD3 smoothing scale σ4 Discount Factor γ SF Factor control starting states ν1 Factor failed rollouts ν2 Factor high return rollouts ν3 Mixing factor ν4 Mixing factor ν5 Number of evaluation rollouts Maximum time to failure early MTFearly Maximum time to failure late MTFlate Number of states MTFnum Control Policy Learning Number of model rollouts Minimal noise scale σmin Maximum noise scale σmax Accurate quantile ζ1 Exceptionally accurate quantile ζ2 Learning rate Polyak averaging factor τ̄ Policy update frequency Policy exploration scale σ2 TD3 smoothing scale σ4 Discount Factor γ Number of environment interactions per Dyna-SAuR iteration
22
Cartpole
Walker
7 4 200 0.0006 0.0007 10 256
7 4 200 0.0006 0.0007 10 256
100 0.001 0.3 0.99 0.01 0.0003 1,000,000 100,000 0.001 2 0.1 0.33 0.1 0.003 0.99 0.05 0.3 0.65 0.5 0.5 2000 10 50 10
500 0.001 0.2 0.99 0.01 0.0003 1,000,000 100,000 0.001 2 1 0.33 0.1 0.003 0.99 0.1 0.45 0.45 0.5 0.5 1000 20 100 5
100 0.001 0.2 0.99 0.01 0.0003 0.001 2 0.1 0.003 0.99 40,000
100 0.001 0.2 0.99 0.01 0.0003 0.001 2 0.1 0.003 0.99 40,000
Safety Augmented Model-Based Reinforcement Learning
Table 2. Hyperparameters for Infoprop on Walker and CartPole environments.
Hyperparameter Model Learning Ensemble size E Number of hidden layers Number of hidden neurons Learning rate Weight decay Patience for early stopping Retrain interval Model Rollouts Accurate quantile ζ1 Exceptionally accurate quantile ζ2 Scaling factor ξ Rollout interval Rollout batch size SAC Agent Number of hidden neurons Number of hidden layers Learning rate SAC target entropy Target update interval Update steps G
Cartpole
Walker
7 4 200 0.001 0.0002 9 40000
7 4 400 0.0006 0.0007 9 40000
0.99 0.01 1 250 100000
0.99 0.01 1 250 100000
1024 2 0.005 0 5 10
1024 2 0.0002 -7 4 10
Table 3. Hyperparameters for PPO-Lagrangian on Walker and CartPole environments.
Hyperparameter Number of hidden neurons Number of hidden layers Steps per epoch Gamma Lambda Cost gamma Cost lambda Penalty learning rate Target Kullback-Leibler divergence Value function learning rate
23
Cartpole 64 2 400 0.99 0.97 0.97 0.97 0.05 0.01 0.001
Walker 128 2 4000 0.99 0.97 0.97 0.97 0.05 0.01 0.001
Safety Augmented Model-Based Reinforcement Learning
Table 4. Hyperparameters for RL-DH on Walker and CartPole environments.
Hyperparameter Safety Filter PPO Agent Number of hidden neurons Number of hidden layers Steps per epoch Gamma Lambda Target Kullback-Leibler divergence Value function learning rate Policy learning rate Environment Steps for Safety Filter Training Clip Ratio Control PPO Agent Number of hidden neurons Number of hidden layers Steps per epoch Gamma Lambda Target Kullback-Leibler divergence Value function learning rate Policy learning rate Clip Ratio
24
Cartpole
Walker
64 2 400 0.99 0.97 0.01 0.001 0.0003 30000 0.2
128 2 4000 0.99 0.97 0.01 0.001 0.0003 4000000 0.2
64 2 400 0.99 0.97 0.01 0.001 0.0003 0.2
128 2 4000 0.99 0.97 0.01 0.001 0.0003 0.2
Safety Augmented Model-Based Reinforcement Learning
E. Definitions and Terminology E.1. Discussion Filter Policy Learning Problem We write the filter policy learning problem µ∗ = arg max{Img(κπµ ) ⊆ ΠV (E)},
(30)
µ∈M
as a shorthand notation for finding a filter policy µ∗ ∈ M, such that Img(κπµ∗ ) ⊆ ΠV (E) and ∄µ ∈ M with Img(κπµ∗ ) ⊊ Img(κπµ ).
(31)
E.2. Safety and Viability The term “safe states” is used inconsistently across the safety literature. For instance, (Cheng et al., 2019) employ it to denote what our work terms viable states, whereas (Brunke et al., 2022) use the term consistently with the definition adopted in this work. Figure 11 illustrates the terminology adopted in this work. As engineers, we define unsafe states SF as states the agent must never reach, e.g., the robot has fallen and lies on the ground. Thus, these failure states are typically easy to define. The set of safe states SS is the complement of the set of unsafe states SF . Consequently, some safe states inevitably lead to failure, e.g., a robot stumbling in a way that it cannot recover but does not yet lie on the ground. Therefore, the more informative distinction is between viable states SV that remain in the set of safe states indefinitely, given a suitable control policy, and unviable states SU that are safe but will fail eventually. This distinction is the non-trivial problem Dyna-SAuR addresses. Figure 11 provides a minimal illustrative example of the distinction. The car on the slope is safe but unviable, as it enters SF under any control sequence, since the engine torque is insufficient to climb the slope.
Unviable
Viable
Unsafe
Safe
Figure 11. Safety vs. viability terminology illustrated by a cart on a sloped track. From an engineering perspective, unsafe states (red) are defined as the undesired states in which the car has crashed; their complement are the safe states (green). At the slope, insufficient torque prevents the cart from returning, so regardless of the control sequence applied, the car inevitably enters the unsafe set in finite time. These states are defined as unviable (orange). Their complement, i.e., states from which a control sequence exists that can keep the system safe, are viable (blue).
25