Bounded Ratio Reinforcement Learning
Yunke Ao ∗ ETH Zurich
Le Chen † MPI for Intelligent Systems
arXiv:2604.18578v1 [cs.LG] 20 Apr 2026
Aline Czarnobai ‡ Dartmouth College
Bruce D. Lee † ETH Zurich
Philipp Fürnstahl Balgrist University Hospital
Assefa S. Wahd ‡ University of Alberta
Bernhard Schölkopf MPI for Intelligent Systems
Andreas Krause ETH Zurich
Abstract Proximal Policy Optimization (PPO) has become the predominant algorithm for onpolicy reinforcement learning due to its scalability and empirical robustness across domains. However, there is a significant disconnect between the underlying foundations of trust region methods and the heuristic clipped objective used in PPO. In this paper, we bridge this gap by introducing the Bounded Ratio Reinforcement Learning (BRRL) framework. We formulate a novel regularized and constrained policy optimization problem and derive its analytical optimal solution. We prove that this solution ensures monotonic performance improvement. To handle parameterized policy classes, we develop a policy optimization algorithm called Bounded Policy Optimization (BPO) that minimizes an advantage-weighted divergence between the policy and the analytic optimal solution from BRRL. We further establish a lower bound on the expected performance of the resulting policy in terms of the BPO loss function. Notably, our framework also provides a new theoretical lens to interpret the success of the PPO loss, and connects trust region policy optimization and the Cross-Entropy Method (CEM). We additionally extend BPO to Group-relative BPO (GBPO) for LLM fine-tuning. Empirical evaluations of BPO across MuJoCo, Atari, and complex IsaacLab environments (e.g., Humanoid locomotion), and of GBPO for LLM fine-tuning tasks, demonstrate that BPO and GBPO generally match or outperform PPO and GRPO in stability and final performance.
1
Introduction
Deep reinforcement learning (DRL) has achieved breakthroughs across diverse domains [27, 11, 17, 19]. Among DRL methods, Proximal Policy Optimization (PPO) [23] remains one of the most widely adopted algorithms. The core design of PPO is motivated by Trust Region Policy Optimization (TRPO, [21]), which constrains policy updates within a “trust region” to ensure stable iterations. By utilizing a first-order approximation of the TRPO objective, PPO achieves the scalability necessary for training modern large-scale models. As a result, PPO and its variant GRPO are now widely applied to tasks ranging from robotics to large language model (LLM) fine-tuning [13, 2, 26]. Despite its empirical success, PPO remains largely heuristic: its clipped objective is not directly derived from the trust-region formulation it was intended to approximate. Instead, the design of ∗
Correspondence to: [email protected] Equal second author contribution ‡ Equal third author contribution †
Preprint.
Figure 1: The Bounded Ratio Reinforcement Learning (BRRL) framework introduces the surrogate policy optimization problem under bounded ratio constraints. Its analytical solution closely relates to the PPO objective function, cross-entropy methods (CEM), and suggests a theoretically grounded policy optimization algorithm with minor changes to PPO: Bounded Policy Optimization (BPO). We observe marked improvements of BPO and its variant Group-relative BPO (GBPO) in the performance and stability for humanoid locomotion and LLM fine-tuning (mathematical reasoning). the PPO objective was primarily driven by experimentation [23, 5]. Furthermore, most existing theoretical analyses of PPO’s performance improvement rely on the original TRPO or policy gradient formulation [21, 12, 4], none of which fully capture the nuances of the first-order loss used in practice. Numerous variants have been recently proposed to improve PPO. Some works focus primarily on algorithm design and report empirical performance gains without formal theoretical contributions [3, 34, 28, 6, 8]. Other works extend PPO to specific domains (e.g., safe RL, non-stationary RL) without modifying the core PPO loss function [1, 14]. There are also PPO variants aiming at improving the PPO loss from a theoretical lens [33, 30, 31, 18]. However, similar to PPO, they also utilize TRPO theory without introducing novel theoretical frameworks or establishing superior performance guarantees. Consequently, there remains a substantial gap between the theoretical foundations and the practical policy optimization algorithms. To address this gap, we introduce the bounded ratio reinforcement learning (BRRL) framework. Instead of constraining policy updates through KL divergence [9] bounds as in TRPO, BRRL imposes bounded ratio constraints on the policy likelihood ratios. This formulation admits an analytic optimal policy, which reveals a simple structure for policy updates. We establish the following contributions using the BRRL framework: • We derive the optimal solution of BRRL and prove its monotonic performance improvement guarantees. We also demonstrate that optimizing the PPO loss approximately pushes the policy towards this analytic optimal solution. • We establish a connection between BRRL and the Cross-Entropy Method (CEM). • We propose Bounded Policy Optimization (BPO), which optimizes an advantage-weighted divergence from the BRRL solution. We also extend BPO to Group-Relative BPO (GBPO), mirroring the extension from PPO to GRPO. • We provide a performance improvement guarantee for the policy attained by BPO in terms of the loss that BPO optimizes. • We demonstrate strong empirical performance of BPO on MuJoCo, Atari, IsaacLab locomotion tasks, and of GBPO for LLM fine-tuning. Overall, BRRL provides a principled perspective on PPO-style algorithms, suggesting that their empirical success arises from approximating an analytically optimal bounded-ratio update. By more directly approximating this analytically optimal bounded-ratio update, BPO achieves improved empirical performance (Figure 1).
2
Notation
Markov Decision Process (MDP): We consider an infinite-horizon MDP defined by the tuple (S, A, P, r, d, γ), where S is the state space, A is the action space, P : S × A × S → R is the transition model, r : S × A × S → R is the reward function, d0 : S → R is the initial state distribution, and γ ∈ (0, 1) is the discount factor. Let π : S × A → R denote the stochastic policy. 2
Figure 2: Illustration of Bounded Ratio RL (BRRL). (Left) Old policy π0 within the parameterized policy class Π. (Middle Left) Construction of a trust region (light green) defined by bounded ratio constraints from Problem (4) or (9). (Middle Right) Estimation of the analytic optimal policy within the trust region (dark green, can be outside Π) using (soft-)median-advantages Ãπ0 (Theorem 4.1). In general, actions with positive (resp. negative) advantages Ãπ0 > 0 (resp. Ãπ0 < 0) yield optimal ratios greater (resp. smaller) than 1. (Right) The updated policy within Π (yellow) is obtained by minimizing a divergence from the estimated optimal policy. We denote rt := r(st , at , st+1 ). The goal of the MDP is to solve the optimization problem "∞ # X t max η(π) := Es0:∞ ,a0:∞ γ rt , s0 ∼ d0 , at ∼ π(at |st ), st+1 ∼ p(st+1 |st , at ), π∈Π
(1)
t=0
which maximizesP the expected discounted return under policy π within the policy class Π. Let us ∞ denote dπ (s) := t=0 γ t P (st = s) as an unnormalized state visitation distribution [21]. Then the objective in (1) can be rewritten as η(π) = Es∼dπ ,a∼π(·|s) [rt ]. Value functionP and advantage: We define the value function of a state s given policy P∞ π as Vπ (s) := ∞ Es0 ,a0 ,...|s0 =s [ t=0 γ t rt ], and the Q-function Qπ (s, a) := Es0 ,a0 ,...|s0 =s,a0 =a [ t=0 γ t rt ], where the actions (excluding the conditioned a0 in the Q-function) are sampled from the policy π. The advantage function is defined as the difference between them Aπ (s, a) := Qπ (s, a) − Vπ (s). As shown in [21], the expected return of the new policy π in (1) with regard to an old policy π0 can be derived as η(π) = η(π0 ) + Es∼dπ ,a∼π(·|s) [Aπ0 (s, a)].
(2)
where the advantage is evaluated under π0 , and the expectation is taken over dπ and π. Surrogate objectives: We denote the surrogate objective optimized by TRPO [21] as Lπ0 (π) := η(π0 ) + Es∼dπ0 ,a∼π(·|s) [Aπ0 (s, a)] = η(π0 ) + Es∼dπ0 ,a∼π0 (·|s) [ρAπ0 (s, a)],
(3)
where ρ = ρ(a|s) := π(a|s)/π0 (a|s) are the importance weights. In contrast to (2), Lπ0 (π) takes the expectation over dπ0 instead of dπ . In TRPO, π is updated to optimize Lπ0 (π) with constrained KL-divergence from π0 .
3
Overview of Contributions
In this section, we present an overview of the contributions within this work, as shown in Figure 2. Bounded ratio RL framework: We consider a policy optimization problem with bounded ratio trust region constraints from an old policy π0 , instead of the KL-divergence constraint of TRPO, as shown in Figure 2 (Middle Left). Specifically, with Lπ0 (π) defined in (3), the problem is expressed as max Lπ0 (π), π
s.t. 1 − ϵ ≤
π(a|s) ≤ 1 + ϵ, ∀ s, a. π0 (a|s)
(4)
Notably, this problem has an analytical optimal solution π ∗ , which in many cases (as detailed in Remark 4.3) can be derived as π ∗ (a|s) = [1 + ϵ · sign(Ãπ0 )] · π0 (a|s),
(5)
where Ãπ0 := Qπ0 (s, a) − µπ0 (s) is the median advantage. In particular, µπ0 (s) denotes the median of Qπ0 (s, a) over π0 , such that for any s ∈ S, Ãπ0 satisfies Ea∼π0 (·|s) [sign(Ãπ0 )] = 0. As 3
a
c
b
d
Figure 3: Loss functions of PPO and bounded-ratio RL. Curves for Ãπ0 > 0 and Ãπ0 < 0 are shown in yellow and blue, respectively. (a) Original PPO loss function. (b) Equivalent loss function of PPO as introduced in Proposition 4.6. (c) Optimal ratios for the optimization problem with bounded ratio constraints in (4). (d) Advantage-weighted TV loss function in BPO, defined in (8). shown in Figure 2 (Middle Right) and Figure 3 (c), this optimal solution can be explained as: if Qπ0 (s, a) is higher than the threshold µπ0 (s), then take the highest probability within the constraint π ∗ (a|s) = (1 + ϵ)π0 (a|s); otherwise, let π ∗ (a|s) = (1 − ϵ)π 0 (a|s). Threshold µπ0 (s) is selected as P the median, s.t. π ∗ is a normalized probability distribution ( a π ∗ (a|s) = 1). A formal theorem on the optimal solution for general cases is provided in Theorem 4.1. Note that Theorem 4.1 can also be extended to problems with asymmetric bounded ratio constraints (cl ≤ π(a|s)/π0 (a|s) ≤ ch ). This asymmetric solution is used to draw a connection to the cross-entropy method (CEM) in Section 4.6. Monotonic performance guarantee: For the cases where optimal policy π ∗ from (5) is optimal, it can be shown to have improved performance over π0 η(π ∗ ) = η(π0 ) + ϵ Es∼dπ∗ ,a∼π0 [sign(Ãπ0 )Ãπ0 ] := η(π0 ) + ϵB,
(6)
where the second term is non-negative and is positive whenever π0 induces non-zero median advantage. For a fixed π0 , we denote this constant improvement term as ϵB. A performance bound for general cases is provided in Theorem 4.2. Though π ∗ is simple to express and provides improvement guarantees, it may not lie in the admissible policy class Π (Figure 2 Middle right). This motivates the design of policy optimization algorithms to minimize divergence between the policy π ∈ Π and π ∗ . Revisiting the PPO loss function: We observe that the PPO loss function approximately drives the policy towards π ∗ in (5). Specifically, as shown in Figure 3 (a-b), optimizing the PPO objective [23] is equivalent to minimizing the expectation of the following loss function evaluated at ρ = π(a|s)/π0 (a|s) ( |Aπ0 | · |ρ − (1 + ϵ · sign(Aπ0 ))|, |ρ − 1| ≤ ϵ, ′ l (ρ) := (7) 0, |ρ − 1| > ϵ. A formal theorem on this equivalence with step-by-step proof is detailed in Section 4.4 and Appendix A.5. At the beginning of the iteration, the ratio always starts from 1, and the PPO loss minimizes an advantage-weighted absolute error between the ratio ρ and the target 1 + ϵsign(Aπ0 ), then it applies zero-gradient after reaching the target. Note that this target ratio closely matches the solution in (5), except that PPO uses the mean advantage Aπ0 , and the BRRL solution is expressed in terms of the median advantage Ãπ0 . Bounded Policy Optimization (BPO): Building on the solution in (5), we introduce a natural PPO variant with the loss function lBP O to directly minimize the advantage weighted total variation from the optimal solution. For the solution in (5), the loss lBP O evaluated under ρ = π(a|s)/π0 (a|s) is lBP O (ρ) := |Aπ0 | · ρ −
π ∗ (a|s) = |Aπ0 | · |ρ − (1 + ϵ · sign(Ãπ0 ))|. π0 (a|s)
(8)
The loss is illustrated in Figure 3 (d). Compared with the PPO loss in (7), this loss function l′ only differs in two ways: (1) a symmetric slope also for |ρ − 1| ≥ ϵ and (2) using Ãπ0 instead of Aπ0 . In practice, this also requires learning an additional median value function alongside the mean value function, though the median can be approximated by the mean to reduce computational overhead. Notably, with this refined loss function, BPO has both theoretical performance guarantees (discussed below) and strong empirical performance, as demonstrated in Section 5. The same loss function can also be adapted for LLM fine-tuning, analogous to how PPO was adapted to GRPO (Section 4.5). 4
BPO performance guarantees: Assuming the optimal solution in (5) is valid, we can express the stepwise improvement in terms of the achieved loss (8). Specifically, we show that π(a|s) η(π) ≥ η(π0 ) + ϵB − Es∼dπ0 ,a∼π0 lBP O − δ(π, π ∗ ), π0 (a|s) ) and reduces where B is defined in (6). Here, δ(π, π ∗ ) is an error term that is related to lBP O ( ππ(a|s) 0 (a|s) ∗ to 0 if we have perfect policy approximation π = π . This theoretical result directly implies that, if our loss function lBP O is sufficiently minimized over states and actions sampled from π0 , and if the policy approximation error is small, we can obtain monotonic performance improvement. The formal result is detailed in Corollary 4.5.
4
Method
We now proceed to present the aforementioned framework of BRRL and its extensions. 4.1
Bounded Ratio RL Framework
Intuitively, for an MDP with finite state and action spaces, Problem (4) is a linear programming problem. Specifically, for a fixed state s, the optimization variable π(a|s) is a finite-dimensional vector. Consequently, the objective function and constraints in Problem (4) are linear in π(a|s). However, for general state and action spaces, the optimal solution of this linear programming problem is difficult to specify analytically. Nevertheless, an additional regularizer allows for the derivation of the general analytical solution. Namely, we consider the following regularized constrained optimization problem: π(a|s) , max Lπ0 (π) − λEs∼dπ0 ,a∼π0 H π π0 (a|s) (9) where H(ρ) := (ρ − 1 + ϵ) log(ρ − 1 + ϵ) + (1 + ϵ − ρ) log(1 + ϵ − ρ). Here the regularizer H(ρ) ∈ [2ϵ log ϵ, 0) decreases as ρ → 1 and increases as ρ → 1 ± ϵ. Moreover, its gradient becomes unbounded near the boundaries 1 ± ϵ, so H provides log barriers for the original bounded ratio constraints 1 − ϵ < ππ(a|s) < 1 + ϵ. The regularizer is weighted by λ. According to 0 (a|s) Fermi-Dirac statistics [10], Problem (9) has a closed-form solution, detailed in the following theorem. Theorem 4.1 (Optimal solution). The optimal policy π ∗ of the problem described in (9) satisfies: !! Ãπ0 ∗ π (a|s) = 1 + ϵ tanh π0 (a|s), Ãπ0 := Qπ0 (s, a) − µπ0 (s), 2λ where Ãπ0 is the soft-median advantage and µπ0 (s) is the soft-median of Qπ0 (s, a) that satisfies !# " Qπ0 (s, a) − µ(s) Ãπ0 = 0 ⇔ µπ0 (s) = arg min Ea∼π0 (·|s) g , Ea∼π0 (·|s) tanh 2λ λ µ(s) x
x
where g : R → R≥0 , g(x) = ln(e− 2 + e 2 ) is a soft absolute function. The detailed proof of Theorem 4.1 is provided in Appendix A.1. Intuitively, the optimal solution assigns a higher ratio to actions with a higher advantage while keeping the ratio between [1 − ϵ, 1 + ϵ]. We can obtain a monotonic performance guarantee for the optimal solution. Theorem 4.2 (Monotonic performance guarantee). The optimal policy in Theorem 4.1 satisfies " ! # Ãπ0 ∗ η(π ) ≥ η(π0 ) + ϵEs∼dπ∗ ,a∼π0 (·|s) tanh Ãπ0 =: η(π0 ) + ϵB, 2λ where Ãπ0 abbreviates Ãπ0 (s, a), B is a non-negative constant given fixed π0 . 5
Ã
π0 The proof of Theorem 4.2 is detailed in Appendix A.2. Note that the term tanh( 2λ )Ãπ0 is always
Ã
π0 non-negative since the signs of tanh( 2λ ) and Ãπ0 are always the same. Therefore, our optimal ∗ policy π guarantees monotonic improvement with an analytical improvement bound, in contrast to the guarantee for TRPO in Theorem 1 of [21]. However, the policy π ∗ may not be a member of the class of parameterized policies Π. Consequently, in Section 4.2 and 4.3, we further develop the policy optimization loss algorithm by minimizing a certain divergence from π to π ∗ . Remark 4.3 (Optimal solution to unregularized Problem (4)). Note that by taking λ → 0 in Theorem 4.1 and Theorem 4.2, one can obtain the optimal ratio and monotonic guarantees for the unregularized Problem (4). In many cases, one can simplify the resulting optimal policy as π ∗ (a|s) = [1 + ϵsign(Ãπ0 )]π0 (a|s) in (5), where µπ0 (s) is the median of Qπ0 (s, a) over π0 (·|s). Such simplification holds if ∀ s, ∃ µ(s), such that
Ea∼π0 (·|s) [sign(Qπ0 (s, a) − µ(s))] = 0.
(10)
Otherwise, the simplified π ∗ can never be normalized. One valid case is a uniform density π0 (·|s) with continuous A and a Q-function Qπ0 (s, a) which is smooth over a. However, there are also counterexamples. Consider, for instance, an MDP with a single state and a discrete action space A = {a1 , a2 }. Assume that Qπ0 (a2 ) > Qπ0 (a1 ), and π0 (a1 ) = 14 , π0 (a2 ) = 34 . Then for any µ ∈ R, condition (10) does not hold. While the simplified interpretation of π ∗ is only valid in special cases, the result of Theorem 4.1 still holds for arbitrarily small λ > 0 and general spaces (see Appendix A.1). 4.2
Alternative Perspective: Minimizing Divergence from Optimal Policy
In this section, we consider the policy optimization problem as minimizing the divergence to the optimal solution, instead of directly applying policy gradient methods. Specifically, given the optimal policy obtained from Theorem 4.1, we can formulate policy optimization as min D(πθ , π ∗ ),
πθ ∈Π
where πθ is the parameterized policy, D is a divergence function such as the KL-divergence, total variation (TV), etc. Specifically, the TV for each state can be expressed as ∗ X π (a|s) πθ (a|s) TV ∗ Dθ (s) := |π (a|s) − πθ (a|s)| = Ea∼π0 (·|s) − . (11) π0 (a|s) π0 (a|s) a We also consider an advantage-weighted TV (ATV) loss function defined as ∗ π (a|s) πθ (a|s) AT V Dθ (s) := Ea∼π0 (·|s) − · |Aπ0 | . π0 (a|s) π0 (a|s)
(12)
Notably, this divergence is directly correlated with the performance improvement of the parameterized policy πθ , as detailed in the following Corollary. Corollary 4.4 (Performance improvement guarantee with policy approximation error). Consider DθAT V defined in (12) with π ∗ given from Theorem 4.1. Then it holds that " ! # Ãπ0 AT V η(πθ ) ≥ η(π0 ) + Es∼dπθ ,a∼π0 (·|s) ϵ tanh Ãπ0 − Dθ (s) , 2λ where Ãπ0 abbreviates Ãπ0 (s, a). The proof of Corollary 4.4 is provided in A.3. Corollary 4.4 shows that by minimizing the loss DθAT V over the state distribution dπθ , we can improve performance w.r.t. policy πθ as long as the parameterized policy class is sufficiently expressive. However, minimizing Es∼dπθ over the policy parameters θ is non-trivial due to the dependence of the state distribution on πθ . Consequently, in practice we only optimize the expectation of the divergence over the old policy, leading to the objectives J AT V (θ) := Es∼dπ0 [DθAT V (s)],
J T V (θ) := Es∼dπ0 [DθT V (s)].
(13)
The following Corollary 4.5 expresses a lower bound on the performance of the policy πθ in terms of these quantities. 6
Corollary 4.5 (Performance improvement iwith loss functions). With B defined in Theoh guarantee rem 4.2, and δ̃ := maxs Ea∼π0 (·|s) tanh
Ãπ0 2λ
η(πθ ) ≥ η(π0 ) + ϵB − J AT V (θ) −
Ãπ0 , it holds that
TV AT V AT V γϵδ̃Dmax Dmax γϵDmax − , J T V (θ) − 1−γ (1 − γ)2 (1 − γ)2
AT V TV where Dmax := maxs DθAT V (s) and Dmax := maxs DθT V (s).
The proof of Corollary 4.5 is detailed in Appendix A.4. Corollary 4.5 decomposes the performance lower bound into a non-negative term, ϵB, and several negative terms which depend on the gap between the optimized policy πθ and the policy π ∗ . The first two of these gap terms depend on J AT V (θ) and J T V (θ). Both of these quantities can be estimated from trajectories collected under π0 and minimized by optimizing θ, motivating a loss defined as a weighted combination of these terms. The other two gap terms are characterized by the worst case divergence of πθ from π ∗ over the state AT V TV space through the quantities Dmax and Dmax . Though these quantities are generally not computable, they can be related to the expected losses over the distribution dπ0 under additional assumptions (e.g., adequate state coverage under dπ0 ). Notably, with perfect policy approximation (πθ = π ∗ ), it holds AT V TV that J AT V (θ) = J T V (θ) = Dmax = Dmax = 0, recovering the original monotonic performance guarantee of Theorem 4.2. In contrast, the bound from TRPO in Theorem 1 of [21] does not contain a positive term. Besides, it penalizes the worst-case divergence between the updated policy and the current policy π0 through ∗ DTmax V (π, π0 ), rather than the approximation error to an ideal solution π . Consequently, this negative term reflects the magnitude of the update away from π0 . It vanishes only in the degenerate case π = π0 (i.e., no policy change), and is generally nonzero whenever a nontrivial policy update occurs. Corollary 4.5 also motivates choosing a small ϵ. For small ϵ > 0, π ∗ remains close to π0 by construction, so matching π ∗ typically requires only a small deviation from a realizable policy in the class (namely π0 ), making the approximation error terms easier to control. Since π0 is realizable in the policy class, the optimal values of J AT V (θ) and J T V (θ) approach zero as ϵ → 0. Therefore, γD AT V
γ δ̃D T V
max max under such regularity conditions, it holds that (1−γ) ≪ B for ϵ sufficiently small, 2 + (1−γ)2 therefore guaranteeing improvement.
4.3
Bounded Policy Optimization
In this section, we present the practical implementation of our algorithm. As in PPO, we use a value network Vϕ to estimate Aπ0 . Specifically, we estimate the return value Rϕ (s, a) using generalized advantage estimation [22], and use it to update the value function by minimizing J V F (ϕ) := Es∼dπ0 ,a∼π0 (·|s) [(Rϕ (s, a) − Vϕ (s))2 ].
(14)
In addition, following Theorem 4.1, we further train a network µψ to minimize the normalization loss Rϕ (s, a) − µψ (s) J M F (ψ) := Es∼dπ0 ,a∼π0 (·|s) λg . (15) λ The practical loss function for θ uses the estimated advantage function to approximate J P (θ) := J AT V (θ) + α1 J T V (θ) with α1 as a tunable weight: " ! # Âπ0 πθ (a|s) P ˆ J (θ) := Es∼dπ0 ,a∼π0 (·|s) 1 + ϵ tanh − · (|Rϕ (s, a) − Vϕ (s)| + α1 ) , 2λ π0 (a|s) (16) where Âπ0 := Rϕ (s, a) − µψ (s). Note that JˆP (θ) is not exactly J P , but the gap can be controlled by minimizing the estimation error of Vϕ and µψ . Our final bounded policy optimization algorithm follows a PPO-style training procedure, summarized in Algorithm 1. 4.4
Revisiting the PPO Objective
In this section, we connect our theory and algorithmic framework to PPO [23]. In PPO, the following surrogate objective function is introduced Es∼dπ0 ,a∼π0 [min {clip (ρ, 1 − ϵ, 1 + ϵ) Aπ0 , ρAπ0 }] := JP P O (θ), (17) 7
Algorithm 1 Bounded policy optimization (BPO) 1: Initialize πθ , Vϕ , µψ , choose a sufficiently small λ 2: for i = 1, 2, ... do 3: Assign π0 ← πθ 4: Run π0 for N steps, and collect the dataset D := {sj , aj , Rj , π0 (aj |sj )}N j=1 . 5: Update θ, ϕ, ψ by minimizing JˆP (θ) + w1 J V F (ϕ) + w2 J M F (ψ), where JˆP , J V F , J M F are defined in (16), (14), (15), and evaluated from D. 6: end for (a|s) is the ratio between the new and old policies, and Aπ0 denotes Aπ0 (s, a). where ρ := ππθ0 (a|s)
We observe a strong correlation between the BPO loss function and the PPO loss function. To show this correlation, we first introduce an equivalent form of the PPO loss in the following proposition. Proposition 4.6. Optimizing the loss function JP P O (θ) in (17) is equivalent to minimizing the following function πθ (a|s) J ′ (θ) := Es∼dπ0 ,a∼π0 (·|s) l′ , π0 (a|s) where ( ′
l (ρ) :=
|Aπ0 | · |ρ − (1 + ϵ · sign(Aπ0 ))|,
|ρ − 1| ≤ ϵ,
0,
|ρ − 1| > ϵ.
Intuitively, this equivalence follows from the fact that adding or subtracting a constant from the objective function does not change the optimal solution. This is illustrated in Fig. 3 (a vs b). A proof is provided in Appendix A.5. On the other hand, as λ → 0, the loss J AT V in (13) can, in many cases (Remark 4.3), be expressed as J AT V (θ) ≈ Es∼dπ0 ,a∼π(·|s) [lBP O (ρ)],
where lBP O (ρ) = |Aπ0 | · |ρ − (1 + ϵ · sign(Ãπ0 ))|.
Thus, the loss lBP O resembles the PPO loss l′ in Proposition 4.6 when |ρ − 1| ≤ ϵ, as detailed in Section 3. For |ρ − 1| > ϵ, BPO penalizes the policy for deviating from the original policy, which encourages satisfaction of the bounded-ratio constraints. This is also partially addressed by the zero gradient of PPO and the target KL divergence mechanism [25], which slows down the update of the new policy if it deviates too far from the original policy (i.e., if the KL divergence between the two surpasses the target KL divergence). Recent PPO variants also utilize similar ideas by introducing negative gradients when |ρ − 1| > ϵ [30, 33], which can be theoretically justified by our framework. Although the exact optimization dynamics of BPO and PPO differ, both follow a common principle: drive the policy ratio from 1 toward the (approximate) analytical optimum of BRRL and then stop. This offers a key insight into the underlying success of PPO-based methods. 4.5
Extension to LLM Fine-Tuning
In the context of LLM fine-tuning, training an additional critic can be computationally expensive. This challenge motivates the design of Group Relative Policy Optimization (GRPO) [26], which estimates advantages relative to a group of concurrent samples rather than utilizing an auxiliary value network. Building on this idea, we introduce Group-relative Bounded Policy Optimization (GBPO), an extension of BPO derived from Theorem 4.1. Specifically, for a given prompt q, the model generates a group of sampled outcomes {o1 , o2 , . . . , oG }. A reward model then assigns a score to each output, denoted by R = {r1 , r2 , . . . , rG }. As in standard GRPO, we estimate advantages using z-scores Ai := ri −mean(R) . As noted in Remark 4.3, when the regularization parameter λ is small, std(R) the implicit baseline µπ0 (q) converges to the median of the Q-values. We therefore also estimate the 8
median-advantage as Ãi := ri −median(R) . The GBPO objective function is then defined as: std(R) " JˆP (θ) = E q∼P (Q) oi ∼π0 (·|q)
|oi | G 1 X 1 X 1 + ϵ tanh G i=1 |oi | t=1
Ãi 2λ
!
# πθ (oi,t |q, oi,<t ) · |Ai,t | , − π0 (oi,t |q, oi,<t )
where t denotes the token index, and Q is the question set. In scenarios where a reward is only provided at the end of the sequence, the step-dependent advantages Ai,t and Ãi,t are equal to the sequence-level Ai and Ãi , respectively. If per-step scores are available, these advantages can be estimated token-wise following the approach in [26]. 4.6
Asymmetric Ratio Constraints and Cross Entropy Method
In this section, we generalize Theorem 4.1 to asymmetric ratio constraints. Similar to (9), we consider ≤ ch , with cl < 1 < ch the regularized problem with general ratio boundaries ∀s, a, cl ≤ ππ(a|s) 0 (a|s) π(a|s) max Lπ0 (π) − λEs∼dπ0 ,a∼π0 H ′ , π π0 (a|s) where
H ′ (ρ) := (ρ − cl ) log(ρ − cl ) + (ch − ρ) log(ch − ρ) + log
ch − 1 ρ. 1 − cl
(18)
Here, the regularizer H ′ still takes its maximum at ρ = 1, and provides log barriers for the asymmetric constraints cl ≤ ρ ≤ ch . The optimal solution is detailed in the following Corollary. Corollary 4.7. (Asymmetric optimal policy) The optimal policy π ∗ of the problem (18) satisfies π ∗ (a|s) ch − cl , Ã′π0 := Qπ0 (s, a) − µ′π0 (s), = cl + c −1 h π0 (a|s) 1 + 1−c exp(−Ã′π0 /λ) l
−1 where µ′π0 (s) is the soft- cchh−c -quantile satisfying l
Qπ0 (s, a) − µ(s) µ′π0 (s) = arg min Ea∼π0 (·|s) g ′ , λ µ(s) c −1 1−c h ch −1 − ch −cll x ′ ′ ch −cl x where g : R → R≥0 , g (x) = ln e + 1−cl e . Similar to Theorem 4.1, these results are closely related to policy optimization with asymmetric clip ratios [29, 32]. A monotonic performance guarantee similar to Theorem 4.2 is provided in −1 Appendix A.7. Moreover, when λ → 0, we also have µ′π0 (s) the exact cchh−c -quantile in many cases, l ch −1 similar to Remark 4.3. Notably, when cl = 0, λ → 0, and the ch −cl -quantile exists for Qπ0 (s, a), we have π ∗ (a|s) = ch π0 (a|s) for Qπ0 (s, a) > µ′π0 (s) and 0 otherwise. This recovers a cross-entropy method (CEM) when π0 is uniform, where the optimal solution at each iteration assigns non-zero l probability to the top c1−c samples. h −cl
5
Experiments
In this section, we present extensive experiments to validate the proposed BPO algorithm. We first benchmark its performance against PPO across standard MuJoCo and Atari environments (Section 5.1). To assess scalability, we evaluate BPO within NVIDIA IsaacLab [15], a high-throughput simulation platform capable of simulating thousands of parallel environments for real-world robotic policy training. Furthermore, we apply our GBPO variant to LLM fine-tuning tasks, and compare it directly against GRPO (Section 5.3). Then, we dive deeper into the analysis of the ratio statistics during training, connecting it to the performance gap between BPO and PPO. Finally, we conduct an ablation study to analyze the sensitivity of training performance to key components, including the loss function, the λ parameter, and various loss coefficients. All hyperparameters are detailed in Appendix A.8 9
Figure 4: BPO versus PPO on MuJoCo and Atari environments. Shaded regions represent the standard deviation across 10 random seeds. In most environments, BPO matches or outperforms PPO. 5.1
Benchmarking with Classical Environments
We compare the performance of BPO and PPO in classical environments. For these experiments, BPO was implemented within the Stable Baselines3 framework, with hyperparameters for all baseline algorithms sourced from RL-Zoo [20]. As shown in Figure 4, BPO performs competitively with or superior to PPO across a range of classical benchmarks. Specifically, in MuJoCo tasks, BPO achieves clear performance gains in the Ant-v4, Hopper-v4, and Humanoid-v4 environments. Training on Humanoid-v4 exhibits high variance for BPO, characterized by significant performance divergence across random seeds. Both PPO and BPO struggle to achieve peak performance in this environment, primarily due to limited sample efficiency. However, as demonstrated in Section 5.2, both methods successfully solve more complex humanoid tasks when provided with sufficient samples. In Atari benchmarks, BPO generally matches PPO’s performance, notably outperforming it in the Asterix environment. We report our benchmarking results against off-policy baselines in MuJoCo and Atari environments in Table 1. While SAC [7] outperforms both PPO and BPO in the Ant-v4 and Humanoid-v4 tasks, it fails to achieve competitive results in Swimmer-v4. In contrast, BPO consistently outperforms PPO in the Ant-v4, Humanoid-v4, and Hopper-v4 environments while remaining competitive in Swimmer-v4. Both BPO and PPO consistently outperform DQN [16] in Atari benchmarks. Table 1: Comparison of converged total rewards between BPO, PPO, and off-policy algorithms. Bolded and underlined numbers indicate the highest and second-highest results across all tested algorithms. For AsterixNoFrameskip-v4, the algorithms are evaluated after the same wall-clock time (12h). Rewards in other environments are evaluated after convergence. Mujoco Envs Ant-v4 Humanoid-v4 Hopper-v4 Swimmer-v4
5.2
BPO 4871.4 1570.4 3505.1 354.6
PPO 4230.1 781.3 2497.7 362.4
SAC 6161.8 6806.4 3015.1 102.7
Atari Envs BreakoutNoFrameskip-v4 PongNoFrameskip-v4 BoxingNoFrameskip-v4 AsterixNoFrameskip-v4
BPO 374.6 20.6 94.7 11247.9
PPO 360.4 20.6 95.7 9471.5
DQN 252.5 20.6 92.5 7122.8
Benchmarking with IsaacLab Environments
In this section, we evaluate the scalability and performance of BPO relative to PPO within the IsaacLab simulation platform. We focus on four challenging locomotion tasks on rough terrain: Go1-rough, Anymal-C-rough, G1-rough, and H1-rough, which require the agents (quadrupeds like Unitree Go1 and Anymal-C or humanoids like Unitree G1 and H1) to maintain stable gaits while tracking target velocities across rough surfaces. Both BPO and PPO were implemented using the RSL-RL framework, utilizing a large-scale parallelization of 4,096 environments per task. 10
Figure 5: BPO versus PPO on IsaacLab environments. Shaded regions represent standard deviation across 5 random seeds. BPO substantially outperforms the baseline on challenging humanoid locomotion tasks while exhibiting more stable training dynamics.
The results in Figure 5 demonstrate that BPO is highly effective in complex robotic locomotion tasks. In particular, on G1-rough, BPO significantly outperforms the baseline to reach a higher performance ceiling. For the Go1-rough and H1-rough environment, BPO also slightly exceeds the final performance of PPO. Notably, across all four benchmarks, BPO exhibits enhanced training stability and smoother dynamics compared to the PPO baseline. 5.3
LLM Fine-Tuning with GBPO
We further evaluate GBPO against GRPO for large language model fine-tuning (Section 4.5). Specifically, we conduct experiments in the Test-Time Reinforcement Learning (TTRL, [35]) framework, fine-tuning the Qwen2.5-Math-1.5B model with GBPO and GRPO on the AIME-TTT and AMC-TTT benchmarks, and then compare their reasoning performance. The empirical results, illustrated in Figure 6, reveal that GBPO can maintain performance gains as the Figure 6: Performance of GRPO (green) and GBPO (blue) number of training epochs and clip for fine-tuning Qwen2.5-Math-1.5B on AIME-TTT and ratio increase. Conversely, GRPO AMC-TTT benchmarks. In the legend, the first and second exhibits instability under these numbers denote the clip ratio and the number of epochs, conditions. These findings highlight respectively. GBPO’s potential as a more robust and stable alternative for the fine-tuning of large-scale models. 5.4
Ratio Statistics Analysis
Figure 7: Analysis of ratio statistics. During the training process, we draw statistics of ratios (π(a|s)/π0 (a|s)) above and below 1.0 separately, corresponding to BPO/PPO_high and BPO/PPO_low. Solid lines and shaded regions represent the mean and standard deviations across 5 random seeds. Dashed black lines highlight 1.0 and clipped ranges for PPO; dashed gray lines show clipped ranges for BPO. 11
We analyze the statistics of importance weights (ratio π(a|s)/π0 (a|s)) during the training process. In MuJoCo environments (using the stable-baselines3 implementation), BPO maintains more stable ratio distributions than PPO, as illustrated in Figure 7. This difference in stability is more obvious in environments where BPO outperforms PPO (e.g., Hopper and Asterix). In IsaacLab environments (utilizing RSL-RL), learning rates are dynamically adjusted to maintain a target KL divergence. As shown in Figure 8, the adapted learning rates for PPO are often lower than those for BPO, suggesting more aggressive ratio updates that surpass the target KL divergence more frequently. The scales of the learning rates differ more in tasks where BPO shows a clear performance improvement (e.g., G1-rough). These findings suggest a strong correlation between Figure 8: Adapted learning rates to match the target KL the stability of ratio distributions and divergence in RSL-RL implementation. overall algorithmic performance. By effectively enforcing this stability, BPO allows for more stable performance improvement. 5.5
Ablation Study
This section presents an ablation study of the impact of the value function, loss function, λ, and the coefficient of TV loss on the performance of the policy, within the G1-rough environment.
Figure 9: Ablation study of BPO components in G1-rough environment. Shaded regions represent standard deviation across 10 random seeds. From left to right: ablation of mean vs median value functions, loss functions, regularization weight λ and total variation (TV) weight α1 . Using the mean value instead of the median value yields similar final performance. The advantage-weighted total variation (ATV) loss provides better performance compared to TV and KL divergence. In general, smaller λ values lead to better performance, although making λ too small (1e−4 ) slightly degrades performance. In practice, including the TV loss does not improve results. Mean vs median value function. We evaluate the performance of the algorithm by substituting median advantages Ãπ0 with the mean advantage Aπ0 . As illustrated in the left panel of Figure 9, this simplification achieves performance comparable to the original BPO. This robustness likely stems from the low practical differences between median and mean values, caused by the specific return distribution and inherent value estimation errors. These results also suggest that this median-to-mean value simplification offers a compelling alternative when the computational overhead of learning the median value is high. Divergence function ablation. As illustrated in the middle-left panel of Figure 9, the ATV loss consistently yields superior performance in both G1-rough and Anymal-C-rough environments. While the standard TV loss facilitates some learning, it fails to match the asymptotic performance of ATV. Conversely, KL divergence proves ineffective and fails to achieve successful policy convergence. Sensitivity to λ. We conduct a hyperparameter sweep for λ in the G1-rough environment. As shown in the middle-right panel of Figure 9 , smaller values of λ generally lead to strong performance. Specifically, increasing λ from 10−3 to 10−2 may slightly improve asymptotic performance, but at 12
the cost of a reduced convergence rate. Conversely, excessively large values of λ prevent the learning process entirely. Impact of TV loss regularization. We study the effect of the TV loss coefficient by incrementally increasing its weight relative to the ATV loss in the G1-rough environment. Although Corollary 4.5 suggests that both terms contribute to performance gains, the results in the right panel of Figure 9 indicate that explicitly adding a TV loss component does not improve performance in practice.
6
Conclusion
We introduced Bounded Ratio Reinforcement Learning (BRRL), a framework for policy optimization under bounded ratio constraints. We showed that the underlying optimization problem admits an analytic solution. Our main finding is that this optimal solution allows interpreting the PPO loss from a new perspective, connects to the cross-entropy method (CEM), and motivates a theoretically grounded variant, Bounded Policy Optimization (BPO). Empirically, BPO is consistently effective across a broad range of tasks, including robotic control and large-model fine-tuning. Despite the extensive evaluation with standard RL benchmarks, extending the experiments towards a broader range of LLM fine-tuning tasks remains a compelling future direction. Other future research directions include enhancing sample efficiency via advanced exploration, extending the framework to constrained MDPs, and adapting the algorithm for fine-tuning generative policies.
7
Acknowledgment
This work is in part supported by the Hasler Foundation ("Learn to learn safely" project, grant number: 21039), Swiss National Science Foundation under NCCR Automation, grant agreement 51NF40 180545, and the ETH AI Center.
References [1] Abdullah Akgül, Gulcin Baykal, Manuel Haußmann, and Melih Kandemir. Overcoming non-stationary dynamics with evidential proximal policy optimization. arXiv preprint arXiv:2503.01468, 2025. [2] OpenAI: Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3–20, 2020. [3] Karl W Cobbe, Jacob Hilton, Oleg Klimov, and John Schulman. Phasic policy gradient. In International Conference on Machine Learning, pages 2020–2027. PMLR, 2021. [4] Leif Doering, Daniel Schmidt, Moritz Melcher, Sebastian Kassing, Benedikt Wille, Tilman Aach, and Simon Weissmann. An approximate ascent approach to prove convergence of ppo. arXiv preprint arXiv:2602.03386, 2026. [5] Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep policy gradients: A case study on ppo and trpo. arXiv preprint arXiv:2005.12729, 2020. [6] Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P3o: Policy-on policy-off policy optimization. In Uncertainty in artificial intelligence, pages 1017–1027. PMLR, 2020. [7] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018. [8] Taisuke Kobayashi. Proximal policy optimization with relative pearson divergence. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 8416–8421. IEEE, 2021. 13
[9] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1):79–86, 1951. [10] L. D. Landau, E. M. Lifshitz, and L. P. Pitaevskii. Statistical Physics: Theory of the Condensed State, volume 9 of Course of Theoretical Physics. Butterworth-Heinemann, Oxford, 1980. [11] Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5(47):eabc5986, 2020. [12] Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural trust region/proximal policy optimization attains globally optimal policy. Advances in neural information processing systems, 32, 2019. [13] Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. Science robotics, 7(62):eabk2822, 2022. [14] Nikola Milosevic, Johannes Müller, and Nico Scherf. Central path proximal policy optimization. arXiv preprint arXiv:2506.00700, 2025. [15] Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Muñoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, et al. Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning. arXiv preprint arXiv:2511.04831, 2025. [16] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. [17] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [18] Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. Rethinking the trust region in llm reinforcement learning. arXiv preprint arXiv:2602.04879, 2026. [19] Ilija Radosavovic, Tete Xiao, Bike Zhang, Trevor Darrell, Jitendra Malik, and Koushil Sreenath. Real-world humanoid locomotion with reinforcement learning. Science Robotics, 9(89):eadi9579, 2024. [20] Antonin Raffin. Rl baselines3 zoo. https://github.com/DLR-RM/rl-baselines3-zoo, 2020. [21] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR, 2015. [22] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. Highdimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015. [23] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [24] Clemens Schwarke, Mayank Mittal, Nikita Rudin, David Hoeller, and Marco Hutter. Rsl-rl: A learning library for robotics research. arXiv preprint arXiv:2509.10771, 2025. [25] Antonio Serrano-Munoz, Dimitrios Chrysostomou, Simon Bøgh, and Nestor AranaArexolaleiba. skrl: Modular and flexible library for reinforcement learning. Journal of Machine Learning Research, 24(254):1–9, 2023. [26] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 14
[27] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354–359, 2017. [28] Charlie B Tan, Edan Toledo, Benjamin Ellis, Jakob N Foerster, and Ferenc Huszár. Beyond the boundaries of proximal policy optimization. arXiv preprint arXiv:2411.00666, 2024. [29] Jiakang Wang, Runze Liu, Lei Lin, Wenping Hu, Xiu Li, Fuzheng Zhang, Guorui Zhou, and Kun Gai. Aspo: Asymmetric importance sampling policy optimization. arXiv preprint arXiv:2510.06062, 2025. [30] Yuhui Wang, Hao He, and Xiaoyang Tan. Truly proximal policy optimization. In Uncertainty in artificial intelligence, pages 113–122. PMLR, 2020. [31] Yuhui Wang, Hao He, Xiaoyang Tan, and Yaozhong Gan. Trust region-guided proximal policy optimization. Advances in Neural Information Processing Systems, 32, 2019. [32] Zhiheng Xi, Xin Guo, Yang Nan, Enyu Zhou, Junrui Shen, Wenxiang Chen, Jiaqi Liu, Jixuan Huang, Zhihao Zhang, Honglin Guo, et al. Bapo: Stabilizing off-policy reinforcement learning for llms via balanced policy optimization with adaptive clipping. arXiv preprint arXiv:2510.18927, 2025. [33] Zhengpeng Xie, Qiang Zhang, and Renjing Xu. Simple policy optimization. arXiv preprint arXiv:2401.16025, 2024. [34] Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. Mastering complex control in moba games with deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6672–6679, 2020. [35] Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, et al. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084, 2025.
15
A
Appendix
A.1
Proof of Theorem 4.1
Proof. Since ∀ s, Ea∼π0 (·|s) [ρ] = 1, we have from Equation (3) Lπ0 (π) = η(π0 ) + Es∼dπ0 ,a∼π0 (·|s) [ρAπ0 (s, a)] = Es∼dπ0 [Vπ0 (s) · Ea∼π0 (·|s) [ρ]] + Es∼dπ0 ,a∼π0 (·|s) [ρAπ0 (s, a)] = Es∼dπ0 ,a∼π0 (·|s) [ρVπ0 (s) + ρAπ0 (s, a)] = Es∼dπ0 ,a∼π0 (·|s) [ρQπ0 (s, a)] The original problem (9) can then be written as max E s∼dπ0 [ρQπ0 (s, a) − λ(ρ − 1 + ϵ) log(ρ − 1 + ϵ) − λ(1 + ϵ − ρ) log(1 + ϵ − ρ)], ρ
a∼π0 (·|s)
with the normalization constraint ∀ s, Ea∼π0 (·|s) [ρ] = 1. This optimization problem can be decomposed into subproblems for each state s. Now, given a fixed state s, we solve the constrained optimization problem using the Lagrangian approach, with a Lagrangian multiplier denoted as −µ(s) L(ρ) :=Ea∼π0 (·|s) [ρQπ0 (s, a) − λ(ρ − 1 + ϵ) log(ρ − 1 + ϵ) − λ(1 + ϵ − ρ) log(1 + ϵ − ρ)] − µ(s)(Ea∼π0 (·|s) [ρ] − 1) =:Ea∼π0 (·|s) [f (ρ)] + µ(s), where f (ρ) := ρQπ0 (s, a) − λ(ρ − 1 + ϵ) log(ρ − 1 + ϵ) − λ(1 + ϵ − ρ) log(1 + ϵ − ρ) − µ(s)ρ. If A is continuous, we apply the calculus of variations
⇒ ⇒ ⇒
⇒
∂ f (ρ) = 0 ∂ρ ∂ (ρQπ0 (s, a) − λ(ρ − 1 + ϵ) log(ρ − 1 + ϵ) − λ(1 + ϵ − ρ) log(1 + ϵ − ρ) − µ(s)ρ) = 0 ∂ρ Qπ0 (s, a) − λ(log(ρ − 1 + ϵ) − log(1 + ϵ − ρ)) − µ(s) = 0 ρ−1+ϵ Qπ0 (s, a) − µ(s) log = 1+ϵ−ρ λ ! Qπ0 (s,a)−µ(s) ϵ exp −ϵ λ Ãπ0 ∗ = 1 + ϵ tanh ρ =1+ Qπ0 (s,a)−µ(s) 2λ 1 + exp λ
(19) λ λ The second derivative of the objective function can be computed as − ρ−1+ϵ − 1+ϵ−ρ , which is ∗ negative for arbitrary 1 − ϵ < ρ < 1 + ϵ. Therefore, ρ is the maximizer of the objective function.
The Lagrangian multiplier µ(s) should be chosen to satisfy the normalization constraint: " !# " !# Ãπ0 Ãπ0 Ea∼π0 (·|s) 1 + ϵ tanh = 1 ⇔ Ea∼π0 (·|s) tanh = 0. 2λ 2λ i h Ãπ0 Note that such µ(s) always exists because for all λ > 0, Ea∼π0 (·|s) 1 + ϵ tanh 2λ is a smooth function of µ with the value range between ϵ. Now we show that the corresponding h 1 − ϵ and 1 +i x x Qπ0 (s,a)−µ(s) µ(s) is also the minimizer of Ea∼π0 (·|s) g , where g := ln(e− 2 + e 2 ): λ " q−u # q−u ∂g q − u e 2λ − e− 2λ q−u Eπ0 = 0 ⇔ Eπ0 = 0 ⇔ E tanh = 0, π0 q−u q−u ∂u λ 2λ 2(e− 2λ + e 2λ ) 16
where Eπ0 abbreviates Ea∼π0 (·|s) . Besides, " u−q # u−q u−q u−q (e 2λ + e− 2λ )2 − (e 2λ − e− 2λ )2 ∂2g 1 ≥ 0. = 2 Ea∼π0 (·|s) u−q u−q ∂2u 4λ 2(e− 2λ + e 2λ )2 Therefore, the optimal µ(s) is the minimizer of g. If A is discrete, for each fixed state s, we denote the vectorized ρ(a), Qπ0 (s, a), H(ρ) and π0 (a|s) as ρ, Q, H, π ∈ R|A| . Then the Lagrangian L(ρ) can be expressed by L(ρ) := π ⊤ (ρ ⊙ Q − λH − µρ) + µ, where ⊙ denotes elementwise product. Applying zero gradient w.r.t. ρ gives (diag(Q) − λdiag(H ′ ) − µI)⊤ π = 0
⇒
π(a|s)(Qπ0 (s, a) − λH ′ (ρ) − µ(s)) = 0, ∀ a,
This gives the same expression as (19), therefore, the following proof steps are the same as the continuous case. A.2
Proof of Theorem 4.2
We start by proving the following Lemma: ∗
Lemma A.1. Define Lππ0 (s) := Ea∼π∗ (·|s) [Qπ0 (s, a)], h i Ãπ0 1 + ϵ tanh 2λ π0 (a|s) from Theorem 4.1, we have " ∗ Lππ0 (s) = Vπ0 (s) + ϵEa∼π0 (·|s)
tanh
Ãπ0 (s, a) 2λ
consider
π ∗ (a|s)
=
#
!
Ãπ0 (s, a) .
∗
Proof. We directly compute Lππ0 (s) as " !! # Ãπ0 (s, a) π∗ Lπ0 (s) = Eπ0 1 + ϵ tanh Qπ0 (s, a) 2λ " ! # Ãπ0 (s, a) = Eπ0 [Qπ0 (s, a)] +ϵEπ0 tanh Qπ0 (s, a) {z } | 2λ Vπ0 (s)
" = Vπ0 (s) + ϵEπ0 tanh " = Vπ0 (s) + ϵEπ0 tanh
Ãπ0 (s, a) 2λ
!
Ãπ0 (s, a) 2λ
!
# · (µ(s) + Ãπ0 (s, a)) #
"
Ãπ0 (s, a) + ϵ Eπ0 |
Ãπ0 (s, a) tanh 2λ {z =0
!# µ(s). }
h i à (s,a) where we abbreviate Ea∼π0 (·|s) with Eπ0 . We have Eπ0 tanh π02λ = 0 because of the normalization constraint for µ in Theorem 4.1. We now prove Theorem 4.2 using matrix representations for MDP with a discrete state and action space. The results for continuous spaces can be extended by using linear operators other than matrices. Proof. (Theorem 4.2) We define rπ ∈ R|S| with rπ (s) := Ea∼π(·|s),s′ ∼P (s′ |s,a) [r(s, a, s′ )]. The P transition kernel Pπ ∈ R|S|×|S| is defined with Pπ (s, s′ ) := a∼π(·|s) π(a|s)p(s′ |s, a). We denote (I − γPπ )−1 := (I + γPπ + γ 2 Pπ2 + ...). Then, given the initial distribution denoted as d ∈ 1 |S| [0, 1]|S| , d(s) := d0 (s), we can express the state visitation distribution dπ ∈ [0, 1−γ ] as d⊤ π = ⊤ −1 d (I − γPπ ) . 17
Let us define Vπ ∈ R|S| , where each component s corresponds to Vπ (s). The definition of the value function implies that Vπ = rπ + γPπ Vπ
(20)
Vπ = (I − γPπ )−1 rπ . We also have η(π) = Es∼dπ ,a∼π(·|s),s′ ∼P (·|s,a) [r(s, a, s′ )] = d⊤ (I − γPπ )−1 rπ = d⊤ Vπ We then define Lππ21 ∈ R|S| with
Lππ21 = rπ2 + γPπ2 Vπ1 ,
(21)
(22)
∗
which aligns with the definition of Lππ0 (s) in Lemma A.1. Let us denote B1 ∈ R|S| with B1 (s) := h i à (s,a) Ea∼π0 (·|s) tanh π02λ Ãπ0 (s, a) . Then the state-wise result Lemma A.1 can be rewritten for the full state space as ∗ Lππ0 = Vπ0 + ϵB1 , (23) where B1 is positive along each of its components. Combining (22) and (23) gives ∗
rπ∗ + γPπ∗ Vπ0 =: Lππ0 = Vπ0 + ϵB1 ⇒ rπ∗ = (I − γPπ∗ )Vπ0 + ϵB1
(24)
∗
On the other hand, applying (20) to π in combination with (24) gives Vπ∗ = (I −γPπ∗ )−1 rπ∗ = (I −γPπ∗ )−1 ((I −γPπ∗ )Vπ0 +ϵB1 ) = Vπ0 +ϵ(I −γPπ∗ )−1 B1 . (25) Finally, we can obtain from (21) η(π ∗ ) := d⊤ Vπ∗ = d⊤ Vπ0 + ϵd⊤ (I − γPπ∗ )−1 B1 = η(π0 ) + ϵEs∼dπ∗ [B1 (s)] Applying the definition of B1 (s) finishes the proof. A.3
Proof of Corollary 4.4
We start by proving the following Lemma on per-state performance improvement. P Lemma A.2. Define Lππθ0 (s) := Ea∼πθ (·|s) [Qπ0 (s, a)], DθAT V (s) := a |(πθ (a|s) − π ∗ (a|s))Aπ0 (s, a)|, we have ! # " Ãπ0 (s, a) πθ Ãπ0 (s, a) − DθAT V (s) Lπ0 (s) ≥ Vπ0 (s) + ϵEa∼π0 (·|s) tanh 2λ ∗
Proof. We first bound |Lππ0 (s) − Lππθ0 (s)| with DθAT V (s) ∗
|Lππ0 (s) − Lππθ0 (s)| =|Ea∼π∗ (·|s) [Qπ0 (s, a)] − Ea∼πθ (·|s) [Qπ0 (s, a)] | =|Ea∼π∗ (·|s) [Qπ0 (s, a)] − Vπ0 (s) − (Ea∼πθ (·|s) [Qπ0 (s, a)] − Vπ0 (s))| =|Ea∼π∗ (·|s) [Aπ0 (s, a)] − Ea∼πθ (·|s) [Aπ0 (s, a)] | ∗ π (a|s) πθ (a|s) = Ea∼π0 (·|s) − Aπ0 (s, a) π0 (a|s) π0 (a|s) ∗ π (a|s) πθ (a|s) ≤Ea∼π0 (·|s) − Aπ0 (s, a) π0 (a|s) π0 (a|s) =DθAT V (s) Then we have from Lemma A.1 " ∗
Lππθ0 (s) ≥ Lππ0 (s) − DθP (s) = Vπ0 (s) + ϵEa∼π0 (·|s) tanh
18
Ãπ0 (s, a) 2λ
!
# Ãπ0 (s, a) − DθAT V (s)
We now prove the Corollary 4.4. Proof. Similar to the proof of 4.2, we start by defining DθAT V ∈ R|S| with each component as DθAT V (s). Then we can express Lemma A.2 in the full state space as Lππθ0 ⪰ Vπ0 + ϵB1 − DθAT V , where ⪰ denotes elementwise ≥. Following similar steps of the proof of Theorem 4.2, we obtain rπθ ⪰ (I − γPπθ )Vπ0 + ϵB1 − DθAT V . −1
Multiplying both sides by (I − γPπθ )
(26)
from left gives
Vπθ ⪰ Vπ0 + (I − γPπθ )−1 (ϵB1 − DθAT V ),
(27)
because all terms of (I − γPπθ )−1 are positive. Multiplying both sides by d⊤ finishes the proof. A.4
Proof of Corollary 4.5
Proof. From the inequality (27), we can obtain Vπθ ⪰Vπ0 + (I − γPπθ )−1 (ϵB1 − DθAT V ) = Vπ0 + ϵ(I − γPπθ )−1 B1 − (I − γPπθ )−1 DθAT V =Vπ0 + ϵ(I − γPπθ )−1 B1 − (I − γPπ0 )−1 DθAT V − ((I − γPπθ )−1 − (I − γPπ0 )−1 )DθAT V =Vπ0 + ϵ(I − γPπθ )−1 B1 − (I − γPπ0 )−1 DθAT V − γ(I − γPπ0 )−1 (Pπθ − Pπ0 ) (I − γPπθ )−1 DθAT V {z } | :=X {z } | :=Y | {z } :=Z
(28) We can first bound X elementwise by AT V X ⪯ (I − γPπθ )−1 Dmax =1·
AT V Dmax , 1−γ
where ⪯ denotes elementwise smaller or equal to. Then we can bound each term Y by X X |Y (s)| =| (πθ (a|s) − π0 (a|s)) p(s′ |s, a)X(s′ )| ≤
a
s′
X
X
|πθ (a|s) − π0 (a|s)|
a
=
p(s′ |s, a) max |X(s′ )| ′ s
s′
X
|πθ (a|s) − π0 (a|s)| max |X(s′ )| ′ s
a
X X |πθ (a|s) − π ∗ (a|s)|) max |X(s′ )| ≤( |π0 (a|s) − π ∗ (a|s)| + ′ s
a
a
|
{z
π ∗ (a|s)
Ea∼π0 (·|s) [| π (a|s) −1|]
}
0
≤(ϵ + DθT V (s)) ·
AT V Dmax 1−γ
Finally, Z satisfies |Z| = |γ(I − γPπ0 )−1 Y | ⪯ γ(I − γPπ0 )−1 (DθT V + ϵ · 1) ·
AT V Dmax 1−γ
(29)
And |d⊤ Z| ≤ γd⊤ (I − γPπ0 )−1 (DθT V + ϵ · 1) · =
AT V Dmax 1−γ
AT V AT V γDmax γϵDmax Es∼dπ0 [DθT V (s)] + 1−γ (1 − γ)2
19
(30)
Now let us consider the second term of (28) as ϵ(I − γPπθ )−1 B1 = ϵ(I − γPπ∗ )−1 B1 − ϵ((I − γPπ∗ )−1 − (I − γPπθ )−1 )B1 . With similar techniques for bounding X, Y and Z from (28) to (29), we have |d⊤ ((I − γPπ∗ )−1 − (I − γPπθ )−1 )B1 | ≤ γd⊤ (1 − γPπ∗ )−1 DθT V · TV ≤ γd⊤ (1 − γPπ∗ )−1 1 · Dmax ·
δ̃ 1−γ
TV γ δ̃Dmax δ̃ = 1−γ (1 − γ)2
Combining this with (30) and (28) finishes the proof. The results for continuous spaces can be extended by using linear operators other than matrices. A.5
Proof of Proposition 4.6
We use A0 to abbreviate Aπ0 . When A0 > 0, the negative PPO objective for a fixed state-action pair can be further expressed as ( −ρA0 , ρ≤1+ϵ PPO l (ρ) = −(1 + ϵ)A0 , ρ > 1 + ϵ ( [(1 + ϵ) − ρ]A0 , ρ ≤ 1 + ϵ ′ PPO ⇒ l (ρ) := l (ρ) + (1 + ϵ)A0 = , 0, ρ>1+ϵ where l′ is constructed from adding a constant (1 + ϵ)A0 to lP P O . Similarly, we construct l′ for A0 ≤ 0: ( −ρA0 , ρ≥1−ϵ lP P O (ρ) = −(1 − ϵ)A0 , ρ < 1 − ϵ ( [(1 − ϵ) − ρ]A0 , ρ ≥ 1 − ϵ ′ PPO , ⇒ l (ρ) := l (ρ) + (1 − ϵ)A0 = 0, ρ<1−ϵ We can rearrange l′ as ( ′
l (ρ) =
|A0 | · |ρ − (1 + sign(A0 ) · ϵ)|,
|ρ − 1| ≤ ϵ
0,
|ρ − 1| ≥ ϵ
Then we have Es∼dπ0 ,a∼π0 (·|s) [l′ (ρ)] = Es∼dπ0 ,a∼π0 (·|s) [lP P O (ρ)] + Es∼dπ0 ,a∼π0 (·|s) [(1 + ϵ)I(A0 > 0)A0 + (1 − ϵ)I(A0 ≤ 0)A0 ], where the second term is a constant independent of ρ. Therefore, l′ is an equivalent loss function to lP P O for solving the optimal ρ. A.6
Proof of Corollary 4.7
Similar to Section A.1, for each fixed s, the Lagrangian of problem (18) can be written as ch − 1 ρ L(ρ) =Ea∼π0 (·|s) ρQπ0 (s, a) − λ(ρ − cl ) log(ρ − cl ) − λ(ch − ρ) log(ch − ρ) − λ log 1 − cl −µ(s)(Ea∼π0 (·|s) [ρ] − 1) 20
For continuous MDP, applying ∂L ∂ρ = 0 gives ch − 1 Qπ0 (s, a) − λ log(ρ − cl ) + λ log(ch − ρ) − λ log − µ(s) = 0 1 − cl ρ − cl ch − 1 ⇔ Qπ0 (s, a) − µ(s) = λ log + log ch − ρ 1 − cl ρ − cl 1 − cl Qπ0 (s, a) − µ(s) = exp ⇔ ch − ρ ch − 1 λ ⇔
ρ=
Qπ0 (s,a)−µ(s) λ Qπ0 (s,a)−µ(s) 1−cl 1 + ch −1 exp λ
l exp cl + ch · c1−c h −1
= cl +
ch − c l µ(s)−Qπ0 (s,a) h −1 1 + c1−c exp λ l
Now we derive the conditions that need to be satisfied by µ′π0 (s). Specifically, having ρ normalized gives c − c h l =1 Ea∼π0 (·|s) cl + µ(s)−Qπ0 (s,a) h −1 1 + c1−c exp λ l c − c h l = 1 − cl ⇔ Ea∼π0 (·|s) µ(s)−Qπ0 (s,a) h −1 1 + c1−c exp λ l µ(s)−Qπ0 (s,a) ch − cl − (1 − cl ) − (ch − 1) exp λ =0 ⇔ Ea∼π0 (·|s) µ(s)−Qπ0 (s,a) h −1 exp 1 + c1−c λ l µ(s)−Qπ0 (s,a) 1 − exp λ =0 ⇔ Ea∼π0 (·|s) µ(s)−Qπ0 (s,a) h −1 1 + c1−c exp λ l ′
′ On the other hand, ∂g ∂x = 0 from g defined in Corollary 4.7 gives c −1 1−c ch −1 chh−cl x ch −1 1−cl − ch −cll x e − · e 1−cl ch −cl =0 Ea∼π0 (·|s) ch −cl ch −1 1−c ch −1 − ch −cll x ch −cl x e + 1−cl e ch −1 1−c − c −cl x ch −cl x h l e − e =0 ⇔ Ea∼π0 (·|s) ch −1 1−cl x h −1 − ch −cl x e ch −cl + c1−c e " #l −x 1−e ⇔ Ea∼π0 (·|s) = 0, h −1 −x 1 + c1−c e l Ã′
where assigning x = λπ0 recovers the normalization condition above. For a discrete MDP, similar to the proof of Theorem 4.2, we can obtain the same Lagrangian as the continuous case. A.7
Monotonic Guarantees for Asymmetric Bounded Ratio RL
Corollary A.3 (Asymmetric monotonic performance guarantee). The optimal policies in Theorem 4.7 satisfy ! −Ã′π /λ 0 1+e Ãπ0 η(π ∗ ) ≥ η(π0 ) + (ch − 1)Es∼dπ∗ ,a∼π0 (·|s) · tanh Ãπ0 ch −1 −Ã′π0 /λ 2λ 1+ e 1−cl
=: η(π0 ) + (ch − 1)B ′ , where Ãπ0 abbreviates Ãπ0 (s, a), B ′ is a non-negative constant given fixed π0 . 21
∗
Proof. We start by deriving Lππ0 (s), similar to Lemma A.1. ∗ ∗ π (a|s) Qπ0 (s, a) Lππ0 (s) = Ea∼π0 (·|s) π0 (a|s) ∗ π (a|s) = Vπ0 (s) + Ea∼π0 (·|s) − 1 Qπ0 (s, a) π0 (a|s) ∗ (31) π (a|s) = Vπ0 (s) + µπ0 (s) · Ea∼π0 (·|s) −1 π0 (a|s) | {z } =0 ∗ π (a|s) + Ea∼π0 (·|s) − 1 Ãπ0 (s, a) π0 (a|s) h ∗ i (a|s) where Ea∼π0 (·|s) ππ0 (a|s) − 1 = 0 is because of the normalization constraints enforced by the ∗
(a|s) definition of µ′π0 (s). We now compute ππ0 (a|s) − 1 from Corollary 4.7:
π ∗ (a|s) ch − cl −1 − 1 =cl + c −1 h π0 (a|s) 1 + 1−c exp(−Ã′π0 /λ) l
=
ch − cl + cl − 1 − (ch − 1) exp(−Ã′π0 /λ) h −1 1 + c1−c exp(−Ã′π0 /λ) l
=(ch − 1) ·
1 − exp(−Ã′π0 /λ) h −1 1 + c1−c exp(−Ã′π0 /λ) l
=(ch − 1) ·
1 + exp(−Ã′π0 /λ)
·
1 − exp(−Ã′π0 /λ)
h −1 exp(−Ã′π0 /λ) 1 + exp(−Ã′π0 /λ) 1 + c1−c l ! 1 + exp(−Ã′π0 /λ) Ã′π0 =(ch − 1) · · tanh h −1 2λ exp(−Ã′π0 /λ) 1 + c1−c l
Apply this into (31) gives " ∗ Lππ0 (s) = Vπ0 (s) + (ch − 1) · Ea∼π0 (·|s)
1 + exp(−Ã′π0 /λ) h −1 1 + c1−c exp(−Ã′π0 /λ) l
· tanh
Ã′π0 2λ
!
# Ã′π0
Following the same steps of proof of Theorem 4.2 in Appendix A.2, we can apply expectations over states and finish the proof. Notably, since
−Ã′π /λ 0 ch −1 −Ã′π /λ 0 1+ 1−c e l
1+e
∗
(a|s) is always positive, ππ0 (a|s) − 1 still has the
same sign as Ã′π0 , which makes B ′ non-negative. With Corollary A.3, one can also derive corresponding corollaries for asymmetric BPO like Corollary 4.4 and 4.5. A.8
Hyperparameters
The hyperparameters for BPO and PPO in RL environments are summarized in Table 2 and Table 3. For the GBPO implementation, we set the discount factor γ = 1 and utilize a mini-batch size of 1. Other hyperparameters are the same as those in the original training scripts provided by TTRL [35]. All the experiments are conducted on 4 x NVIDIA H100 GPUs. We set the group size to 32 and the maximum sequence length to 4,096 tokens. Empirical Tuning Observations: While BPO can generally be initialized using hyperparameters tuned for PPO, specific adjustments often yield superior performance. Empirically, we found that increasing the clip ratio by 0.1 and doubling the number of epochs (e.g., from 5 to 10) can sometimes enhance stability and results. Furthermore, BPO exhibits a higher sensitivity to the entropy coefficient; in many environments, an entropy weight 10−1 smaller than the optimal PPO setting is sufficient to maintain adequate exploration without destabilizing the policy. 22
Table 2: Hyperparameters of BPO for benchmarking environments. We take λ = 0.001, α1 = 0 and w1 = w2 = 0.5 across all environments. The GAE-λ is set to 0.95 for all environments except 0.98 for Swimmer. ADP abbreviates adaptive learning rates based on the KL-divergence, according to [24]. Envs Atari Ant-v4 Humanoid-v4 Hopper-v4 Swimmer-v4 Go1-Rough Anymal-C G1-Rough H1-Rough
batch size 256 256 128 32 256 24576 24576 24576 24576
clip 0.3 0.3 0.2 0.25 0.1 0.3 0.25 0.2 0.2
ent_coef 0.001 0 0 0 0 0.001 0 0 0
gamma 0.98 0.99 0.99 0.999 0.9999 0.99 0.99 0.99 0.99
lr 2.5e−4 1e−4 1e−4 9.808e−5 3e−4 ADP ADP ADP ADP
n_epochs 5 10 5 10 10 10 5 10 5
n_steps 128 2048 512 512 1024 24 24 24 24
n_envs 8 1 1 4 4 4096 4096 4096 4096
Table 3: Hyperparameters of PPO for benchmarking environments, based on RL-Zoo [20]. The GAEλ is set to 0.95 for all environments except 0.98 for Swimmer, 0.99 for Hopper, 0.9 for Humanoid, and 0.8 for Ant. ADP abbreviates adaptive learning rates based on the KL-divergence, according to [24]. Envs Atari Ant-v4 Humanoid-v4 Hopper-v4 Swimmer-v4 Go1-Rough Anymal-C G1-Rough H1-Rough
batch size 256 32 256 32 256 24576 24576 24576 24576
clip 0.2 0.1 0.3 0.25 0.1 0.2 0.2 0.2 0.2
ent_coef 0.01 4.96e−7 0.00238 0 0 0.01 0.005 0.008 0.01
gamma 0.98 0.98 0.98 0.999 0.9999 0.99 0.99 0.99 0.99
23
lr 2.5e−4 1.9e−5 3.57e−5 9.808e−5 6e−4 ADP ADP ADP ADP
n_epochs 4 10 5 10 10 5 5 5 5
n_steps 128 512 512 512 1024 24 24 24 24
n_envs 8 1 1 4 4 4096 4096 4096 4096