An Agency-Transferring Model-Free Policy Enhancement Technique Preprint, compiled June 9, 2026 Anton Bolychev∗
Georgiy Malaniya∗
Center for Engineering Systems and Sciences [email protected]
Center for Engineering Systems and Sciences [email protected]
Sinan Ibrahim
Pavel Osinenko†
Center for Engineering Systems and Sciences [email protected]
Central University; Center for Engineering Systems and Sciences; Sirius University of Science and Technology [email protected]
arXiv:2606.09825v1 [cs.LG] 8 Jun 2026
Abstract Training reinforcement learning (RL) policies from scratch is costly: it requires careful reward and environment design, extensive tuning, and substantial computation. Yet many control problems already have a functional but suboptimal policy available as a baseline. This paper proposes a method for embedding such a baseline into the RL training process, simultaneously improving training efficiency relative to from-scratch methods and producing a learning policy that outperforms the baseline. At each step, the method arbitrates between the baseline policy and a trainable learning policy, initially relying strongly on the baseline policy and then progressively transferring agency to the learning policy. By the end of training, the learning policy is a standalone neural network that operates without baseline policy support. The paper formalizes what it means for the baseline policy to be functional: under this policy, the agent reaches a goal set and remains there with high probability. The proposed arbitration mechanism is designed to exploit this property during training, yielding high goal-reaching rates right from the beginning of training. A theoretical analysis provides a formal interpretation of this behavior under stated assumptions and extends it to the final baseline-free regime, where explicit lower bounds are derived for the goal-reaching probability of the standalone learning policy. Empirical results on continuous-control benchmarks show that the proposed method achieves returns that match or exceed those of competitive approaches, while maintaining the highest goal-reaching rates throughout training among the compared methods—including in the final stage, where the learning policy operates without any baseline support. Keywords Reinforcement learning · Policy arbitration · Policy switching
1
Introduction
Reinforcement learning (RL) has repeatedly demonstrated that a single framework can master an impressive spectrum of complex control problems. Flagship achievements include AlphaGo and AlphaZero’s dominance in Go, chess, and shogi [1]; OpenAI Five’s grand-master-level play in Dota 2 [2]; and AlphaStar’s success in StarCraft II [3]. In robotics, RL enabled a dexterous hand to solve a Rubik’s Cube [4] and has powered a variety of manipulation and locomotion systems [5]. Yet training an RL agent from scratch is still tricky. The outcome hinges on a number of small decisions — how one clips rewards, sets learning rates, or normalizes inputs — and on a toolkit of hard-won tricks [6]. Libraries such as Stable-Baselines3 [7] and CleanRL [8] bundle these tricks, but they do not lift all the weight. One still needs to pick the right hyperparameters, design the training environment carefully [9], and secure enough compute for long runs on high-end hardware. RL is powerful, yet getting to a working implementation can be timeconsuming and may require a great deal of effort even for an experienced practitioner.
ered in which a functional policy is available as a baseline that completes the task but does not achieve the desired performance. Rather than training a reinforcement learning policy from scratch, this paper proposes a method for embedding prior knowledge, in the form of the baseline policy, into the RL training process. This approach simultaneously improves training efficiency compared to vanilla from-scratch methods such as SAC [10], PPO [11], and TD3 [12], and yields a policy that outperforms the baseline.
Problem settings with an available but suboptimal baseline policy arise naturally across many application domains. A logistics company may rely on heuristic routing algorithms and seek to improve delivery efficiency; a financial trader may aim to refine a profitable but coarse rule-based strategy; a robotics engineer may use a controller that ensures stability but fails to optimize energy usage; or a game AI developer may start with rule-based agents and strive for superhuman performance in complex multi-agent settings. When accurate modeling is difficult, as in financial trading, vehicle routing, or video game AI, the available policy may come from hand-crafted routines, engineering heuristics, textbook techniques, or other applicationThese practical difficulties, together with the effectiveness of specific sources. Such solutions are unlikely to maximize metreinforcement learning itself, constitutes the central motivation rics of interest, just as in related approaches that also embed a of the present paper. A class of control problems is consid- baseline policy into the learning process the design of that baseline policy is not the primary concern, with the focus instead * The first two authors contributed equally. placed on how it is integrated into training. One such related †
Corresponding author.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
2
approach is residual reinforcement learning (residual RL) [13– tion module determines whether the executed action is supplied 16], which serves as the closest comparison baseline in this pa- by the learning policy or by the baseline policy. Initially, it favors the baseline policy. As training progresses, it increasingly per and is discussed in more detail in Section 1.2. favors the learning policy, gradually reducing dependence on The presence of approaches such as residual RL highlights that the baseline. Upon training completion, the learning policy, embedding prior knowledge via a pre-existing baseline policy represented by a standalone neural network, operates indepenfalls within a well-established class of reinforcement learning dently. methods. The contribution of the present paper therefore lies in the development of a reinforcement learning method that lever- Formally, at each training step t, the arbitration module can be ages a pre-existing baseline policy, together with its empirical represented by a mixing coefficient αt ∈ [0, 1]. This coefficient validation within an established and well-studied problem set- determines how the action At executed at step t is sampled from a probabilistic mixture of the baseline policy πb and the learning ting. policy πθt with weights θt at time step t: 1.1 High-level overview of the proposed method A ∼ (1 − α ) πb (• | S ) + α πθt (• | S ), (1) t
In standard reinforcement learning, an agent interacts with an environment over discrete time steps. At step t, the agent observes the current state S t and selects an action At according to its policy π(• | S t ), which is formally a conditional probability distribution over the action space. The environment executes that action, returns a reward, and produces the next state. The overall objective is to maximize cumulative reward. This objective can be pursued with a variety of standard RL algorithms, including canonical methods such as SAC [10], PPO [11], and TD3 [12]. The more formal setup used here is introduced later in Section 2.3. A schematic overview of this standard interaction loop is depicted in Figure 1 below.
Agent
Given current state St , the agent samples action At ∼ π(• | St ) from its policy π
Environment
Æ Environment executes At , returns reward r(St , At ) and new state St+1 back to agent
Figure 1: Standard reinforcement learning interaction loop. The agent aims to identify a policy maximizes the expected P that t discounted cumulative reward ∞ t=0 γ r(S t , At ), where γ ∈ (0, 1) is the discount factor and a hyperparameter of the problem. The formal problem statement is given in Section 2.3 and (4). The proposed method can be built on top of any appropriate RL backbone algorithm, including SAC [10], PPO [11], and TD3 [12]. At a high level, such backbone algorithms improve a policy through repeated interaction with the environment. They collect transition data generated by the agent and use these data to train an auxiliary model, commonly called a critic, whose role is to estimate how favorable particular actions or states are in terms of expected future reward. The policy is then updated in a direction favored by these estimates, so that the critic provides the learning signal used to improve decision making. Accordingly, the backbone fully determines the updates of the learning policy, critic, and any other trainable quantities. The proposed method leaves this learning mechanism intact and augments it with an arbitration module that governs two policies: the learning policy, represented by the trainable policy network, and a baseline policy. At each time step, the arbitra-
t
t
t
t
θt
where both π (• | S t ) and π (• | S t ) are, in general, stochastic distributions over the action space conditioned on the current state S t . The key design problem of the proposed method is therefore the construction of the coefficient αt , which in effect constitutes the arbitration module itself. The coefficient αt can be represented as a Bernoulli random variable. It is equal to one almost surely if the critic value of the candidate learning action exceeds the current episode-local critic value Q†t by the margin ν. Here, Q†t is the maximum critic value observed in the current episode up to the margin ν, where ν is also an algorithmic hyperparameter. Here and below, this critic is denoted by Qw , where w collects its trainable parameters. Otherwise, it is sampled with success probability prel λt−τ , where τ is the beginning time of the current episode and t is the current training time step. Equivalently, αt is obtained by an ordinary logical OR between the critic-trigger event and the random relaxation event: i h θ αt = I Qw (S t , Aπt ) ≥ Q†t + ν ∨ Ut ≤ prel λt−τ . b
Here, Ut ∼ Uniform[0, 1] and I denotes the indicator function. The parameters prel and λ are fixed within an episode and updated only between episodes, increasing monotonically until both become equal to one. A high-level overview is shown in Figure 2, and a detailed description is given in Section 3; see in particular Section 3.5 and Algorithm 2. The activation probability P [αt = 1] is designed to remain relatively small on average during the early stages of training, so that most actions are still selected from the baseline policy. As training progresses, this activation probability is designed to increase on average according to a prescribed schedule until, after a finite transition time T tran , it becomes equal to 1. This terminal case corresponds to both prel and λ reaching their maximum possible value of 1, so that αt = 1 almost surely. As a result, all subsequent actions are selected from the learning policy and the baseline policy is no longer invoked. In tasks where solving the environment amounts to driving the agent to a goal set and keeping it there, the proposed method offers an additional advantage. In this setting, the design of the mixing coefficient αt allows the method to retain the beneficial influence of the baseline policy from the very beginning of training, provided that the baseline policy itself is able to reach the goal, albeit in a generally suboptimal manner. Experimental findings in Section 5.3 indicate that, compared with the other evaluated algorithms, this is associated with consistently high goal-reaching rates throughout training. Moreover, under the stated assumptions, this phenomenon admits a theoretical
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique Initialize environment and RL backbone; set algorithm parameters prel , λ, and ν; set Q† ← −∞
Sample actions from baseline and learning policies learning policy (NN)
baseline policy
learning action Atπ
θ
baseline action Atπ
Select action given critic Qw , benchmark Q† , episode start τ , and Ut ∼ Uniform[0, 1]
αt = 1 by ordinary OR: θ Qw (St , Atπ ) ≥ Q†t + ν or Ut ≤ prel λt−τ
Execute learning action
b
otherwise αt = 0
Execute baseline action
Update benchmark Q† : if θ Qw (St , Aπt ) ≥ Q†t + ν, θ set Q† ← Qw (St , Aπt )
Update critic and learning policy when scheduled by RL backbone algorithm
no
Is episode complete?
yes Reset episode state, update episode start τ , set Q† ← −∞, and move prel and λ closer to 1 to give more agency to the learning policy
Figure 2: Diagram of the proposed method. The diagram shows action sampling from the baseline and learning policies, the arbitration rule, the episode-local benchmark update, and the scheduled updates of the RL backbone. A detailed version of the algorithm illustrated here is provided in Algorithm 2 within Section 3.
interpretation of why it may happen that goal-reaching performance remains strong, as observed in the experimental findings, during the early and intermediate stages of learning (i.e., for t < T tran ); see Section 4. The uniform result in the main body strengthens this analysis under additional assumptions on the baseline policy that quantify how quickly trajectories approach the goal set from a given initial state; see Definition 6 and Theorem 2.
3
policy operates autonomously (i.e., for t ≥ T tran ). Experimental findings in Section 5.3 suggest that it does, with the proposed method attaining the highest goal-reaching rates among the compared approaches. This question is also addressed formally in 4.3 through a transfer analysis between two regimes: the baseline-assisted regime, in which baseline actions may still be selected, and the baseline-free regime, in which actions are generated exclusively by the learning policy. Under the stated assumptions, this analysis yields an explicit lower bound on the probability that the standalone learning policy reaches the goal region and remains there. In qualitative terms, the result shows that some performance degradation may occur once the baseline is removed, but that this degradation is not arbitrary and can be characterized within the proposed framework. The analysis therefore clarifies the conditions under which the loss after full transfer of agency remains small, and when the standalone learning policy can still be expected to retain strong goal-reaching behavior.
1.2
Related work
A number of methods leverage existing policies into the training process. The existing policy can be available in two forms: as recorded traces (or demonstrations) of its behavior (trajectories, state–action pairs, videos, etc.), or as a callable policy that can be queried on the environment at any time step. A large body of work operates in the first regime. Many well-established methods do not follow the classical RL setting at all, instead seeking a policy that reproduces or improves upon existing demonstrations of expert behavior through supervised or self-supervised objectives. VPT [17] recovers policies from unlabeled internet video via an inverse-dynamics model. GAIL [18] does employ RL, but only as an intermediate mechanism: the core learning signal comes from a discriminator trained to distinguish expert from generated behavior, with RL serving to optimize the policy against this surrogate reward. A separate line of work takes full RL training as the backbone and injects demonstrations directly into it. DQfD [19] permanently seeds the replay buffer with expert transitions, conducts an offline warm-up phase, and augments the loss with supervised classification and regularization terms; DAPG [20] pre-trains the policy via behavioral cloning and adds a geometrically decaying demonstration-weighted auxiliary loss to the policy gradient. All of the methods above operate on a fixed, pre-collected dataset of demonstrations. The proposed approach belongs to a different regime: prior knowledge is available as a callable baseline policy that can be queried at every time step during training, and the method acts at the level of action selection without modifying the backbone RL algorithm’s loss functions or gradients. Methods that share this live-policy regime are therefore the appropriate point of comparison. A widely adopted representative of this regime is residual reinforcement learning (residual RL) [13–16].
In residual RL, the control action is decomposed into the algeHowever, the baseline policy is fully phased out by design once braic sum of a pre-designed controller (i.e., the baseline policy) the transition time T tran is reached. Another question addressed and a learned residual policy trained via reinforcement learnhere is whether such performance persists once the learning ing. Formally, given the current state S t , the executed action At
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique is defined as
θ
At = Aπt + Aπt , b
• as shown in Section 5, the proposed method preserves high goal-reaching rates throughout training, including the initial and intermediate stages of learning. In Section 4, theoretical insight is provided into this phenomenon. In contrast, the canonical formulations of Residual RL and from-scratch reinforcement learning algorithms such as PPO [11], TD3 [12], and SAC [10] were not originally designed to admit theoretical interpretations of this kind. Moreover, in practice, their performance during the initial and intermediate stages of training is often poor, as the underlying learning policies remain significantly undertrained in these stages.
(2)
where Aπt ∼ πb (• | S t ) is sampled from the baseline policy πb , which is generally a probability distribution over the action θ space conditioned on the current state S t , and Aπt ∼ πθt (• | S t ) is sampled from the learning (residual) policy πθt with parameters θt at time step t. b
Residual RL has demonstrated superior performance, particularly in domains of complex robotic manipulation. Experiments show that learning a residual on top of hand-designed or modelpredictive controllers yields substantial improvements in challenging manipulation tasks involving partial observability, sensor noise, model misspecification, and controller miscalibration [14]. Complementary work applies residual RL to real-world robot manipulation with contact and friction dynamics [13], extends the paradigm to learning from demonstrations with highdimensional visual inputs [15], and demonstrates generalization to new challenges such as autonomous vehicle control by combining prior knowledge with residual learning [16]. 1.2.1
Comparison to Residual Reinforcement Learning
Residual RL shares several key methodological similarities with the proposed method, making it a fair and meaningful point of comparison:
4
1.2.2
Safe, Shielded, and Switching Control Constitute a Different Problem Setting
The proposed method differs fundamentally from safe reinforcement learning, shielded RL, and classical switching controllers. Safe and shielded RL approaches introduce persistent safety mechanisms—such as hard constraints, safety critics, or action filters—that remain active throughout deployment (and/or training) to enforce constraint satisfaction [21– 23, 25, 28, 29]. In contrast, the proposed method does not impose safety constraints nor guarantee constraint satisfaction; instead, it leverages a functional baseline policy as a temporary competence scaffold during training. Unlike shielded reinforcement learning, where unsafe actions are deterministically overridden at runtime by a formally synthesized shield [24], the result of the proposed approach is a standalone neural network, and no actions are overridden during execution. Finally, while switching and hybrid control rely on pre-defined statedependent or rule-based mode selection [26, 27, 30, 31], the proposed method performs learning-adaptive arbitration driven by critic improvement and probabilistic scheduling, enabling a smooth, data-driven handover of control authority rather than a fixed or permanent switching structure.
• it requires a comparable set of prerequisites. Specifically, it explicitly embeds a baseline policy into the training process; • it can be instantiated on top of any widely used offpolicy actor–critic reinforcement learning algorithm (similarly, the proposed method can be applied on top of vanilla RL algorithms such as SAC [10], PPO [11], and TD3 [12]); • it has an implementation complexity comparable to the proposed method, as both approaches require only a minimal modification of the action computation while leaving the core training components of the un- 1.2.3 Comparative Summary derlying reinforcement learning algorithm (i.e., the The related approaches discussed above can be summarized critic and policy update routines) unchanged. by five functional features that determine how external compeHowever, it is important to emphasize that residual RL is dif- tence enters the learning problem and whether it remains part of the final controller. Table 1 uses these features to place the ferent from the proposed method in several key ways: proposed method relative to the main families of related work. • during deployment, Residual RL continues to rely The table separates required resources and dependencies from on the baseline policy, whereas in the proposed ap- theoretical support. The columns have the following interpreproach the baseline is completely removed through a tation: deliberately designed arbitration module that transfers agency from the baseline to the learning policy; • Recorded external behavior marks methods that use • Residual RL computes actions as an algebraic sum of demonstrations, traces, logs, or offline trajectories the baseline and residual policies: from another source. This feature captures approaches where prior competence is present only through data πb πθ πb b πθ θ At = At +At , where At ∼ π (• | S t ), At ∼ π (• | S t ). and the generating policy or controller cannot be (3) queried during training. In contrast, the proposed method employs a switching mechanism in which the executed action is sampled • Callable external policy/controller during training from an algebraic mixture of two action distributions marks methods that can query an external policy, concorresponding to the baseline and the learning policy: troller, shield, safety filter, or prior while training the At ∼ (1 − αt ) πb (• | S t ) + αt πθt (• | S t ),
learning policy.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
5
Table 1: Functional comparison of related method families. The first four columns report required resources and deployed dependencies: “Yes” indicates that the resource is usually required, and a centered dot indicates that it is not. The final column reports theoretical support: a check mark indicates that formal closed-loop behavior analysis is provided. Required resources and dependencies
Method family
Standard model-free RL [10–12] Demonstration-based imitation/RL [17–20] Safe, shielded, CBF/CLF, and switching control [21–27] Residual RL [13–16] Proposed method
Theoretical support
Recorded external behavior
Callable external policy/controller during training
External policy/ controller at deployment
Model or constraint oracle
Formal analysis of closed-loop behavior (stability/safety/ reachability/constraints)
· Yes · · ·
· · Yes Yes Yes
· · Yes Yes ·
· · Yes · ·
· · ✓ · ✓
• External policy/controller required at deployment marks methods whose deployed system still needs that external component after training. • Model or constraint oracle required marks methods that rely on dynamics, a learned model, a constraint checker, a control barrier or Lyapunov function, a reachability oracle, or a comparable external formal object. • Formal analysis of closed-loop behavior marks method families that provide an analysis of stability, safety, constraint satisfaction, recoverability, reachability, or goal-reaching behavior under the induced policy or controller. Generic optimization or convergence analyses are not counted here unless they characterize the behavior of the controlled closed-loop system.
Definition 1 (Stationary Policy). A stationary policy π(• | s) : S → R≥0 maps each state s to a probability density over action space A, independent of time. Definition 2 (Non-stationary Policy). A non-stationary policy π(• | s, t) : S × N → R≥0 maps each state-time pair (s, t) to a probability density over action space A. Let Πstat and Πnstat denote the sets of stationary and nonstationary policies, respectively. Note that Πstat ⊆ Πnstat . 2.3
Reinforcement Learning Objective
The standard RL objective seeks a policy π ∈ Πnstat that maximizes the expected discounted return: ∞ X t J(π) = E γ r(S t , At ) , (4) t=0
2
Problem Statement and Notation
2.1
Markov Decision Process
where γ ∈ (0, 1] is the discount factor, and the trajectory {(S t , At )}∞ t=0 is generated according to: S 0 ∼ p0 (•), At ∼ π(• | S t , t), S t+1 ∼ p(• | S t , At ).
(5)
The environment is modeled as a Markov Decision Process For a fixed stationary policy π ∈ Πstat , the state-value function is defined by (MDP) defined by the tuple (S, A, p, p0 , r), where: ∞ X t π • S is the state space (assumed to be a Banach space (6) v (s) = E γ r(S t , At ) | S 0 = s . with norm ∥•∥ for theoretical completeness); t=0 • A is the action space (assumed to be a compact topo- Similarly, the corresponding action-value function (Q-function) logical space); is defined by ∞ • p(• | s, a) : S × S × A → R≥0 is the transition probX t π ability density function for the sampling of the next Q (s, a) = E γ r(S t , At ) | S 0 = s, A0 = a . (7) state S ′ given the current state s and the action a, i. e., t=0 S ′ ∼ p(• | s, a). The theoretical analysis assumes In both definitions, the trajectory evolves according to A ∼ t the existence of an upper semi-continuous function π(• | S ) and S t t+1 ∼ p(• | S t , At ) after the stated initial condip̄ : S × A → R≥0 such that for any s ∈ S and a ∈ A, tion. the next state S ′ sampled from p(• | s, a) satisfies P ∥S ′ ∥ ≤ p̄(s, a) = 1, effectively bounding the sys2.4 Goal Set and Goal-Reaching Property tem’s one-step transition magnitude; • p0 (•) : S → R≥0 is the initial state distribution, where For theoretical analysis and experimental validation, the conS 0 ∼ p0 (•); cept of a goal set is introduced and the goal-reaching property is formalized. • r(s, a) : S × A → R is the reward function. Definition 3 (Goal Set). The goal set G ⊆ S is a closed subset of the state space representing the desired target region. The 2.2 Policy Definitions environment is considered solved when the system reaches G A distinction is drawn between two types of policies: and stays there.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique The goal set need not always be supplied as an independent object. It may be specified directly, for instance as a target region, or it may be induced by the reward function. When the reward is maximal at a target state or target region and decreases away from it, a suitable neighborhood of the reward maximizer defines the goal region. In this sense, the reward function itself can determine what it means for the environment to be solved, while the goal-reaching property formalized in Definition 4 specifies whether a policy reliably reaches that region.
3
Remark 1 (Practical Interpretation). The ε-improbable goalreaching property is not required for applying the proposed approach. Rather, it identifies a class of baseline policies for which the empirically observed goal-reaching behavior of the proposed method can be related to the theoretical analysis developed below.
3.2
6
Approach
The approach leverages an existing baseline policy to guide the training process while gradually transitioning to an autonomous learning policy. Upon training completion, the learning policy becomes a standalone neural network that operates independently without relying on the baseline policy. This autonomy is achieved by integrating the baseline policy into the learning process of the backbone RL algorithm used to train the learnGeneral boundedness assumption 1. When the goal region ing policy. The core principle is to make extensive use of the is understood as the region where the task is solved, it would baseline policy during early training stages and progressively be unnatural for rewards away from any neighborhood of that transfer agency to the learning policy. region to be unbounded from above. Such rewards would make The proposed approach is RL-algorithm-agnostic and any stanthe objective favor behavior unrelated to solving the task. Thus, dard RL algorithm can be used as its backbone, including in goal-reaching problems of this type, it is natural to assume TD3 [12], PPO [11], DDPG [32], and SAC [10]. The experthat the reward is bounded from above: there exists rmax < ∞ iments discussed in Section 5 employ TD3 and SAC; the TD3 such that r(s, a) ≤ rmax for all (s, a) ∈ S × A. backbone is also consistent with the implementation choices in residual RL [13]. To analyze policy performance with respect to goal-reaching, The remainder of this section is organized into several subthe following notation is introduced: sections, each introducing one of the main constituents from • S tπ (s0 ) denotes the state at time t under policy π ∈ which the proposed method, formally specified in Algorithm 2, is built. Algorithm 1 should not be interpreted as a separate alΠnstat with initial state S 0 = s0 ∈ S; gorithm executed independently of the proposed method: it is ′ • dG (s) = inf s′ ∈G ∥s − s ∥ denotes the distance from state invoked inside Algorithm 2, line 4, and forms part of that algos to the goal set G. rithm. This decomposition is adopted for clarity and to reduce Definition 4 (ε-improbable goal-reaching property). A policy the cognitive load on the reader. π ∈ Πnstat satisfies the ε-goal-reaching property if for any initial state s0 ∈ S, 3.1 Baseline Policy π P lim dG (S t (s0 )) = 0 ≥ 1 − ε, The baseline policy πb ∈ Πstat is a functional but suboptimal t→∞ policy that successfully completes the task. While it is assumed where ε ∈ [0, 1) is the failure probability tolerance. that πb satisfies the ε-improbable goal-reaching property for theoretical analysis, this is not a strict requirement in practice. This formalization makes it possible to demonstrate that the The property serves to formalize the notion of a “functional” or approach described in Section 3 preserves high goal-reaching “working” policy. rates during the initial stages of learning. The learning policy πθ is a policy parameterized by weights θ (e.g. artificial neural network) that is trained in order to outperform the baseline policy. 3.3 2.5
Additional Notation • Throughout the paper, I{•} denotes indicator function, i. e., ( 1, if • holds, I{•} B 0, otherwise. • For any vector x = (x1 , ..., xn ) ∈ Rn , ∥x∥2 denotes the P 1/2 Euclidean norm, ∥x∥ B ni=1 xi2 . • For any subsets X1 , X2 ⊆ Rn , d(X1 , X2 ) denotes the Euclidean distance between the subsets, d(X1 , X2 ) B inf x1 ∈X1 ,x2 ∈X2 ∥x1 − x2 ∥2 . • In the algorithm listings only, ← denotes deterministic assignment and f denotes stochastic assignment. In particular, X f pdf(•) means that X is sampled from the specified density pdf(•).
Learning Policy
Backbone RL Algorithm
The backbone RL algorithm is the underlying reinforcement learning method used to train πθ . Algorithm 2 is flexible and permits the usage of an arbitrary RL algorithm as the backbone. An RL backbone is characterized by when it performs critic and policy updates and by the corresponding update operators. Accordingly, it is assumed that each backbone is specified by four routines: • is critic update time, perform critic update determine whether a critic update should be performed at time step t and implement that update; they are invoked in Algorithm 1, lines 18-20, • is policy update time, perform policy update determine whether a policy update should be performed at time step
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
7
t and implement that update; they are invoked in and otherwise falling back to the baseline action (line 7), where Algorithm 1, lines 21-23. Ut ∼ Uniform[0, 1]. The next state is then obtained as S t+1 f p(• | S t , At ). For formal notation, the critic weights are denoted by w and the learning policy weights by θ. Algorithm 1 and Algorithm 2 Finally, the episode-local benchmark is updated as follows action exceeds the current benchare written for a backbone with a Q-function-based critic. The (lines 15–17): if the learning † † wt πθ mark by at least ν, i.e., Q (S t , At ) ≥ Qt + ν, then Qt+1 is set value-function version is obtained by a direct notational re† † wt πθ placement, as discussed in Remark 3. Accordingly, the critic to Q (S t , At ); otherwise Qt+1 ← Qt . approximation at training time step t is denoted by Qwt (s, a), where wt ∈ W represents the critic weights at time step t and Intuition. The purpose of the arbitration module is to drive W is the space of all critic weights. The learning policy at time the trajectory toward directions in which the critic estimate inθ step t is similarly denoted as πθt . creases. At time t, the quantity Qwt S t , Aπt is interpreted as an Remark 2 (Upper-consistent critic). For a feasible and well- estimate of expected return for taking the learning action in the posed discounted-control problem, the objective functional in current state. Therefore, when θ (4) has a maximal theoretically attainable value. Denote this Qwt S t , Aπt ≥ Q†t + ν, ⋆ scalar value by Q . This value gives the theoretical upper scale for any policy-induced critic value. From a theoretical the critic increase is treated as significant, which is interpreted standpoint, it is therefore natural to parameterize the critic ap- as a strong signal that the agent is moving in a promising direcproximation class used during training so that every admissible tion; the learning action in this case is accepted deterministicritic is uniformly bounded from above by this scale. Since Q⋆ cally. If the condition is violated, the action can still be accepted is generally unknown, a computable scale can be chosen with within-episode decaying probability prel λt−τ . This mechaP∞upper ⋆ k max from Assumption 1: Q ≤ Q̄ := k=0 γ r . This can be en- nism can be interpreted as controlled risk-taking, enabling poforced, for example, by clipping critic outputs from above at Q̄ tential performance gains later in the episode. or by using a final activation of the form Q̄ − |x|. The enforced This acceptance logic is directly analogous to simulated ancritic then satisfies Qw (s, a) ≤ Q̄ for all (s, a, w). nealing neighbor selection: improving candidates are accepted deterministically, while non-improving candidates may still be 3.4 Hyperparameters accepted with a probability that decreases as the temperature is lowered (i.e., over the annealing schedule). The algorithm makes use of four hyperparameters beyond those of the backbone: Algorithm 1 Intra-Episode Training • prel 0 ∈ [0, 1] (initial): base probability of selecting the learning policy over the baseline policy, • λ0 ∈ (0, 1] (initial): decay factor that reduces reliance on the baseline policy within episodes, • ν > 0: minimum improvement threshold required for updating Q†t to a new maximum, where Q†t is the best critic value observed so far in the current episode, • T tran : the number of steps it takes to fully transfer agency to the learning policy, 3.5
Arbitration Module
The arbitration module is implemented in Algorithm 1, lines 4–11. Let t denote the current training time step and τ the time when the current episode began. Thus, t − τ + 1 is the number of time steps that have passed since the beginning of the current episode. The maximum critic value observed in the current episode Q†t is tracked online and initialized at the beginning of every episode as Q†τ = −∞. At each time step t, the algorithm computes a candidate learnθ ing action Aπt f πθt (• | S t ) and a candidate baseline action b Aπt f πb (• | S t ) (lines 4–5). It selects between them by accepting the learning action whenever θ
Qwt (S t , Aπt ) ≥ Q†t + ν
or
Ut ≤ prel λt−τ ,
1: Input: global time t, state S t , critic weights wt , learning-
policy weights θt , baseline policy πb , schedule parameters prel , λ, threshold ν. † 2: τ ← t, Qt ← −∞ 3: repeat θ 4: Aπt f πθt (• | S t ) b 5: Aπt f πb (• | S t ) 6: Ut f Uniform[0, 1] θ 7: if Qwt (S t , Aπt ) ≥ Q†t + ν or Ut ≤ prel λt−τ then θ 8: At ← Aπt 9: else b 10: At ← Aπt 11: end if 12: S t+1 f p(• | S t , At ) 13: Store the transition for the backbone RL algorithm. 14: Q†t+1 ← Q†t , wt+1 ← wt , θt+1 ← θt θ 15: if Qwt (S t , Aπt ) ≥ Q†t + ν then θ 16: Q†t+1 ← Qwt (S t , Aπt ) 17: end if 18: if is critic update time(t) then 19: wt+1 ← perform critic update(wt ) 20: end if 21: if is policy update time(t) then 22: θt+1 ← perform policy update(θt ) 23: end if 24: t ←t+1 25: until episode complete(t) 26: Return: t, S t , wt , θt , episode length T = t − τ.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique Algorithm 2 Inter-Episode Training 1: Hyperparameters: initial schedule prel 0 , λ0 , threshold ν,
transition horizon T tran , and all hyperparameters of the backbone RL algorithm. 2: Initialize: global time t ← 0, schedule parameters prel ← prel 0 , λ ← λ0 , initial critic weights w0 , initial learning-policy weights θ0 , and S 0 f p0 (•). 3: while training is not complete do 4: Run one episode by calling the intra-episode rollout routine defined by Algorithm 1 (denoted EpRollout below): (t, S t , wt , θt , T ) ← EpRollout(t, S t , wt , θt , πb , prel , λ, ν). 5: Reset the environment for the next episode: S t f p0 (•). PT −1 k 6: Compute η ← min 1, Tt−1 and χ0 ← prel k=0 λ0 . 0 tran 7: Compute χ ← χ0 + η(T − χ0 ). rel 8: Set prel ← prel 0 + η(1 − p0 ). P rel T −1 ′ k 9: Set λ ← solve χ = p (λ ) . k=0 ′
8
Taking lower bound is obtained: hP expectations,θ ian adjustable T +τ−1 π rel PT −1 t E t=τ I{At = At } ≥ p t=0 λ . The strategy behind these updates works by making this lower bound grow linearly across training episodes: 1. prel is incrementally increased after each episode until it reaches unity (line 8), 2. the following equation is solved for λ to equate the lower bound to the target value χ (lines 7–9): P −1 t prel Tt=0 λ = χ, where χ increases linearly from a small initial value to T over the duration of training (lines 6–7). It is recommended to initialize prel 0 ∈ [0.8, 1.0] and choose λ0 prel
PT −1
λt
such that 0 Tt=0 0 ≈ 0.2. This choice ensures a strong reliance on the baseline policy at the early stages of training, while preλ ∈[0,1] venting excessive dominance that would otherwise cause the 10: end while collected training data to consist almost exclusively of transitions generated by the baseline policy. An example of the resulting profile is illustrated in Figure 7: prel increases linearly Remark 3. The listings in Algorithms 1 and 2 are stated in the toward one, whereas λ follows the nonlinear profile induced by Q-function form because it is more general for the present pur- solving the lower-bound equation in lines 7–9. pose. Indeed, the analogous listing for a value-function critic is obtained by the literal replacement of every occurrence of 4 Theoretical Analysis Qwt (s, a) in the listings with vwt (s). 4.1 3.6
Goal-Reaching Analysis
Transition Scheduling Strategy
The analysis begins by clarifying the policy notation used The transition scheduling strategy is implemented in Algo- throughout this section. For each training step t, let πt denote the non-stationary executed policy induced by Algorithm 1. rithm 2, lines 6–9. That is, πt is the composite training-time policy that, at each rel The hyperparameters p and λ fundamentally represent the de- decision step, executes either the learning-policy action or the gree of trust in the learning policy’s capabilities. When these baseline-policy action according to the arbitration rule of Algovalues are relatively low, the mechanism exhibits limited con- rithm 1. fidence in the learning policy and maintains strong reliance on the baseline policy for guidance. Conversely, as prel and λ ap- The notion of high goal-reaching rates is interpreted as folproach unity, the system demonstrates increasing trust in the lows. During the initial training phase, that is, for t < T tran , the learning policy, invoking the baseline policy less frequently. majority of episodes terminate by reaching the goal set G. When both parameters reach their target values prel = 1 and An important remark is required at this point. Reaching the λ = 1, the probabilistic selection reduces to Ut ≤ 1, which is goal set G generally requires a sufficient number of interaction always satisfied, effectively recovering the pure backbone RL steps. Therefore, each episode must be long enough to allow algorithm with no baseline policy involvement. the policy π to reach G. Consequently, any theoretical intert
The core idea underlying the entire scheme is to begin training with relatively low values of prel and λ, then systematically increase them throughout the training process until they reach unity. This way agency is gradually transferred from the baseline to the learning policy. Initially, the algorithm exhibits strong reliance on the baseline policy, but as training progresses, the learning policy enhances its exploration capabilities and performance, warranting increased trust and autonomy.
pretation of goal-reaching behavior necessarily presumes that the episode length is sufficiently large relative to the time scale on which πt approaches the goal set. For a practical heuristic for choosing such a horizon, see Remark 5. This issue is addressed more explicitly in Theorem 2, which relies on a quantitative characterization of how trajectories under the baseline policy approach the goal set from arbitrary initial states.
Scope of the result. To avoid unnecessary technical complications, a setting is considered in which the policy πt generated by Algorithm 1 is allowed to take as many steps as needed within a single episode to reach the goal set G. In Theorem 1, a setting is considered, in which the dynamics evolve within an episode of unbounded length; consequently, termination or Let T denote the episode length. The probabilistic selec- truncation does not occur once the policy πt reaches G. Equivtion mechanism guarantees that in each episode, the learn- alently, in the notation of Algorithm 1, episode complete is P +τ−1 ing policy is selected at least Tt=τ I{Ut ≤ prel λt−τ } times. taken to be identically false. Thus, Theorem 1 below should While one cannot directly control the exact number of learning policy actions per episode, a lower bound on this quantity can be systematically managed through the hyperparameter update strategy outlined in lines 6-9. Consider the update strategy in more detail.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
9
Remark 6 (On critic consistency in practice). The criticconsistency requirement in item 1 of Theorem 1 is used to keep the theoretical interpretation clean. It should not be read as a fragile practical prerequisite for the goal-reaching phenomenon. In implementation, even if the critic is not explicitly constrained according to Remark 2, critic estimates typically do not keep increasing indefinitely along an episode. Once the critic values saturate, or otherwise stop producing marginimproving updates, the critic-triggered preference for the learning policy is no longer activated at every step. The arbitration mechanism can then select the baseline policy again. When the 1. The critic approximation Qw (s, a) is chosen consis- baseline policy is capable of recovering the system, these intertently with Remark 2. ventions can correct poor learning-policy actions and drive the trajectory back toward the goal set. This is why goal-reaching 2. The decay factor λ satisfies λ < 1 at the beginning of behavior can still be observed empirically without explicitly enthe episode (which holds for t < T tran ). forcing the critic-consistency construction. b 3. The baseline policy π satisfies the ε-improbable goalreaching property. 4.2 Uniform Goal-Reaching Property
not be read as any kind of guarantee. Instead, the theorem offers a theoretical interpretation: it identifies a mechanism by which the policy πt inherits the asymptotic ε-improbable goalreaching property of the baseline πb , and thereby explains why, in practice, high goal-reaching rates are observed for t < T tran . This interpretive framing is maintained throughout the paper — in the statement and proof of the theorem, as well as in the surrounding discussion. Theorem 1. Consider an episode of unbounded length generated by πt . Suppose that:
Then the policy πt generated by Algorithm 1 also satisfies the ε-improbable goal-reaching property. Proof idea. The arbitration rule can select the learning policy only through two mechanisms: the critic-improvement condition and the stochastic relaxation condition. The criticimprovement condition can fire only finitely many times, because each such event increases the episode-local benchmark Q†t by at least ν, while the critic is uniformly bounded above. The stochastic relaxation condition also fires only finitely many P rel t times almost surely, since ∞ p λ < ∞ when λ < 1. t=0 Hence, after a finite random time, the composite policy coincides with the baseline policy. Since the baseline satisfies the εimprobable goal-reaching property from arbitrary initial states, the same property is inherited by πt .
Control-theoretic analyses often pay particular attention to formal uniform results. To address this aspect, this section states a stronger uniform version of the goal-reaching interpretation used in Section 4. The result concerns reaching time, uniform overshoot boundedness, and the distribution of the reaching time. It is not used in the experiments and should be viewed as an additional theoretical result included for completeness. Nevertheless, the assumptions entering the result are also explained from an implementation viewpoint, including how they can be made feasible in practice and how they may be enforced when applying the theorem.
For the uniform result, the stochastic relaxation term in the intra-episode mechanism of Algorithm 1 is analyzed with an additional critic-monotonicity gate. Concretely, the stochastic relaxation factor λt−τ prel appearing in the algorithm listing is replaced, in the analysis, by λt−τ ρrel t , where Proof. See A.1. n o rel wt πθ wτ πθ ρrel (8) t := p · I Q (S t , At ) ≥ Q (S τ , Aτ ) . Remark 4. The above result concerns the composite trainingtime policy πt . The goal-reaching analysis is extended to t−τ rel the standalone neural-network policy πθ after training in 4.3, Equivalently, the analyzed factor is λ p times the indicator in (8). Here, τ is the beginning of the current episode. The where Theorem 3 provides an explicit bound on the degradarel rel coefficient ρ coincides with p under normal conditions but t tion of the goal-reaching probability in terms of the expected becomes zero when the current episode trajectory follows an θ distance between trajectories sampled by πt and π . unfavorable path, i. e., when the critic value drops below its iniRemark 5 (Practical horizon heuristic). The proof also gives a tial level. The gate prevents the stochastic branch from activatsimple way to estimate how much additional episode length is ing the learning policy in such cases and is used only to make needed before the baseline-dominated behavior becomes visi- the uniform argument explicit. ble. Even if the learning policy is poor, it is still activated by The exposition begins with a few auxiliary definitions. Definithe stochastic relaxation branch in expectation at least tion 6 refines Definition 4 by requiring a class-KL certificate ∞ rel X p rel t (Definition 5) that bounds the decay of the distance to the goal p λ = 1 − λ over time. Definition 7 fixes the terminology for superlevel sets. t=0 times. Thus, as a practical rule of thumb, the episode hori- Definition 8 then introduces a concept that is uncommon in the zon should allow the baseline policy enough effective steps to literature—functions with bounded superlevel sets. Although reach the goal even after roughly this many learning-policy in- this definition is not frequently encountered, it is in fact closely terventions. Equivalently, when the baseline needs about T b related to the well-known concept of radial unboundedness (see steps to reach G, one should choose the episode horizon no- Definition 9). Specifically, a function has bounded superlevel ticeably larger than T b + prel /(1 − λ) during the early training sets if and only if its negative is radially unbounded, or if a simphase. The next section provides a constructive uniform result ple logarithmic transformation of it is radially unbounded (see that makes this type of horizon reasoning formal, but it requires Proposition 1). additional assumptions; the estimate above is intended only as Definition 5 (Class-KL function). A function β : R≥0 × R≥0 → R≥0 belongs to class KL if the following properties hold: for a quick heuristic.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
10
As for the practical choice of κlow and κhigh , Remark 2 provides P∞ the natural upper scale Q̄ := k=0 γk rmax . Thus, the upper envelope can be chosen as κhigh (s) ≡ Q̄. If the rewards are bounded on both sides, the lower envelope can be chosen analogously. When rmin ≤ r(s, a) ≤ rmax , one can simply define P k min κlow (s) = ∞ γ r − ∥s − sG ∥2 and κhigh (s) ≡ Q̄, where sG k=0 is the center of the goal set G. In this common case, Assumption (A1 ) imposes no additional restriction: it merely formalizes the natural boundedness of the critic that follows from bounded Definition 7 (Superlevel set). Let X be a set, let f : X → R, rewards and discounting. and let a ∈ R. The a-superlevel set of f on X is the set If the rewards are unbounded from below (for example, for negative quadratic costs), one can estimate empirical bounds by {x ∈ X | f (x) ≥ a}. sampling several random trajectories under the initial policy, Definition 8. Let X be a metric space and f : X → R. The evaluating Qw0 (st , at ), and then defining κlow using the empirifunction f is said to have bounded superlevel sets if, for every cal minimum. These initial estimates can then be refined adapa ∈ f (X) = { f (x) | x ∈ X}, the a-superlevel set of f on X is tively as training progresses. Such constructions ensure that bounded in X. the critic operates within a numerically stable and well-defined Definition 9. A function f : Rn → R is radially unbounded if range, making Assumption (A1 ) both theoretically sound and straightforward to implement in practice. lim∥x∥→+∞ f (x) = +∞. each fixed t ≥ 0, the mapping r 7→ β(r, t) is continuous, satisfies β(0, t) = 0, and is strictly increasing on R≥0 ; and for each fixed r ≥ 0, the mapping t 7→ β(r, t) is nonincreasing and satisfies limt→∞ β(r, t) = 0. Definition 6. A policy π ∈ Πnstat is said to satisfy the uniform ε-improbable goal-reaching property if there exists a function β ∈ KL such that, for all initial states s0 ∈ S, P dG (S tπ (s0 )) ≤ β(dG (s0 ), t) for all t ≥ 0 ≥ 1 − ε.
Proposition 1 (Equivalent characterizations of functions with Remark 7 (On critic envelope constraints). Assumption (A1 ) bounded superlevel sets). Let f : Rn → R. The following should be understood as a design constraint on the critic class, not as a claim that an unconstrained neural-network critic austatements are equivalent: tomatically satisfies such bounds during training. The motiva(i) f (x) has bounded superlevel sets; tion for using state-dependent comparison envelopes is standard in Lyapunov analysis: continuous positive definite func′ (ii) lim f (x) = inf f (x ) ∈ [−∞, +∞), and f (x) > ′ n tions can be bounded from below and above by class-K func∥x∥→∞ x ∈R ′ n tions, and if they are radially unbounded the bounds can be inf f (x ) for all x ∈ R ; x′ ∈Rn chosen from class-K∞ [33, Lemma 4.3]. Thus, for an ideal value- or Lyapunov-like object, a role that the optimal critic (iii) − f (x) is radially unbounded, or there exists a finite inf inf n inf x∈R f (x) =: f ∈ R such that − log f (x) − f is can play under the usual control interpretation, the existence of comparison-function envelopes is a classical fact. In the radially unbounded. present algorithm this fact is used as motivation for restricting the critic approximator by construction. For example, with reProof. See A.2. min wards bounded below , one may use the same envelopes P∞by rk min low as above, κ (s) = γ r −∥s− sG ∥2 and κhigh (s) ≡ Q̄, and k=0 There are two technical assumptions needed to prove Thelow w high orem 2: Assumption (A1 ) and Assumption (A2 ). Assump- enforce κ (s) ≤ Q (s, a) ≤ κ (s) through output clipping or final layer. The bounded-superlevel-set property retion (A2 ) is simply a requirement that the baseline policy satisfy a bounded low quired of κ then follows from the quadratic term in ∥s − sG ∥2 . Definition 6. Assumption (A1 ) requires additional discussion. Consequently, Assumption (A1 ) is a feasible critic-design reIn particular, it is necessary to ensure that the critic function quirement rather than an automatic property of an arbitrary Qw (s, a) remains uniformly bounded between two continuous trained neural critic. envelope functions for all states, actions, and critic parameters: Theorem 2. Consider the intra-episode process generated by Algorithm 1, with the stochastic relaxation term interpreted κlow (s) ≤ Qw (s, a) ≤ κhigh (s), ∀s ∈ S, a ∈ A, w ∈ W. through the critic-monotonicity gate ρrel in (8). Without loss There are several practical ways to guarantee this property. One of generality set τ = 0. Let this process tbe initialized at s with 0 option is to explicitly clip the critic outputs, dG (s0 ) ≤ d◦ , where d◦ ∈ R>0 is arbitrary. Qw (s, a) ← clip Qw (s, a), κlow (s), κhigh (s) , Assume that: which enforces the desired bounds by construction. Equiva- (A ) The function Qw (s, a) admits lower and upper bound1 lently, the last critic layer can be parameterized through any ing continuous functions κlow (s) and κhigh (s), respectively, suitable bounded activation and then linearly transformed to where κlow (s) also has bounded superlevel sets: the interval [κlow (s), κhigh (s)]; examples include a linearly transformed tanh activation or a bounded folding map such as x 7→ κlow (s) ≤ Qw (s, a) ≤ κhigh (s). arccos(cos x). Alternatively, one can regularize the critic pafor all s ∈ S, a ∈ A and w ∈ W. rameters w (for instance, using spectral normalization or weight decay) so that the critic values cannot diverge outside the adb missible range. Both approaches make Assumption (A1 ) easy (A2 ) π satisfies the uniform ε-improbable goal-reaching property with certificate β ∈ KL (see Definition 6). to satisfy in practice while preserving continuity and stability of the critic, ensuring the boundedness conditions required for Then the following claims hold: the proof below.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
11
(C1 ) (ε-improbable uniform overshoot boundedness) There Finally, using a standard decomposition β(d, t) ≤ κ(d)ξ(e−t ) exists δ(d◦ ) ∈ R>0 such that with κ, ξ ∈ K∞ , define ( & !!') πt ◦ P dG S t (s0 ) ≤ δ(d ) for all t ≥ 0 ≥ 1 − ε. d∗ τb (d◦ , d∗ ) := max 1, − log ξ−1 . (16) κ(dmax (d◦ )) ∗ (C2 ) (ε-improbable uniform reaching time) For each d ∈ (0, d◦ ), there is an almost surely finite random time The random relaxation activations are controlled by T (d◦ , d∗ ) ∈ R≥0 such that T rel := inf{t ≥ 0 : Uk ≥ λk prel for all k ≥ t}, (17) πt ∗ ◦ ∗ ∞ P dG S t (s0 ) ≤ d for all t ≥ T (d , d ) ≥ 1 − ε. h i Y (1 − λk prel ). (18) P T rel ≤ t = (C3 ) (Reaching time distribution) There exist natural numbers k=t τ(d◦ ) and τb (d◦ , d∗ ) such that for all t ∈ Z≥0 , Thus the learning policy can interrupt the baseline no more than ∞ h i Y τ(d◦ ) + T rel times, and each interruption returns the state to a P T (d◦ , d∗ ) ≤ (τ(d◦ ) + t) τb (d◦ , d∗ ) = 1 − λk prel . region from which τb (d◦ , d∗ ) baseline steps suffice, with probak=t bility at least 1 − ε, to remain within d∗ of the goal. This gives Q∞ where k=t 1 − λk prel → 1 as t → ∞. T (d◦ , d∗ ) = τ(d◦ ) + T rel τb (d◦ , d∗ ), (19) from which the overshoot, reaching-time, and distributional Furthermore, τ(d◦ ), τb (d◦ , d∗ ), and δ(d◦ ) are given explicitly in claims follow. (12), (16), and (15), respectively. Interpretation. Item (C1 ) is the bounded-excursion claim. Starting from any state with initial distance at most d◦ , the closed-loop trajectory remains inside a finite operational tube of radius δ(d◦ ) with probability at least 1 − ε. This claim does not say that the trajectory is already near the goal. It says that the learning-policy insertions cannot make the process escape arbitrarily far before recovery by the baseline policy. Item (C2 ) is the eventual-settling claim. For any requested final tolerance d∗ < d◦ , there is a finite random time T (d◦ , d∗ ) after which the trajectory stays inside the d∗ -neighborhood of the goal, again with probability at least 1 − ε. Thus, temporary deviations are allowed before T (d◦ , d∗ ), but after that time the closed-loop behavior is interpreted as settled near the goal.
Proof. See A.3. Extension to value functions. The preceding analysis applies equally when the critic is a state–value function. To make this explicit, replace Qw (s, a) by vw (s) throughout the gated intra-episode process considered in Theorem 2 and track the reference level v†t instead of Q†t . Equivalently, this is the special case in which the critic is independent of the action argument. The structural assumptions translate directly (Assumption (A1 ) → Assumption (Av1 ), Assumption (A2 ) → Assumption (Av2 )), either by viewing vw as an aggregation of Qw or by postulating envelopes for vw itself. With this substitution, the definitions in the proof of Theorem 2 remain unchanged, so the overshoot bound, the uniform reaching time, and the distributional statement follow by the same argument. These facts are collected in Corollary 1 below. Corollary 1. Consider the value-critic analogue of the gated intra-episode process in Theorem 2, initialized at s0 with dG (s0 ) ≤ d◦ , where d◦ ∈ R>0 is arbitrary.
Item (C3 ) explains how large this settling time can be. It states that the time needed to reach and remain inside the d∗ -neighborhood can be computed constructively from the quantities appearing in the theorem. In practice, this reaching time is a random variable because the relaxation mechanism is stochastic. However, iits distribution is explicit: h Q Assume that: ◦ ∗ k rel P T (d , d ) ≤ (τ(d◦ ) + t)τb (d◦ , d∗ ) = ∞ k=t (1 − λ p ). Thus, for any selected probability level, this formula specifies a (Av1 ) The function vw (s) admits lower and upper bounding time by which the closed-loop process has reached the d∗ continuous functions κlow (s) and κhigh (s), respectively, neighborhood of the goal set. where κlow (s) also has bounded superlevel sets: Proof idea.
κlow (s) ≤ vw (s) ≤ κhigh (s).
The proof is constructive. First define vmin (d◦ ) := min{κlow (s) : dG (s) ≤ d◦ },
(9)
V(d◦ ) := {s : κlow (s) ≥ vmin (d◦ )},
(10)
for all s ∈ S and w ∈ W. (Av2 ) πb satisfies the uniform ε-improbable goal-reaching property with certificate β ∈ KL (see Definition 6).
vmax (d◦ ) := max{κhigh (s) : s ∈ V(d◦ )}, (11) % $ max ◦ Then the following claims hold: min ◦ v (d ) − v (d ) . (12) τ(d◦ ) := 1 + ν (C1 ) (ε-improbable uniform overshoot boundedness) There exists δ(d◦ ) ∈ R>0 such that The gated relaxation ensures that the learning policy is in voked only inside V(d◦ ). Consequently, the critic-improvement P dG S tπt (s0 ) ≤ δ(d◦ ) for all t ≥ 0 ≥ 1 − ε. ◦ branch can fire at most τ(d ) times. Next set uniform reaching time) For each d∗ ∈ dp̄ (d◦ ) := sup{ p̄(s, a) : s ∈ V(d◦ ), a ∈ A}, (13) (C2 ) (ε-improbable ◦ (0, d ), there is an almost surely finite random time dmax (d◦ ) := max{d◦ , dp̄ (d◦ )}, (14) T (d◦ , d∗ ) ∈ R≥0 such that δ(d◦ ) := β(dmax (d◦ ), 0). (15) P dG S πt (s0 ) ≤ d∗ for all t ≥ T (d◦ , d∗ ) ≥ 1 − ε. t
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
12
(C3 ) (Reaching time distribution) There exist natural numbers The two deployment regimes are formalized as the operation of two distinct policies. R1 corresponds to the policy πt generτ(d◦ ) and τb (d◦ , d∗ ) such that for all t ∈ Z≥0 , ated by the frozen fallback instance of Algorithm 2, while R2 ∞ h i Y corresponds to the pure neural-network instance of the same P T (d◦ , d∗ ) ≤ (τ(d◦ ) + t) τb (d◦ , d∗ ) = 1 − λk prel . algorithm, denoted by πθ . Both policies are initialized at a k=t common state s0 ∈ S and produce respective random state θ θ Q∞ sequences (S 0πt , S 1πt , . . .) ∈ ST and (S 0π , S 1π , . . .) ∈ ST , with k rel where k=t 1 − λ p → 1 as t → ∞. θ π S 0π = S 0t = s0 , where T ∈ N ∪ {∞} is the episode length. ◦ b ◦ ∗ ◦ Furthermore, τ(d ), τ (d , d ), and δ(d ) are given explicitly by the same formulas (12), (16), and (15), respectively. 4.3.2 Expected trajectory distance Hereafter, the Python-like shorthand X0:T denotes the sequence Proof idea. The proof repeats the constructive argument for (X0 , X1 , . . . , XT −1 ) (or (X0 , X1 , . . . ) when T = ∞), so that, for w w Theorem 2 after the literal substitutions Q (s, a) 7→ v (s) and πt πθ instance, the expressions above contract to S 0:T and S 0:T . Q†t 7→ v†t . The gated relaxation coefficient is replaced by The strategy for answering the central question—what can be prel I{vwt (S t ) ≥ vw0 (S 0 )}, said about the ability of the learned policy πθ to reach the goal set G when deployed on its own—is straightforward: if the traand the same formulas for τ(d◦ ), τb (d◦ , d∗ ), δ(d◦ ), and T (d◦ , d∗ ) jectories S πθ and S πt are close to each other, then one can 0:T 0:T in (9)–(19) apply. πθ estimate how close S 0:T is to the goal set. The first step is πθ therefore to quantify the distance between the trajectories S 0:T Proof. See A.4. θ πt πt π and S 0:T , S 0:T . A natural single-rollout distance is dST S 0:T := sup0≤t<T ∥S tπ − S tπt ∥, the worst-case state deviation over the episode. Since both state sequences are random processes, this supremum is itself a random variable. To obtain a scalar distance, the two processes are realized on a fixed common probability space and the expected sup-distance is used. For simplicity, this paper uses the independent-rollout convention: conditional on the common initial state, the rollout under πθ and the rollout under πt are generated independently. θ
4.3
Goal-Reaching Transfer to the Neural Policy
A key feature of the policy produced by Algorithm 2 is its reliance on the baseline fallback mechanism during training, particularly in the initial phase where λ < 1. As established in Theorems 1 and 2 and corollary 1, this mechanism admits a theoretical interpretation that explains the high goal-reaching rates observed in practice, consistent with the experimental findings reported in Section 5. A natural question arises: once the learned neural-network policy πθ is deployed without baseline intervention—that is, with the schedule parameters set to their terminal values (λ = 1, prel = 1), so that every action is sampled directly from πθ (• | S t )—what can be said about the ability to reach the goal set G? This section provides a formal answer to this question. 4.3.1
Deployment setting and trajectory distance
Suppose that at training step t the learning policy has weights θt = θ. Both deployment regimes below are obtained from Algorithm 1 by initializing a new episode at deployment time and using the rollout routine only for action selection: learning is disabled, so is critic update time and is policy update time always return False. Thus the critic weights w, learning-policy weights θ, and the schedule parameters used in action selection remain fixed throughout deployment. (R1 ) Policy πt with baseline fallback — the action-selection mechanism of Algorithm 2 is retained with the current training-time schedule values λ < 1 and prel ≤ 1. The baseline policy πb may therefore still be invoked as a fallback whenever the critic condition is not met.
Definition 10 (Independent-rollout trajectory distance). Let π1 , π2 be two (generally non-stationary) policies acting from a common initial state s0 ∈ S under the transition kernel p. Fix πk a horizon T ∈ N ∪ {∞}. For each k ∈ {1, 2}, denote by S 0:T the k random trajectory generated by π . The two trajectories are sampled independently. The trajectory distance is " # T 1 2 π1 π2 D (π , π ) := E sup ∥S t − S t ∥ . (20) 0≤t<T
The independence convention is not essential for Theorem 3. The proof of Theorem 3 only uses a fixed joint realization of the two state sequences and an upper bound on the corresponding expected sup-distance. Thus, the same theorem would remain valid under a shared-noise realization, an independent-noise realization, or any other prescribed joint construction, provided DT is replaced by the expected trajectory distance under that construction. The independence convention is used only to make the quantity in (20) unambiguous and easy to estimate.
A concrete and readily implementable estimate can be obtained as follows. Consider N independent pairs of trajectories, indexed by n = 1, . . . , N. Within each pair, one trajectory is generated by the learning policy πθ (with λ = 1, prel = 1) and the other by the deployment policy πt from R1 (with the (R2 ) Pure neural-network mode — all schedule parameters actual schedule parameters), using independent simulator ranare set to their terminal values (λ = 1, prel = 1), so every domness after theθ shared initial state. Denote the resulting state πt , n π ,n and S 0:T , respectively. The expected trajecaction is sampled directly from πθ (• | S t ) without any sequences by S 0:T tory distance in (20) can then be estimated by the Monte Carlo baseline fallback.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
13
Proof. See A.5.
average bTN := D
N 1 X
θ
sup S π , n − S tπt , n , N n=1 0≤t<T t
(21) Remark 8 (Optimal slack). The bound (23) involves a tradeoff between the neighborhood size d∗ + δ and the probability √ T θ which is a consistent estimator of D (π , πt ) under the loss ∆T /δ. Setting δ = ∆T yields i h p p independent-rollout convention. P τT (d∗ + ∆T , πθ ) < T ≥ 1 − ε − ∆T , Contaminated-Zone AUV Navigation MC estimate of DT
2.5
prel = 1
which converges to 1 − ε as ∆T → 0.
2.0
5
1.5
The proposed approach is validated in two environments:
1.0 0.5 0.0
Experiments
0.96
0.97
0.98 λ
0.99
1. Contaminated-Zone Autonomous Underwater Vehicle (AUV) Navigation (Section 5.1): the agent must reach a target location while avoiding contaminated areas.
1.00
Figure 3: Monte Carlo estimate of the trajectory distance in 2. Treasure-Collecting Robot (Section 5.2): the agent (21) for the Contaminated-Zone AUV Navigation environment. must reach a target region while collecting a highThe estimate is computed from independent rollout pairs of reward treasure along the way. the saved learning-policy checkpoint at the baseline-removal point and the corresponding policy with baseline fallback, us- The choice of environments is deliberate. To demonstrate the ing prel = 1 and varying the within-episode decay factor λ. key advantages of the proposed method, the evaluation task should have a goal set with a clear operational meaning and a baseline policy that can be specified explicitly. Many stan4.3.3 Goal-reaching transfer theorem dard Gymnasium benchmarks are less suitable for this purpose: Theorem 3 (Goal-reaching transfer via trajectory distance). in complex locomotion tasks, such as humanoid control, conFix a horizon T ∈ N ∪ {∞} and an initial state s0 ∈ S. Consider structing an explicit baseline policy is itself a difficult problem; regimes R1 and R2 , both executed from s0 . Define the settling in simpler benchmarks, such as Pendulum, Inverted Pendulum, or Mountain Car, the goal-reaching structure is too elementary time ( ) to stress the proposed mechanism; and in many generic bench τT (d, π) := min 0 ≤ t′ < T : sup dG S kπ ≤ d (22) marks the goal set is not as explicitly interpretable. These two t′ ≤k<T environments were chosen to expose the intended setting: goal ′ (set τT = ∞ if no such t exists). Assume that: regions are explicit, baseline policies are available in closed form, and the reward structure still leaves room for nontrivial ∗ (A1 ) P [τT (d , πt ) < T ] ≥ 1 − ε, i. e., the deployment pollearning behavior. They are also specified transparently: the ∗ icy πt produced by regime R1 settles within the d dynamics, reward functions, goal regions, and baseline polineighborhood of G and remains there until the end of cies are given explicitly. This makes it possible to introduce the episode with probability at least 1 − ε; demonstration metrics, such as goal-reaching and constraintavoidance rates, that directly reflect the behavior targeted by (A2 ) DT (πθ , πt ) ≤ ∆T for some ∆T ≥ 0. the method. Such metrics are complementary to cumulative reThen the following claim holds: ward, whose numerical value is often difficult to interpret on its own. (C1 ) (Settling transfer to regime R2 ) For every δ > 0: The section first describes the two evaluation environments and h i ∆T P τT (d∗ + δ, πθ ) < T ≥ 1 − ε − . (23) their associated tasks. It then reports the performance of the δ standard TD3 and SAC methods, along with their residual RL Proof idea. Realize the trajectory of the fallback deployment policy πt and the trajectory of the pure neural-network policy πθ independently, as in Definition 10. Their expected sup-norm distance is then controlled by DT (πθ , πt ). On the event that the two trajectories remain within distance δ of each other over the horizon T , the triangle inequality implies that every trajectory of πt that settles inside the d∗ -neighborhood of G yields a trajectory of πθ that settles inside the (d∗ + δ)-neighborhood. The only loss comes from the event that the two jointly realized trajectories separate by more than δ; Markov’s inequality bounds this probability by DT (πθ , πt )/δ, and hence by ∆T /δ. Combining this loss with the assumed settling probability 1 − ε for πt gives (23).
variants and the proposed approach instantiated on top of the same TD3 and SAC backbones, as summarized in Section 5.3. 5.1
Contaminated-Zone AUV Navigation
Environment Description. The Contaminated-Zone AUV Navigation task shown in the left panel of Figure 4 is governed by the following system of differential equations: ẋ = v x
ẏ = vy
ϑ̇ = ω
ω̇ = doffsetI ·Flat
Flong cos ϑ−Flat sin ϑ−Cd ∥v∥v x m Flong sin ϑ+Flat cos ϑ−Cd ∥v∥vy −mg v̇y = m
v̇ x =
(24)
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
14
where the state vector s = (x, y, ϑ, v x , vy , ω) ∈ R6 represents 5.2 Treasure-Collecting Robot the AUV’s position coordinates (x, y), orientation angle ϑ, linear velocities (v x , vy ), and angular velocity ω. The control input Environment Description. The robot moves with a constant a = (Flong , Flat ) ∈ [−1, 1] × [−0.5, 0.5] is two-dimensional, rep- linear speed. Its dynamics are given by resenting the longitudinal and lateral thrust forces. ẋ = v cos ϑ, ẏ = v sin ϑ, ϑ̇ = ω,
The system parameters are: mass m = 1, moment of inertia 2 I = 0.1, gravitational acceleration g = 0.5, drag coefficient where (x, y) ∈ [0, 1] are the robot’s coordinates, ϑ ∈ [−π, π) is Cd = 0.05, and lateral offset distance doffset = 0.2 for torque the robot’s heading, v ≡ 0.15m/s denotes the robot’s constant linear speed, and ω ∈ [−π, π) is the angular velocity serving as generation. the control input. Objective and Reward Function. The reward function encourages the AUV to reach the target while penalizing excessive velocities and contaminated area intrusion: v2y (y − 4)2 x2 v2x ω2 r(s, a) = − − − − − − 5 · I{(x, y) ∈ C} 4 4 20 20 100 Here the two interpretations coincide: the goal set is specified explicitly as G = {(x, y) ∈ R2 | ∥x∥ < 0.4 and y ≥ 4}, and it also corresponds to the high-reward target area at the water surface. The contaminated region is a parabolic area C = {(x, y) ∈ R2 | (y−2)2 x 0.81 + 0.36 ≤ 1} that the agent must avoid, depicted in green in Figure 4. The episode length is 1500 steps. Every step is integrated with a time step of 0.02 seconds.
During the episode the robot must collect a treasure whose position (xtr , ytr ) ∈ [0, 1]2 is known to the robot. The treasure is considered collected when the robot enters a neighborhood of radius 0.05 around the treasure, i.e., 2 2 x − xtr + y − ytr < 0.052 .
The treasure moves stochastically in the workspace. Let (xttr , ytrt ) and (vtx , vyt ) denote its position and velocity at time t, and let ∆t = 0.05 s. Its initial velocity (v0x , vy0 ) has a random heading and fixed speed vmax = 0.12. At each time step, the velocity is perturbed by random variables ϵtx , ϵty ∼ N(0, 0.032 ), which are independent across components and time, and the resulting velocity is clipped to magnitude vmax . The velocity and The initial state distribution is uniform position updates are given by
Initial Conditions. over the domain: p0 (•) ∼ Uniform [−2, 2] × [0, 4/3] × [9π/20, 11π/20] (25) × [−0.2, 0.2] × [−0.2, 0.2] × [−0.2, 0.2] , ensuring the AUV starts in the lower portion of the environment with moderate initial velocities and near-vertical orientation.
vtx + ϵtx √ x x2
x vt+1 =
max 1,
y
y
(vt +ϵt ) +(vt +ϵt )2 vmax
vyt + ϵty √ x x2
! vyt+1 = max 1,
y
y
(vt +ϵt ) +(vt +ϵt )2 vmax
!
tr xt+1 = xttr + ∆t vtx ytrt+1 = ytrt + ∆t vyt . The update is followed by reflection at the workspace boundBaseline Policy Design. The baseline policy employs two PD aries for xtr and ytr , implemented as component-wise veloct+1 t+1 controllers with coordinate transformation to direct the AUV ity sign flips. toward the goal. While this approach successfully reaches the goal set G, it does not account for the contaminated re- After collecting the treasure, the robot must reach a goal region gion, which often results in suboptimal trajectories through that is specified explicitly and is also favored by the distancethe penalty area. A sample trajectory demonstrating this be- to-goal term in the reward. The goal set is defined as o n havior is shown in the right panel of Figure 4. The com G = (x, y) ∈ [0, 1]2 x − xg 2 + y − yg 2 < 0.052 , plete implementation can be found in the repository: https: //github.com/aidagroup/calf-enhance. An animated visuwhere (xg , yg ) = (0, 0.5) is the goal position. alization comparing trajectory behaviors of the baseline policy and the final trained policy produced by the proposed method is also available in the repository.
Figure 4: Visualizations for the Contaminated-Zone AUV Navigation task. Left: environment layout. The contaminated region is highlighted in green, and the goal is located at the water surface at (0, 4). Right: sample trajectory of the baseline policy. The baseline reaches the goal region but passes through the contaminated area, illustrating its suboptimal behavior.
Figure 5: Visualizations of the Treasure-Collecting Robot task. Left: representative behavior of the nominal policy. The black marker on the left boundary indicates the goal, and the orange disk denotes the collectible treasure. The baseline policy steers directly toward the goal and does not intercept the treasure. Right: desired behavior induced by the learned policy. The learned policy first deviates toward the collectible treasure and then returns toward the goal, instead of following a direct goalseeking trajectory.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
15
only the method-specific parameters are selected. ν = 0.01 is set to a small value, so that the critic gate does not become s = (x, y, cos ϑ, sin ϑ, xtr , ytr , I tr ), overly conservative. The relaxation-schedule parameters prel 0 PT −1 t tr where I ∈ {0, 1} is a binary indicator of treasure availability: and λ0 are chosen so that prel t=0 λ0 /T ≈ 0.2. This yields 0 ( prel 0 = 0.8 and λ0 = 0.995 for the Contaminated-Zone AUV 1, if the treasure is available for collection, tr I = Navigation environment, and prel 0 = 0.9 and λ0 = 0.96 for the 0, if the treasure has already been collected. Treasure-Collecting Robot environment. The transition time The dynamics are integrated using an explicit Euler scheme at T tran is set to 2.7M; the baseline policy is never invoked after 20 Hz. Each episode lasts at most 1000 steps (50 s) and termi- this point, and the learning policy operates independently from nates early if the goal set G is reached. Representative rollouts 2.7M to 3.0M timesteps. for this task are shown in Figure 5. 5.3.1 Evaluation Reward Function and Objective. The reward function is deSeveral types of evaluation are introduced for better underfined as standing the performance of the proposed method across both TD3 and SAC backbones: r(s, a) = − (xg , yg ) − (x, y) 2 n o tr tr tr + 50 · I · I (x, y) − (x , y ) 2 < 0.05 , • Learning curves (episode return). Episode return is defined as the cumulative sum of rewards within an where ∥•∥2 denotes the Euclidean norm. episode. Learning curves report episode return verThus, the task rewards both collecting the treasure and reaching sus the total number of environment timesteps and are the goal as quickly as possible. presented in Figures 8 and 10. The state vector is 7-dimensional and is defined as
Initial Conditions. The initial state is sampled according to the following distribution: p0 (•) ∼ Uniform [0.7, 0.9] × [0.1, 0.9] × {−1}× {0} × [0.15, 0.85] × [0.05, 0.95] × {1} Baseline Policy Design. The baseline employs a simple geometric steering law that turns the robot toward the provided target point (goal). From the observation [x, y, cos ϑ, sin ϑ, . . .], the controller reconstructs the current heading as ϑ = atan2(sin ϑ, cos ϑ) and computes the desired heading to the goal as ϑ⋆ = atan2(yg − y, xg − x). The wrapped angle error is e = (ϑ⋆ − ϑ + π) mod 2π − π, and the action is the clipped proportional control ω = clip(kturn e, −ωmax , ωmax ). The parameters are set to kturn = 1 and ωmax = π. When the robot is (numerically) at the goal, ϑ⋆ is set to ϑ to avoid an ill-defined direction. This baseline reliably reaches the goal but optimizes only the goal-reaching objective; it is not designed to capture high-reward treasure, which results in suboptimal behavior with respect to the full reward function. This behavior is illustrated in the left panel of Figure 5. 5.3
Experimental results
Experimental Setup. The proposed method is evaluated with two backbones, TD3 and SAC, along with the corresponding vanilla and residual variants for each backbone. All algorithms are trained for 3M environment steps across ten independent random seeds for each algorithm-environment pair to ensure statistical reliability. The TD3 and SAC implementations are both sourced from the CleanRL library [8] and use the same environment interfaces and evaluation protocol. No backbonespecific hyperparameter tuning is performed: for both TD3 and SAC, the default CleanRL hyperparameters are used and kept fixed within each backbone family. For the proposed method,
• Goal reaching during training. To demonstrate the effectiveness of the proposed method in terms of goalreaching performance throughout training, a rolling goal-reaching rate is computed for each random seed. Specifically, for every completed episode, the binary goal-reaching indicator is averaged over the most recent 75 episodes in the same seed, using all available earlier episodes at the beginning of training. The resulting per-seed rolling curves are aligned on a regular timestep grid, and the median across seeds together with the interquartile range is then reported at each grid point. • Final-stage metrics. Since episode return often provides limited interpretability in reinforcement learning, additional task-oriented metrics are reported for both environments in Table 2 and Table 3. These metrics are computed over the final stage of training, i.e., for timesteps from 2.7M to 3.0M (after the transition time T tran = 2.7M). The following metrics are used: – Goal-reaching rate (both environments): percentage of episodes in which the goal set is reached. Higher values are better (best is 100%). – Treasure-collection rate (Treasure-Collecting Robot): percentage of episodes in which the treasure is collected. Since successful completion requires both collecting the treasure and reaching the goal, Table 3 reports both rates. A combined success score is also reported, defined as the average of the goal-reaching and treasure-collection rates. Higher values are better (best is 100%). – Avoidance score (Contaminated-Zone AUV Navigation): maximum penetration depth into C during an episode, max d (xt , yt ), C̄ , t∈episode
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
16
Fraction of learning policy calls to total episode steps (in %)
where d(·, ·) denotes the Euclidean distance, C̄ is Goal-reaching. Algorithm 2 achieves the highest goalthe closed complement of C, and (xt , yt ) is the reaching rates across all baselines in both environments. agent position at time t. Lower values are better • During training, the proposed method outperforms all (best is 0). other algorithms evaluated alongside it in terms of goal-reaching rate (see Figures 9 and 11). In particuDiagnostic plots. To illustrate how control is gradually translar, the goal-reaching rate is already high at the beginferred from the baseline policy to the learning policy, two dining of training. This is due to the design of the proagnostics are reported. The first is the fraction of actions seposed method, which effectively embeds the baseline lected by the learning policy per episode. The second is the policy into RL training process. A formal interpretarel evolution of the schedule parameters p and λ. Figure 6 shows tion of this behavior is provided in Section 4. Equivthe learning-policy call fraction during training, averaged over alently, the influence of the learning policy is small at ten independent random seeds, while Figure 7 shows the correthe beginning of training and then increases at a modsponding transition schedule. erate pace (approximately linearly on average; see the right panel of Figure 4). Baseline disabled • After the transition (t ≥ T train = 2.7M), when the Contaminated-Zone AUV Treasure-Collecting Robot 100 baseline influence becomes exactly zero, the learning policy—implemented as a standalone neural network 80 — operates independently and still attains the high60 est goal-reaching rates. This holds in comparison with the corresponding residual and vanilla backbone base40 lines, as reported in Table 2 and Table 3. 20 1
2
3 ×106
Step
1
2 Step
3 ×106
Figure 6: Evolution of the fraction of learning policy calls per episode during training. Results are averaged over ten independent random seeds, with shaded regions indicating standard deviation. At the beginning of training, the baseline policy produces most of the actions, so the learning-policy call fraction is low. As training progresses, agency is gradually transferred to the learning policy, with the learning-policy call fraction increasing approximately linearly—neither too abruptly nor too slowly. The vertical dashed line marks the Baseline disabled point, after which this fraction reaches 100% and the baseline policy is no longer called (Algorithm 2).
prel
p
Baseline disabled
rel
λ
λ
1.00 1.000
0.998
0.90
λ
prel
0.95
The slight decrease in goal-reaching rate observed near the end of training is consistent with the intended transition mechanism. During the early and intermediate stages, the baseline policy still has substantial influence and can often recover trajectories in which the learning policy has moved the system away from an easy path to the goal. Near the transition time, this corrective influence becomes negligible: the learning policy is selected almost always, and within a finite episode there may be too little baseline intervention left to compensate for occasional mistakes. This finite-horizon effect is precisely why the early-stage goal-reaching theorem in Section 4 is used as an interpretation rather than as a direct performance guarantee: in experiments, goal reaching is always judged over a finite episode length, so any empirical claim that a policy reaches the goal necessarily depends on whether the observation horizon is long enough for the relevant recovery mechanism to act. The same point is reflected in the trajectory-distance transfer analysis in Theorem 3: after moving from the baselinesupported regime to the baseline-free regime, the finite-horizon goal-reaching probability may degrade by a term controlled by the distance between the corresponding trajectory distributions.
Nevertheless, training on trajectories that predominantly reach the goal has a lasting effect. After the baseline is removed, the standalone learning policy achieves higher final goal-reaching 0.80 0.994 metrics than the other evaluated algorithms. This is the empir1 2 3 1 2 3 ×10 ×10 Step Step ical advantage targeted by the proposed method: the baseline rel Figure 7: Evolution of the schedule parameters p and λ dur- shapes the training distribution toward successful trajectories, ing training in the Contaminated-Zone AUV Navigation envi- and this influence persists even in the final baseline-free regime. ronment. The curves show the logged schedule values; since the schedule is deterministic, the same values are obtained for Final-stage metrics. Beyond goal-reaching, the proposed all ten independent random seeds. Both parameters increase approach performs well on task-specific metrics (see Tamonotonically toward their terminal value of one; the vertical ble 2 and Table 3). In particular, the avoidance score (Contaminated-Zone AUV Navigation) and the treasure collecdashed line marks the Baseline disabled point. tion rate (Treasure-Collecting Robot) are both strong. Combined with the best goal-reaching performance, this indicates that the proposed method finds better overall behaviors in both 5.3.2 Results cases: it reliably reaches the goal set while also optimizing the task objective (avoiding the contaminated region and collectThe results support the following observations. 0.996
0.85
6
6
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
0
Ours (TD3 backbone)
Baseline disabled
Treasure-Collecting Robot 40
−1000
20 0
−2000
−20
−3000
−40
−4000
1
2 Step
3 ×106
−60
1
2 Step
3 ×106
Figure 8: Episode return comparison for TD3-based methods. Results are averaged over ten independent random seeds for each algorithm-environment pair, with standard deviation bands. The vertical dashed line marks the Baseline disabled point.
Rolling 20-Episode Return
SAC
Residual SAC 0
Ours (SAC backbone)
Contaminated-Zone AUV
Baseline disabled
Treasure-Collecting Robot 40
−1000
20 0
−2000
−20
−3000
−40
−4000
1
2 Step
3 ×106
−60
1
2 Step
3 ×106
TD3
Rolling 75-episode goal-reaching rate (%) Median with interquantile range
Residual TD3
Contaminated-Zone AUV
Residual TD3
Ours (TD3 backbone)
Contaminated-Zone AUV
Baseline disabled
Treasure-Collecting Robot
100 80 60 40 20 0
1
2 Step
3 ×106
1
2 Step
3 ×106
Figure 9: Goal-reaching rate comparison for TD3-based methods. The solid line denotes the median across ten independent random seeds, while the shaded region denotes the interquartile range. The vertical dashed line marks the Baseline disabled point. SAC
Rolling 75-episode goal-reaching rate (%) Median with interquantile range
Rolling 20-Episode Return
TD3
17
Residual SAC
Ours (SAC backbone)
Contaminated-Zone AUV
Baseline disabled
Treasure-Collecting Robot
100 80 60 40 20 0
1
2 Step
3 ×106
1
2 Step
3 ×106
Figure 10: Episode return comparison for SAC-based methods. Results are averaged over ten independent random seeds for each algorithm-environment pair, with standard deviation bands. The vertical dashed line marks the Baseline disabled point.
Figure 11: Goal-reaching rate comparison for SAC-based methods. The solid line denotes the median across ten independent random seeds, while the shaded region denotes the interquartile range. The vertical dashed line marks the Baseline disabled point.
Table 2: Final-stage metrics in the Contaminated-Zone AUV Navigation environment. TD3- and SAC-based methods are reported in the same table. Results correspond to episodes occurring between 2.7M and 3.0M environment steps and are aggregated over ten independent random seeds. Goal reached (%) denotes the percentage of episodes in which the agent successfully reaches the goal set G within the episode horizon. Avoidance score is defined as the maximum penetration depth into contaminated region C during an episode, maxt∈episode d((xt , yt ), C̄), where d(·, ·) is the Euclidean distance and C̄ is the closed complement of C. Lower avoidance score indicates safer behavior (less intrusion into C), while higher goal-reaching rate indicates better task success. All values are shown as mean ± standard deviation.
Table 3: Final-stage metrics in the Treasure-Collecting Robot environment. TD3- and SAC-based methods are reported in the same table. Results correspond to episodes occurring between 2.7M and 3.0M environment steps and are aggregated over ten independent random seeds. Goal reached (%) denotes the percentage of episodes in which the robot reaches the goal set G within the episode horizon. Treasure collected (%) denotes the percentage of episodes in which the robot successfully collects the required treasure at least once during the episode. Since the task requires achieving both sub-goals, the Combined metric is defined as 1 2 (Goal reached + Treasure collected). Higher values indicate better performance. All values are shown as mean ± standard deviation.
Method
Goal reached (%) Avoidance score
Method
Goal reached (%)
Treasure collected (%)
Combined
Ours (TD3 backbone) Residual TD3 TD3
99.15 ± 0.58 95.60 ± 3.75 76.35 ± 25.47
0.0100 ± 0.0031 0.0207 ± 0.0132 0.0150 ± 0.0104
Ours (TD3 backbone) Residual TD3 TD3
94.11 ± 3.85 76.15 ± 6.01 31.69 ± 13.27
99.78 ± 0.10 96.95 ± 1.92 99.27 ± 0.76 87.71 ± 2.92 99.89 ± 0.14 65.79 ± 6.67
Ours (SAC backbone) Residual SAC SAC
99.45 ± 0.50 99.10 ± 1.07 90.25 ± 29.43
0.0052 ± 0.0031 0.0115 ± 0.0076 0.0065 ± 0.0048
Ours (SAC backbone) 85.05 ± 11.01 Residual SAC 66.62 ± 18.70 SAC 11.04 ± 10.36
99.56 ± 0.35 92.31 ± 5.53 98.86 ± 1.09 82.74 ± 9.48 99.78 ± 0.30 55.41 ± 5.22
100 ± 0
0.54 ± 0.18
Baseline policy
Baseline policy
100 ± 0
17.30 ± 37.84 58.65 ± 18.92
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
18
ing the treasure, respectively), outperforming the correspond- In addition to the return curves, the analysis also examines how often the algorithm falls back to the baseline policy. This effect ing vanilla and residual backbone baselines. is particularly pronounced in the Treasure-Collecting Robot enCumulative reward and training. The learning curves in vironment: when tracking is disabled, the number of baseline Figures 8 and 10 show that the proposed approach achieves ei- policy calls increases substantially (see Figure 13). This behavther superior returns or, in the worst case, comparable returns. ior is consistent with the intended role of tracking: by recognizing confident improvements via the benchmark, the method Conclusion. Overall, the proposed approach effectively im- more readily commits control to the learning policy; without proves the RL training procedure by producing a standalone this mechanism, the acceptance decisions become more conneural policy that outperforms both from-scratch methods and servative on average, triggering baseline interventions more fredomain-specific residual methods that also embed a baseline quently and slowing down effective transfer of agency.
Ablation and sensitivity studies
6.1
Effect of ν: critic-value tracking
The intuition behind tracking the best critic value Q†t is discussed in Section 3.5; this section empirically evaluates its practical contribution. To this end, the tracking mechanism is ablated by disabling the deterministic acceptance condition based on the improvement margin. Concretely, the default setting ν = 0.01 is replaced by ν = ∞, which makes the condition Q
wt
θ (S t , Aπt ) ≥ Q†t + ν
Ours (TD3 backbone) (default, ν = 0.01) Baseline disabled
0
Contaminated-Zone AUV
50
Treasure-Collecting Robot
25 −1000
0
−2000
−25
−3000
1
2 Step
3
−50
1
×106
2 Step
3 ×106
Figure 12: Comparison of training performance with criticvalue tracking enabled (ν = 0.01) versus disabled (ablation with ν = ∞). The proposed method is evaluated on top of a TD3 backbone. The vertical dashed line marks the Baseline disabled point in Algorithm 2. Throughout training, tracking is consistently beneficial, achieving superior or comparable returns relative to the ablated variant. Ours (TD3 backbone) (ν = ∞)
Ours (TD3 backbone) (default, ν = 0.01) Baseline disabled
Fraction of learning policy calls to total episode steps (in %)
6
Ours (TD3 backbone) (ν = ∞)
Rolling 40-Episode Return
policy into the RL training process. Moreover, the empirical results show that, when the baseline policy is capable of solving the task, the proposed method can reach the goal from the very beginning of training, even when the underlying learning policy is still completely undertrained. This yields stronger goalreaching capability than the other evaluated approaches. The vanilla TD3 and SAC baselines are trained without access to the baseline policy, providing reference points that make the cost of learning from scratch explicit under the same training budget. Their underperformance relative to the residual variants and to the variants equipped with the proposed approach is therefore expected, as the latter incorporate prior knowledge from the baseline policy into the learning process. As noted in the introduction, tuning RL methods can be difficult: achieving strong performance often requires careful reward shaping and extensive hyperparameter tuning. The experiments suggest that when a working baseline policy is already available, it can be leveraged to substantially simplify this process and yield a policy that not only solves the task but also does so more efficiently than a policy trained from scratch.
Treasure-Collecting Robot 100 80 60 40 20
never satisfied. As a result, the algorithm never performs “cer0.5 1.0 1.5 2.0 2.5 3.0 tain” acceptance based on outrunning the episode-local bench×106 Step mark and instead always operates in the probabilistic accepFigure 13: Fraction of learning policy calls during training with tance regime. critic-value tracking enabled (ν = 0.01) versus disabled (ablaThe learning curves for this ablation are shown in Fig- tion with ν = ∞). The vertical dashed line marks the Baseline ure 12. Across all considered environments, enabling tracking disabled point in Algorithm 2. Critic-value tracking substanyields consistently better or, at worst, comparable performance tially increases the learning-policy call fraction, especially in throughout training. In contrast, removing tracking leads to a the Treasure-Collecting Robot environment, indicating a more systematic degradation of returns and typically increases vari- confident and efficient transfer of control. ability, indicating that the benchmark-based test provides a stabilizing effect in practice. While the method remains operational without tracking (i.e., it does not collapse), its sample- 6.2 Effect of T tran : baseline-removal time efficiency and final performance are noticeably reduced, which supports the conclusion that critic-value tracking is an essential The next experiment evaluates how the time at which the basecomponent of the proposed approach. line policy is fully removed affects performance. The baseline-
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique removal time T tran exposes a trade-off in agency transfer. Early removal can make the transition too abrupt: the learned policy may not yet have accumulated sufficient closed-loop experience to sustain the task without baseline support, leading to degraded goal-reaching quality, as shown in Figure 14. Later baseline removal provides a longer protected learning phase, improves the resulting outcome, and makes the agency transfer smoother in the final results. Ttran = 0.45M Ttran = 1.20M
Ttran = 1.95M
Ours (TD3 backbone) (default, p0 = 0.8, λ0 = 0.995) Baseline disabled
Ours (TD3 backbone)
Rolling 40-Episode Return
0 −1000 −2000 −3000 −4000
1
2
3 ×106
Step
100 80 60 40 20 0
1
2 Step
3 ×106
Figure 14: Rolling goal-reaching rate for the baseline-removaltime ablation of the proposed approach on top of a TD3 backbone in the Contaminated-Zone AUV Navigation environment. The plotted metric is the goal-reaching rate, reported as a rolling 75-episode median across five independent random seeds, with interquartile bands. Dashed vertical lines mark the corresponding Baseline disabled times T tran . All runs use the default configuration of the proposed approach on top of a TD3 backbone, with only T tran varied.
Figure 15: Return sensitivity of the proposed method on top of a TD3 backbone in the Contaminated-Zone AUV Navigation environment under a relaxation schedule initialized too close to one, p0 = 1.0, λ0 = 0.9995, versus the default schedule. For the default TD3-backbone curve, T tran = 2.7M, p0 = 0.8, and λ0 = 0.995. Curves show rolling 40-episode returns averaged over five independent random seeds, with standard deviation bands. The vertical dashed line marks the default Baseline disabled point. Ours (SAC backbone) (default, p0 = 0.9, λ0 = 0.96) Ours (SAC backbone) (p0 = 0.8, λ0 = 0.995) Baseline disabled Rolling 75-episode rate (%)
Rolling 75-episode goal-reaching rate (%)
Ours (TD3 backbone) (p0 = 1.0, λ0 = 0.9995)
default (Ttran = 2.70M)
Baseline-removal time
6.3
19
Effect of p0 and λ0 : relaxation schedule
Ours (SAC backbone) 100 75 50 25 0
0.5
1.0
1.5 Step
2.0
2.5
Finally, the sensitivity of the method to the within-episode relaxation schedule, controlled by the initial probability p0 and the initial decay factor λ0 , is evaluated. Larger values make the algorithm rely more strongly on the learning policy from the beginning of training, whereas smaller values keep the system closer to the baseline policy for longer.
3.0 ×106
Figure 16: Rolling goal-reaching sensitivity of the proposed approach on top of a SAC backbone in the Treasure-Collecting Robot environment under the default setting versus the more conservative p0 = 0.8, λ0 = 0.995 setting. Curves show rolling 75-episode medians across ten independent random seeds, with interquartile bands. The vertical dashed line marks the default If the initial trust in the learning policy is too high, the sched- Baseline disabled point. ule continues to amplify this trust during training, leaving less room for baseline support in the early learning stages. This can lead to worse final performance. This effect is demonstrated by comparing the proposed method on top of a TD3 backbone un- A Proofs der the aggressive relaxation schedule p0 = 1.0, λ0 = 0.9995 against the default schedule (p0 = 0.8, λ0 = 0.995) in the A.1 Proof of Theorem 1 Contaminated-Zone AUV Navigation environment. The return curves in Figure 15 show that starting with excessively large p0 Proof. In the present proof, an episode is considered for which and λ0 worsens performance. λ < 1 at its beginning. Since the analysis is restricted to an A similar degradation can be observed in the TreasureCollecting Robot environment in terms of the goal-reaching rate (Figure 16). There, the default schedule (p0 = 0.9, λ0 = 0.96) is compared against the alternative schedule (p0 = 0.8, λ0 = 0.995), which assigns substantially larger cumulative trust to the learning policy. Indeed, the infinite-horizon schedule mass is p0 /(1 − λ0 ) = 22.5 for the default setting, but p0 /(1 − λ0 ) = 160 for the alternative setting, i.e., more than seven times larger. As a practical rule of thumb, following Section 3.6, p0 is iniP −1 t tialized in [0.8, 1.0], and λ0 is chosen so that pT0 Tt=0 λ0 ≈ 0.2.
intra-episode setting, the beginning time τ = 0 may be set without loss of generality. By construction of Algorithm 2, the parameter λ remains constant throughout the episode.
The critical idea is to prove that the number of times P the learning policy is triggered is bounded above, i. e., ∞ t=0 I{At = πθ πθ wt πθ At } < ∞. Moreover, At = At if Q (S t , At ) ≥ Q†t + ν or P πθ Ut ≤ prel λt . Thus, ∞ t=0 I{At = At } is bounded above by ∞ X t=0
θ
I{Qwt (S t , Aπt ) ≥ Q†t + ν} +
∞ X t=0
I{Ut ≤ prel λt }
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
20
Although Q†0 = −∞ by initialization, the subsequent value Q†1 that f (x) − f inf > 0 for all x, i.e., f (x) > f inf . Moreover, is finite since the condition in line 15 is satisfied at t = 0. Since − log( f (x) − f inf ) → +∞ implies f (x) − f inf → 0+ as ∥x∥ → ∞, the sequence {Q†t }t≥1 is non-decreasing and can only increase by hence f (x) → f inf . This is precisely statement (ii). P wt πθ increments of at least ν, it follows that: ∞ t=0 I{Q (S t , At ) ≥ Q̄−Q†
Q†t + ν} ≤ ν 1 + 1. P The sum ∞ prel λt } is bounded by the Borel-Cantelli t=0 I{U t ≤ i P∞ h prel lemma since t=0 P Ut ≤ prel λt = 1−λ < ∞. Therefore, the total number of times the learning policy is triggered is bounded above. Consequently, there exists a time t0 such that for all t ≥ t0 , only the baseline policy is executed. The ε-improbable goalreaching property of the resulting algorithm follows from the εimprobable goal-reaching property of the baseline policy.
A.3
Proof of Theorem 2
Proof. The proof begins by introducing the definitions (26)– (31) for the quantities τ(d◦ ), τb (d◦ , d∗ ), and δ(d◦ ). The proof then explains, step by step, how these definitions guarantee Claims (C1 ), (C2 ), and (C3 ), making clear how each object is constructed from scratch. Specifically, the following quantities are defined: vmin (d◦ ) := min{κlow (s) : dG (s) ≤ d◦ }
A.2
Proof of Proposition 1
V(d ) := {s ∈ S : κ ◦
Proof. Let f inf := inf x∈Rn f (x) ∈ [−∞, +∞). The proof shows (i) ⇔ (ii) ⇔ (iii) by implications. (i) ⇒ (ii).
Assume f (x) has bounded superlevel sets.
Case 1: f inf = −∞. Fix any M > 0. Since f inf = −∞, there exists a ∈ f (Rn ) with a ≤ −M. Boundedness of the a-superlevel set gives R > 0 such that ∥x∥ > R ⇒ f (x) < a ≤ −M. Hence lim∥x∥→∞ f (x) = −∞ = f is automatic when f inf = −∞.
inf
. The inequality f (x) > f
inf
Case 2: f inf > −∞. If f inf ∈ f (Rn ), the f inf -superlevel set would equal Rn , contradicting boundedness. Thus f (x) > f inf for all x,
v
(26)
◦
(d )}
(27)
◦
(d ) := max{κ (s) : s ∈ V(d )} $ max ◦ % v (d ) − vmin (d◦ ) τ(d◦ ) := 1 + ν
(28)
max
◦
low
min
(s) ≥ v
high
dp̄ (d◦ ) := sup{ p̄(s, a) : s ∈ V(d◦ ), a ∈ A} dmax (d◦ ) := max d◦ , dp̄ (d◦ ) δ(d ) := β(d ◦
max
◦
(d ), 0).
(29) (30) (31) (32)
In the equations above, vmin (d◦ ) and vmax (d◦ ) bound critic values, the superlevel set V(d◦ ) defines the operational domain of the baseline policy, and τ(d◦ ) bounds the number of criticimprovement iterations, while dp̄ (d◦ ) bounds the maximum one-step transition from any state within the superlevel set V(d◦ ), and dmax (d◦ ) provides a composite bound incorporating both initial distance and transition magnitudes.
Let a ∈ f (Rn ) be arbitrary with f inf < a. Because f has Further, τb (d◦ , d∗ ) is defined as the minimum time steps for bounded super-level sets, there is an R > 0 such that f (x) ≥ the baseline policy to drive any state with dG (s) ≤ dmax (d◦ ) a =⇒ ∥x∥ ≤ R, or equivalently, ∥x∥ > R =⇒ f (x) < a. to within d∗ of the goal: n l mo Because f inf = inf f (Rn ) and f (x) > f inf for every x ∈ Rn , one d∗ τb (d◦ , d∗ ) := max 1, − log ξ−1 κ(dmax , (33) can pick a arbitrarily close to f inf . (d◦ )) (ii) ⇒ (i). Assume lim∥x∥→∞ f (x) = f inf := inf f (Rn ) and f (x) > f inf for all x.
where κ, ξ ∈ K∞ are functions such that β(d, t) ≤ κ(d)ξ(e−t ) for all d ≥ 0, t ≥ 0 (a standard decomposition of KL functions, see [34, Lemma 8]).
Let a ∈ f (Rn ). Since f (Rn ) ⊂ ( f inf , +∞), it follows that a > A series of results is now established that together proves the f inf . By the limit assumption, there exists R > 0 such that theorem’s claims: ∥x∥ > R ⇒ f (x) < a. Thus {x : f (x) ≥ a} ⊆ {x : ∥x∥ ≤ R}, θ Lemma 1. Whenever At = Aπt , it holds that S t ∈ V(d◦ ). which is bounded. Hence f has bounded superlevel sets. (ii) ⇒ (iii).. If f inf = −∞, then lim∥x∥→∞ f (x) = −∞, so − f (x) → +∞ and − f is radially unbounded. If f ∈ R, then f (x) > f for all x and f (x) → f ∥x∥ → ∞. Hence f (x) − f inf → 0+ and so − log f (x) − f inf −→ +∞ as ∥x∥ → ∞, inf
inf
i.e., − log( f (x) − f inf ) is radially unbounded.
inf
Proof. The πθ is chosen in two cases: θ
(1) When Q†t + ν < Qwt (S t , Aπt ): At initialization, Q†0 is inias tialized to −∞. On the first iteration, it updates to Q† = 1 θ Qw0 (S 0 , Aπ0 ). Moreover, Q†t is nondecreasing by construction. Hence, for t ≥ 1, θ
Q†t ≥ Qw0 (S 0 , Aπ0 ) ≥ κlow (S 0 ) ≥ vmin (d◦ ),
and therefore S t ∈ V(d◦ ) for all t ≥ 1. By Assumption (A1 ), the (iii) ⇒ (ii).. If − f (x) is radially unbounded, then f (x) → −∞ θ critic satisfies κlow (S 0 ) ≤ Qw0 (S 0 , Aπ0 ), and since dG (S 0 ) ≤ d◦ , as ∥x∥ → ∞, so f inf = −∞ and f (x) > f inf holds trivially. it follows that κlow (S 0 ) ≥ vmin (d◦ ), implying S 0 ∈ V(d◦ ). Otherwise, assume there exists a finite f inf = inf f (Rn ) ∈ t rel rel R such that − log( f (x) − f inf ) is radially unbounded. For (2) When θ Ut ≤ λ ρt : θBy definition, ρt = 0 whenever inf wt π w0 π − log( f (x) − f ) to be well-defined everywhere, it must hold Q (S t , At ) < Q (S 0 , A0 ), which can occur only for t ≥ 1.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
21
Q P∞ k rel In such a case, the learning-policy branch is not executed, since Finally, ∞ k=t (1 − λ p ) → 1 as t → ∞ because k=t log(1 − P θ † ∞ k rel t the condition Qt + ν < Qwt (S t , Aπt ) cannot hold, because for all λ p ) → 0 as t → ∞, which follows from t=0 λ prel < ∞. t ≥ 1, θ θ Q†t ≥ Qw0 (S 0 , Aπ0 ) ≥ vmin (d◦ ). Lemma 4. If S t ∈ V(d◦ ) and At = Aπt , then: rel wt πθ w0 πθ When ρt > 0, it follows that Q (S t , At ) ≥ Q (S 0 , A0 ) ≥ (i) The next state satisfies ∥S ∥ ≤ dp̄ (d◦ ) almost surely. t+1 vmin (d◦ ), and therefore S t ∈ V(d◦ ). (ii) Both s0 and all states with ∥s∥ ≤ dp̄ (d◦ ) satisfy dG (s) ≤ dmax (d◦ ). Lemma 2. The quantity τ(d◦ ) in (29) bounds the number of times the critic value can significantly improve: Proof. (i) By the definition of p̄ (from the system assumption) θ ∞ X and dp̄ (d◦ ) in (30), when S t ∈ V(d◦ ) and At = Aπt , it holds that θ † wt π ◦ θ Nv† := I{Qt + ν < Q (S t , At )} ≤ τ(d ) ∥S t+1 ∥ ≤ p̄(S t , Aπt ) ≤ dp̄ (d◦ ) almost surely. t=0
(ii) By definition, dG (s0 ) ≤ d◦ ≤ dmax (d◦ ). For any state with p̄ ◦ p̄ ◦ max ◦ Proof. The quantity τ(d◦ ) is well-defined because vmin (d◦ ) and ∥s∥ ≤ d (d ), it follows that dG (s) ≤ ∥s∥ ≤ d (d ) ≤ d (d ). max ◦ v (d ) are finite. This follows from the compactness of the relevant superlevel set V(d◦ ) and the continuity of κlow (s) and κhigh (s) (Assumption (A1 )). Lemma 5. For any state s0 with dG (s0 ) ≤ dmax (d◦ ): h i b At t = 0, Q†0 is initialized to −∞, so the update condition Q†t + (i) P dG (S tπ (s0 )) ≤ d∗ for all t ≥ τb (d◦ , d∗ ) ≥ 1 − ε. θ h i ν < Qwt (S t , Aπt ) is satisfied trivially, resulting in an unbounded πb ◦ θ initial jump to Q†1 = Qw0 (S 0 , Aπ0 ). For all subsequent iterations (ii) P dG (S t (s0 )) ≤ δ(d ) for all t ≥ 0 ≥ 1 − ε. (t ≥ 1), each occurrence of this condition increases Q†t by a finite jump of at least ν. Proof. By Assumption (A2 ), h i By Lemma 1, such updates can occur only when S t ∈ V(d◦ ), b θ θ P dG (S tπ (s0 )) ≤ β(dG (s0 ), t) for all t ≥ 1 − ε where Qwt (S t , Aπt ) ≤ vmax (d◦ ). Since Q†1 = Qw0 (S 0 , Aπ0 ) ≥ vmin (d◦ ), the number of finite jumps that can occur is bounded by ⌊(vmax (d◦ ) − vmin (d◦ ))/ν⌋. Including the initial unbounded (i) From the definition of τb (d◦ , d∗ ) in (33), it follows that update, the total number of possible improvements is therefore β(dG (s0 ), t) ≤ d∗ for all t ≥ τb (d◦ , d∗ ) when dG (s0 ) ≤ dmax (d◦ ). $ max ◦ % v (d ) − vmin (d◦ ) (ii) For any t ≥ 0, it holds that β(dG (s0 ), t) ≤ β(dmax (d◦ ), 0) = = τ(d◦ ), 1+ δ(d◦ ) when dG (s0 ) ≤ dmax (d◦ ). ν which completes the proof. Conclusion of the proof of Theorem 2. From Lemmas 2 and 3, the total number of times the baseline policy is chosen is at most rel (i) the total number of random acceptances: τ(d◦ ) + T rel almost surely. P∞ T bounds t rel rel t=0 I{U t < λ p } ≤ T . From Lemmas 1 and 4, whenever the algorithm switches to the rel baseline policy, the state satisfies dG (s) ≤ dmax (d◦ ). (ii) T is almost surely finite. h i Q Q∞ k rel From Lemma 5, after running the baseline policy for τb (d◦ , d∗ ) (iii) For all t ∈ Z≥0 , P T rel ≤ t = ∞ k=t (1−λ p ), and k=t (1− steps from any such state, the system stays within d∗ of the goal λk prel ) → 1 as t → ∞. thereafter with probability at least 1 − ε. Lemma 3. Let T rel := inf{t ≥ 0 : Uk ≥ λk prel ∀k ≥ t}.
rel Proof. (i) Since ρrel t ≤ p for all t: ∞ X t=0
I{Ut < λt ρrel t }≤
∞ X t=0
I{Ut < λt prel } =: N prel
Therefore, define the reaching time as T (d◦ , d∗ ) := (τ(d◦ ) + T rel )τb (d◦ , d∗ )
(34)
All three claims now follow:
PT rel −1 Claim (C1 ) follows from Lemma 5(ii), showing that the maxiFurthermore, N prel ≤ t=0 1 = T rel , since by definition of T rel , mum deviation from the goal is bounded by δ(d◦ ). the event Ut < λt prel cannot occur for t ≥ T rel . Claim (C2 ) follows as the baseline policy is used at most τ(d◦ )+ (ii) The Borel-Cantelli lemma [35] ensures that N prel is almost T rel times, and τb (d◦ , d∗ ) baseline steps suffice to maintain the P t rel surely finite because ∞ < ∞, which implies that T rel is system within d∗ of the goal with probability 1 − ε, whether t=0 λ p almost surely finite as well. starting from the initial state or after baseline policy use. rel t rel (iii) The event {T ≤ t} occurs if and only if Uk ≥ λ p for all Claim (C3 ) follows directly from Lemma 3(iii) and the definik ≥ t. Since the Uk are independent, it follows that: tion of T (d◦ , d∗ ) in (34). h i Q h i Q t rel k rel P T rel ≤ t = ∞ = ∞ The proof of Theorem 2 is complete. k=t P U k ≥ λ p k=t (1 − λ p )
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique A.4
Proof of Corollary 1
22
Georgiy Malaniya: Methodology, Formal analysis, Software, Validation, Writing – original draft.
Proof. Take the proof of Theorem 2 and perform the following Sinan Ibrahim: Software, Visualization, Data curation. replacements everywhere: Qw (s, a) → vw (s), Q†t → v†t , and rel rel wt w0 ρt → p I{v (S t ) ≥ v (S 0 )}, while relabeling Assumptions Pavel Osinenko: Conceptualization, Methodology, Supervi(A1 ) and (A2 ) to (Av1 ) and (Av2 ); with these substitutions, every sion, Writing – review and editing. definition, lemma, and bound is unchanged, and the corollary follows.
Declaration of competing interest
A.5
Proof of Theorem 3 πt πθ and S 0:T The state processes S 0:T are sampled indepen-
Proof. dently, as in Definition 10.
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Step 1 (geometric argument). Define the tube event and its comAcknowledgements plement: ( ) ( ) θ θ Research reported in this publication was financially supported Eδ := sup ∥S tπ − S tπt ∥ ≤ δ , Eδc := sup ∥S tπ − S tπt ∥ > δ . by the Russian Science Foundation (RSF) grant No. 25-210≤t<T 0≤t<T 00872. n o Claim: {τT (d∗ , πt ) < T } ∩ Eδ ⊆ τT (d∗ + δ, πθ ) < T . Indeed, suppose τT (d∗ , πt ) = t∗ < T . Then for every t ∈ Data availability {t∗ , . . . , T −1}, dG S tπt ≤ d∗ . On Eδ , for every such t: θ All simulation code, source code required to reproduce the ex θ dG S tπ ≤ ∥S tπ − S tπt ∥ + dG S tπt ≤ δ + d∗ , perimental runs, the resulting run data, and scripts used to generate the figures are available in the project repository: https: so τT (d∗ + δ, πθ ) ≤ t∗ < T . //github.com/aidagroup/calf-enhance. Step 2 (probability estimate). Using the inclusion from Step 1: h i References P τT (d∗ + δ, πθ ) < T ≥ P {τT (d∗ , πt ) < T } ∩ Eδ . [1] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, Decomposing by Eδ : M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, ∗ T. Graepel, T. Lillicrap, K. Simonyan, D. Hassabis, A P {τT (d , πt ) < T } ∩ Eδ = general reinforcement learning algorithm that masters i h P τT (d∗ , πt ) < T − P {τT (d∗ , πt ) < T } ∩ Eδc chess, shogi, and go through self-play, Science 362 (6419) h i (2018) 1140–1144. ≥ P τT (d∗ , πt ) < T − P Eδc . (35) [2] OpenAI, :, C. Berner, G. Brockman, B. Chan, V. Cheung, P. Dzbiak, C. Dennison, D. Farhi, Q. Fischer, S. Hashme, Step 3 (Markov bound on tube exit). By Markov’s inequality: C. Hesse, R. Jozefowicz, S. Gray, C. Olsson, J. Pachocki, " # M. Petrov, H. P. d. O. Pinto, J. Raiman, T. Salimans, h i θ J. Schlatter, J. Schneider, S. Sidor, I. Sutskever, J. Tang, P Eδc = P sup ∥S tπ − S tπt ∥ > δ 0≤t<T F. Wolski, S. Zhang, Dota 2 with large scale deep reinh i θ forcement learning (2019). arXiv:1912.06680. E sup0≤t<T ∥S tπ − S tπt ∥ DT (πθ , πt ) ∆T URL https://arxiv.org/abs/1912.06680 ≤ = ≤ . (36) δ δ δ [3] O. Vinyals, et al., Grandmaster level in StarCraft II using multi-agent reinforcement learning, Nature 575 (7782) Step 4 (combine). Substituting Assumption (A1 ) and (36) (2019) 350–354. into (35): [4] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, h i ∆T B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, P τT (d∗ + δ, πθ ) < T ≥ P τT (d∗ , πt ) < T − δ R. Ribas, et al., Solving rubik’s cube with a robot hand, ∆T arXiv preprint arXiv:1910.07113 (2019). ≥ (1 − ε) − . (37) δ [5] H. Surmann, C. Jestel, R. Marchel, F. Musberg, H. Elhadj, M. Ardani, Deep reinforcement learning for real auCRediT authorship contribution statement tonomous mobile robot navigation in indoor environments (2020). arXiv:2005.13857. Anton Bolychev and Georgiy Malaniya contributed equally to [6] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, F. Janoos, this work. L. Rudolph, A. Madry, Implementation matters in deep rl: A case study on ppo and trpo, in: International Anton Bolychev: Formal analysis, Investigation, Software, ValConference on Learning Representations, 2020. idation, Visualization, Writing – original draft.
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique
23
URL https://openreview.net/forum?id= [19] T. Hester, M. Vecerı́k, O. Pietquin, M. Lanctot, T. Schaul, r1etN1rtPB B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Arnold, J. Agapiou, J. Z. Leibo, A. Gruslys, [7] A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, Deep q-learning from demonstrations, in: Proceedings N. Dormann, Stable-baselines3: Reliable reinforcement of the 32nd AAAI Conference on Artificial Intelligence, learning implementations, Journal of Machine Learning 2018, pp. 3223–3230. Research 22 (268) (2021) 1–8. [8] S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, [20] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, S. Levine, Learning complex dexK. Mehta, J. G. Araújo, Cleanrl: High-quality singleterous manipulation with deep reinforcement learning file implementations of deep reinforcement learning algoand demonstrations, in: Robotics: Science and Systems rithms, Journal of Machine Learning Research 23 (274) (RSS), 2018. (2022) 1–18. [21] J. Garcı́a, F. Fernández, A comprehensive survey on safe [9] T. Eimer, M. Lindauer, R. Raileanu, Hyperparameters in reinforcement learning, Journal of Machine Learning reinforcement learning and how to tune them, in: ProResearch 16 (42) (2015) 1437–1480. ceedings of the 40th International Conference on Machine URL https://jmlr.org/papers/v16/garcia15a. Learning (ICML), Vol. 202, PMLR, 2023, pp. 14811– html 14835. [10] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft actor- [22] J. Achiam, D. Held, A. Tamar, P. Abbeel, Constrained policy optimization, in: Proceedings of the 34th International critic: Off-policy maximum entropy deep reinforcement Conference on Machine Learning (ICML), 2017, pp. 22– learning with a stochastic actor, in: Proceedings of 31. the 35th International Conference on Machine Learning (ICML), Vol. 80 of Proceedings of Machine Learning Re- [23] Y. Chow, O. Nachum, E. Duenez-Guzman, search, 2018, pp. 1861–1870. M. Ghavamzadeh, A lyapunov-based approach to safe reinforcement learning, in: Advances in [11] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, Neural Information Processing Systems (NeurIPS), O. Klimov, Proximal policy optimization algorithms., Vol. 31, 2018, pp. 8092–8101. arXiv:1805.07708, CoRR abs/1707.06347 (2017). doi:10.48550/arXiv.1805.07708. URL http://dblp.uni-trier.de/db/journals/ URL https://papers.nips.cc/ corr/corr1707.html#SchulmanWDRK17 paper_files/paper/2018/hash/ [12] S. Fujimoto, H. van Hoof, D. Meger, Addressing func4fe5149039b52765bde64beb9f674940-Abstract. tion approximation error in actor-critic methods, in: Prohtml ceedings of the 35th International Conference on Machine [24] M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, Learning (ICML), Vol. 80 of Proceedings of Machine S. Niekum, U. Topcu, Safe reinforcement learning via Learning Research, 2018, pp. 1587–1596. shielding, in: Proceedings of the 32nd AAAI Conference [13] T. Johannink, S. Bahl, A. Nair, J. Luo, A. Kumar, on Artificial Intelligence (AAAI), 2018, pp. 2669–2678. M. Loskyll, J. A. Ojea, E. Solowjow, S. Levine, Residual [25] A. D. Ames, X. Xu, J. W. Grizzle, P. Tabuada, Control reinforcement learning for robot control (2018). arXiv: barrier function based quadratic programs for safety 1812.03201. critical systems, IEEE Transactions on Automatic ConURL https://arxiv.org/abs/1812.03201 trol 62 (8) (2017) 3861–3876. arXiv:1609.06408, [14] T. Silver, K. Allen, J. Tenenbaum, L. Kaelbling, Residual doi:10.1109/TAC.2016.2638961. policy learning (2019). arXiv:1812.06298. URL https://doi.org/10.1109/TAC.2016. URL https://arxiv.org/abs/1812.06298 2638961 [15] M. Alakuijala, G. Dulac-Arnold, J. Mairal, J. Ponce, [26] D. Liberzon, Switching in Systems and Control, SysC. Schmid, Residual reinforcement learning from demontems & Control: Foundations & Applications, Birkhäuser strations (2021). arXiv:2106.08050. Boston, 2003. doi:10.1007/978-1-4612-0017-8. URL https://arxiv.org/abs/2106.08050 URL https://link.springer.com/book/10.1007/ 978-1-4612-0017-8 [16] Z. Sheng, Z. Huang, S. Chen, Traffic expertise meets residual rl: Knowledge-informed model-based residual [27] M. S. Branicky, Multiple lyapunov functions and other reinforcement learning for cav trajectory control, Comanalysis tools for switched and hybrid systems, IEEE munications in Transportation Research 4 (2024) 100142. Transactions on Automatic Control 43 (4) (1998) 475– doi:10.1016/j.commtr.2024.100142. 482. doi:10.1109/9.664150. URL https://doi.org/10.1109/9.664150 [17] B. Baker, I. Akkaya, P. Zhokov, J. Huizinga, J. Tang, A. Ecoffet, B. Houghton, R. Sampedro, J. Clune, Video [28] H. Bharadhwaj, A. Kumar, N. Rhinehart, S. Levine, pretraining (VPT): Learning to act by watching unlabeled F. Shkurti, A. Garg, Conservative safety critics for online videos, in: Advances in Neural Information Proexploration, in: International Conference on Learning cessing Systems, Vol. 35, 2022. Representations (ICLR), 2021. arXiv:2010.14497, [18] J. Ho, S. Ermon, Generative adversarial imitation learning, in: Advances in Neural Information Processing Systems 29 (NeurIPS), 2016, pp. 4565–4573.
doi:10.48550/arXiv.2010.14497. URL https://openreview.net/forum?id= iaO86DUuKi
Preprint – An Agency-Transferring Model-Free Policy Enhancement Technique [29] G. Dalal, K. Dvijotham, M. Vecerik, T. Hester, C. Paduraru, Y. Tassa, Safe exploration in continuous action spaces, CoRR abs/1801.08757 (2018). doi:10.48550/ arXiv.1801.08757. URL https://arxiv.org/abs/1801.08757 [30] A. S. Morse, Supervisory control of families of linear setpoint controllers—part i: Exact matching, IEEE Transactions on Automatic Control 41 (10) (1996) 1413–1431. doi:10.1109/9.539424. URL https://doi.org/10.1109/9.539424 [31] A. S. Morse, Supervisory control of families of linear setpoint controllers—part ii: Robustness, IEEE Transactions on Automatic Control 42 (11) (1997) 1500–1515. doi: 10.1109/9.649687. URL https://doi.org/10.1109/9.649687 [32] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, D. Wierstra, Continuous control with deep reinforcement learning, in: Proceedings of the 4th International Conference on Learning Representations (ICLR), 2016, arXiv:1509.02971. [33] H. K. Khalil, Nonlinear Systems, 3rd Edition, Prentice Hall, 2002. [34] E. D. Sontag, Comments on integral variants of ISS, Systems and Control Letters 34 (1-2) (1998) 93–100. doi: 10.1016/S0167-6911(98)00007-1. [35] P. Billingsley, Probability and Measure, 3rd Edition, John Wiley & Sons, 1995.
24