When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
Shuning Shang∗, Hubert Strauss∗, Stanley Wei, Sanjeev Arora, Noam Razin
arXiv:2604.25872v1 [cs.LG] 28 Apr 2026
Princeton Language and Intelligence, Princeton University
Abstract Training language models via reinforcement learning often relies on imperfect proxy rewards, since ground truth rewards that precisely define the intended behavior are rarely available. Standard metrics for assessing the quality of proxy rewards, such as ranking accuracy, treat incorrect rewards as strictly harmful. In this work, however, we highlight that not all deviations from the ground truth are equal. By theoretically analyzing which outputs attract probability during policy gradient optimization, we categorize reward errors according to their effect on the increase in ground truth reward. The analysis establishes that reward errors, though conventionally viewed as harmful, can also be benign or even beneficial by preventing the policy from stalling around outputs with mediocre ground truth reward. We then present two practical implications of our theory. First, for reinforcement learning from human feedback (RLHF), we develop reward model evaluation metrics that account for the harmfulness of reward errors. Compared to standard ranking accuracy, these metrics typically correlate better with the performance of a language model after RLHF, yet gaps remain in robustly evaluating reward models. Second, we provide insights for reward design in settings with verifiable rewards. A key theme underlying our results is that the effectiveness of a proxy reward function depends heavily on its interaction with the initial policy and learning algorithm.
1
Introduction
Training language models via reinforcement learning commonly relies on imperfect proxy rewards. This reliance is largely unavoidable since specifying ground truth rewards that exactly capture the intended behavior is rarely feasible. For example, in reinforcement learning from human feedback (RLHF) [58], learned reward models serve as proxies for a ground truth reward that is assumed to govern human preferences. Moreover, so-called verifiable rewards [37, 26] are often also imperfect proxies: rewards for mathematical reasoning usually do not verify intermediate steps and can suffer from incorrect parsing [31, 85], and rewards for code generation are based on unit tests, which are inherently incomplete [45, 26, 57]. Despite the widespread use of proxy rewards, there is limited understanding of how discrepancies between the proxy and ground truth rewards affect the reinforcement learning process. Arguably the most well-known potential consequence of such discrepancies is reward hacking, where maximizing the proxy reward results in poor ground truth performance [4, 77, 59, 21, 32, 18, 36]. Reflecting this concern, standard metrics for assessing the quality of a proxy reward function, including ranking accuracy [38] and mean squared error [29], consider deviations from the ground truth reward as strictly harmful. However, this coarse treatment overlooks the possibility that different erroneous proxy rewards may influence the learning process in distinct ways. In this work, we highlight that not all reward errors are equal, or even necessarily harmful. Formally, reward error refers to cases where rP (x, y) ̸= rG (x, y), for a proxy reward function rP , ground truth reward function rG , and one or more input-output pairs (x, y). Focusing on policy gradient—the ∗
Equal contribution.
How does Harmful Error
influence the increase in ground truth reward?
Benign Error
Beneficial Error
I. High proxy reward for an output
I. Incorrect proxy reward of a non-
I. Low proxy reward for an output
with low ground truth reward
optimal output is lower than the initial
with mediocre ground truth reward
expected proxy reward Reason: reward hacking
Reason: y does not attract probability
Reason: prevents policy from concentrating and stalling on y
Conditions Conditions
Conditions
II. Mediocre proxy reward for an output with low ground truth reward Reason: causes the policy
to concentrate and stall on y Conditions
II. Incorrect proxy reward for an output with low probability under the initial policy Reason: negligible effect on outcome
Notation - input-output pair - initial policy - proxy reward
Conditions
- ground truth reward - optimal output
Figure 1: Reward error categorization overview. We categorize reward errors—cases where the proxy and ground truth rewards disagree on one or more input-output pairs—according to their effect on the increase in ground truth reward under policy gradient optimization (Section 3). Aside from being harmful, we prove that reward errors can also be benign or even beneficial. This categorization depends on the interplay between the proxy reward function, initial policy, and policy gradient updates. Sections 4 and 5 explore practical implications of our theory for reward model evaluation in RLHF and reward design in settings with verifiable rewards.
predominant approach for training language models via reinforcement learning—we theoretically characterize the effect of reward errors on the increase in ground truth reward (Section 3). In particular, we categorize reward errors into harmful, benign, or beneficial. Harmful errors, aside from causing reward hacking, can slow down the rate at which the ground truth reward increases. Benign errors have negligible impact on optimization. Lastly, and perhaps most strikingly, we prove that some errors are beneficial, accelerating the increase in ground truth reward beyond what is achieved when optimizing the ground truth reward directly. See Figure 1 for an overview of our categorization. We derive this categorization by analyzing which outputs attract probability during policy gradient optimization for linear softmax policies. The main theoretical result, which underlies the existence of beneficial reward errors, shows that outputs with mediocre reward can attract probability and stall policy gradient in their vicinity for an arbitrarily long time. Thus, assigning low proxy reward to outputs with mediocre ground truth reward can accelerate the increase in ground truth reward by steering the policy away from such outputs. The analysis is corroborated by controlled experiments. As a practical application of our theory, we consider the problem of reward model evaluation for RLHF (Section 4). Reward models are primarily evaluated through ranking accuracy [38, 98, 46, 20, 47], which treats all incorrect rankings of outputs as equally harmful. Based on our categorization of reward errors, we develop ranking accuracy variants that account for the harmfulness of an incorrect ranking to policy gradient optimization. Experiments show that these harm-aware metrics typically correlate better with the performance of a language model after RLHF, across datasets and model families (Llama [24], OLMo [57], and Qwen [95]). Yet, despite these gains, the correlation can still be weak, showcasing challenges in robustly evaluating reward models. Beyond reward model evaluation, we explore implications of our theory for reward design in settings with verifiable rule-based rewards (Section 5). Namely, we demonstrate that rewarding partially correct outputs (cf. [62, 82]) can be detrimental for optimization if the initial policy is noticeably more likely to produce partially correct outputs than fully correct ones. A central theme of this work is that the effectiveness of proxy rewards cannot be assessed solely by how much they deviate from the ground truth. It is essential to consider the interplay between the proxy reward function, initial policy, and learning algorithm. Our results take a step towards characterizing principles for proxy reward evaluation and design that account for this interplay. We hope that they will inspire further research in this direction, given the growing usage of reinforcement learning for training language models in complex environments. 2
2
Preliminaries
We use πθ to denote a language model, parameterized by θ, that maps an input (i.e., prompt) x ∈ X to a distribution over outputs y ∈ Y, where X and Y are the input and output spaces, respectively. We will also refer to πθ as a policy. 2.1
Reinforcement Learning for Language Models
Reinforcement learning is a key component of language model post-training pipelines [58, 57, 26, 95]. In many cases, it is difficult to specify a ground truth reward function rG : X × Y → [−1, 1] that exactly defines the intended objective. Thus, a proxy reward function rP : X × Y → [−1, 1] is used instead.2 A prominent example is reinforcement learning from human feedback (RLHF) [58], where rP is a learned reward model and rG is a reward function assumed to capture human preferences. Perhaps less obvious examples arise in environments with so-called verifiable rewards, such as mathematical reasoning and code generation. An ideal reward function would distinguish fully correct outputs from partially correct or incorrect ones. Yet, rewards in math environments typically do not verify intermediate steps and can suffer from incorrect parsing [37, 31, 85], and rewards for code generation rely on unit tests, which are fundamentally limited [26, 57]. Given a proxy reward function rP , the language model πθ is usually optimized via policy gradient methods (e.g., PPO [74], RLOO [2], and GRPO [75]). This amounts to maximizing through gradient updates the expected proxy reward Ex∼D,y∼πθ (·|x) [rP (x, y)], where D is a distribution over inputs. Optionally, a KL regularization term may be included to penalize divergence from some reference policy (cf. [99, 81, 58]). The premise is that, if rP is a good proxy for rG , increasing expected proxy reward should also increase the expected ground truth reward Ex∼D,y∼πθ (·|x) [rG (x, y)]. 2.2
Evaluating the Quality of Proxy Reward Functions
Assessing the quality of a proxy reward function rP is a problem of both practical and theoretical interest [23, 93, 78, 32, 38]. This problem is particularly salient in the context of RLHF, where ranking accuracy (Definition 1) is the primary metric used to evaluate reward models [38, 46, 20, 98, 47]. A main reason for its adoption in practice is that it does not require direct access to the ground truth rG , relying instead on preference labels that can be obtained through human annotation. − Definition 1. For a set S containing preference examples (x, y + , y1− , . . . , yK ), where x ∈ X is an − + − + input and y , y1 , . . . , yK ∈ Y are outputs satisfying rG (x, y ) > maxk∈[K] rG (x, yk− ), the ranking accuracy of rP is given by: 1 X Acc(rP ; S) := 1 maxk∈[K] rP (x, yk− ) < rP (x, y + ) , + ,{y − } (x,y )∈S |S| k k∈[K] where 1[·] is an indicator function and [K] := {1, . . . , K}. Ranking accuracy, along with other metrics for evaluating proxy rewards in the literature [23, 93, 78, 29], rests on a common assumption: deviations of rP from rG are strictly harmful. Moreover, it treats all incorrect rankings of outputs as equal. Our aim is to show that this view is too coarse, as it disregards the fact that different reward errors (i.e., cases where the proxy and ground truth rewards disagree on one or more input-output pairs) affect the policy optimization process in distinct ways.
3
Categorization of Reward Errors: Harmful, Benign, and Beneficial
Reward errors are conventionally viewed as harmful due to the risk of reward hacking—if an output with low ground truth reward is assigned a high proxy reward, maximizing the proxy reward may lead to poor ground truth performance [4, 77, 59, 21, 32, 18]. However, we prove that aside from being harmful, reward errors can also be benign or even beneficial. Specifically, by theoretically analyzing which outputs attract probability during policy gradient optimization, we categorize reward errors according to their effect on the increase in ground truth reward; see Figure 1 for an overview. Section 3.1 presents the technical setting, after which Section 3.2 characterizes two types of benign reward errors by showing that outputs with low proxy reward or low probability under the initial 2
As typical in language model applications, we consider rewards defined over complete outputs.
3
policy negligibly affect the outcome of policy gradient. Section 3.3 then delivers our main theoretical result, establishing the existence of beneficial reward errors. It identifies that outputs with mediocre proxy reward can attract probability and trap policy gradient for an arbitrarily long time. Thus, assigning low proxy reward to outputs with mediocre ground truth reward can accelerate the increase in ground truth reward by steering the policy away from such outputs. Lastly, the attraction to mediocre outputs reveals an additional type of harmful reward error (Section 3.4). Namely, suppose that an output with low ground truth reward is incorrectly assigned a higher proxy reward. Even if the proxy reward is not high enough to induce reward hacking, this error may still slow down the increase in ground truth reward. Controlled experiments support our theoretical findings. 3.1
Technical Setting
Policy parameterization. We consider bandit environments with a single input (i.e., context) x ∈ X , finite set of outputs Y, and linear softmax policies [1, 53, 68, 42, 19]. Each output y ∈ Y is associated with a feature vector ϕ(y) ∈ RD \ {0} and the policy πθ is defined by: exp ⟨ϕ(y), θ⟩ , πθ (y) := P z∈Y exp ⟨ϕ(z), θ⟩ where θ ∈ RD . For conciseness, in this section we omit the input x ∈ X from our notation. Despite the apparent simplicity of linear softmax policies, they induce non-concave proxy and ground truth objectives for rP , rG : Y → [−1, 1]: VP (θ) := Ey∼πθ [rP (y)] , VG (θ) := Ey∼πθ [rG (y)] , whose optimization properties are not well understood [51, 1, 40, 53, 42]. Optimization. Language model post-training is often done with small learning rates (cf. [94, 37, 57]). Accordingly, and following prior theoretical analyses of gradient-based optimization with small learning rates (e.g., [72, 92, 65, 87, 66, 67, 71, 35, 86, 12, 79, 8, 63]), we analyze gradient flow: d θt = ∇VP (θt ) , t ≥ 0 , dt
(1)
where θt denotes the parameters at time t of training and πθ0 is the initial policy.3 Limitations. The technical setting involves three main simplifications: (i) it considers linear softmax policies; (ii) it assumes access to exact gradients of the expected proxy reward VP ; and (iii) it focuses on bandit environments with a single input. As Sections 4 and 5 demonstrate empirically, although these simplifications abstract away complexities that may stem from using a neural network architecture, sample-based estimates of ∇VP (θ), or the interaction between inputs, they allow deriving insights that apply to practical settings. 3.2
Benign Errors
3.2.1
Proxy Reward Is Lower Than the Initial Expected Proxy Reward
We begin by making a straightforward, yet important observation: outputs with proxy reward lower than the initial expected proxy reward, VP (θ0 ), generally do not attract probability. This becomes apparent by examining the optimization dynamics for the logit ⟨ϕ(y), θt ⟩ of an output y ∈ Y. Proposition 1 (Proof deferred to Appendix B.2). Suppose that we maximize via gradient flow the expected proxy reward VP (Equation (1)). For any output y ∈ Y and time t ≥ 0 it holds that: X d 2 ⟨ϕ(y), θt ⟩ = πθt (y)AP (y; θt ) · ∥ϕ(y)∥ + πθt (z)AP (z; θt ) · ⟨ϕ(z), ϕ(y)⟩ , z∈Y\{y} | | {z } {z } dt contribution due to y
contribution due to other output
where AP (z; θt ) := rP (z) − VP (θt ) is the advantage of z ∈ Y under rP and πθt . Since VP (θt ) is non-decreasing,4 if rP (y) < VP (θ0 ), then the contribution of y to its own logit dynamics is negative for all t ≥ 0, and is more negative the lower rP (y) is compared to VP (θt ). The 3 4
Our theoretical results can be translated to gradient ascent with a small learning rate via Corollary 4 in [17]. d The fact that VP (θt ) is monotonically non-decreasing follows from dt VP (θt ) = ∥∇VP (θt )∥2 ≥ 0.
4
contribution due to other outputs counteracts this negative push only if the proxy reward function and output features are misaligned, in the sense that there exist outputs with reward higher than VP (θ0 ) whose features have a large inner product with ϕ(y). Thus, as long as the feature geometry reasonably accords with the proxy rewards, an output y with rP (y) < VP (θ0 ) will not attract probability. This implies that the following reward error type is benign. Benign Error I: Incorrect proxy reward is lower than the initial expected proxy reward. If an output y is assigned an incorrect proxy reward rP (y) ̸= rG (y) satisfying rP (y) < VP (θ0 ), then it will generally not attract probability. As a result, unless y achieves maximal ground truth reward, this reward error usually does not harm the increase in ground truth reward. 3.2.2
Incorrect Proxy Reward for Improbable Outputs
The logit dynamics in Proposition 1 further reveals that the contribution of an output y is dampened by its probability πθt (y). This suggests that the optimization trajectory is not significantly affected by incorrect rewards assigned to outputs with low probability under the initial policy πθ0 . Proposition 2 formalizes this prospect. Namely, suppose that rP and rG differ only on a set of outputs with low initial probability. Proposition 2 shows that policies learned by maximizing these reward functions remain close in total variation distance throughout training. This is in line with common intuition, by which low probability outputs have little influence since they are unlikely to be sampled when constructing sample-based gradient estimates. However, we prove a stronger claim: even in an idealized setting with access to exact gradients—where all outputs factor into the update—the contribution of low probability outputs is still negligible. This is a consequence of the optimization dynamics induced by policies that produce a distribution over outputs via the softmax function. Proposition 2. Suppose that the proxy reward function rP and ground truth reward function rG differ only on a set of outputs Z ⊆ Y, i.e., rP (y) = rG (y) for all y ∈ Y \ Z. Denote by θt and θtG the parameters at time t ≥ 0 when maximizing via gradient flow (Equation (1)) the expected reward with respect to rP and rG , respectively, starting from the same initial parameters θ0 = θ0G . For any time 2 T ≥ 0 and approximation level ϵ > 0, if πθ0 (Z) ≤ ∆Z exp(10B 2 T ) · ϵ, with B := maxy∈Y ∥ϕ(y)∥ and ∆Z := maxy∈Z |rP (y) − rG (y)|, then for all t ∈ [0, T ]: TV πθt , πθtG ≤ ϵ , where TV(πθt , πθtG ) := 21 ∥πθt − πθtG ∥1 is the total variation distance between πθt and πθtG . Proof sketch (full proof in Appendix B.3). We prove that πθt (Z) remains low for all t ∈ [0, T ], under both rP and rG , by showing that the rate at which πθt (Z) grows is small whenever πθt (Z) is low. Since rP and rG differ only on Z and, for softmax policies, the contribution of outputs in Z to the gradient is scaled by their own probability, this implies that the distance between ∇VP (θ) and ∇VG (θ) is small at any θ visited during training (under either rP or rG ). The fact that πθt and πθtG stay close in total variation distance then follows through standard arguments on the stability of gradient flow over smooth objectives to gradient perturbations. Proposition 2 yields an additional type of benign reward error. Benign Error II: Incorrect proxy reward for an output with low probability under πθ0 . If an output y is improbable under πθ0 , then assigning it an incorrect proxy reward rP (y) ̸= rG (y) negligibly affects the outcome of policy gradient. 3.3
Beneficial Errors That Prevent Attraction to Mediocre Outputs
Section 3.2 showed that outputs with low proxy reward or low probability under the initial policy do not attract probability. Where then does probability mass go during policy gradient? It is helpful to first consider the case of orthonormal feature vectors {ϕ(y)}y∈Y . In this setting, the logit dynamics of an output y ∈ Y simplifies to (cf. Proposition 1): d ⟨ϕ(y), θt ⟩ = πθt (y)AP (y; θt ) . dt 5
As evident from the equation above, the logit of y grows whenever its advantage AP (y; θt ) = rP (y) − VP (θt ) is positive. Though, since the advantage is scaled by the output probability πθt (y), policy gradient can be drawn to outputs with mediocre proxy reward at the expense of outputs with higher proxy reward, if the former are more likely under the policy. This phenomenon was empirically observed in [50]. However, the analysis of [50] only showed that if the policy already assigns high probability to a suboptimal output, escaping that region takes a long time, leaving open the question of when and to what extent the policy becomes highly concentrated on such outputs to begin with. We prove that the attraction to mediocre outputs can be severe, stalling optimization for an arbitrarily long time (Theorem 1). As we establish below, this implies that the following reward error type is beneficial for improving the rate at which the ground truth reward increases. Beneficial Error I: Low proxy reward for an output with mediocre ground truth reward. Let ymed be an output with mediocre ground truth reward, i.e., VG (θ0 ) < rG (ymed ) < rG (y⋆ ) where y⋆ := argmaxy∈Y rG (y) and VG (θ0 ) := Ey∼πθ0 [rG (y)]. If the probability of ymed under πθ0 is higher than that of y⋆ , then assigning low proxy reward to ymed can accelerate the increase in ground truth reward by steering the policy away from it. Formally, we compare the optimization trajectory across two settings. In the first setting, we maximize directly a ground truth reward function rG that assigns mediocre reward to an output ymed . In the second setting, we maximize a proxy reward function rP that is identical to rG , except that it assigns low reward to ymed . Theorem 1 establishes that, if πθ0 (y⋆ ) is sufficiently small relative to πθ0 (ymed ), the time required to achieve high ground truth reward when maximizing rG directly can be arbitrarily larger than the time required when maximizing rP . This gap in optimization time arises because, when maximizing rG , the policy initially concentrates its probability mass on ymed and stalls in its vicinity. By contrast, when maximizing rP , the policy directly shifts its probability mass towards y⋆ . For conciseness, Theorem 1 provides an abridged version of the result; see Appendix A for the detailed theorem statement. Figure 2 corroborates Theorem 1 by demonstrating empirically the effect of mediocre outputs: the smaller πθ0 (y⋆ ) is relative to πθ0 (ymed ), the longer policy gradient is delayed by an attraction to ymed (when ymed is assigned mediocre proxy reward). Theorem 1 (Abridged version of Theorem 2). Assume that the feature vectors {ϕ(y)}y∈Y are orthonormal and that Y = {y⋆ , ymed }∪Ybad , where y⋆ = argmaxy∈Y rG (y) is an output with maximal ground truth reward, ymed has mediocre ground truth reward satisfying VG (θ0 ) = Ey∼πθ0 [rG (y)] < rG (ymed ) < rG (y⋆ ), and every output in Ybad ⊂ Y has ground truth reward lower than VG (θ0 ). Let ϵ ∈ 0, rG (y⋆ ) − rG (ymed ) and denote by t⋆ the initial time at which VG (θt ) ≥ rG (y⋆ ) − ϵ when maximizing the expected ground truth reward directly via gradient flow (i.e., Equation (1) with VG in place of VP ). Furthermore, let tno−med be the analogous time when maximizing a proxy reward ⋆ function rP that is identical to rG , except that it assigns to ymed minimal reward, starting from the same initial parameters θ0 . For any T ≥ 0, if πθ0 (y⋆ ) is sufficiently small, then: t⋆ − tno−med ≥T, ⋆ i.e., the gap between t⋆ and tno−med can be arbitrarily large. ⋆ Proof sketch (full proof of Theorem 2 is in Appendix B.4). Consider first the case of maximizing rG d directly. The logit dynamics of an output y in this case is given by dt ⟨ϕ(y), θt ⟩ = πθt (y)AG (y; θt ), where AG (y; θt ) = rG (y) − VG (θt ) is the advantage of y under rG and πθt (cf. Proposition 1). If πθ0 (ymed ) is substantially higher than πθ0 (y⋆ ), the logit of the mediocre output ymed initially grows much faster than that of the optimal output y⋆ , despite ymed having a smaller advantage than y⋆ . This causes the probability of y⋆ to decrease. By carefully analyzing the optimization trajectory, we show that πθt (y⋆ ) continues to decrease at least until πθt (ymed ) is close to one, in the sense that πθt (ymed ) ≥ 1 − Θ(πθ0 (y⋆ )14/13 ). We then use the fact that the gradient vanishes whenever the policy is highly concentrated on a single output (cf. [50, 1, 68, 69]) to prove that escaping the vicinity of ymed requires at least Ω(πθ0 (y⋆ )−14/13 ) time, and so t⋆ is at least of that order. By contrast, when maximizing rP , the advantage of all outputs except y⋆ is negative throughout training since rP assigns them a low reward. Thus, the policy directly shifts its probability mass towards y⋆ and achieves ϵ-optimal ground truth reward at time tno−med , which we prove is at most O(πθ0 (y⋆ )−1 ). ⋆ no−med This implies that the gap between t⋆ and t⋆ becomes arbitrarily large as πθ0 (y⋆ ) → 0. 6
πθt (ymed )
Output Probability
πθ0 (y ) = 0.15
Output Probability
Case II: rP (ymed ) = −1
Case I: rP (ymed ) = 0.8
πθt (y )
πθt (other outputs)
πθ0 (y ) = 0.1
πθ0 (y ) = 0.05
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0
5000
10000
15000
Training Step
20000
0.0
0
5000
10000
15000
Training Step
20000
0.0
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0
5000
10000
15000
Training Step
20000
0.0
0
5000
10000
15000
Training Step
20000
0.0
0
5000
0
5000
10000
15000
20000
10000
15000
20000
Training Step
Training Step
Figure 2: Attraction to mediocre outputs can impede policy gradient optimization. Plotted is the evolution of output probabilities during policy gradient in settings corresponding to Theorem 1: a linear softmax policy with orthonormal output features, trained using exact gradients of the expected proxy reward. The ground truth reward rG assigns a maximal reward of 1 to y⋆ , a mediocre reward of 0.8 to ymed , and a low reward of −1 to all remaining outputs. In the top row, we use rP = rG . Bottom row shows experiments in identical settings, except that rP (ymed ) = −1. In line with Theorem 1, when ymed receives mediocre proxy reward, the policy initially concentrates its probability mass on ymed and stagnates. By contrast, when ymed is assigned low proxy reward, the policy directly increases the probability of y⋆ . The gap in optimization time between the two cases becomes larger as πθ0 (y⋆ ) decreases (left to right). Appendix C provides experiments demonstrating similar trends with sample-based gradients instead of exact gradients. See Appendix D for additional implementation details.
Influence of feature similarity. Under the assumption of orthonormal feature vectors, Theorem 1 shows that outputs with mediocre reward can prolong the time required to reach higher-quality outputs. In Appendix A, we analyze how the feature geometry governs the extent to which mediocre outputs impede optimization. Specifically, we prove that when ⟨ϕ(y⋆ ), ϕ(ymed )⟩ < 0, an output ymed with mediocre reward delays optimization even further and may prevent the policy from ever reaching the optimal output y⋆ . Conversely, when ⟨ϕ(y⋆ ), ϕ(ymed )⟩ > 0, attraction to ymed does not necessarily obstruct convergence to y⋆ . This highlights that whether outputs with mediocre reward are problematic for optimization also depends on their similarity to higher-quality outputs. 3.4
Harmful Errors Beyond Reward Hacking
To conclude the categorization we revisit the most well-known type of reward error (cf. [4, 77, 18]). Harmful Error I: High proxy reward for an output with low ground truth reward. Let ybad be an output with low ground truth reward and denote y⋆ := argmaxy∈Y rG (y). If rP (ybad ) > rP (y⋆ ), then maximizing rP can lead to reward hacking. While assigning high proxy reward to an output ybad with low ground truth reward is obviously undesirable, the attraction to mediocre outputs characterized in Section 3.3 reveals a more subtle harmful reward error. Even if the proxy reward assigned to ybad is not high enough to cause reward hacking, it may still impede optimization. This follows as a corollary of Theorem 1; see Corollary 1 in Appendix A for the formal statement. Harmful Error II: Mediocre proxy reward for an output with low ground truth reward. Let ybad be an output with low ground truth reward whose probability under πθ0 is higher than the probability of y⋆ := argmaxy∈Y rG (y). Then, assigning a mediocre proxy reward to ybad , satisfying VP (θ0 ) < rP (ybad ) < rP (y⋆ ) where VP (θ0 ) := Ey∼πθ0 [rP (y)], can slow down the increase in ground truth reward by causing the policy to stall in the vicinity of ybad .
7
4
Application I: Harm-Aware Accuracy for Reward Model Evaluation
To demonstrate the potential utility of our theory, we consider the problem of reward model evaluation for RLHF. Existing benchmarks primarily evaluate reward models via ranking accuracy (see Section 2). A limitation of ranking accuracy, uncovered in Section 3, is that it treats all incorrect rankings of outputs as equally harmful, while some deviations from the ground truth reward are benign or even beneficial for policy gradient optimization. Indeed, recent work showed that ranking accuracy often does not correlate well with the performance of a language model after RLHF [9, 89, 69]. Motivated by this observation, we develop ranking accuracy variants that account for the harmfulness of an incorrect ranking to policy gradient. Compared to standard ranking accuracy, our harm-aware variants usually correlate better with language model performance, across datasets and model families (Llama [24], OLMo [57], and Qwen [95]). However, our experiments also show that the correlation can still be weak, highlighting persistent challenges in robustly evaluating reward models. For brevity, we defer some experiments and implementation details to Appendices C and D, respectively. 4.1
Harm-Aware Ranking Accuracy
− Let S be a dataset of preference examples (x, y + , y1− , . . . , yK ), where x is an input and the output − − + y is preferred over y1 , . . . , yK according to a ground truth reward function rG . In practice, we typically do not have access to rG . It is therefore impossible to determine for every incorrect ranking whether it is harmful or not based on the categorization in Section 3. Nevertheless, the categorization allows us to identify certain types of incorrect rankings that are unlikely to be harmful.
For a reward model rP , let VP (x; θ) := Ey∼πθ (·|x) [rP (x, y)] be the expected proxy reward achieved by a language model πθ given an input x. If rP ranks a dispreferred output yk− above y + , but rP (x, yk− ) < VP (x; θ), then this error may be benign since yk− does not attract probability (Benign Error I); it may even be beneficial when y + is of mediocre quality (Beneficial Error I). Accordingly, we define the harm-aware ranking accuracy (HAcc) by making a simple modification to ranking accuracy: HAcc does not penalize incorrect rankings whenever all dispreferred outputs receive proxy reward below an empirical estimate of VP (x; θ), denoted VsP (x; θ): h i 1 X HAcc(rP ; S) := 1 max rP (x, yk− ) < max rP (x, y + ), VsP (x; θ) . − + (x,y ,{yk }k∈[K] )∈S |S| k∈[K] Furthermore, reward errors involving outputs that are improbable under πθ are also benign (Benign Error II). Thus, in addition to ranking accuracy (Acc) and HAcc, we considerQweighted variants (Acc-W K sθ (y + |x) k=1 π sθ (yk− |x)/Z(S; πθ ), and HAcc-W) that multiply the contribution of an example by π 1/|y| sθ (y|x) := πθ (y|x) where π and Z(S; πθ ) is the normalization constant ensuring weights sum to sθ (y|x) is equivalent to length normalizing the log probability of y. We apply this one. The usage of π transformation for numerical stability and to avoid biasing against long sequences. Computational cost. Unlike standard ranking accuracy, the proposed harm-aware variants depend on the language model (i.e., policy) πθ . This dependence is desirable: our analysis, together with prior work [69], shows that the effectiveness of a reward model is inherently policy-dependent. The drawback is a modest computational overhead. Namely, these variants require sampling from πθ to estimate VsP (x; θ) and computing the probabilities of outputs in S. In practice, this overhead is small relative to the cost of running policy gradient and is further reduced by reusing samples and probabilities when evaluating multiple reward models. 4.2
Comparison of Ranking Accuracy Variants
Main setting. We use four language models of different families and types: Llama-3.2-3B-Instruct, Llama-3.2-1B-Instruct [24], OLMo-2-1B-SFT [56], and Qwen3-1.7B-Base [95]. Each language model is trained via the RLOO [34, 2] policy gradient method over prompts from the UltraFeedback dataset [14], using 13 reward models chosen to span a diverse range of RewardBench2 scores [47] (see Table 1 in Appendix D for the full list).5 We then compare ranking accuracy with its harm-aware variants (computed for the initial language models) based on their predictiveness of which reward 5
We adopt RLOO because it is more resource efficient than PPO [74] and has shown competitive results [2]. We also experimented with GRPO [75] and observed analogous results.
8
Spearman Correlation
0.8 0.6 0.4 0.2
Regret in Choosing Best Reward Model Acc Acc-W HAcc HAcc-W
0.39
0.31 0.32 0.15 0.12 -0.11
Llama-3B Instruct
Llama-1B Instruct
0.29
0.34
0.29
0.12
0.04
0.0 0.2
0.34
0.34
25.8%
17.6%
16.1%
3.1%
-W Acc
25.8%
17.6%
16.1%
3.1%
60%
cc HA
0.0%
0.0%
22.4%
3.1%
40%
W
0.0%
0.0%
16.1%
3.1%
Llama-3B Instruct
Llama-1B Instruct
OLMo-1B SFT
Qwen-1.7B Base
-0.07 -0.21
ccHA
-0.19
OLMo-1B SFT
Qwen-1.7B Base
100%
Acc
80%
Regret
Correlation With Ground Truth Reward Increase 1.0
20% 0%
Figure 3: Harm-aware ranking accuracy variants are more predictive of which reward model leads to better language model performance. For each language model, we run policy gradient (specifically, RLOO) using 13 different reward models on prompts from the UltraFeedback dataset, and compute (per language and reward model) the mean ground truth reward increase based on three separate runs. Compared to standard ranking accuracy and its weighted counterpart (Acc and Acc-W), the harm-aware variants (HAcc and HAcc-W) correlate better with ground truth reward increase and lead to lower regret in selecting the best reward model (see Section 4.2 for the definition of regret). Yet, the correlation remains below 0.4 and can even be negative, highlighting that robust reward model evaluation remains an open challenge. See Section 4 for further details.
model leads to better language model performance. Following prior work [21, 13, 84, 6, 9, 89], we simulate ground truth rewards with a reward model (ArmoRM [88]), excluded from the ones used for training. This ground truth model is used for evaluating both language model performance and reward model accuracy. Unless stated otherwise, rewards and accuracy values are measured on examples from the policy gradient training set.6 Additional settings. Appendix C includes experiments showing similar trends: (i) when computing reward model accuracy values on RewardBench2 instead of the policy gradient training set; (ii) over the WildChat-IF dataset [37] instead of UltraFeedback; (iii) using a different ground truth reward model; and (iv) when evaluating language model performance by win-rates according to a frontier (GPT) judge model, as opposed to by ground truth reward increase. Result I: Harm-aware ranking accuracy variants are more predictive of which reward model leads to better language model performance. For each language model and ranking accuracy variant, Figure 3 presents the Spearman correlation between reward model accuracy and ground truth reward increase. We also report the regret incurred when choosing the most accurate reward model, defined as 100(Rbest − Rchosen )/Rbest , where Rbest is the largest ground truth reward increase due to policy gradient, across all reward models, and Rchosen is the increase under the chosen reward model. Compared to standard ranking accuracy, we find that HAcc typically correlates better with ground truth reward increase and leads to lower regret. On the other hand, we observe that weighting examples by output probabilities is not consistently helpful. This suggests that the contribution of low probability outputs to standard ranking accuracy is not a primary cause for its limited correlation with language model performance (for the considered dataset and models). Result II: Robust reward model evaluation remains an open challenge. Despite the improvement provided by harm-aware ranking accuracy, the correlation between a ranking accuracy variant and ground truth reward increase remains below 0.4, and in some settings is even negative. As we discuss in Section 7, this indicates that robustly evaluating reward models may require looking beyond adjustments to ranking accuracy.
5
Application II: When Should Partially Correct Outputs Be Rewarded?
Aside from reward model evaluation for RLHF (Section 4), we explore implications of our theory for reward design in settings with verifiable rule-based rewards. In tasks such as instruction following or code generation, an output must satisfy a set of constraints or unit tests to be considered correct [62, 26, 57]. This raises the question of whether partially correct outputs should be rewarded. For example, if an output satisfies one out of two desired constraints, should it receive some partial reward (e.g., 0.5) or should all non-fully correct outputs receive zero reward? 6 We mainly consider the ground truth reward increase on training examples as it directly reflects policy gradient optimization, which is the focus of our work, without conflating it with generalization. Nonetheless, as reported in Appendix C, we found the reward increase on test examples to be nearly identical.
9
Binary (Full-Correctness) Rewards
Partial Rewards
Gap in Learnability
Constraint Pair B:
No Gap in Learnability
Probability of Success
Constraint Pair A:
Probability of Success
(0.5 for each satisfied constraint)
(1 if both constraints satisfied, 0 otherwise)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
constraint 1 constraint 2 both
0.2 0.0
0
200
400
600
Training Step
800
1000
0.4 0.0
1.0
1.0
0.8
0.8
0.6
0.6
0.4
constraint 1 constraint 2 both
0.2 0.0
0
200
400
600
Training Step
800
1000
constraint 1 constraint 2 both
0.2 0
200
400
600
Training Step
0.4
1000
constraint 1 constraint 2 both
0.2 0.0
800
0
200
400
600
Training Step
800
1000
Figure 4: Rewarding partially correct outputs can impede policy gradient optimization. We train Qwen31.7B using GRPO on two instruction following datasets, where the prompts in each dataset include a pair of constraints from IFBench that an output must satisfy to be considered correct (all prompts within a dataset share the same constraint pair). Plotted are probabilities of satisfying the constraints, averaged over ten training steps for clarity; unsmoothed values are shown in semi-transparent lines. As suggested by our theory (Section 3), when the initial probability of satisfying one constraint is noticeably higher than the probability of satisfying the other, rewarding partial correctness can cause the policy to learn only the easier constraint. In this case, rewarding only fully correct outputs can lead to faster learning of both constraints. However, when both constraints are initially satisfied with similar probability, both reward designs tend to work well. See Section 5 for further details.
Since the goal is to produce fully correct outputs, it is natural to view the ground truth reward function as giving a reward of 1 to fully correct outputs and 0 to all others. The question above then becomes a matter of proxy reward design. Our analysis (Section 3) suggests that when the language model (i.e., initial policy) is noticeably more likely to produce partially correct outputs than fully correct ones, rewarding partial correctness can cause it to stall on these mediocre outputs (Harmful Error II). Thus, as long as there is a non-zero probability of producing fully correct outputs at initialization, in such cases it may be more effective to only reward full correctness. We demonstrate this prospect empirically using instruction following tasks from IFBench [62]. Setting. We construct instruction following datasets by taking prompts from UltraFeedback and appending to them two instructions, each specifying a constraint from IFBench (e.g., “the response must start with a verb”). All prompts within a dataset share the same constraints, and the datasets differ in which constraints they contain. We then train Qwen3-1.7B on these datasets using GRPO [75] with two proxy reward designs. In the first, we reward partial correctness by giving a reward of 0.5 for each satisfied constraint. In the second, the reward is binary: fully correct outputs receive a reward of 1, and all other outputs receive a reward of 0. See Appendix C for experiments with additional language models and Appendix D for further implementation details. Results. Figure 4 shows that, if the initial probability of satisfying one constraint noticeably exceeds the probability of satisfying the other, rewarding partial correctness may cause the policy to learn only the easier constraint, while binary (full-correctness) rewards can enable the policy to learn both constraints.7 If instead the constraints are initially satisfied with similar probability, both reward designs tend to work well. Taken together, these findings yield a practical insight: rewarding partially correct outputs is not universally beneficial or harmful. Choosing suitable proxy rewards in verifiable settings requires considering both the task structure and language model capabilities. 7
We note, however, that the probability of initially satisfying a constraint is not the sole factor determining its ease of learnability. As Appendix C.3 shows, in cases where one constraint is initially satisfied with substantially higher probability than the other, it is also possible for both constraints to be learned quickly under partial rewards (even when learning fails under binary rewards).
10
6
Related Work
Reinforcement learning with proxy rewards. Manually handcrafted and automatically learned proxy rewards have a long history across reinforcement learning applications (e.g., [49, 64, 76, 80, 97, 7, 11, 61, 16]), far too broad to fully cover here. In the context of language models, which are the focus of this work, proxy rewards are widely used both for alignment with human preferences (i.e., RLHF; [99, 58, 24, 70, 83]) and for verifiable tasks such as mathematical problem solving, code generation, and instruction following [37, 26, 57, 31, 85, 62, 48]. Yet, despite the ubiquity of imperfect proxy rewards in practice, there is limited theoretical understanding of what makes a good proxy reward function. We take a step towards addressing this gap. What makes a good proxy reward function? This question has roots in the reward shaping literature [55]. There, prior theoretical work characterized reward transformations that preserve the order of policies [55, 5, 25, 78] and demonstrated that reward shaping can improve the efficiency of stylized learning algorithms or value-based methods in long-horizon environments [39, 15, 10, 16, 27]. Furthermore, [23, 93, 78] derived measures for comparing two reward functions (e.g., a proxy and ground truth reward function) that account for said reward transformations, which leave policy ordering unchanged. Similarly to our work, these lines of research can be interpreted as identifying cases where deviating from a ground truth reward function is in some sense benign or beneficial. However, they disregard the impact of such deviations on policy gradient optimization. As Section 3 proves, reward errors that modify policy ordering can still be benign or beneficial. Conversely, reward errors that preserve policy ordering may be harmful and slow down optimization. Perhaps most closely related to our work is [69]. While reward models for RLHF are mainly evaluated via ranking accuracy (cf. Section 2.2), [69] established that other aspects can determine how good a reward model is. In particular, they showed that when the reward model induces low reward variance, policy gradient suffers from a flat objective landscape that hinders optimization. Our technical approach resembles that of [69], which also analyzes the rate of ground truth reward increase under proxy reward functions. Though, unlike [69], we consider how different types of reward errors influence optimization beyond their effect on reward variance (see Remark 1 in Appendix A for further discussion on this point). This allows us to develop harm-aware ranking accuracy variants and provide insights for reward design in settings with verifiable rewards. Theoretical analyses of policy gradient optimization. For softmax policies, i.e., policies that produce a distribution over outputs via the softmax function, the reward maximization objective is non-concave even under tabular or linear parameterizations [1]. Characterizing the complex optimization dynamics of policy gradient methods in such settings has therefore remained an active area of research. Existing analyses showed that optimization can fail due to the gradient vanishing in long-horizon environments with sparse rewards [1, 40], when the policy is nearly deterministic [3, 28, 73, 50, 51, 1, 22], or, more generally, when the reward variance is low [68, 69]. Particularly relevant to our work, [50] observed that policy gradient can be attracted to mediocre outputs at the expense of higher-quality ones, but left open the question of when and to what extent this phenomenon can impede optimization. Theorem 1 shows that the attraction to mediocre outputs can be severe, stalling optimization for an arbitrarily long time. Our results also complement existing convergence guarantees [51, 52, 54, 33, 44, 42]. These guarantees depend on inf t≥0 πθt (y⋆ )—the infimal probability assigned to the optimal output during training. We identify that the attraction to mediocre outputs can drive this quantity arbitrarily low.
7
Conclusion
Imperfect proxy rewards are widely used for language model post-training, and reinforcement learning more broadly. Yet, the understanding of how discrepancies between the proxy and ground truth rewards affect the learning process remains limited. Such reward errors are conventionally viewed as harmful due to the risk of reward hacking. In this work, however, we proved that aside from being harmful, reward errors can also be benign or even beneficial for policy gradient optimization. Specifically, our theoretical analysis categorizes prominent reward error types into harmful, benign, or beneficial, and characterizes the mechanisms behind their distinct effects on optimization. We then demonstrated that this categorization yields practical insights. For RLHF, it motivated harm-aware reward model evaluation metrics that are more predictive of language model performance 11
than standard ranking accuracy. For environments with verifiable rewards, it revealed that rewarding partially correct outputs can impede learning rather than help it. A key theme underlying these results is that the effectiveness of proxy rewards depends heavily on their interplay with the initial policy and learning algorithm. As elaborated below, we hope that our work will inspire further research on proxy reward evaluation and design that accounts for this interplay. 7.1
Limitations and Future Work
Theoretical analysis: beyond bandit environments. Our theory (Section 3) focuses on bandit environments since, in language model applications, rewards are often available only for complete outputs. Extending the analysis to longer-horizon settings, corresponding to multi-turn dialogue or environments with process rewards, may reveal additional ways in which reward errors shape learning. It would also be valuable to understand how reward errors interact across inputs, especially given recent evidence that learnability via policy gradient can depend on the data composition [63, 30]. Towards robust reward model evaluation. Although the harm-aware ranking accuracy variants proposed in Section 4 improve upon standard ranking accuracy, our results suggest that even these metrics capture only part of what makes an effective reward model. We outline two potential barriers towards robust reward model evaluation, which future work can attempt to address. First, output rankings provide only coarse information about the ground truth reward function. This seems to place an inherent ceiling on metrics that solely rely on such data. Second, reward model evaluation benchmarks often lack coverage of relevant inputs and outputs. This limitation is reflected both in recent efforts to revisit existing benchmarks [98, 89, 47] and in our experiments, where evaluating reward models on RewardBench2 is markedly less predictive of language model performance than evaluating them on examples drawn from the policy gradient training set (Figure 8 in Appendix C). Reward design in verifiable settings. We identified a drawback of rewarding partially correct outputs: it can hinder optimization by causing the policy to stall on such suboptimal outputs (Section 5). However, there are cases where rewarding partial correctness is beneficial. In particular, [82] showed that when the initial policy produces fully correct outputs with near-zero probability, by first using partial rewards and then switching to binary (full-correctness) rewards, one can achieve better performance than using binary rewards from the start. We therefore view analyzing adaptive proxy reward schemes that vary across inputs and training steps as a promising direction for future work.
Acknowledgments and Disclosure of Funding We thank Eshbal Hezroni for aid in preparing illustrative figures. SW is supported by an NSF Graduate Research Fellowship. SA acknowledges funding from ONR, Schmidt Science, and OpenAI. NR is supported in part by the Zuckerman STEM Leadership Program.
References [1] Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. The Journal of Machine Learning Research, 22(1):4431–4506, 2021. [2] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740, 2024. [3] Zafarali Ahmed, Nicolas Le Roux, Mohammad Norouzi, and Dale Schuurmans. Understanding the impact of entropy on policy optimization. In International conference on machine learning, pages 151–160. PMLR, 2019. [4] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. [5] John Asmuth, Michael L Littman, and Robert Zinkov. Potential-based shaping in model-based reinforcement learning. In AAAI, pages 604–609, 2008.
12
[6] Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, et al. Infalign: Inference-aware language model alignment. arXiv preprint arXiv:2412.19792, 2024. [7] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019. [8] Alberto Bietti, Joan Bruna, and Loucas Pillaud-Vivien. On learning gaussian multi-index models with gradient flow part i: General properties and two-timescale learning. Communications on Pure and Applied Mathematics, 78(12):2354–2435, 2025. [9] Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, and Xiaoyu Shen. The accuracy paradox in rlhf: When better reward models don’t yield better language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. [10] Ching-An Cheng, Andrey Kolobov, and Adith Swaminathan. Heuristic-guided reinforcement learning. Advances in Neural Information Processing Systems, 34:13550–13563, 2021. [11] Hao-Tien Lewis Chiang, Aleksandra Faust, Marek Fiser, and Anthony Francis. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 4(2):2007–2014, 2019. [12] Hung-Hsu Chou, Johannes Maly, and Holger Rauhut. More is less: inducing sparsity via overparameterization. Information and Inference: A Journal of the IMA, 12(3):1437–1460, 2023. [13] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. In International Conference on Learning Representations, 2024. [14] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. In International Conference on Machine Learning, 2024. [15] Falcon Dai and Matthew Walter. Maximum expected hitting cost of a markov decision process and informativeness of rewards. Advances in Neural Information Processing Systems, 2019. [16] Rati Devidze, Parameswaran Kamalaruban, and Adish Singla. Exploration-guided reward shaping for reinforcement learning under sparse rewards. Advances in Neural Information Processing Systems, 2022. [17] Omer Elkabetz and Nadav Cohen. Continuous vs. discrete optimization of deep neural networks. Advances in Neural Information Processing Systems, 2021. [18] Lukas Fluri, Leon Lang, Alessandro Abate, Patrick Forré, David Krueger, and Joar Skalse. The perils of optimizing learned reward functions: Low training error does not guarantee low regret. In International Conference on Machine Learning, 2025. [19] Dylan J Foster, Zakaria Mhammedi, and Dhruv Rohatgi. Is a good foundation necessary for efficient reinforcement learning? the computational role of the base model in exploration. In Proceedings of Thirty Eighth Conference on Learning Theory, 2025. [20] Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. How to evaluate reward models for rlhf. In International Conference on Learning Representations, 2025. [21] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023. [22] Shivam Garg, Samuele Tosatto, Yangchen Pan, Martha White, and Rupam Mahmood. An alternate policy gradient estimator for softmax policies. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, pages 6630–6689, 2022. [23] Adam Gleave, Michael Dennis, Shane Legg, Stuart Russell, and Jan Leike. Quantifying differences in reward functions. In International Conference on Learning Representations, 2021. [24] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AlDahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [25] Marek Grzes. Reward shaping in episodic reinforcement learning. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, 2017.
13
[26] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [27] Abhishek Gupta, Aldo Pacchiano, Yuexiang Zhai, Sham Kakade, and Sergey Levine. Unpacking reward shaping: Understanding the benefits of reward engineering on sample complexity. Advances in Neural Information Processing Systems, 2022. [28] Daniel Hennes, Dustin Morrill, Shayegan Omidshafiei, Remi Munos, Julien Perolat, Marc Lanctot, Audrunas Gruslys, Jean-Baptiste Lespiau, Paavo Parmas, Edgar Duenez-Guzman, et al. Neural replicator dynamics. arXiv preprint arXiv:1906.00190, 2019. [29] Audrey Huang, Adam Block, Qinghua Liu, Nan Jiang, Akshay Krishnamurthy, and Dylan J Foster. Is best-of-n the best of them? coverage, scaling, and optimality in inference-time alignment. In International Conference on Machine Learning, 2025. [30] Yu Huang, Zixin Wen, Yuejie Chi, Yuting Wei, Aarti Singh, Yingbin Liang, and Yuxin Chen. On the learning dynamics of rlvr at the edge of competence. arXiv preprint arXiv:2602.14872, 2026. [31] Yuzhen Huang, Weihao Zeng, Xingshan Zeng, Qi Zhu, and Junxian He. Pitfalls of rule-and model-based verifiers–a case study on mathematical reasoning. arXiv preprint arXiv:2505.22203, 2025. [32] Jacek Karwowski, Oliver Hayman, Xingjian Bai, Klaus Kiendlhofer, Charlie Griffin, and Joar Skalse. Goodhart’s law in reinforcement learning. In International Conference on Learning Representations, 2024. [33] Sara Klein, Simon Weissmann, and Leif Döring. Beyond stationarity: Convergence analysis of stochastic softmax policy gradient methods. In International Conference on Learning Representations, 2024. [34] Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! Deep Reinforcement Learning Meets Structured Prediction ICLR Workhsop, 2019. [35] Vignesh Kothapalli, Tom Tirer, and Joan Bruna. A neural collapse perspective on feature evolution in graph neural networks. Advances in Neural Information Processing Systems, 2023. [36] Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. In International Conference on Learning Representations, 2025. [37] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. [38] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: NAACL, 2025. [39] Adam Laud and Gerald DeJong. The influence of reward on the speed of reinforcement learning: An analysis of shaping. In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 440–447, 2003. [40] Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Softmax policy gradient methods can take exponential time to converge. In Conference on Learning Theory. PMLR, 2021. [41] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023. [42] Max Qiushi Lin, Jincheng Mei, Matin Aghaei, Michael Lu, Bo Dai, Alekh Agarwal, Dale Schuurmans, Csaba Szepesvari, and Sharan Vaswani. Rethinking the global convergence of softmax policy gradient with linear function approximation. arXiv preprint arXiv:2505.03155, 2025. [43] Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352, 2025. [44] Jiacai Liu, Wenye Li, and Ke Wei. Elementary analysis of policy gradient methods. arXiv preprint arXiv:2404.03372, 2024. [45] Jiate Liu, Yiqin Zhu, Kaiwen Xiao, QIANG FU, Xiao Han, Yang Wei, and Deheng Ye. RLTF: Reinforcement learning from unit test feedback. Transactions on Machine Learning Research, 2023.
14
[46] Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. Rm-bench: Benchmarking reward models of language models with subtlety and style. In International Conference on Learning Representations, 2025. [47] Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A Smith, Hannaneh Hajishirzi, and Nathan Lambert. Rewardbench 2: Advancing reward model evaluation. arXiv preprint arXiv:2506.01937, 2025. [48] Md Rayhanul Masud, Azmine Toushik Wasi, Salman Rahman, and Md Rizwan Parvez. Reward engineering for reinforcement learning in software tasks. arXiv preprint arXiv:2601.19100, 2026. [49] Maja J Mataric. Reward functions for accelerated learning. In Machine learning proceedings 1994, pages 181–189. Elsevier, 1994. [50] Jincheng Mei, Chenjun Xiao, Bo Dai, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Escaping the gravitational pull of softmax. Advances in Neural Information Processing Systems, 2020. [51] Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In International Conference on Machine Learning, pages 6820–6829. PMLR, 2020. [52] Jincheng Mei, Yue Gao, Bo Dai, Csaba Szepesvari, and Dale Schuurmans. Leveraging non-uniformity in first-order non-convex optimization. In International Conference on Machine Learning, pages 7555–7564. PMLR, 2021. [53] Jincheng Mei, Bo Dai, Alekh Agarwal, Mohammad Ghavamzadeh, Csaba Szepesvári, and Dale Schuurmans. Ordering-based conditions for global convergence of policy gradient methods. Advances in Neural Information Processing Systems, 2023. [54] Jincheng Mei, Zixin Zhong, Bo Dai, Alekh Agarwal, Csaba Szepesvari, and Dale Schuurmans. Stochastic gradient succeeds for bandits. In International Conference on Machine Learning, pages 24325–24360. PMLR, 2023. [55] Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pages 278–287, 1999. [56] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, et al. 2 olmo 2 furious. arXiv preprint arXiv:2501.00656, 2024. [57] Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, et al. Olmo 3. arXiv preprint arXiv:2512.13961, 2025. [58] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 2022. [59] Richard Yuanzhe Pang, Vishakh Padmakumar, Thibault Sellam, Ankur Parikh, and He He. Reward gaming in conditional text generation. In The 61st Annual Meeting Of The Association For Computational Linguistics, 2023. [60] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017. [61] Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG), 40(4):1–20, 2021. [62] Valentina Pyatkin, Saumya Malik, Victoria Graf, Hamish Ivison, Shengyi Huang, Pradeep Dasigi, Nathan Lambert, and Hannaneh Hajishirzi. Generalizing verifiable instruction following. In Advances in Neural Information Processing Systems, 2025. [63] Yuval Ran-Milo, Yotam Alexander, Shahar Mendel, and Nadav Cohen. Outcome-based rl provably leads transformers to reason, but only with the right data. arXiv preprint arXiv:2601.15158, 2026. [64] Jette Randløv and Preben Alstrøm. Learning to drive a bicycle using reinforcement learning and shaping. In International Conference on Machine Learning, 1999.
15
[65] Noam Razin and Nadav Cohen. Implicit regularization in deep learning may not be explainable by norms. In Advances in Neural Information Processing Systems, 2020. [66] Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in tensor factorization. In International Conference on Machine Learning, 2021. [67] Noam Razin, Asaf Maman, and Nadav Cohen. Implicit regularization in hierarchical tensor factorization and deep convolutional neural networks. In International Conference on Machine Learning, 2022. [68] Noam Razin, Hattie Zhou, Omid Saremi, Vimal Thilak, Arwen Bradley, Preetum Nakkiran, Joshua M. Susskind, and Etai Littwin. Vanishing gradients in reinforcement finetuning of language models. In International Conference on Learning Representations, 2024. [69] Noam Razin, Zixuan Wang, Hubert Strauss, Stanley Wei, Jason D Lee, and Sanjeev Arora. What makes a reward model a good teacher? an optimization perspective. In Advances in Neural Information Processing Systems, 2025. [70] Noam Razin, Yong Lin, Jiarui Yao, and Sanjeev Arora. Why is your language model a poor implicit reward model? In International Conference on Learning Representations, 2026. [71] Itay Safran, Gal Vardi, and Jason D Lee. On the effective number of linear regions in shallow univariate relu networks: Convergence guarantees and implicit bias. Advances in Neural Information Processing Systems, 2022. [72] Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Representations, 2014. [73] Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: a source of plateaus in deep reinforcement learning. arXiv preprint arXiv:1904.11455, 2019. [74] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [75] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [76] Satinder Singh, Richard L Lewis, Andrew G Barto, and Jonathan Sorg. Intrinsically motivated reinforcement learning: An evolutionary perspective. IEEE Transactions on Autonomous Mental Development, 2 (2):70–82, 2010. [77] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 2022. [78] Joar Skalse, Lucy Farnik, Sumeet Ramesh Motwani, Erik Jenner, Adam Gleave, and Alessandro Abate. Starc: A general framework for quantifying differences between reward functions. In International Conference on Learning Representations, 2024. [79] Yonatan Slutzky, Yotam Alexander, Noam Razin, and Nadav Cohen. The implicit bias of structured state space models can be poisoned with clean labels. In Advances in Neural Information Processing Systems, 2025. [80] Jonathan Sorg, Richard L Lewis, and Satinder Singh. Reward design via online gradient ascent. Advances in Neural Information Processing Systems, 23, 2010. [81] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. In Advances in Neural Information Processing Systems, 2020. [82] Yiyou Sun, Yuhan Cao, Pohao Huang, Haoyue Bai, Hannaneh Hajishirzi, Nouha Dziri, and Dawn Song. Rl grokking recipe: How does rl unlock and transfer new algorithms in llms? In International Conference on Learning Representations, 2026. [83] Gokul Swamy, Sanjiban Choudhury, Wen Sun, Zhiwei Steven Wu, and J Andrew Bagnell. All roads lead to likelihood: The value of reinforcement learning in fine-tuning. In International Conference on Learning Representations, 2026. [84] Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.08448, 2024.
16
[85] Leitian Tao, Ilia Kulikov, Swarnadeep Saha, Tianlu Wang, Jing Xu, Sharon Li, Jason E Weston, and Ping Yu. Hybrid reinforcement: When reward is sparse, it’s better to be dense. arXiv preprint arXiv:2510.07242, 2025. [86] Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed. Perturbation analysis of neural collapse. In International Conference on Machine Learning, pages 34301–34329. PMLR, 2023. [87] Gal Vardi and Ohad Shamir. Implicit regularization in relu networks with the square loss. In Conference on Learning Theory, pages 4224–4258. PMLR, 2021. [88] Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845, 2024. [89] Xueru Wen, Jie Lou, Yaojie Lu, Hongyu Lin, Xing Yu, Xinyu Lu, Ben He, Xianpei Han, Debing Zhang, and Le Sun. Rethinking reward model evaluation: Are we barking up the wrong tree? In International Conference on Learning Representations, 2025. [90] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992. [91] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. [92] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory (COLT), pages 3635–3673, 2020. [93] Blake Wulfe, Ashwin Balakrishna, Logan Ellis, Jean Mercat, Rowan McAllister, and Adrien Gaidon. Dynamics-aware comparison of learned reward functions. In International Conference on Learning Representations, 2022. [94] Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. Is dpo superior to ppo for llm alignment? a comprehensive study. arXiv preprint arXiv:2404.10719, 2024. [95] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. [96] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. [97] Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Advances in neural information processing systems, 31, 2018. [98] Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, et al. Rmb: Comprehensively benchmarking reward models in llm alignment. In International Conference on Learning Representations, 2025. [99] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
17
A
Detailed Theorem Statements and Influence of Feature Similarity on Attraction to Mediocre Outputs
In this appendix, we present Theorem 2—the detailed statement of Theorem 1—which establishes the existence of beneficial reward errors. In particular, Theorem 2 shows that mediocre outputs can attract probability and stall policy gradient optimization for an arbitrarily long time. Thus, assigning low proxy reward to outputs with mediocre ground truth reward can be beneficial for accelerating the increase in ground truth reward. As a corollary (Corollary 1), we also obtain that assigning mediocre proxy reward to outputs with low ground truth reward can impede optimization, even if it does not induce reward hacking. This formalizes Harmful Error II from Section 3.4. Both Theorem 2 and Corollary 1 consider orthonormal output feature vectors. In Appendix A.1, we relax this assumption and analyze how the similarity between output features can exacerbate or attenuate the attraction to mediocre outputs. We refer the reader to Section 3.1 for the technical setting of the analysis. Our aim is to characterize the phenomenon where mediocre outputs attract probability at the expense of higher-quality outputs. Thus, we consider a setting in which Y contains an output ymed with ground truth reward lower than rG (y⋆ ), where y⋆ = argmaxy∈Y rG (y) is the optimal output, but higher than the ground truth reward of all remaining outputs Ybad ⊂ Y. For simplicity, we assume all outputs in Ybad have the same ground truth reward. Our analysis extends straightforwardly to the case where outputs in Ybad have different rewards, as long as they are sufficiently lower than rG (ymed ). Assumption 1 (Reward structure). The set of outputs can be decomposed as Y = {y⋆ , ymed } ∪ Ybad such that rG (y⋆ ) > rG (ymed ) > 0 ≥ maxy∈Ybad rG (y). Furthermore, all outputs in Ybad ⊂ Y have the same ground truth reward, i.e., rG (y) = rG (y ′ ) for all y, y ′ ∈ Ybad . Suppose that we can maximize the ground truth reward directly (i.e., rP = rG ). In this case, as discussed in Section 3.3, an output ymed with mediocre (ground truth and proxy) reward can attract probability if rG (ymed ) > VG (θ0 ) = Ey∼πθ0 [rG (y)] and the probability of ymed under πθ0 is higher than the probability of y⋆ . This condition is formalized by assuming that πθ0 (y⋆ ) is not too high and that πθ0 (Ybad ) is neither too low (otherwise, VG (θ0 ) will be above rG (ymed )) nor too high (otherwise, πθ0 (ymed ) may not be sufficiently higher than πθ0 (y⋆ )). See Lemma 14 in Appendix B for a proof that Assumptions 1 and 2 ensure rG (ymed ) > VG (θ0 ). Assumption 2 (Initial policy). The initial policy πθ0 satisfies the following conditions. • The initial probability of y⋆ is not too high: ( πθ0 (y⋆ ) < M · min 0.01,
21 1100(∆1 + ∆2 )
7/3
)13/14 , ∆22
,
where ∆1 := rG (y⋆ ) − rG (ymed ), ∆2 := rG (ymed ) − rG (ybad ) for some ybad ∈ Ybad , and M := min 1, 0.05 · rG (ymed )2/7 (∆1 + ∆2 )−1 . • The initial probability of Ybad is neither too low nor too high: 2πθ0 (y⋆ )7/13 max , 0.05 · rG (ymed ) ≤ πθ0 (Ybad ) ≤ 0.1 · rG (ymed ) . ∆2 M 7/13 Under Assumptions 1 and 2, we prove that incorrectly assigning low reward to ymed is beneficial for increasing the ground truth reward. Specifically, we compare the optimization trajectory when maximizing rG directly with the trajectory obtained when maximizing a proxy reward function rP that is identical to rG , except that it assigns low reward to the mediocre output ymed . Theorem 2 establishes that the time required to achieve high ground truth reward when maximizing rG directly can be arbitrarily larger than when maximizing rP . This gap in optimization time arises because, when maximizing rG , the policy first concentrates its probability mass on ymed and stalls in its vicinity. By contrast, when maximizing rP , the policy directly shifts its probability mass towards y⋆ . Theorem 2 (Detailed version of Theorem 1). Suppose that the feature vectors {ϕ(y)}y∈Y are orthonormal and that the ground truth reward function rG and initial policy parameters θ0 uphold Assumptions 1 and 2. Furthermore, let rP be a proxy reward function identical to rG , except that it assigns ymed a low reward rP (ymed ) = miny∈Ybad rG (y). For any ϵ ∈ (0, ∆1 ), where ∆1 := rG (y⋆ ) − rG (ymed ), the following hold. 18
• Case I: Maximizing rG . If gradient flow is used to maximize the expected reward with respect to rG (Equation (1) with VG in place of VP ), then the initial time at which VG (θt ) ≥ rG (y⋆ ) − ϵ, denoted t⋆ , is lower bounded as follows: √ √ 2 2M 14/13 ∆1 rG (ymed )(1 − e− 2(∆1 −ϵ) ) t⋆ ≥ · πθ0 (y⋆ )−14/13 = Ω πθ0 (y⋆ )−14/13 , (∆1 + ∆2 )(1 + 8∆1 ) where ∆2 := rG (ymed ) − rG (ybad ) for some ybad ∈ Ybad and M is as defined in Assumption 2. • Case II: Maximizing rP . In contrast, denote by tno−med the initial time at which VG (θt ) ≥ ⋆ rG (y⋆ ) − ϵ when gradient flow is used to maximize the expected reward with respect to rP . Then, tno−med is upper bounded by a quantity that grows asymptotically slower as πθ0 (y⋆ ) → 0 ⋆ compared to the lower bound on t⋆ in Case I: tno−med ≤ ⋆
2 (rG (y⋆ ) + 1) · πθ0 (y⋆ )−1 = O πθ0 (y⋆ )−1 , 2 ϵ (rP (y⋆ ) − VP (θ0 ))
Furthermore, at any time t greater than or equal to the right-hand side in the inequality above, we have πθt (y⋆ ) ≥ 1 − ϵ/(∆1 + ∆2 ). Thus, for any arbitrarily large T ≥ 0, if πθ0 (y⋆ ) is sufficiently small, then t⋆ − tno−med ≥ T. ⋆ A proof sketch is provided after the statement of Theorem 1; see Appendix B.4 for the full proof. Remark 1. Recently, [68, 69] identified that if the reward function r used for training induces low reward variance for the initial policy, i.e., if Vary∼πθ0 [r(y)] ≈ 0, then policy gradient suffers from slow optimization (see Theorem 1 in [69]). It is therefore natural to wonder whether rP facilitates faster optimization than rG in Theorem 2 due to inducing a higher initial reward variance. Interestingly, the opposite is true: the initial reward variance is actually higher under rG . The slow optimization under rG is not caused by low initial reward variance, but rather by a collapse in reward variance during training, as the policy becomes concentrated on the mediocre output ymed . Although rP initially induces lower reward variance than rG , it guides the policy directly towards y⋆ , avoiding this intermediate variance collapse. Formal statement behind Harmful Error II. As a corollary of Theorem 2, by swapping the proxy and ground truth reward functions, we obtain that assigning mediocre proxy reward to an output with low ground truth reward can be harmful due to impeding optimization, even if it does not induce reward hacking. That is, Corollary 1 formalizes Harmful Error II from Section 3.4. Corollary 1. Consider the setting and assumptions of Theorem 2, with Assumptions 1 and 2 interpreted with rP in place of rG . Furthermore, let rG be identical to rP , except that it assigns ymed a low reward rG (ymed ) = miny∈Ybad rP (y). For any ϵ ∈ (0, ∆1 ), where ∆1 := rP (y⋆ ) − rP (ymed ), the following hold. • Case I: Maximizing rP . If gradient flow is used to maximize the expected reward with respect to rP (Equation (1)), then the initial time at which VG (θt ) ≥ rG (y⋆ ) − ϵ, denoted t⋆ , is lower bounded as follows: √ √ 2 2M 14/13 ∆1 rP (ymed )(1 − e− 2(∆1 −ϵ) ) · πθ0 (y⋆ )−14/13 = Ω πθ0 (y⋆ )−14/13 , t⋆ ≥ (∆1 + ∆2 )(1 + 8∆1 ) where ∆2 := rP (ymed ) − rP (ybad ) for some ybad ∈ Ybad and M is as defined in Assumption 2. • Case II: Maximizing rG . In contrast, denote by tno−med the initial time at which VG (θt ) ≥ ⋆ rG (y⋆ ) − ϵ when gradient flow is used to maximize the expected reward with respect to rG (Equation (1) with VG in place of VP ). Then, tno−med is upper bounded by a quantity that grows ⋆ asymptotically slower as πθ0 (y⋆ ) → 0 compared to the lower bound on t⋆ in Case I: tno−med ≤ ⋆
2 (rG (y⋆ ) + 1) · πθ0 (y⋆ )−1 = O πθ0 (y⋆ )−1 , 2 ϵ (rG (y⋆ ) − VG (θ0 ))
Furthermore, at any time t greater than or equal to the right-hand side in the inequality above, we have πθt (y⋆ ) ≥ 1 − ϵ/(∆1 + ∆2 ). 19
Thus, for any arbitrarily large T ≥ 0, if πθ0 (y⋆ ) is sufficiently small, then t⋆ − tno−med ≥ T. ⋆ Proof. The result follows by applying Theorem 2 while swapping the roles of rG and rP . For Case I (i.e., when maximizing rP ), Theorem 2 yields a lower bound on the time until VP (θt ) ≥ rP (y⋆ ) − ϵ. Since rG (y⋆ ) = rP (y⋆ ) and VG (θ) ≤ VP (θ) for all θ (recall, rG is identical to rP , except that it assigns a lower reward to ymed ), this immediately implies that the same lower bound holds for t⋆ —the time until VG (θt ) ≥ rG (y⋆ ) − ϵ. For Case II (i.e., when maximizing rG ), Theorem 2 implies that πθt (y⋆ ) ≥ 1 − ϵ/(∆1 + ∆2 ) at any time t greater than or equal to the desired upper bound on tno−med . Hence, because ground truth rewards lie within the interval [rG (ybad ), rG (y⋆ )] and ⋆ ∆1 + ∆2 = rG (y⋆ ) − rG (ybad ), where ybad ∈ Ybad , at any such time VG (θt ) ≥ rG (y⋆ ) − ϵ. This establishes the upper bound on tno−med . ⋆ A.1
Influence of Feature Similarity
We now analyze how the similarity (or dissimilarity) of output features affects the extent to which mediocre outputs impede optimization. Namely, under conditions analogous to those of Theorem 2, we characterize how the conclusion of Theorem 2 changes when ϕ(y⋆ ) and ϕ(ymed ) are not orthogonal (recall that y⋆ is the output with maximal ground truth reward and ymed is an output with mediocre ground truth reward). The analysis reveals that when ⟨ϕ(y⋆ ), ϕ(ymed )⟩ < 0, the mediocre output ymed delays optimization even further and can prevent the policy from ever reaching the optimal output y⋆ . Conversely, when ⟨ϕ(y⋆ ), ϕ(ymed )⟩ > 0, attraction to ymed does not necessarily impede optimization and, if ϕ(y⋆ ) and ϕ(ymed ) are extremely similar, assigning ymed a mediocre proxy reward may be required for enabling convergence to y⋆ . Appendices A.1.1 and A.1.2 deliver the formal results for the cases of negative and positive inner products, respectively. A.1.1
Negative Inner Product Between ϕ(y⋆ ) and ϕ(ymed )
We consider the ground truth reward function rG described in Assumption 1 (i.e., the ground truth from the orthonormal features setting considered in Theorem 2) and make the following Assumption 3 on the initial policy probabilities. Assumption 3 is identical to Assumption 2 up to constants that depend on the norms of feature vectors. Similar to Assumption 2, Assumption 3 guarantees that rG (ymed ) > VG (θ0 ) (see Lemma 15) and that πθ0 (ymed ) is substantially higher than πθ0 (y⋆ ). Assumption 3 (Initial policy). The initial policy πθ0 satisfies the following conditions. • The initial probability of y⋆ is not too high: 7/313/14 2 ∥ϕ(ymed )∥ 21∥ϕ(ymed )∥ ′ πθ0 (y⋆ ) < min M · min 0.01, , πθ0 (ymed ) , 1100B 2 (∆1 + ∆2 ) ∥ϕ(y⋆ )∥ where ∆1 := rG (y⋆ ) − rG (ymed ), ∆2 := rG (ymed ) − rG (ybad ) for some ybad ∈ Ybad , B := 2 maxy∈Y ∥ϕ(y)∥, and M ′ := min 1, ∥ϕ(ymed )∥ rG (ymed )2/7 (40B 2 (∆1 + ∆2 ))−1 . • The initial probability of Ybad is neither too low nor too high: 2πθ0 (y⋆ )7/13 max , 0.05 · r (y ) ≤ πθ0 (Ybad ) ≤ 0.1 · rG (ymed ) . G med ∆2 M ′7/13 We focus here on the case of ⟨ϕ(y⋆ ), ϕ(ymed )⟩ < 0. To isolate the effect of ⟨ϕ(y⋆ ), ϕ(ymed )⟩ on the optimization dynamics from interactions between other feature vectors, we require that, for any output in Ybad , its feature vector is orthogonal to the feature vectors of all other outputs. We also make the mild assumption that ∥ϕ(ymed )∥ ≥ ∥ϕ(ybad )∥ for all ybad ∈ Ybad . Assumption 4 (Output feature vectors). The feature vectors of y⋆ and ymed have negative inner product, i.e., ⟨ϕ(y⋆ ), ϕ(ymed )⟩ < 0, and ⟨ϕ(ybad ), ϕ(y)⟩ = 0 for all ybad ∈ Ybad , y ∈ Y \ {ybad }. Furthermore, ∥ϕ(ymed )∥ ≥ ∥ϕ(ybad )∥ for all ybad ∈ Ybad . If we maximize rG directly, ymed attracts probability at the expense of y⋆ even more strongly than in the case of orthonormal feature vectors. The reason for this stronger attraction is that, as long as the advantage AG (ymed ; θt ) := rG (ymed ) − VG (θt ) is positive, ⟨ϕ(y⋆ ), ϕ(ymed )⟩ pushes the logit of y⋆ downwards (see Proposition 1). As Theorem 3 shows, this yields a larger separation between 20
t⋆ —the time needed to achieve high ground truth reward when maximizing rG directly—and the corresponding time tno−med when maximizing a proxy reward function rP that assigns low reward to ⋆ ymed . In particular, the lower bound in Theorem 3 on t⋆ grows asymptotically faster as πθ0 (y⋆ ) → 0 compared to the lower bound in Theorem 2. On the other hand, the upper bound on tno−med can ⋆ remain roughly the same since AP (ymed ; θt ) < 0 throughout optimization when ymed is assigned a low proxy reward, in which case ⟨ϕ(y⋆ ), ϕ(ymed )⟩ only pushes the logit of y⋆ further upwards. Theorem 3. Suppose that the ground truth reward function rG , initial policy parameters θ0 , and output feature vectors {ϕ(y)}y∈Y uphold Assumptions 1, 3, and 4. Let rP be a proxy reward function identical to rG , except that it assigns ymed a low reward rP (ymed ) = miny∈Ybad rG (y). For any ϵ ∈ (0, ∆1 ), where ∆1 := rG (y⋆ ) − rG (ymed ), the following hold. • Case I: Maximizing rG . If gradient flow is used to maximize the expected reward with respect to rG (Equation (1) with VG in place of VP ), then the initial time at which VG (θt ) ≥ rG (y⋆ ) − ϵ, denoted t⋆ , is lower bounded as follows: rG (ymed )M ′14/13+α (1 − e−2(∆1 −ϵ) ) −14/13−α −14/13−α t⋆ ≥ · π (y ) , = Ω π (y ) θ ⋆ θ ⋆ 0 0 4B 2 (∆1 + ∆2 )K where ∆2 := rG (ymed ) − rG (ybad ) for some ybad ∈ Ybad . The constants B and M ′ are asde2 fined in Assumption 3, α := − ⟨ϕ(y⋆ ), ϕ(ymed )⟩ / 26(∥ϕ(ymed )∥ − 0.5 ⟨ϕ(y⋆ ), ϕ(ymed )⟩) ∈ (0, 1/13), and 13α/7 26 ∥ϕ(ymed )∥2 πθ0 (ymed )2 K := 1 + . rG (ymed )2 8(∥ϕ(y⋆ )∥2 − ⟨ϕ(y⋆ ), ϕ(ymed )⟩)AG (y⋆ ; θ0 ) • Case II: Maximizing rP . In contrast, denote by tno−med the initial time at which VG (θt ) ≥ ⋆ rG (y⋆ ) − ϵ when gradient flow is used to maximize the expected reward with respect to rP . Then, tno−med is upper bounded by a quantity that grows asymptotically slower as πθ0 (y⋆ ) → 0 ⋆ compared to the lower bound on t⋆ in Case I: tno−med ≤ ⋆
2 (rG (y⋆ ) + 1) −1 = O πθ0 (y⋆ )−1 , 2 · πθ0 (y⋆ ) 2 ϵ rP (y⋆ ) − VP (θ0 ) ∥ϕ(y⋆ )∥
Thus, for any arbitrarily large T > 0, if πθ0 (y⋆ ) is sufficiently small, then t⋆ − tno−med ≥ T. ⋆ Proof sketch (full proof in Appendix B.5). The proof follows a line similar to that of Theorem 2. The key difference is that, due to the negative inner product between ϕ(y⋆ ) and ϕ(ymed ), the logit dynamics of y⋆ now contains an additional term that depends on ⟨ϕ(y⋆ ), ϕ(ymed )⟩. Specifically, when maximizing rG directly (Case I), the logit dynamics of y⋆ is given by (cf. Proposition 1): d 2 ⟨ϕ(y⋆ ), θt ⟩ = πθt (y⋆ )AG (y⋆ ; θt ) · ∥ϕ(y⋆ )∥ + πθt (ymed )AG (ymed ; θt ) · ⟨ϕ(y⋆ ), ϕ(ymed )⟩ . dt Thus, as long as AG (ymed ; θt ) > 0, the second term on the right-hand side is negative and pushes the probability of y⋆ further downwards compared to the case of orthonormal feature vectors. We show that this implies that the policy will become even more concentrated on ymed , and so will take a longer time to escape the vicinity of ymed . On the other hand, when maximizing rP (Case II), the advantage AP (ymed ; θt ) is negative throughout optimization since rP assigns ymed a low reward. Hence, the contribution of ymed to the logit dynamics of y⋆ can only push the logit of y⋆ further upwards compared to the orthonormal feature vectors case. This straightforwardly allows deriving an upper bound on tno−med that is roughly the same as the one in Theorem 2. ⋆ When ⟨ϕ(y⋆ ), ϕ(ymed )⟩ < 0 the policy may fail to ever assign high probability to y⋆ . Theorem 3 indicates that the lower the inner product between ϕ(y⋆ ) and ϕ(ymed ) is, the longer the policy stalls in the vicinity of the mediocre output ymed when maximizing rG . One may wonder if the policy always escapes the attraction of ymed and reaches y⋆ as optimization progresses. We show that the answer is no. For certain initial policies and output feature vectors, Proposition 3 proves that the policy never achieves a ground truth reward above rG (ymed ), and thus never assigns high probability to y⋆ . Notably, this failure occurs despite the existence of policy parameters that assign y⋆ a probability arbitrarily close to one. Proposition 3 is obtained by adapting Proposition 13 from [42] to our setting. 21
Proposition 3. Suppose that the ground truth reward function rG upholds Assumption 1. Furthermore, let ybad ∈ Ybad , define ∆1 := rG (y⋆ ) − rG (ymed ) and ∆2 := rG (ymed ) − rG (ybad ), and assume that the initial policy parameters θ0 and output feature vectors {ϕ(y)}y∈Y satisfy the following conditions, alongside Assumption 4. • πθ0 (ymed ) > πθ0 (y) for all y ∈ Y \ {ymed }; • πθ0 (y⋆ ) ≤
∆2 πθ0 (Ybad ) ; ∆1
2 • and θ0 = C · (ϕ(ybad ) − ϕ(y⋆ )) for some C > max 0, − ln(ζ)/∥ϕ(ybad ) − ϕ(y⋆ )∥ , where ζ :=
ln πθ0 (ymed ) − ln πθ0 (ybad ) VG (θ0 ) − rG (ybad ) · > 0. ln πθ0 (ymed ) − ln πθ0 (y⋆ ) rG (y⋆ ) − VG (θ0 )
If gradient flow is used to maximize the expected reward with respect to rG (Equation (1) with VG in place of VP ), then VG (θt ) < rG (ymed ) for all t ≥ 0. Proof sketch (full proof in Appendix B.6). At initialization, the policy favors ybad over y⋆ in the sense that πθ0 (ybad ) > πθ0 (y⋆ ) and πθ0 (y⋆ )/πθ0 (ybad ) < ζ. The proof follows by showing that the suppressive effect of ymed on y⋆ , which stems from ymed having a positive initial advantage and the inner product of ϕ(y⋆ ) and ϕ(ymed ) being negative, is strong enough to ensure that the policy maintains this relationship between ybad and y⋆ throughout optimization. In other words, the policy never assigns substantially more probability to y⋆ than to ybad . This in turn implies that the expected ground truth reward remains below rG (ymed ) for all t ≥ 0. Remark 2. Consider the policy parameters θ(β) = β · ϕ(y⋆ ), for β ∈ R. Under the conditions of Proposition 3 on {ϕ(y)}y∈Y , taking β → ∞ implies πθ(β) (y⋆ ) → 1 since ⟨ϕ(y⋆ ), ϕ(ymed )⟩ < 0 and ⟨ϕ(y⋆ ), ϕ(y)⟩ = 0 for all y ∈ Ybad . Thus, as mentioned in the text preceding Proposition 3, this result shows that even though there exist policy parameters that assign y⋆ a probability arbitrarily close to one, policy gradient fails to reach them. Remark 3. For completeness, we provide an example of output feature vectors satisfying Assumption 4 for which the conditions in Proposition 3 hold. Consider for simplicity a case with only three outputs, y⋆ , ymed , and ybad . Let p ϕ(y⋆ ) = e1 , ϕ(ymed ) = −ρ · e1 + 1 − ρ2 · e2 , ϕ(ybad ) = λ · e3 , for 0 < λ2 < ρ < 1 and orthonormal vectors e1 , e2 , e3 ∈ RD . In this case, the inner products between the feature vectors are given by: ⟨ϕ(y⋆ ), ϕ(ymed )⟩ = −ρ < 0
,
⟨ϕ(ybad ), ϕ(ymed )⟩ = ⟨ϕ(ybad ), ϕ(y⋆ )⟩ = 0 .
Thus, taking θ0 = C · (ϕ(ybad ) − ϕ(y⋆ )) for some C > 0, the initial logits assigned to y⋆ , ymed , and ybad by θ0 are: ⟨ϕ(y⋆ ), θ0 ⟩ = −C
,
⟨ϕ(ymed ), θ0 ⟩ = Cρ
,
⟨ϕ(ybad ), θ0 ⟩ = Cλ2 .
This implies that πθ0 (ymed ) > πθ0 (ybad ) > πθ0 (y⋆ ). In particular, ymed has the highest initial probability, as required by the first condition in Proposition 3. For the second and third conditions, notice that: ζ := and
ρ − λ2 VG (θ0 ) − rG (ybad ) ln πθ0 (ymed ) − ln πθ0 (ybad ) VG (θ0 ) − rG (ybad ) · = · >0 ln πθ0 (ymed ) − ln πθ0 (y⋆ ) rG (y⋆ ) − VG (θ0 ) ρ+1 rG (y⋆ ) − VG (θ0 ) πθ0 (y⋆ ) = exp −C(1 + λ2 ) . πθ0 (ybad )
2 Hence, for any C > max 0, − ln(ζ)/∥ϕ(ybad ) − ϕ(y⋆ )∥ , ln(∆1 /∆2 ) we get that: πθ0 (y⋆ ) ≤
∆2 πθ0 (ybad ) , ∆1
meaning all conditions in Proposition 3 are satisfied. 22
A.1.2
Positive Inner Product Between ϕ(y⋆ ) and ϕ(ymed )
Moving to the case where ⟨ϕ(y⋆ ), ϕ(ymed )⟩ > 0, we again consider the ground truth reward function described in Assumption 1 and impose the same assumptions on the initial policy as in the negative inner product case (i.e., Assumption 3). Furthermore, we make the following assumption on the structure of output features, analogous to Assumption 4. Assumption 5 (Output feature vectors). The feature vectors of y⋆ and ymed have positive inner product, i.e., ⟨ϕ(y⋆ ), ϕ(ymed )⟩ > 0, and ⟨ϕ(ybad ), ϕ(y)⟩ = 0 for all ybad ∈ Ybad , y ∈ Y \ {ybad }. Suppose that we maximize rG directly. In contrast to the case of ⟨ϕ(y⋆ ), ϕ(ymed )⟩ < 0, when the advantage AG (ymed ; θt ) = rG (ymed ) − VG (θt ) is positive, the mediocre output ymed contributes a positive term πθt (ymed )AG (ymed ; θt ) · ⟨ϕ(y⋆ ), ϕ(ymed )⟩ to the logit dynamics of y⋆ (see Proposition 1). Intuitively, this implies that the attraction to ymed need not impede optimization: even if probability mass initially moves towards ymed , this movement can simultaneously facilitate the growth of πθt (y⋆ ). However, once VG (θt ) exceeds rG (ymed ), the advantage of ymed becomes negative. As a result, the contribution of ymed to the logit of y⋆ reverses sign and starts pushing it downwards. This creates an intricate dynamics, where to enable the growth of πθt (y⋆ ) towards one, 2 the contribution of y⋆ to its own logit dynamics, πθt (y⋆ )AG (y⋆ ; θt ) · ∥ϕ(y⋆ )∥ , needs to overcome the negative contribution of ymed . 2
To enable a clean characterization, we focus on a regime where ∥ϕ(y⋆ )∥ > ⟨ϕ(y⋆ ), ϕ(ymed )⟩ and 2 ⟨ϕ(y⋆ ), ϕ(ymed )⟩ > ∥ϕ(ymed )∥ , leaving treatment of other regimes to future work. As established in Theorem 4, under this condition the mediocre output ymed does not harm optimization. Specifically, the time needed to achieve high ground truth reward when maximizing rG directly has the same asymptotic dependence on πθ0 (y⋆ ) as the upper bounds in Theorems 2 and 3, which apply to cases where ymed is assigned low proxy reward. Furthermore, if ϕ(y⋆ ) and ϕ(ymed ) are highly similar, in the sense that their inner product is above a certain threshold, then assigning ymed a mediocre reward can be necessary for optimization to succeed. In such cases, we prove that if one maximizes a proxy reward function rP that assigns low reward to ymed , the policy fails to achieve near-optimal ground truth reward, and thus never assigns high probability to y⋆ . Remark 4. Theorem 4 implies that, when the feature vectors of y⋆ and ymed are highly similar, the type of reward error caused by assigning ymed a low proxy reward changes from beneficial to harmful. By the same reasoning, in the case where ymed has low ground truth reward, assigning it a mediocre proxy reward is beneficial rather than harmful. Theorem 4. Suppose that the ground truth reward function rG , initial policy parameters θ0 , and output feature vectors {ϕ(y)}y∈Y uphold Assumptions 1, 3, and 5. Furthermore, assume that 2 2 2 ∥ϕ(ymed )∥ < ⟨ϕ(y⋆ ), ϕ(ymed )⟩ < ∥ϕ(y⋆ )∥ < 100∥ϕ(ymed )∥ , and let rP be a proxy reward function identical to rG , except that it assigns ymed a low reward rP (ymed ) ≤ miny∈Ybad rG (y). For any ϵ ∈ (0, ∆1 ), where ∆1 := rG (y⋆ ) − rG (ymed ), the following hold. • Case I: Maximizing rG . If gradient flow is used to maximize the expected reward with respect to rG (Equation (1) with VG in place of VP ), then the initial time at which VG (θt ) ≥ rG (y⋆ ) − ϵ, denoted t⋆ , is upper bounded as follows: t⋆ ≤
(rG (y⋆ ) + 1)
2
ϵ2 ∆1 ∥ϕ(y⋆ ) − ϕ(ymed )∥
2 · πθ0 (y⋆ )
−1
= O πθ0 (y⋆ )−1 .
• Case II: Maximizing rP . In contrast, if gradient flow is used to maximize the expected reward with respect to rP and ⟨ϕ(y⋆ ), ϕ(ymed )⟩ is sufficiently high, in the sense that 2
⟨ϕ(y⋆ ), ϕ(ymed )⟩ >
2
πθ0 (y⋆ )AP (y⋆ ; θ0 )∥ϕ(y⋆ )∥ − πθ0 (ymed )AP (ymed ; θ0 )∥ϕ(ymed )∥ , πθ0 (y⋆ )AP (y⋆ ; θ0 ) − πθ0 (ymed )AP (ymed ; θ0 )
then VG (θt ) < rG (y⋆ ) − (rG (y⋆ ) − rG (ymed ))πθ0 (ymed ) for all t ≥ 0. Proof sketch (full proof in Appendix B.7). When maximizing rG directly (Case I), the logit dynamics of y⋆ is given by (cf. Proposition 1): d 2 ⟨ϕ(y⋆ ), θt ⟩ = πθt (y⋆ )AG (y⋆ ; θt ) · ∥ϕ(y⋆ )∥ + πθt (ymed )AG (ymed ; θt ) · ⟨ϕ(y⋆ ), ϕ(ymed )⟩ . dt 23
Output Probability
πθt (y )
1.0
®
φ(y ), φ(ymed ) = 0 1.0
0.5 0.0
πθt (ymed )
πθt (other outputs) ®
1.0
0.5
0
10000
20000
Training Step
30000
0.0
φ(y ), φ(ymed ) < 0
®
φ(y ), φ(ymed ) > 0
0.5
0
10000
20000
Training Step
30000
0.0
0
10000
20000
Training Step
30000
Figure 5: Feature similarity affects whether mediocre outputs impede policy gradient optimization. Plotted is the evolution of output probabilities during policy gradient in settings corresponding to Theorems 2, 3, and 4 (left to right plots). We train linear softmax policies using exact gradients of the expected ground truth reward rG , which assigns a maximal reward of 1 to y⋆ , a mediocre reward of 0.8 to ymed , and a low reward of −1 to the remaining outputs. In all experiments, initial policy probabilities and feature vector norms are identical, with πθ0 (y⋆ ) = 0.05, πθ0 (ymed ) = 0.5, ∥ϕ(y⋆ )∥ = 1.5, and ∥ϕ(ymed )∥ = 1. The feature vectors of other outputs (i.e., not y⋆ and ymed ) have unit norm and are mutually orthogonal, as well as orthogonal to ϕ(y⋆ ) and ϕ(ymed ). What √ varies across the experiments √ is the value of ⟨ϕ(y⋆ ), ϕ(ymed )⟩, which is equal to 0 in the leftmost plot, −3/2 2 in the middle plot, and 3/2 2 in the rightmost plot. In line with our theory, when ϕ(y⋆ ) and ϕ(ymed ) are orthogonal or have a negative inner product, the policy initially concentrates its probability mass on ymed and stagnates, with the attraction to ymed being stronger in the negative inner product case. By contrast, when ϕ(y⋆ ) and ϕ(ymed ) have a positive inner product, the attraction to ymed is milder and does not obstruct the policy from assigning high probability to y⋆ . See Appendix D for additional implementation details.
The advantage of the mediocre output ymed is initially positive, i.e., AG (ymed ; θ0 ) > 0. Thus, since ⟨ϕ(y⋆ ), ϕ(ymed )⟩ > 0, whenever ymed attracts probability it also pushes the logit of y⋆ upwards. In particular, the conditions on the output feature vectors ensure that πθt (y⋆ ) is monotonically nondecreasing, as opposed to the orthonormal feature vectors case (Theorem 2) where πθt (y⋆ ) initially decreases due to the increase in the logit of ymed . As a result, the policy does not become highly concentrated on ymed and t⋆ can be upper bounded by a quantity that has the same asymptotic dependence on πθ0 (y⋆ ) as the upper bound in Theorem 2, which applies to cases where ymed is assigned low proxy reward. On the other hand, when maximizing rP (Case II), the advantage of ymed is negative throughout optimization since rP assigns ymed a low reward. Hence, ymed contributes negatively to the logit dynamics of y⋆ . We show that this suppressive effect of ymed on y⋆ prevents the policy from assigning y⋆ a higher probability than ymed . Specifically, πθt (y⋆ )/πθt (ymed ) is monotonically non-increasing for all t ≥ 0, and initially πθ0 (y⋆ ) < πθ0 (ymed ). This in turn implies that the expected ground truth reward is bounded away from rG (y⋆ ) even as t → ∞.
A.1.3
Empirical Demonstration
Figure 5 empirically demonstrates how the similarity between output features affects the extent to which mediocre outputs impede optimization. Consistent with our theoretical analysis (Theorems 2, 3, and 4), when ϕ(y⋆ ) and ϕ(ymed ) have a negative inner product, attraction to ymed is more severe than in the orthogonal case, causing the policy to stall near ymed for longer. Conversely, when ϕ(y⋆ ) and ϕ(ymed ) have a positive inner product, the attraction to ymed is milder and does not obstruct the policy from assigning high probability to y⋆ .
B
Deferred Proofs
B.1
Notation
We use AG (y; θ) := rG (y) − VG (θ) to denote the advantage of an output y ∈ Y under the ground truth reward function rG and the policy πθ . Similarly, we let AP (y; θ) := rP (y) − VP (θ) denote the advantage of y under rP and πθ . Furthermore, ∥·∥ denotes the Euclidean norm and ∥·∥1 the ℓ1 norm. For matrices, ∥·∥2 stands for the spectral norm. 24
B.2
Proof of Proposition 1
Fix an output y ∈ Y. By Equation (1) and the chain rule we have that d d ⟨ϕ(y), θt ⟩ = ϕ(y), dt θt = ⟨ϕ(y), ∇VP (θt )⟩. dt P By Lemma 10 we know that ∇VP (θ) = z∈Y πθ (z)AP (z; θ) · ϕ(z). Plugging this gradient expression into the logit dynamics above yields: D E X X d ⟨ϕ(y), θt ⟩ = ϕ(y), πθt (z)AP (z; θt ) · ϕ(z) = πθt (z)AP (z; θt )⟨ϕ(z), ϕ(y)⟩. z∈Y z∈Y dt Separating the term corresponding to z = y from the rest concludes the proof: X d 2 ⟨ϕ(y), θt ⟩ = πθt (y)AP (y; θt ) · ∥ϕ(y)∥ + πθt (z)AP (z; θt ) · ⟨ϕ(z), ϕ(y)⟩. z∈Y\{y} dt
B.3
Proof of Proposition 2
If ∆Z = 0, then rP = rG , and so trivially θt = θtG and TV πθt , πθtG = 0 for all t ≥ 0. Hence, in the remainder of the proof we assume ∆Z > 0. We begin by showing that πθT (Z) remains low until time T , regardless of which reward function is used for optimization. Lemma 1. Suppose that gradient flow is used to maximize the expected reward with respect to r : Y → [−1, 1], starting from θ0 . Then, for all t ∈ [0, T ] it holds that: πθt (Z) ≤ πθ0 (Z) · exp 4B 2 T ≤
2 · ϵ. ∆Z exp(6B 2 T )
Proof. For simplicity of notation, let fy (θ) := ⟨ϕ(y), θ⟩ denote the logit of y ∈ Y under the policy parameters θ. By Proposition 1, the fact that rewards are bounded within [−1, 1], and the d fy (θt ), for any y ∈ Y and t ∈ [0, T ], as follows: Cauchy-Schwarz inequality, we can upper bound dt d fy (θt ) = dt =
d ϕ(y), dt θt
X
πθt (z)(r(z) − V (θt ))⟨ϕ(y), ϕ(z)⟩
z∈Y
≤2
X
≤2
X
z∈Y z∈Y
≤ 2B 2
πθt (z)|⟨ϕ(y), ϕ(z)⟩|
(2)
πθt (z)∥ϕ(y)∥∥ϕ(z)∥
X z∈Y
πθt (z)
2
= 2B , where V (θ) := Ez∼πθ [r(z)] is the expected reward with respect to r and πθ , and recall that B = d maxy∈Y ∥ϕ(y)∥. This leads to an upper bound on dt πθt (y): d d πθt (y) = πθt (y) ln πθt (y) dt dt
exp(fu (θt )) u∈Y d d = πθt (y) dt fy (θt ) − Eu∼πθt dt fu (θt ) d d ≤ πθt (y) dt fy (θt ) + Eu∼πθt dt fu (θt ) d d ≤ πθt (y) dt fy (θt ) + Eu∼πθt dt fu (θt ) = πθt (y)
X
d d dt fy (θt ) − dt ln
≤ 4B 2 πθt (y). 25
Here, in the first and second inequalities we apply the triangle inequality, and the last inequality is from Equation (2). Summing over all y ∈ Z, we therefore have that for all t ∈ [0, T ]: d πθ (Z) ≤ 4B 2 πθt (Z). dt t By Grönwall’s inequality, this implies that for all t ∈ [0, T ]: 2 · ϵ, πθt (Z) ≤ πθ0 (Z) · exp(4B 2 t) ≤ πθ0 (Z) · exp(4B 2 T ) ≤ ∆Z exp(6B 2 T ) 2 where the last transition is by the assumption that πθ0 (Z) ≤ ∆Z exp(10B 2T ) · ϵ
With Lemma 1 in place, we turn to bound the deviation between θt and θtG at time t ∈ [0, T ]. Examining their time derivatives, by the triangle inequality we get that d G d dt θt − dt θt
= ∥∇VG (θtG ) − ∇VP (θt )∥ ≤ ∥∇VG (θtG ) − ∇VG (θt )∥ + ∥∇VG (θt ) − ∇VP (θt )∥ . {z } | {z } | I1
I2
2
For I1 , Lemma 3 shows that ∇VG is 6B -Lipschitz, for B = maxy∈Y ∥ϕ(y)∥. Thus, I1 ≤ 6B 2 θtG − θt . For I2 , by examining the gradient expression derived in Lemma 10 we can see that ∇VG (θt ) − ∇VP (θt ) = Ey∼πθt [(rG (y) − VG (θt ) − rP (y) + VP (θt )) · ϕ(y)] = Ey∼πθt [(rG (y) − rP (y)) · ϕ(y)] − (VG (θt ) − VP (θt )) · ϕ̄θt = Ey∼πθt [(rG (y) − rP (y)) · ϕ(y)] − Ey∼πθt (rG (y) − rP (y)) · ϕ̄θt = Ey∼πθt (rG (y) − rP (y)) · ϕ(y) − ϕ̄θt , where ϕ̄θt := Ez∼πθt [ϕ(z)]. By the triangle inequality and the fact that rG (y) = rP (y) for all y ∈ Y \ Z, this leads to the following upper bound on I2 : I2 = ∥∇VG (θt ) − ∇VP (θt )∥ ≤ Ey∼πθt |rG (y) − rP (y)| · ϕ(y) − ϕ̄θt ≤ 2B∆Z πθt (Z), where ∆Z = maxy∈Z |rG (y) − rP (y)|. Thus, applying the bound from Lemma 1 on πθt (Z), we arrive at: 4Bϵ . I2 ≤ exp(6B 2 T ) Combining the upper bounds on I1 and I2 , we get: 4Bϵ 2 d G d θtG − θt + . (3) dt θt − dt θt ≤ 6B exp(6B 2 T ) Since h(t) := θtG − θt is absolutely continuous with respect to t over [0, T ], so is ∥h(t)∥. As a result, for almost every t ∈ [0, T ], d G d G d θ − θt ≤ dt θt − dt θt . dt t Thus, by Equation (3), for almost every t ∈ [0, T ] it holds that d G 4Bϵ ∥θt − θt ∥ ≤ 6B 2 ∥θtG − θt ∥ + . dt exp(6B 2 T ) Grönwall’s inequality, along with the fact that θ0G = θ0 , then yields for all t ∈ [0, T ]: ∥θtG − θt ∥ ≤
4Bϵ exp(6B 2 T ) − 1 2ϵ ϵ · ≤ ≤ . 2 2 exp(6B T ) 6B 3B B
Finally, since TV(πθ , πθ′ ) ≤ B∥θ − θ′ ∥ (Lemma 2) for any θ, θ′ ∈ RD , we may conclude that for all t ∈ [0, T ]: TV πθt , πθtG ≤ B∥θtG − θt ∥ ≤ ϵ.
26
B.3.1
Auxiliary Lemmas for the Proof of Proposition 2
Lemma 2. For any θ, θ′ ∈ RD it holds that TV(πθ , πθ′ ) ≤ B∥θ − θ′ ∥, where B = maxy∈Y ∥ϕ(y)∥. Proof. Define the curve θ(s) := θ′ + s · (θ − θ′ ) for s ∈ [0, 1]. By the fundamental theorem of calculus it holds that Z 1
J(θ(s))(θ − θ′ ) ds,
πθ − π θ ′ = 0
⊤
where J(ξ) is the Jacobian of πξ at ξ ∈ RD , with rows J(ξ)y,: = (∇πξ (y)) . Taking ℓ1 norms and using the induced operator norm ∥ · ∥2→1 gives Z 1 Z 1 ′ ′ ∥πθ − πθ′ ∥1 ≤ ∥J(θ(s))(θ − θ )∥1 ds ≤ ∥θ − θ ∥ · ∥J(θ(s))∥2→1 ds. (4) 0
0
D
It therefore suffices to show that ∥J(ξ)∥2→1 ≤ 2B for all ξ ∈ R . For linear softmax policies, ∇πξ (y) = πξ (y)(ϕ(y) − ϕ̄ξ ), where ϕ̄ξ :=
P
y∈Y πξ (y)ϕ(y). Furthermore, from the definition of norm ∥ · ∥2→1 , we have:
∥J(ξ)∥2→1 = sup∥v∥=1 ∥J(ξ)v∥1 = sup∥v∥=1
X y∈Y
|⟨∇πξ (y), v⟩|,
and we can upper bound |⟨∇πξ (y), v⟩|, for y ∈ Y and v ∈ RD with ∥v∥ = 1, by: |⟨∇πξ (y), v⟩| = πξ (y) ϕ(y) − ϕ̄ξ , v ≤ πξ (y) · ϕ(y) − ϕ̄ξ · ∥v∥ ≤ πξ (y) · 2B. The first inequality is from the Cauchy-Schwarz inequality and the second is due to ∥ϕ(y)∥ ≤ B for all y ∈ Y. Summing over all y ∈ Y then yields: X X |⟨∇πξ (y), v⟩| ≤ 2Bπξ (y) = 2B. y∈Y
y∈Y
Since this holds for all v ∈ RD with ∥v∥ = 1, we get the desired bound on ∥J(ξ)∥2→1 : ∥J(ξ)∥2→1 ≤ 2B. Plugging this into Equation (4) concludes the proof: 1 1 TV πθ , πθ′ = ∥πθ − πθ′ ∥1 ≤ ∥θ − θ′ ∥ · 2 2
Z 1
2B ds = B∥θ − θ′ ∥.
0
Lemma 3. For any θ, θ′ ∈ RD it holds that ∥∇VG (θ) − ∇VG (θ′ )∥ ≤ 6B 2 ∥θ − θ′ ∥, where B = maxy∈Y ∥ϕ(y)∥. Proof. By Lemma 10, we may write ∇VG (θ) as: ∇VG (θ) = Ey∼πθ [AG (y; θ)ϕ(y)] = Ey∼πθ [rG (y)ϕ(y)] − VG (θ) · ϕ̄θ , where ϕ̄θ := Ey∼πθ [ϕ(y)]. Furthermore, notice that for linear softmax policies and y ∈ Y, ∇πθ (y) = πθ (y)(ϕ(y) − ϕ̄θ ). Now, for convenience of notation denote a(θ) := Ey∼πθ [rG (y)ϕ(y)] and b(θ) := ϕ̄θ , and let Ja (θ), Jb (θ) ∈ RD×D be the Jacobians of a(θ) and b(θ) at θ, respectively. We have: X Ja (θ) = rG (y)ϕ(y)(∇πθ (y))⊤ y∈Y X = rG (y)ϕ(y)πθ (y)(ϕ(y) − ϕ̄θ )⊤ y∈Y = Ey∼πθ rG (y)ϕ(y)(ϕ(y) − ϕ̄θ )⊤ . 27
Similarly, X
Jb (θ) =
y∈Y
ϕ(y)∇πθ (y)⊤ = Ey∼πθ ϕ(y)(ϕ(y) − ϕ̄θ )⊤ .
Thus, the Hessian ∇2 VG (θ) is given by ∇2 VG (θ) = Ja (θ) − b(θ)∇VG (θ)⊤ − VG (θ) · Jb (θ) = Ey∼πθ rG (y)ϕ(y)(ϕ(y) − ϕ̄θ )⊤ − ϕ̄θ ∇VG (θ)⊤ − Ey∼πθ VG (θ)ϕ(y)(ϕ(y) − ϕ̄θ )⊤ = Ey∼πθ AG (y; θ)ϕ(y)(ϕ(y) − ϕ̄θ )⊤ − ϕ̄θ ∇VG (θ)⊤ . To upper bound the spectral norm of the Hessian, i.e., ∥∇2 VG (θ)∥2 , we bound the two terms separately. For the first term, Ey∼πθ [AG (y; θ)ϕ(y)(ϕ(y) − ϕ̄θ )⊤ ], we have: ⊤ ≤ Ey∼πθ AG (y; θ)ϕ(y)(ϕ(y) − ϕ̄θ )⊤ 2 Ey∼πθ AG (y; θ)ϕ(y)(ϕ(y) − ϕ̄θ ) 2 (5) ≤ Ey∼πθ 2∥ϕ(y)∥∥ϕ(y) − ϕ̄θ ∥ ≤ 4B 2 . The first inequality is from |AG (y; θ)| ≤ 2 and the second inequality is from B = maxy∈Y ∥ϕ(y)∥. For the second term, ϕ̄θ ∇VG (θ)⊤ , we have: ϕ̄θ ∇VG (θ)⊤ 2 = ϕ̄θ ∥∇VG (θ)∥ ≤ B · ∥Ey∼πθ [AG (y; θ)ϕ(y)]∥ ≤ 2B 2 .
(6)
Combining Equation (5) and Equation (6), we obtain: ∇2 VG (θ) 2 ≤ 4B 2 + 2B 2 = 6B 2 , and so ∇VG is 6B 2 -Lipschitz. B.4
Proof of Theorem 2
For convenience, we start by introducing the following helper variable: 14/13 πθ0 (y⋆ ) γ := , M where recall that M := min 1, 0.05 · rG (ymed )2/7 (∆1 + ∆2 )−1 , with ∆1 := rG (y⋆ ) − rG (ymed ) and ∆2 := rG (ymed ) − rG (ybad ) for some ybad ∈ Ybad . We can then straightforwardly state the conditions of Assumption 2 in terms of γ. Lemma 4. Under Assumptions 1 and 2, the following properties hold: 7/3 1. γ ≤ min 0.01, 1100(∆211 +∆2 ) , ∆22 ; ( 2. πθ0 (y⋆ ) ≤ min 3. max
n √
2/7 G (ymed ) γ 13/14 , r20(∆ · γ 13/14 1 +∆2 )
2 γ ∆2 , 0.05 · rG (ymed )
o
) ; and
≤ πθ0 (Ybad ) ≤ 0.1 · rG (ymed ).
Proof. All three conditions are obtained directly from Assumption 2 by substituting the definition of M and noticing that πθ0 (y⋆ ) = M γ 13/14 . Note that Assumption 1 on the ground truth reward structure ensures that ∆1 and ∆2 are positive, so all quantities in the conditions above are welldefined. B.4.1
Proof of Case I
Suppose that gradient flow is used to maximize the expected reward with respect to rG . We define tγ as the initial time at which VG (θt ) reaches rG (ymed ) − γ, i.e.: tγ := min{t ≥ 0 : VG (θt ) ≥ rG (ymed ) − γ}, 28
and analyze the time interval [0, tγ ]. In Lemma 5 and Proposition 4, we show that throughout this time interval, πθt (ymed ) monotonically increases while πθt (Ybad ) and πθt (y⋆ ) monotonically decrease. Moreover, we prove that until tγ the policy becomes highly concentrated on ymed , which in turn implies through Lemma 11 that VG (θt ) remains stuck near ymed for a long time. This will yield the desired lower bound on t⋆ —the initial time at which VG (θt ) is ϵ-optimal. Note that if VG (θt ) < rG (ymed ) for all t ≥ 0, then the lower bound on t⋆ trivially holds as VG (θt ) never reaches rG (y⋆ ) − ϵ. Thus, throughout the proof of this case we can assume that there exists a time at which VG (θt ) ≥ rG (ymed ). Towards showing that πθt (ymed ) increases over [0, tγ ] while πθt (y⋆ ) decreases, the following lemma d proves that dt πθt (Ybad ) < 0 when VG (θt ) < rG (ymed ). d Lemma 5. At any time t ≥ 0, if VG (θt ) < rG (ymed ), then dt πθt (Ybad ) < 0.
Proof. For any z ∈ Ybad , by Lemma 13: X d πθt (z) = πθt (z) πθt (z)AG (z; θt ) − πθt (y)2 AG (y; θt ) , y∈Y dt Summing the equation above over all z ∈ Ybad , we arrive at: X X X X d πθt (z) = πθt (y)2 AG (y; θt ) . πθt (z)2 AG (z; θt ) − πθt (z) dt z∈Ybad
z∈Ybad
z∈Ybad
y∈Y
Thus, d πθt (Ybad ) dt X = πθt (z)2 AG (z; θt ) z∈Ybad
−
X
X πθt (z) πθt (z)2 AG (z; θt ) + πθt (ymed )2 AG (ymed ; θt ) + πθt (y⋆ )2 AG (y⋆ ; θt )
z∈Ybad
=
X | −
z∈Ybad
X πθt (z) AG (z; θt ) 1 − πθt (z) z∈Ybad z∈Ybad {z } 2
I1
X |
πθt (z) πθt (ymed )2 AG (ymed ; θt ) + πθt (y⋆ )2 AG (y⋆ ; θt ) . z∈Ybad {z } I2
When maximizing rG via gradient flow, VG (θt ) is monotonically increasing since d 2 VG (θt ) = ∥∇VG (θt )∥ ≥ 0, dt for all t ≥ 0. Hence, AG (z; θt ) ≤ 0 for all z ∈ Ybad and t ≥ 0, because outputs in Ybad have minimal ground truth reward. Furthermore, as long as VG (θt ) < rG (ymed ), it holds that AG (ymed ; θt ) > 0 d and AG (y⋆ ; θt ) > 0. This implies that I1 < 0 and I2 > 0, so dt πθt (Ybad ) = I1 − I2 < 0. Using Lemma 5, we can now prove that until tγ the probability of ymed monotonically increases while the probability of y⋆ monotonically decreases, and characterize how small πθt (y⋆ ) becomes. This is a core component in the proof of Theorem 2. Proposition 4. For all t ∈ [0, tγ ] it holds that: d • dt πθt (y⋆ ) < 0; and d • dt πθt (ymed ) > 0.
Furthermore, mint∈[0,tγ ] πθt (y⋆ ) = πθtγ (y⋆ ) ≤ 29
πθ0 (ymed )2 γ . 8AG (y⋆ ; θ0 )
d d d Proof. Notice that when dt πθt (y⋆ ) < 0, we must have dt πθt (ymed ) > 0. This is because dt πθt (y⋆ )+ d d d d d π (y ) + π (Y ) = π (Y) = 1 = 0, and so when π (y ) < 0: bad dt θt med dt θt dt θt dt dt θt ⋆
d d d d πθt (ymed ) = − πθt (y⋆ ) − πθt (Ybad ) > − πθt (Ybad ) > 0, dt dt dt dt where the second inequality follows from Lemma 5. Hence, for proving monotonicity of πθt (y⋆ ) and d πθt (ymed ), it suffices to show that dt πθt (y⋆ ) < 0 for all t ∈ [0, tγ ]. √ √ To do this, we let t γ be the initial time at which VG (θt ) ≥ rG (ymed ) − γ, i.e.: √ t√γ := min{t ≥ 0 : VG (θt ) ≥ rG (ymed ) − γ}. √ From Lemma 14, VG (θ0 ) < rG (ymed ) − γ. Hence, t√γ > 0. Furthermore, since VG (θt ) is √ continuous in t, at t√γ it holds that VG (θt√γ ) = rG (ymed ) − γ. The proof proceeds by showing d πθt (y⋆ ) < 0 up to time t√γ and upper bounding πθt√γ (y⋆ ) by πθ0 (ymed )2 γ/8AG (y⋆ ; θ0 ). We that dt will then establish that πθt (y⋆ ) continues decreasing until time tγ , from which it follows that the upper bound on πθt√γ (y⋆ ) transfers to πθtγ (y⋆ ). d d Step 1: monotonicity in [0, t√γ ]. We show that dt πθt (y⋆ ) < 0 and hence dt πθt (ymed ) > 0 ′ for all t ∈ [0, t√γ ]. Assume by way of contradiction that there exists a time t ≤ t√γ at which d dt πθt (y⋆ ) t=t′ ≥ 0 and denote by τ the initial such time, i.e.: d τ := min t ∈ [0, t√γ ] : dt πθt (y⋆ ) ≥ 0 . d d πθt (y⋆ ) < 0 and dt πθt (ymed ) > 0, and so πθτ (y⋆ ) ≤ For all t ∈ [0, τ ), we therefore have that dt πθ0 (y⋆ ) and πθτ (ymed ) ≥ πθ0 (ymed ). We now upper bound πθτ (Ybad )2 under this assumption. Notice that X AG (ymed ; θτ ) = (1 − πθτ (ymed ))rG (ymed ) − πθτ (y⋆ )rG (y⋆ ) − πθτ (z)rG (z) z∈Ybad
≥ −∆1 πθτ (y⋆ ) + ∆2 πθτ (Ybad ). Thus, AG (ymed ; θτ ) + ∆1 πθτ (y⋆ ) 1.3AG (ymed ; θτ ) ≤ . ∆2 ∆2 √ The second inequality is from ∆1 πθτ (y⋆ ) ≤ 2πθ0 (y⋆ ) ≤ 2γ 13/14 ≤ 0.3 γ ≤ 0.3AG (ymed ; θτ ) for γ ≤ 0.01 (Lemma 4). Then, we have: πθτ (Ybad ) ≤
1.69AG (ymed ; θτ )2 ∆22 7πθ0 (ymed )2 AG (ymed ; θτ ) 2AG (ymed ; θτ ) < · 8∆2 ∆2 πθ0 (ymed )2 2 7πθ0 (ymed ) AG (ymed ; θτ ) 2AG (ymed ; θ0 ) ≤ · , 8∆2 ∆2 πθ0 (ymed )2
πθτ (Ybad )2 ≤
where the second inequality is due to 2×7 > 1.69×8 and the third inequality is from AG (ymed ; θτ ) ≤ AG (ymed ; θ0 ) (under gradient flow VG (θt ) is monotonically non-decreasing). Focusing on the second 2AG (ymed ;θ0 ) term on the right-hand side, we prove that ∆ 2 ≤ 1. This follows from: 2 πθ0 (ymed ) X AG (ymed ; θ0 ) = (1 − πθ0 (ymed ))rG (ymed ) − πθ0 (y⋆ )rG (y⋆ ) − πθ0 (z)rG (z) z∈Ybad
≤ (1 − πθ0 (ymed ))rG (ymed ) + πθ0 (Ybad ) ≤ 0.2rG (ymed ) + 0.1rG (ymed ) = 0.3rG (ymed ) 0.82 rG (ymed ) 2 ∆2 πθ0 (ymed )2 . ≤ 2 <
30
Here, the first inequality is from rG (z) ≥ −1 for z ∈ Ybad and rG (y⋆ ) ≥ 0; the second inequality is from Items 2 and 3 in Lemma 4, which imply πθ0 (ymed ) ≥ 0.8; and the last inequality is from 2AG (ymed ;θ0 ) rG (ybad ) ≤ 0 and πθ0 (ymed ) ≥ 0.8. Therefore, ∆ 2 ≤ 1, which yields 2 πθ (ymed ) 0
πθτ (Ybad )2 ≤
7πθ0 (ymed )2 AG (ymed ; θτ ) . 8∆2
(7)
d πθt (y⋆ ) t=τ , based on the expression derived in Lemma 13: We now upper bound dt
X d = πθτ (y⋆ ) πθτ (y⋆ )AG (y⋆ ; θτ ) − πθτ (y)2 AG (y; θτ ) πθt (y⋆ ) y∈Y dt t=τ = πθτ (y⋆ ) πθτ (y⋆ ) − πθτ (y⋆ )2 AG (y⋆ ; θτ ) − πθτ (ymed )2 AG (ymed ; θτ ) X − πθτ (z)2 AG (z; θτ ) z∈Ybad ≤ πθτ (y⋆ ) πθτ (y⋆ ) − πθτ (y⋆ )2 AG (y⋆ ; θτ ) − πθτ (ymed )2 AG (ymed ; θτ ) √ X πθτ (z)2 , + (∆2 − γ) z∈Ybad
√ √ where the inequality is by −AG (z; θτ ) ≤ −AG (z; θt√γ ) ≤ rG (ymed ) − γ − rG (ybad ) = ∆2 − γ. d We continue upper bounding dt πθt (y⋆ ) t=τ as follows: d ≤ πθτ (y⋆ ) πθτ (y⋆ ) − πθτ (y⋆ )2 AG (y⋆ ; θτ ) − πθτ (ymed )2 AG (ymed ; θτ ) πθt (y⋆ ) dt t=τ √ X + (∆2 − γ) πθτ (z)2 z∈Ybad ≤ πθτ (y⋆ ) πθ0 (y⋆ ) − πθ0 (y⋆ )2 AG (y⋆ ; θτ ) − πθ0 (ymed )2 AG (ymed ; θτ ) √ + (∆2 − γ)πθτ (Ybad )2 ≤ πθτ (y⋆ ) πθ0 (y⋆ ) − πθ0 (y⋆ )2 AG (y⋆ ; θτ ) − (1/8)πθ0 (ymed )2 AG (ymed ; θτ ) √ ≤ πθτ (y⋆ ) πθ0 (y⋆ ) − πθ0 (y⋆ )2 AG (y⋆ ; θ0 ) − (1/8)πθ0 (ymed )2 γ . The second inequality is from πθτ (y⋆ ) − πθτ (y⋆ )2 ≤ πθ0 (y⋆ ) − πθ0 (y⋆ )2 (this holds since 2 P P πθτ (y⋆ ) ≤ πθ0 (y⋆ ) < 0.5), πθτ (ymed ) ≥ πθ0 (ymed ), and z∈Ybad πθt (z)2 ≤ z∈Ybad πθt (z) . √ The third inequality is from (∆2 − γ)πθτ (Ybad )2 ≤ ∆2 πθτ (Ybad )2 ≤ 87 ·πθ0 (ymed )2 AG (ymed ; θτ ) (Equation (7)). The last inequality is from the fact that VG (θt ) is√monotonically increasing, so AG (y⋆ ; θτ ) ≤ AG (y⋆ ; θ0 ) and AG (ymed ; θτ ) ≥ AG (ymed ; θt√γ ) = γ. From Lemma 4, we further obtain that: πθ0 (y⋆ ) ≤
√ √ √ γ · γ 3/7 0.82 γ πθ (ymed )2 γ rG (ymed )2/7 γ 13/14 ≤ < ≤ 0 . 20(∆1 + ∆2 ) 20AG (y⋆ ; θ0 ) 8AG (y⋆ ; θ0 ) 8AG (y⋆ ; θ0 )
Here, the first inequality is from Item 2 in Lemma 4; the second inequality is from rG (ymed ) ≤ 1 and AG (y⋆ ; θ0 ) ≤ ∆1 + ∆2 ; the third inequality is from γ 3/7 < 1.6; and the last inequality is from πθ0 (ymed ) = 1 − πθ0 (y⋆ ) − πθ0 (Ybad ) ≥ 1 − γ 13/14 − 0.1rG (ymed ) ≥ 0.8. Therefore, √ πθ0 (y⋆ ) − πθ0 (y⋆ )2 AG (y⋆ ; θ0 ) − (1/8)πθ0 (ymed )2 γ < 0, d which implies that dt πθt (y⋆ ) t=τ < 0. However, this contradicts the definition of τ , according to d d d which dt πθt (y⋆ ) t=τ ≥ 0. Hence, dt πθt (y⋆ ) < 0 and dt πθt (ymed ) > 0 for all t ∈ [0, t√γ ].
Step 2: upper bound on πθt√γ (y⋆ ) and monotonicity in [t√γ , tγ ]. Next, we prove that πθt (y⋆ ) keeps decreasing and πθt (ymed ) keeps increasing on the time interval [t√γ , tγ ]. This requires deriving 31
d d upper bounds on dt πθt (y⋆ ) and dt πθt (ymed ) for t ∈ [0, t√γ ] that allow quantifying how small the probability of y⋆ becomes until time t√γ . d πθt (y⋆ ) in [0, t√γ ]. In this step, we prove that for all t ∈ [0, t√γ ]: Step 2.1: upper bound on dt d πθ (y⋆ ) ≤ −0.3 · πθt (y⋆ )πθt (ymed )2 AG (ymed ; θt ). dt t
First, by Lemma 13, for t ∈ [0, t√γ ] it holds that: d πθ (y⋆ ) dt t X = πθt (y⋆ ) πθt (y⋆ )AG (y⋆ ; θt ) − πθt (y)2 AG (y; θt ) y∈Y X = πθt (y⋆ ) πθt (y⋆ ) − πθt (y⋆ )2 AG (y⋆ ; θt ) − πθt (ymed )2 AG (ymed ; θt ) − πθt (z)2 AG (z; θt ) z∈Ybad
X ≤ πθt (y⋆ ) πθ0 (y⋆ ) − πθ0 (y⋆ )2 AG (y⋆ ; θt ) − πθt (ymed )2 AG (ymed ; θt ) − πθt (z)2 AG (z; θt ) , z∈Ybad
{z
|
}
G
(8) where the inequality is due to πθt (y⋆ ) being monotonically decreasing in [0, t√γ ] and πθ0 (y⋆ ) < 0.5 2 (the function g(p) = p − p is monotonically increasing over [0, 0.5]). We proceed by establishing that G ≤ −0.3πθt (ymed )2 AG (ymed ; θt ) through the following computations: G + 0.3πθt (ymed )2 AG (ymed ; θt ) X = −0.7πθt (ymed )2 AG (ymed ; θt ) + πθ0 (y⋆ ) − πθ0 (y⋆ )2 AG (y⋆ ; θt ) − πθt (z)2 AG (z; θt ) z∈Ybad 2
≤ −0.7πθt (ymed ) AG (ymed ; θt ) + πθ0 (y⋆ )AG (y⋆ ; θt ) −
X
2
πθt (z) AG (z; θt )
z∈Ybad
X
= −0.7πθt (ymed )2 (rG (ymed ) − VG (θt )) + πθ0 (y⋆ )AG (y⋆ ; θt ) +
πθt (z)2 (VG (θt ) − rG (z))
z∈Ybad 2
= −0.7πθt (ymed )
(1 − πθt (ymed ))rG (ymed ) − πθt (y⋆ )rG (y⋆ ) −
X z∈Ybad
πθt (z)rG (ybad )
+ πθ0 (y⋆ )AG (y⋆ ; θt ) X + πθt (z)2 (1 − πθt (z))(−rG (z)) + πθt (y⋆ )rG (y⋆ ) + πθt (ymed )rG (ymed ) z∈Ybad X X 2 ′ ′ + πθt (z) πθt (z )rG (z ) z∈Ybad z ′ ∈Ybad \{z} = −0.7πθt (ymed )2 πθt (Ybad )∆2 − πθt (y⋆ )∆1 + πθ0 (y⋆ )AG (y⋆ ; θt ) X + πθt (z)2 πθt (ymed )∆2 + πθt (y⋆ )(∆1 + ∆2 ) z∈Ybad ≤ −0.7πθt (ymed )2 πθt (Ybad )∆2 − πθt (y⋆ )∆1 + πθ0 (y⋆ )AG (y⋆ ; θt ) + πθt (Ybad )2 πθt (ymed )∆2 + πθt (y⋆ )(∆1 + ∆2 ) ≤ πθt (Ybad )∆2 −0.7πθt (ymed )2 + πθt (Ybad )πθt (ymed ) | {z } I1
2
+ πθ0 (y⋆ ) 0.7πθt (ymed ) ∆1 + πθt (Ybad )2 (∆1 + ∆2 ) + AG (y⋆ ; θt ) . | {z } I2
P The first inequality is from 1 − πθ0 (y⋆ ) ≤ 1; the second inequality is due to z∈Ybad πθt (z)2 ≤ πθt (Ybad )2 ; and the last inequality is from πθt (y⋆ ) ≤ πθ0 (y⋆ ). We now upper bound I1 and I2 separately. Specifically, we will prove the following two inequalities: I1 ≤ −0.2πθt (Ybad )πθt (ymed )2 ∆2 , I2 ≤ 0.2πθt (Ybad )πθt (ymed )2 ∆2 . 32
For I1 , note that it suffices to show that 0.5πθt (ymed ) ≥ πθt (Ybad ). Indeed, from Lemma 4 we know that πθ0 (Ybad ) < 0.2 and γ < 0.05, so πθ0 (y⋆ ) ≤ γ 13/14 < 0.13 and πθ0 (ymed ) = 1 − πθ0 (y⋆ ) − πθ0 (Ybad ) > 0.67. In turn, this implies that πθt (Ybad ) ≤ πθ0 (Ybad ) < 0.5πθ0 (ymed ) ≤ 0.5πθt (ymed ). Hence, I1 ≤ −0.2πθt (Ybad )πθt (ymed )2 ∆2 . As for I2 , we have: 0.7πθt (ymed )2 ∆1 + πθt (Ybad )2 (∆1 + ∆2 ) + AG (y⋆ ; θt ) ≤ 0.7∆1 + 0.1(∆1 + ∆2 ) + AG (y⋆ ; θ0 ) ≤ 2(∆1 + ∆2 ). Here, the first inequality is from πθt (Ybad )2 ≤ πθ0 (Ybad )2 ≤ πθ0 (Ybad ) ≤ 0.1 and VG (θt ) ≥ VG (θ0 ), and the second inequality is from AG (y⋆ ; θ0 ) ≤ ∆1 +∆2 . Meanwhile, since AG (ymed ; θt ) ≥ √ γ, it follows that: X AG (ymed ; θt ) = (1 − πθt (ymed ))rG (ymed ) − πθt (y⋆ )rG (y⋆ ) − πθt (z)rG (z) z∈Ybad
= −∆1 πθt (y⋆ ) + ∆2 πθt (Ybad ). √ Thus, −∆1 πθt (y⋆ ) + ∆2 πθt (Ybad ) ≥ γ, which implies that √ √ γ + ∆1 πθt (y⋆ ) γ ≥ . (9) πθt (Ybad ) ≥ ∆2 ∆2 Since πθ0 (ymed ) > 0.67 (as we showed above when bounding I1 ), from Lemma 4 we know that 7/3 γ ≤ πθ0 (ymed )14/3 /(10(∆1 + ∆2 )) . This leads to: √ γπθ0 (ymed )2 13/14 πθ0 (y⋆ ) ≤ γ ≤ , 10(∆1 + ∆2 ) which yields: I2 ≤ 2(∆1 + ∆2 )πθ0 (y⋆ ) ≤ 0.2πθt (Ybad )πθt (ymed )2 ∆2 . Therefore, I1 + I2 ≤ 0 and we can conclude that G ≤ −0.3πθt (ymed )2 AG (ymed ; θt ). Going back to d Equation (8), we obtain the desired upper bound on dt πθt (y⋆ ) for t ∈ [0, t√γ ]: d πθ (y⋆ ) ≤ −0.3 · πθt (y⋆ )πθt (ymed )2 AG (ymed ; θt ). dt t d d Step 2.2: upper bound on dt πθt (ymed ) in [0, t√γ ]. Step 1 showed that dt πθt (ymed ) > 0 for all t ∈ [0, t√γ ]. Here, we prove that πθt (ymed ) does not grow too fast by establishing that for any such t:
d πθ (ymed ) ≤ 2.1πθt (ymed )2 (1 − πθt (ymed ))AG (ymed ; θt ). dt t d We first upper bound dt πθt (ymed ), based on the expression derived in Lemma 13:
d πθ (ymed ) dt t
X = πθt (ymed ) πθt (ymed )AG (ymed ; θt ) − πθt (y)2 AG (y; θt ) y∈Y 2 = πθt (ymed ) πθt (ymed ) − πθt (ymed ) AG (ymed ; θt ) − πθt (y⋆ )2 AG (y⋆ ; θt ) X − πθt (z)2 AG (z; θt ) z∈Ybad X ≤ πθt (ymed ) πθt (ymed ) − πθt (ymed )2 AG (ymed ; θt ) − πθt (z)2 AG (z; θt ) z∈Ybad X 2 = πθt (ymed ) πθt (ymed ) − πθt (ymed ) AG (ymed ; θt ) + (∆2 − AG (ymed ; θt )) πθt (z)2 z∈Ybad
≤ πθt (ymed ) πθt (ymed ) − πθt (ymed )2 AG (ymed ; θt ) + (∆2 − AG (ymed ; θt ))πθt (Ybad )2 ≤ πθt (ymed ) 1 − πθt (ymed ) πθt (ymed )AG (ymed ; θt ) + (∆2 − AG (ymed ; θt ))πθt (Ybad ) . {z } | G
(10) 33
P The first inequality is from AG (y⋆ ; θt ) ≥ 0; the second inequality is due to z∈Ybad πθt (z)2 ≤ πθt (Ybad )2 ; and the last inequality is by πθt (Ybad ) ≤ 1 − πθt (ymed ). Next, we upper bound the term G from the inequality above, proving that G ≤ 1.1πθt (ymed )AG (ymed ; θt ). Equivalently, we show that 1.1πθt (ymed )AG (ymed ; θt ) − G ≥ 0. This follows from the following computations: 1.1πθt (ymed )AG (ymed ; θt ) − G = 1.1πθt (ymed )AG (ymed ; θt ) − (∆2 − AG (ymed ; θt ))πθt (Ybad ) = 1.1πθt (ymed ) rG (ymed ) − VG (θt ) − πθt (Ybad ) VG (θt ) − rG (ybad ) X = 1.1πθt (ymed ) (1 − πθt (ymed ))rG (ymed ) − πθt (y⋆ )rG (y⋆ ) − πθt (z)rG (z) z∈Ybad
− πθt (Ybad ) (1 − πθt (Ybad ))(−rG (ybad )) + πθt (y⋆ )rG (y⋆ ) + πθt (ymed )rG (ymed ) = 1.1πθt (ymed ) πθt (Ybad )∆2 − πθt (y⋆ )∆1 − πθt (Ybad ) πθt (ymed )∆2 + πθt (y⋆ )(∆1 + ∆2 ) = πθt (Ybad ) 1.1πθt (ymed )∆2 − πθt (ymed )∆2 − πθt (y⋆ ) 1.1πθt (ymed )∆1 + πθt (Ybad )(∆1 + ∆2 ) = 0.1πθt (Ybad )∆2 πθt (ymed ) − πθt (y⋆ ) 1.1πθt (ymed )∆1 + πθt (Ybad )(∆1 + ∆2 ) . {z } | | {z } I1
I2
It remains to show that I2 ≤ 0.1πθt (ymed )πθt (Ybad )∆2 = I1 . Focusing on I2 , we have: 1.1πθt (ymed )∆1 + πθt (Ybad )(∆1 + ∆2 ) < 1.1πθt (ymed )∆1 + 0.4πθt (ymed )(∆1 + ∆2 ) ≤ 1.5πθt (ymed )(∆1 + ∆2 ), where the first inequality follows from Lemma 4, which along with the monotonicity of πθt (Ybad ) (Lemma 5) and πθt (ymed ), implies that πθt (Ybad ) ≤ πθ0 (Ybad ) < 0.4πθ0 (ymed ) ≤ 0.4πθt (ymed ). 7/3 Furthermore, because γ ≤ 50(∆11+∆2 ) (Lemma 4) and πθt (y⋆ ) monotonically decreases over √ [0, t γ ], we can bound πθt (y⋆ ) as: √ γ . πθt (y⋆ ) ≤ πθ0 (y⋆ ) ≤ γ 13/14 ≤ 50(∆1 + ∆2 ) √ √ γ By Equation (9), πθt (Ybad ) ≥ ∆2 whenever AG (ymed ; θt ) ≥ γ. Using this fact, we obtain: I2 ≤ 1.5πθt (ymed )πθt (y⋆ )(∆1 + ∆2 ) ≤ 0.1πθt (ymed )πθt (Ybad )∆2 = I1 . Thus, G ≤ 1.1πθt (ymed )AG (ymed ; θt ) for G defined in Equation (10) and we can conclude that: d πθ (ymed ) ≤ 2.1πθt (ymed )2 (1 − πθt (ymed ))AG (ymed ; θt ). dt t Step 2.3: upper bound on πθt√γ (y⋆ ). To summarize, we showed that for all t ∈ [0, t√γ ]: d πθ (y⋆ ) ≤ −0.3 · πθt (y⋆ )πθt (ymed )2 AG (ymed ; θt ), dt t d πθ (ymed ) ≤ 2.1 · πθt (ymed )2 (1 − πθt (ymed ))AG (ymed ; θt ). dt t Since both sides of the first inequality are negative and both sides of the second inequality are positive, we can divide them to get: d 1 πθt (y⋆ ) dt πθt (y⋆ ) ≤− · . d 7 1 − πθt (ymed ) π (y ) dt θt med
We can rewrite this inequality as: d dt πθt (y⋆ )
πθt (y⋆ )
≤
d πθt (ymed ) 1 − dt 1 d (1 − πθt (ymed )) · = · dt , 7 1 − πθt (ymed ) 7 1 − πθt (ymed )
34
from which it follows that: d 1 d ln πθt (y⋆ ) ≤ · ln(1 − πθt (ymed )). dt 7 dt Integrating both sides then leads to: ln πθt (y⋆ )
t√γ
≤
0
t√γ 1 . · ln(1 − πθt (ymed )) 7 0
Thus, πθt√γ (y⋆ ) πθ0 (y⋆ )
≤
1 − πθt√γ (ymed )
!1/7
1 − πθ0 (ymed )
,
and so, πθt√γ (y⋆ ) ≤
πθ0 (y⋆ )(1 − πθt√γ (ymed ))1/7 (1 − πθ0 (ymed ))1/7
.
(11)
√
Next, we prove that 1 − πθt√γ (ymed ) ≤ the case. We would therefore have that:
γ+πθ0 (y⋆ ) rG (ymed ) . Assume by way of contradiction that this is not
X VG (θt√γ ) = πθt√γ (ymed )rG (ymed ) + πθt√γ (y⋆ )rG (y⋆ ) + πθt√γ (z)rG (z) z∈Ybad √ γ + πθ0 (y⋆ ) √ < 1− rG (ymed ) + πθ0 (y⋆ ) ≤ rG (ymed ) − γ, rG (ymed ) where the inequality is by rG (z) ≤ 0 for all z ∈ Ybad and rG (y⋆ ) ≤ 1. However, √ this contradicts the definition of t√γ , which is the initial time at which VG (θt ) ≥ rG (ymed ) − γ. Hence, it must be that: √ γ + πθ0 (y⋆ ) 1 − πθt√γ (ymed ) ≤ . rG (ymed ) Plugging this upper bound into Equation (11), we get: 1/7 √ πθ0 (y⋆ ) γ + πθ0 (y⋆ ) πθt√γ (y⋆ ) ≤ 1/7 rG (ymed )(1 − πθ0 (ymed )) √ πθ0 (y⋆ )(1.3 γ)1/7 ≤ (0.05 · rG (ymed )2 )1/7 1/7 √ 26 γ = πθ0 (y⋆ ) · , rG (ymed )2
(12)
where the second inequality is from 1 − πθ0 (ymed ) ≥ πθ0 (Ybad ) ≥ 0.05 · rG (ymed ) and πθ0 (y⋆ ) ≤ √ γ 13/14 ≤ 0.3 γ (see Lemma 4). Step 2.4: completing the proof of monotonicity in [t√γ , tγ ] Finally, we conclude the proof of Proposition 4 by considering the optimization dynamics over [t√γ , tγ ] and showing that πθt (y⋆ ) continues to decrease during this time interval. d Assume by way of contradiction that there exists t′ ∈ [0, tγ ] such that dt πθt (y⋆ ) t=t′ ≥ 0. Let τ be d the initial time in [0, tγ ] at which dt πθt (y⋆ ) ≥ 0, i.e.: d τ := min t ∈ [0, tγ ] : dt πθt (y⋆ ) ≥ 0 . d d πθt (y⋆ ) < 0, and so dt πθt (ymed ) > 0, for all t ∈ [0, τ ). We also know that if τ By definition, dt d √ exists, then τ ≥ t γ since dt πθt (y⋆ ) < 0 for all t ∈ [0, t√γ ]. This implies that πθτ (y⋆ ) ≤ πθt√γ (y⋆ ) and πθτ (ymed ) ≥ πθ0 (ymed ). Next, we establish that under the assumption above
πθτ (Ybad )2 ≤
7πθ0 (ymed )2 AG (ymed ; θτ ) . 8∆2 35
To see this, notice that: AG (ymed ; θτ ) = (1 − πθτ (ymed ))rG (ymed ) − πθτ (y⋆ )rG (y⋆ ) −
X z∈Ybad
πθτ (z)rG (z)
= −∆1 πθτ (y⋆ ) + ∆2 πθτ (Ybad ). Thus, πθτ (Ybad ) =
AG (ymed ; θτ ) + ∆1 πθτ (y⋆ ) 1.3AG (ymed ; θτ ) ≤ . ∆2 ∆2
The inequality above follows from Item 2 in Lemma 4 since πθ0 (y⋆ ) ≤
rG (ymed )2/7 γ 13/14 0.3γ 13/14 rG (ymed )2/7 γ 13/14 ≤ ≤ , 20(∆1 + ∆2 ) 20∆1 ∆1 (26/rG (ymed )2 )1/7
which combined with Equation (12) yields: √ ∆1 πθτ (y⋆ ) ≤ ∆1 πθt√γ (y⋆ ) ≤ ∆1 πθ0 (y⋆ ) · (26 γ/rG (ymed )2 )1/7 ≤ 0.3γ ≤ 0.3AG (ymed ; θτ ). It therefore holds that: 1.69AG (ymed ; θτ )2 ∆22 7πθ0 (ymed )2 AG (ymed ; θτ ) 2AG (ymed ; θτ ) < · 8∆2 ∆2 πθ0 (ymed )2 2 7πθ0 (ymed ) AG (ymed ; θτ ) 2AG (ymed ; θ0 ) ≤ , · 8∆2 ∆2 πθ0 (ymed )2
πθτ (Ybad )2 ≤
where the third inequality is from AG (ymed ; θτ ) ≤ AG (ymed ; θ0 ) (the advantage of an output is monotonically non-increasing since VG (θt ) is non-decreasing under gradient flow). We now upper bound the second term on the right-hand side in the inequality above by one. Examining AG (ymed ; θ0 ): AG (ymed ; θ0 ) = (1 − πθ0 (ymed ))rG (ymed ) − πθ0 (y⋆ )rG (y⋆ ) −
X z∈Ybad
πθ0 (z)rG (z)
≤ (1 − πθ0 (ymed ))rG (ymed ) + πθ0 (Ybad ) ≤ 0.2rG (ymed ) + 0.1rG (ymed ) = 0.3rG (ymed ) 0.82 rG (ymed ) 2 21∆2 πθ0 (ymed )2 ≤ . 44 <
Here, the first inequality is from rG (z) ≥ −1; the second inequality is from Lemma 4 since πθ0 (Ybad ) ≤ 0.1rG (ymed ) ≤ 0.1 and πθ0 (ymed ) ≥ 0.8; and the last inequality is from rG (ybad ) ≤ 0 2AG (ymed ;θ0 ) and πθ0 (ymed ) ≥ 0.8. This implies that ∆ 2 ≤ 1, and so: 2 πθ (ymed ) 0
πθτ (Ybad )2 ≤
7πθ0 (ymed )2 AG (ymed ; θτ ) . 8∆2 36
(13)
d Let us now consider dt πθt (y⋆ ) t=τ and obtain a contradiction by showing that it is negative. We start with the following computations: d πθ (y⋆ ) dt t t=τ X = πθτ (y⋆ ) πθτ (y⋆ )AG (y⋆ ; θτ ) − πθτ (y)2 AG (y; θτ ) y
πθτ (y⋆ ) − πθτ (y⋆ ) AG (y⋆ ; θτ ) − πθτ (ymed )2 AG (ymed ; θτ ) X − πθτ (z)2 AG (z; θτ ) z∈Ybad ≤ πθτ (y⋆ ) πθt√γ (y⋆ ) − πθt√γ (y⋆ )2 AG (y⋆ ; θτ ) − πθ0 (ymed )2 AG (ymed ; θτ ) X − πθτ (z)2 AG (z; θτ ) z∈Ybad ≤ πθτ (y⋆ ) πθt√γ (y⋆ ) − πθt√γ (y⋆ )2 AG (y⋆ ; θ0 ) − πθ0 (ymed )2 AG (ymed ; θτ ) X + (∆2 − γ) πθτ (z)2 z∈Ybad ≤ πθτ (y⋆ ) πθt√γ (y⋆ ) − πθt√γ (y⋆ )2 AG (y⋆ ; θ0 ) − πθ0 (ymed )2 AG (ymed ; θτ ) + (∆2 − γ)πθτ (Ybad )2 ≤ πθτ (y⋆ ) πθt√γ (y⋆ ) − πθt√γ (y⋆ )2 AG (y⋆ ; θ0 ) − (1/8)πθ0 (ymed )2 AG (ymed ; θτ ) ≤ πθτ (y⋆ ) πθt√γ (y⋆ ) − πθt√γ (y⋆ )2 AG (y⋆ ; θ0 ) − (1/8)πθ0 (ymed )2 γ ≤ πθτ (y⋆ ) πθt√γ (y⋆ )AG (y⋆ ; θ0 ) − (1/8)πθ0 (ymed )2 γ = πθτ (y⋆ )
2
The first inequality is from πθτ (y⋆ ) − πθτ (y⋆ )2 ≤ πθt√γ (y⋆ ) − πθt√γ (y⋆ )2 (this holds since πθτ (y⋆ ) ≤ πθt√γ (y⋆ ) ≤ πθ0 (y⋆ ) < 0.5); the second inequality is from AG (y⋆ ; θτ ) ≤ AG (y⋆ ; θ0 ) P and −AG (z; θτ ) ≤ ∆2 − γ for all z ∈ Ybad ; the third inequality is due to z∈Ybad πθτ (z)2 ≤ πθτ (Ybad )2 ; the fourth inequality is from Equation (13); the fifth inequality is from AG (ymed ; θτ ) ≥ AG (ymed ; θtγ ) = γ; and the last inequality is by 1 − πθt√γ (y⋆ ) ≤ 1. Now, by Equation (12) and Item 2 in Lemma 4, we have √ πθt√γ (y⋆ ) ≤ πθ0 (y⋆ ) · (26 γ/rG (ymed )2 )1/7 1/7 √ 26 γ γ 13/14 rG (ymed )2/7 · ≤ 20(∆1 + ∆2 ) rG (ymed )2 1/7 √ 26 γ 0.82 γ 13/14 rG (ymed )2/7 < · rG (ymed )2 8 · 261/7 (∆1 + ∆2 ) 2 πθ (ymed ) γ ≤ 0 . 8AG (y⋆ ; θ0 ) Therefore: d πθt (y⋆ ) ≤ πθτ (y⋆ ) πθt√γ (y⋆ )AG (y⋆ ; θ0 ) − (1/8)πθ0 (ymed )2 γ dt t=τ πθ0 (ymed )2 γ < πθτ (y⋆ ) · AG (y⋆ ; θ0 ) − (1/8)πθ0 (ymed )2 γ 8AG (y⋆ ; θ0 ) = 0, d which contradicts the fact that τ is the initial time at which dt πθt (y⋆ ) t=τ ≥ 0. Hence, it must be d that dt πθt (y⋆ ) < 0 for all t ∈ [0, tγ ]. Additionally, by Lemma 5 we know that πθt (Ybad ) is also d decreasing in the time interval [0, tγ ], and so dt πθt (ymed ) > 0 over [0, tγ ].
37
d d To conclude, we showed that dt πθt (y⋆ ) < 0 and dt πθt (ymed ) > 0 for all t ∈ [0, tγ ]. Thus, mint∈[0,tγ ] πθt (y⋆ ) = πθtγ (y⋆ ). Furthermore, since
πθt√γ (y⋆ ) ≤
πθ0 (ymed )2 γ 8AG (y⋆ ; θ0 )
and t√γ < tγ , it holds that πθtγ (y⋆ ) < πθt√γ (y⋆ ) ≤
πθ0 (ymed )2 γ , 8AG (y⋆ ; θ0 )
completing the proof of Proposition 4. Based on Proposition 4, we can now complete the proof of Case I. Proof of Theorem 2, Case I. By Proposition 4, as long as VG (θt ) ≤ rG (ymed ) − γ, we know that πθt (y⋆ ) is monotonically decreasing and πθt (ymed ) is monotonically increasing. Recall that tγ denotes the initial time at which VG (θt ) ≥ rG (ymed ) − γ. Since VG (θt ) is continuous in t, at time tγ it must be that VG (θtγ ) = rG (ymed ) − γ and tγ is the initial time at which this equality holds. We first prove that at tγ the policy is highly concentrated on ymed by upper bounding 1 − πθtγ (ymed ). This follows from X VG (θtγ ) = πθtγ (ymed )rG (ymed ) + πθtγ (y⋆ )rG (y⋆ ) + πθtγ (z)rG (z) = rG (ymed ) − γ, z∈Ybad
which implies: X 1 πθtγ (z)rG (z) γ + πθtγ (y⋆ )rG (y⋆ ) + z∈Ybad rG (ymed ) 1 γ ≤ γ+ rG (ymed ) 8∆1 (1 + 8∆1 )γ . = 8∆1 rG (ymed )
1 − πθtγ (ymed ) =
π
(y
(14)
)2 γ
θ0 med γ (Proposition 4). The inequality is due to πθtγ (y⋆ ) ≤ 8A ≤ 8∆ G (y⋆ ;θ0 ) 1
Now, the fact that the policy assigns high probability to ymed at tγ allows us to lower bound t⋆ —the initial time at which VG (θt ) ≥ rG (y⋆ ) − ϵ. For any time T ≥ tγ , we have: Z T d θt dt ∥θT − θtγ ∥ = dt tγ Z T d θt dt ≤ dt tγ Z T = ∥∇VG (θt )∥dt tγ
Z T ≤
2(∆1 + ∆2 )(1 − πθt (ymed ))dt,
(15)
tγ
where the last inequality is by Lemma 11, with B = 1 since the feature vectors are orthonormal. We proceed by upper bounding the rate at which 1 − πθt (ymed ) increases (i.e., the rate at which πθt (ymed ) decreases). From the chain rule we get: d d (1 − πθt (ymed )) = ∇(1 − πθt (ymed )), θt dt dt = ⟨∇(1 − πθt (ymed )), ∇VG (θt )⟩ D E X = ∇(πθt (y⋆ ) + πθt (z)), ∇VG (θt ) . z∈Ybad
38
For all y ∈ Y, the gradient of πθt (y) with respect to θt can be computed as follows: X ∇πθt (y) = πθt (y) ϕ(y) − πθt (z)ϕ(z) = πθt (y) ϕ(y) − ϕ̄θt , z∈Y P where ϕ̄θt := z∈Y πθt (z)ϕ(z). Thus: X d (1 − πθt (ymed )) = ⟨∇πθt (y⋆ ), ∇VG (θt )⟩ + ⟨∇πθt (z), ∇VG (θt )⟩ z∈Ybad dt X = πθt (y⋆ ) ϕ(y⋆ ) − ϕ̄θt , ∇VG (θt ) + πθt (z) ϕ(z) − ϕ̄θt , ∇VG (θt ) z∈Ybad
≤ (1 − πθt (ymed )) supy∈Y ∥ϕ(y) − ϕ̄θt ∥∥∇VG (θt )∥.
(16)
Notice that since the feature vectors are orthonormal, for any y ∈ Y: X ∥ϕ(y) − ϕ̄θt ∥2 = 1 − 2πθt (y) + πθt (z)2 z∈Y X = (1 − πθt (y))2 + πθt (z)2 z∈Y\{y} X 2 2 ≤ (1 − πθt (y)) + πθt (z) z∈Y\{y}
2
= 2(1 − πθt (y)) . √ Thus, ∥ϕ(y) − ϕ̄θt ∥ ≤ 2(1 − πθt (y)) ≤ 2. Plugging this bound into Equation (16) and applying Lemma 11 with B = 1 to upper bound ∥∇VG (θt )∥ then yields: √ d (1 − πθt (ymed )) ≤ 2 2(∆1 + ∆2 )(1 − πθt (ymed ))2 . dt This implies that d √ (1 − πθt (ymed )) 1 d 2(∆1 + ∆2 ). = − dt ≥ −2 dt 1 − πθt (ymed ) (1 − πθt (ymed ))2 √
Integrating both sides from tγ to t, for any t ≥ tγ , leads to: √ 1 1 ≥ −2 2(∆1 + ∆2 )(t − tγ ). − 1 − πθt (ymed ) 1 − πθtγ (ymed ) Hence, for any t ≥ tγ such that √ 2 2(∆1 + ∆2 )(1 − πθtγ (ymed ))(t − tγ ) < 1, it holds that 1 − πθt (ymed ) ≤
√
1 − πθtγ (ymed )
1 − 2 2(∆1 + ∆2 )(1 − πθtγ (ymed ))(t − tγ ) Consequently, for any T ≥ tγ satisfying √ 2 2(∆1 + ∆2 )(1 − πθtγ (ymed ))(T − tγ ) < 1,
.
we can plug this upper bound into Equation (15) and obtain: Z T ∥θT − θtγ ∥ ≤ 2(∆1 + ∆2 ) (1 − πθt (ymed ))dt tγ
Z T −tγ
√
= 2(∆1 + ∆2 )
1 − πθtγ (ymed )
1 − 2 2(∆1 + ∆2 )(1 − πθtγ (ymed ))s 1 1 √ = √ ln . 2 1 − 2 2(∆1 + ∆2 )(1 − πθtγ (ymed ))(T − tγ )
ds
0
By Lemma 12, VG is 1-Lipschitz with respect to θ, and so for any T ≥ tγ : 1 1 √ |VG (θT ) − VG (θtγ )| ≤ ∥θT − θtγ ∥ ≤ √ ln . 2 1 − 2 2(∆1 + ∆2 )(1 − πθtγ (ymed ))(T − tγ ) (17) 39
If t⋆ = ∞, then the desired lower bound holds trivially. Hence, assume t⋆ < ∞. By the definition of t⋆ and continuity of VG (θt ), VG (θt⋆ ) = rG (y⋆ ) − ϵ. In particular, t⋆ > tγ since VG (θt⋆ ) = rG (y⋆ ) − ϵ > rG (ymed ) > VG (θtγ ). Therefore, VG (θt⋆ ) − VG (θtγ ) = rG (y⋆ ) − ϵ − rG (ymed ) + γ = ∆1 − ϵ + γ ≥ ∆1 − ϵ. √ If 2 2(∆1 + ∆2 )(1 − πθtγ (ymed ))(t⋆ − tγ ) ≥ 1, then directly √
1 1 − e− 2(∆1 −ϵ) t⋆ − t γ ≥ √ ≥ √ , 2 2(∆1 + ∆2 )(1 − πθtγ (ymed )) 2 2(∆1 + ∆2 )(1 − πθtγ (ymed )) √
since 1 − e− 2(∆1 −ϵ) ≤ 1. Otherwise, we apply the bound in Equation (17) with T = t⋆ to obtain: 1 1 √ ∆1 − ϵ ≤ VG (θt⋆ ) − VG (θtγ ) ≤ √ ln , 2 1 − 2 2(∆1 + ∆2 )(1 − πθtγ (ymed ))(t⋆ − tγ ) which implies √
1 − e− 2(∆1 −ϵ) t⋆ − tγ ≥ √ . 2 2(∆1 + ∆2 )(1 − πθtγ (ymed )) Therefore, in either case: t⋆ ≥ t⋆ − tγ
√
1 − e− 2(∆1 −ϵ) ≥ √ 2 2(∆1 + ∆2 )(1 − πθtγ (ymed )) √ √ 2 2∆1 rG (ymed )(1 − e− 2(∆1 −ϵ) ) ≥ (∆1 + ∆2 )(1 + 8∆1 )γ √ √ 2 2M 14/13 ∆1 rG (ymed )(1 − e− 2(∆1 −ϵ) ) · πθ0 (y⋆ )−14/13 ≥ (∆1 + ∆2 )(1 + 8∆1 ) = Ω πθ0 (y⋆ )−14/13 , where the second inequality follows from Equation (14). This completes the proof for Case I of Theorem 2. B.4.2
Proof of Case II
Suppose that gradient flow is used to maximize the expected reward with respect to rP , which assigns ymed a low reward rP (ymed ) = miny∈Ybad rG (y). In this case, we show that πθt (y⋆ ) increases already from t = 0 and that tno−med = O(πθ0 (y⋆ )−1 ), where recall that tno−med is the initial time ⋆ ⋆ at which VG (θt ) ≥ rG (y⋆ ) − ϵ when maximizing rP . Proof of Theorem 2, Case II. Notice that VG (θ) ≥ VP (θ) for all θ ∈ RD since rG (y) ≥ rP (y) for all y ∈ Y. Thus, when VP (θt ) ≥ rG (y⋆ ) − ϵ it also holds that VG (θt ) ≥ rG (y⋆ ) − ϵ. We can therefore focus on VP and consider the time it takes until it reaches a value of rG (y⋆ ) − ϵ. In particular, denote by tno−med the initial time at which VP (θt ) ≥ rG (y⋆ ) − ϵ, i.e.: P tno−med := min{t ≥ 0 : VP (θt ) ≥ rG (y⋆ ) − ϵ}. P The discussion above implies that tno−med ≤ tno−med , and so it suffices to upper bound tno−med . ⋆ P P rG (y⋆ )+1−ϵ For simplicity of notation, denote ρ := rG (y⋆ )+1 . Our first claim is that when πθt (y⋆ ) ≥ ρ, we must have VP (θt ) ≥ rG (y⋆ ) − ϵ. This can be seen via the following calculation: VP (θt ) ≥ πθt (y⋆ )rP (y⋆ ) − (1 − πθt (y⋆ )) = πθt (y⋆ )(rG (y⋆ ) + 1) − 1 ≥ rG (y⋆ ) − ϵ. 40
This implies that πθt (y⋆ ) ≤ ρ for all t ∈ [0, tno−med ]. P We prove that T :=
1 1 1 − ≥ tno−med , P (1 − ρ)2 AP (y⋆ ; θ0 ) πθ0 (y⋆ ) ρ
from which the desired upper bound on tno−med immediately follows by tno−med ≤ tno−med and ⋆ ⋆ P substituting the value of ρ. Assume by way of contradiction that tno−med > T . Let us lower bound the rate at which πθt (y⋆ ) P no−med increases over [0, tP ]. Notice first that VP (θ0 ) > rP (y) for all y ∈ Y \ {y⋆ } since all such y have the same proxy reward value, which is lower than rP (y⋆ ) = rG (y⋆ ). Now, consider the time derivative of πθt (y⋆ ), starting from the expression derived in Lemma 13: d πθ (y⋆ ) dt t X = πθt (y⋆ ) πθt (y⋆ )AP (y⋆ ; θt ) − πθt (y)2 AP (y; θt ) y∈Y X = πθt (y⋆ ) πθt (y⋆ ) − πθt (y⋆ )2 AP (y⋆ ; θt ) − πθt (ymed )2 AP (ymed ; θt ) − πθt (z)2 AP (z; θt ) z∈Ybad
≥ πθt (y⋆ )2 (1 − πθt (y⋆ ))AP (y⋆ ; θt ). Here, the inequality is from AP (y; θt ) ≤ AP (y; θ0 ) ≤ 0 for all y ∈ Y \ {y⋆ }. As we proved above, πθt (y⋆ ) ≤ ρ for all t ∈ [0, tno−med ]. Thus for such t it holds that: P d πθ (y⋆ ) ≥ (1 − ρ)AP (y⋆ ; θt )πθt (y⋆ )2 . dt t Next, we show that AP (y⋆ ; θt ) ≥ (1 − ρ)AP (y⋆ ; θ0 ). To prove this, observe that since rP (y⋆ ) = rG (y⋆ ) and VP (θ0 ) ≥ −1, ϵ ϵ rG (y⋆ ) + 1 − ϵ rG (y⋆ ) − ϵ − VP (θ0 ) =1− ≤1− = = ρ. rP (y⋆ ) − VP (θ0 ) rP (y⋆ ) − VP (θ0 ) rP (y⋆ ) + 1 rG (y⋆ ) + 1 Using this fact, we have that: AP (y⋆ ; θt ) rP (y⋆ ) − VP (θt ) VP (θt ) − VP (θ0 ) rG (y⋆ ) − ϵ − VP (θ0 ) = =1− ≥1− ≥ 1 − ρ. AP (y⋆ ; θ0 ) rP (y⋆ ) − VP (θ0 ) rP (y⋆ ) − VP (θ0 ) rP (y⋆ ) − VP (θ0 ) Therefore, we can plug in this bound to get: d πθ (y⋆ ) ≥ (1 − ρ)2 AP (y⋆ ; θ0 )πθt (y⋆ )2 . dt t Dividing both sides by πθt (y⋆ )2 , integrating from 0 to t, and rearranging terms results in the following lower bound: πθ0 (y⋆ ) πθt (y⋆ ) ≥ . 1 − (1 − ρ)2 AP (y⋆ ; θ0 )πθ0 (y⋆ ) · t At time t = T for 1 1 1 − , T = (1 − ρ)2 AP (y⋆ ; θ0 ) πθ0 (y⋆ ) ρ this lower bound on πθT (y⋆ ) implies that πθT (y⋆ ) ≥ ρ, and so VP (θT ) ≥ rG (y⋆ )−ϵ, in contradiction to our assumption that tno−med > T . Therefore, P 2 (rG (y⋆ ) + 1) · πθ0 (y⋆ )−1 = O πθ0 (y⋆ )−1 . 2 ϵ (rP (y⋆ ) − VP (θ0 )) Moreover, notice that VP (θt ) is monotonically non-decreasing when maximizing rP via gradient flow 2 d since dt VP (θt ) = ∥∇VP (θt )∥ ≥ 0 for all t ≥ 0. Thus, for all t ≥ T ≥ tno−med : P
tno−med ≤ tno−med ≤T ≤ ⋆ P
ϵ ≥ rG (y⋆ ) − VP (θt ) = (1 − πθt (y⋆ ))(∆1 + ∆2 ), from which we can conclude that for all t ≥ T : ϵ πθt (y⋆ ) ≥ 1 − . ∆1 + ∆ 2
41
B.5
Proof of Theorem 3
For convenience of notation, we start by introducing the following helper variables: γ :=
πθ0 (y⋆ ) M′
14/13
2
,
s := ⟨ϕ(y⋆ ), ϕ(ymed )⟩
,
h(s) :=
∥ϕ(ymed )∥ − s 2
7 ∥ϕ(ymed )∥ − s/2
, (18)
2 where recall that M ′ := min 1, ∥ϕ(ymed )∥ rG (ymed )2/7 (40B 2 (∆1 + ∆2 ))−1 , with ∆1 := rG (y⋆ ) − rG (ymed ), ∆2 := rG (ymed ) − rG (ybad ) for some ybad ∈ Ybad , and B := maxy∈Y ∥ϕ(y)∥. We can then straightforwardly state the conditions of Assumption 3 in terms of γ. Lemma 6. Under Assumptions 1 and 3, the following properties hold:
1. γ ≤ min 0.01, ( 2. πθ0 (y⋆ ) ≤ min 3. max
n √
21∥ϕ(ymed )∥2 1100B 2 (∆1 +∆2 )
7/3 ; )
2 2/7 med )∥ med )∥ rG (ymed ) · γ 13/14 , ∥ϕ(y γ 13/14 , ∥ϕ(y40B 2 (∆ +∆ ) ∥ϕ(y⋆ )∥ πθ0 (ymed ) 1 2
2 γ ∆2 , 0.05 · rG (ymed )
o
; and
≤ πθ0 (Ybad ) ≤ 0.1 · rG (ymed ).
Proof. All three conditions are obtained directly from Assumption 3 by substituting the definition of M ′ and noticing that πθ0 (y⋆ ) = M ′ γ 13/14 . Note that Assumption 1 on the ground truth reward structure ensures ∆1 and ∆2 are positive, so all quantities in the conditions above are well-defined.
B.5.1
Proof of Case I
Suppose that gradient flow is used to maximize the expected reward with respect to rG . We define tγ as the initial time at which VG (θt ) reaches rG (ymed ) − Cγ 13/14+h(s)/2 , i.e.: n o tγ := min t ≥ 0 : VG (θt ) ≥ rG (ymed ) − Cγ 13/14+h(s)/2 , (19) 13α/7 2 where C := 26/rG (ymed )2 for α := −s/ 26(∥ϕ(ymed )∥ − 0.5s) ∈ (0, 1/13). As in the proof of Theorem 2, the key step is to show that on [0, tγ ] the probability of y⋆ decreases while the probability of ymed increases. We begin with a monotonicity lemma showing that, whenever VG (θt ) < rG (ymed ), a decrease in πθt (y⋆ ) implies an increase in πθt (ymed ). We then prove that πθt (y⋆ ) indeed decreases throughout [0, tγ ], which in turn yields monotonic growth of πθt (ymed ) on this interval. The negative inner product between ϕ(y⋆ ) and ϕ(ymed ) strengthens this effect relative to the case of orthonormal features (Theorem 2) by pulling probability away from y⋆ more aggressively and concentrating the policy more strongly around ymed . Combining this stronger concentration with the gradient-norm bound from Lemma 11 will allow establishing the desired lower bound on t⋆ . Note that if VG (θt ) < rG (ymed ) for all t ≥ 0, then the lower bound on t⋆ holds trivially, since VG (θt ) never reaches rG (y⋆ ) − ϵ. Thus, throughout the proof of this case, we may assume that there exists a time at which VG (θt ) ≥ rG (ymed ). d d Lemma 7. At any time t ≥ 0 such that VG (θt ) < rG (ymed ), if dt πθt (y⋆ ) < 0, then dt πθt (ymed ) > 0.
Proof. Fix any time t ≥ 0 such that VG (θt ) < rG (ymed ). By Lemma 13, instantiated with r = rG and y = y⋆ , we have D E X X d πθt (y⋆ ) = πθt (y⋆ ) ϕ(y⋆ ) − πθt (y)ϕ(y), πθt (y)AG (y; θt )ϕ(y) , y∈Y y∈Y dt 42
Since ⟨ϕ(z), ϕ(y)⟩ = 0 for all z ∈ Ybad , y ∈ Y \ {z} (Assumption 4), we may write: h d πθt (y⋆ ) = πθt (y⋆ ) πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) dt + πθt (ymed )AG (ymed ; θt ) s − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) i X − πθt (z)2 AG (z; θt )∥ϕ(z)∥2 z∈Ybad h ≥ πθt (y⋆ ) πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) i + πθt (ymed )AG (ymed ; θt ) s − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) . d Here, the inequality is due to AG (z; θt ) ≤ 0 for all z ∈ Ybad . Thus, when dt πθt (y⋆ ) < 0, it must hold that: πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) < πθt (ymed )AG (ymed ; θt ) − s + ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) ,
or equivalently, −s + ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) πθt (y⋆ )AG (y⋆ ; θt ) < , πθt (ymed )AG (ymed ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed )
(20)
since ∥ϕ(y⋆ )∥2 −∥ϕ(y⋆ )∥2 πθt (y⋆ )−sπθt (ymed ) > 0 and πθt (ymed )AG (ymed ; θt ) > 0. We continue by upper bounding the right-hand side in the inequality above via the following claim: −s + ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) ∥ϕ(ymed )∥2 − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) ≤ . 2 2 ∥ϕ(y⋆ )∥ − ∥ϕ(y⋆ )∥ πθt (y⋆ ) − sπθt (ymed ) ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − s + sπθt (ymed ) (21) Observe that the denominators of both sides are positive. Hence, upon multiplying both sides by the denominators and subtracting common terms, to show that Equation (21) holds it suffices to prove: s2 (1 − πθt (ymed ) − πθt (y⋆ )) ≤ ∥ϕ(y⋆ )∥2 ∥ϕ(ymed )∥2 (1 − πθt (ymed ) − πθt (y⋆ )). 2
2
2
Indeed, this is true because s2 = ⟨ϕ(y⋆ ), ϕ(ymed )⟩ ≤ ∥ϕ(ymed )∥ ∥ϕ(y⋆ )∥ . Combining Equation (20) and Equation (21), we thus get: πθt (y⋆ )AG (y⋆ ; θt ) −s + ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) < πθt (ymed )AG (ymed ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) ∥ϕ(ymed )∥2 − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) ≤ , ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − s + sπθt (ymed ) which implies: πθt (ymed )AG (ymed ; θt )(∥ϕ(ymed )∥2 − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ )) − πθt (y⋆ )AG (y⋆ ; θt )(∥ϕ(y⋆ )∥2 πθt (y⋆ ) − s + sπθt (ymed )) > 0.
(22)
Next, we analyze the time derivative of πθt (ymed ), which is (see Lemma 13): h d πθt (ymed ) = πθt (ymed ) πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥2 − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) dt + πθt (y⋆ )AG (y⋆ ; θt ) s − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) i X − πθt (z)2 AG (z; θt )∥ϕ(z)∥2 z∈Ybad h ≥ πθt (ymed ) πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥2 − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) i + πθt (y⋆ )AG (y⋆ ; θt ) s − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) > 0. Here, the first inequality is due to AG (z; θt ) ≤ 0 for all z ∈ Ybad , and the second inequality is from d Equation (22). Putting it all together, we have shown that if VG (θt ) < rG (ymed ) and dt πθt (y⋆ ) < 0, d then necessarily dt πθt (ymed ) > 0. 43
Using Lemma 7, we can now prove that until tγ the probability of ymed monotonically increases while the probability of y⋆ monotonically decreases, and characterize how small πθt (y⋆ ) becomes. This is a core component in the proof of Theorem 3. Proposition 5. For all t ∈ [0, tγ ] it holds that: d • dt πθt (y⋆ ) < 0; and d • dt πθt (ymed ) > 0.
Furthermore, 13
mint∈[0,tγ ] πθt (y⋆ ) = πθtγ (y⋆ ) ≤ where C := 26/rG (ymed )2
13α/7
h(s)
C∥ϕ(ymed )∥2 πθ0 (ymed )2 γ 14 + 2 , 8(∥ϕ(y⋆ )∥2 − s)AG (y⋆ ; θ0 )
, as introduced in Equation (19), and s := ⟨ϕ(y⋆ ), ϕ(ymed )⟩ and
∥ϕ(ymed )∥2 −s h(s) := 7 ∥ϕ(y are as defined in Equation (18). 2 ( med )∥ −s/2)
Proof. Since tγ is the initial time at which VG (θt ) ≥ rG (ymed ) − Cγ 13/14+h(s)/2 , Lemma 7 implies d d that for all t ∈ [0, tγ ], if dt πθt (y⋆ ) < 0, then dt πθt (ymed ) > 0. Hence, for establishing that d d d π (y ) < 0 and π (y ) > 0 over [0, t ], it suffices to prove that dt πθt (y⋆ ) < 0. γ dt θt ⋆ dt θt med √ To that end, we let t√γ be the initial time at which VG (θt ) ≥ rG (ymed ) − γ, i.e.: √ t√γ := min{t ≥ 0 : VG (θt ) ≥ rG (ymed ) − γ}. √ From Lemma 15, VG (θ0 ) < rG (ymed )− γ, so t√γ > 0. Furthermore, since VG (θt ) is continuous in √ d πθt (y⋆ ) < t, at t√γ it holds that VG (θt√γ ) = rG (ymed ) − γ. The proof proceeds by showing that dt √ 0 up to time t γ and upper bounding πθt√γ (y⋆ ). Lastly, we will establish that πθt (y⋆ ) continues decreasing until time tγ . d d Step 1: monotonicity in [0, t√γ ]. We show that dt πθt (y⋆ ) < 0, and so dt πθt (ymed ) > 0, for √ all t ∈ [0, t γ ]. Assume by way of contradiction that there exists a time t′ ≤ t√γ at which d dt πθt (y⋆ ) t=t′ ≥ 0 and denote by τ the initial such time, i.e.: d τ := inf t ∈ [0, t√γ ] : dt πθt (y⋆ ) ≥ 0 . d d For all t ∈ [0, τ ), we therefore have that dt πθt (y⋆ ) < 0 and dt πθt (ymed ) > 0, which imply πθτ (y⋆ ) ≤ πθ0 (y⋆ ) and πθτ (ymed ) ≥ πθ0 (ymed ), respectively. d We now show that dt πθt (Ybad ) < 0 for all t ∈ [0, τ ). By Lemma 13, the time derivative of πθt (z) for z ∈ Ybad can be written as: d πθt (z) = πθt (z)2 AG (z; θt )∥ϕ(z)∥2 − πθt (z) πθt (z)2 AG (z; θt )∥ϕ(z)∥2 dt + πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) + πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed ) .
Summing over all z ∈ Ybad , we have: X d πθt (Ybad ) = πθt (z)2 AG (z; θt )∥ϕ(z)∥2 1 − πθt (z) z∈Ybad dt | {z } I1
− πθt (Ybad ) πθt (ymed )AG (ymed ; θt )(∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ )) + πθt (y⋆ )AG (y⋆ ; θt )(∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed )) ,
44
where we denote by I2 the second term in the expression above (excluding the minus sign). Note that I1 ≤ 0 from the fact that AG (z; θt ) ≤ 0 for all z ∈ Ybad (outputs in Ybad have the lowest ground truth reward among all outputs). We now show that I2 > 0. It suffices to prove that: 0 < πθt (ymed )AG (ymed ; θt )(∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ))
(23)
+ πθt (y⋆ )AG (y⋆ ; θt )(∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed )).
If ∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed ) ≥ 0, which is equivalent to the second term on the right-hand side being non-negative, then: πθt (ymed )AG (ymed ; θt )(∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ )) + πθt (y⋆ )AG (y⋆ ; θt )(∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed )) ≥ πθt (ymed )AG (ymed ; θt )(∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ )) ≥ πθt (ymed )AG (ymed ; θt )(∥ϕ(ymed )∥2 πθ0 (ymed ) + sπθ0 (y⋆ )) ≥ πθt (ymed )AG (ymed ; θt )∥ϕ(ymed )∥ ∥ϕ(ymed )∥πθ0 (ymed ) − ∥ϕ(y⋆ )∥πθ0 (y⋆ ) > 0, where the second inequality is from πθt (y⋆ ) ≤ πθ0 (y⋆ ) and πθt (ymed ) ≥ πθ0 (ymed ) for all t ∈ [0, τ ), the third inequality is from Cauchy-Schwarz since −s ≤ ∥ϕ(y⋆ )∥∥ϕ(ymed )∥, and the last inequality is from Item 2 in Lemma 6. Overall, this implies that if ∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed ) ≥ 0, then I1 − I2 < 0. We now consider the case of ∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed ) < 0, noting that this inequality is equivalent to the second term in Equation (23) being negative. We show that Equation (23) holds by proving the following equivalent inequality: πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) ≤ . πθt (ymed )AG (ymed ; θt ) −∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) To see that this inequality holds, we lower bound the right-hand side by: −s + ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) ≥ . −∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) Since both denominators are positive in this case, multiplying both sides of the inequality by them and canceling common terms, we arrive at the following equivalent form of the inequality above: s2 πθt (ymed ) ≤ ∥ϕ(y⋆ )∥2 ∥ϕ(ymed )∥2 πθt (ymed ), 2
2
2
which straightforwardly holds because s2 = ⟨ϕ(y⋆ ), ϕ(ymed )⟩ ≤ ∥ϕ(ymed )∥ ∥ϕ(y⋆ )∥ . Together with Equation (20), this implies that: −s + ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) πθt (y⋆ )AG (y⋆ ; θt ) < πθt (ymed )AG (ymed ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) ≤ . −∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed )
(24)
Thus, Equation (23) holds, and so I2 > 0, also when ∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed ) < 0. Altogether, d we may conclude that dt πθt (Ybad ) = I1 − I2 < 0 for all t ∈ [0, τ ). Next, we prove that πθτ (Ybad )2 ≤ as:
7∥ϕ(ymed )∥2 πθ0 (ymed )2 AG (ymed ;θτ ) . We can write AG (ymed ; θτ ) 8 maxz∈Ybad ∥ϕ(z)∥2 ∆2
AG (ymed ; θτ ) = (1 − πθτ (ymed ))rG (ymed ) − πθτ (y⋆ )rG (y⋆ ) −
X z∈Ybad
πθτ (z)rG (z)
= −∆1 πθτ (y⋆ ) + ∆2 πθτ (Ybad ). Thus, πθτ (Ybad ) =
AG (ymed ; θτ ) + ∆1 πθτ (y⋆ ) 1.3AG (ymed ; θτ ) ≤ . ∆2 ∆2 45
√ The inequality is due to ∆1 πθτ (y⋆ ) ≤ 2πθ0 (y⋆ ) ≤ 2γ 13/14 ≤ 0.3 γ ≤ 0.3AG (ymed ; θτ ), where √ 2γ 13/14 ≤ 0.3 γ since γ ≤ 0.01. Then, 1.69AG (ymed ; θτ )2 ∆22 7∥ϕ(ymed )∥2 πθ0 (ymed )2 AG (ymed ; θτ ) 2 maxz∈Ybad ∥ϕ(z)∥2 AG (ymed ; θτ ) < · 8 maxz∈Ybad ∥ϕ(z)∥2 ∆2 ∥ϕ(ymed )∥2 ∆2 πθ0 (ymed )2 2 2 7∥ϕ(ymed )∥ πθ0 (ymed ) AG (ymed ; θτ ) 2AG (ymed ; θ0 ) ≤ , · 8 maxz∈Ybad ∥ϕ(z)∥2 ∆2 ∆2 πθ0 (ymed )2
πθτ (Ybad )2 ≤
where the second inequality is by 1.69×8 < 2×7 and the last inequality is by maxz∈Ybad ∥ϕ(z)∥2 ≤ ∥ϕ(ymed )∥2 (Assumption 4) and AG (ymed ; θτ ) ≤ AG (ymed ; θ0 ). Hence, it suffices to prove that 2AG (ymed ;θ0 ) ∆2 πθ0 (ymed )2 ≤ 1. Indeed, it holds that: X AG (ymed ; θ0 ) = (1 − πθ0 (ymed ))rG (ymed ) − πθ0 (y⋆ )rG (y⋆ ) − πθ0 (z)rG (z) z∈Ybad
≤ (1 − πθ0 (ymed ))rG (ymed ) + πθ0 (Ybad ) ≤ 0.2rG (ymed ) + 0.1rG (ymed ) 0.82 rG (ymed ) 2 ∆2 πθ0 (ymed )2 ≤ , 2 where the first inequality is from rG (z) ≥ −1; the second inequality is from Lemma 6, according to which πθ0 (Ybad ) ≤ 0.1rG (ymed ) ≤ 0.1 and πθ0 (ymed ) ≥ 0.8; and the last inequality is from 2AG (ymed ;θ0 ) rG (ybad ) ≤ 0 and πθ0 (ymed ) ≥ 0.8. Thus, ∆ 2 ≤ 1, from which it follows that: 2 πθ (ymed ) <
0
7∥ϕ(ymed )∥2 πθ0 (ymed )2 AG (ymed ; θτ ) πθτ (Ybad )2 ≤ , 8 maxz∈Ybad ∥ϕ(z)∥2 ∆2 as desired. d πθt (y⋆ ) t=τ , based on the expression derived in Lemma 13: We proceed by analyzing dt
d πθ (y⋆ ) dt t ht=τ = πθτ (y⋆ ) πθτ (y⋆ )AG (y⋆ ; θτ ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθτ (y⋆ ) − sπθτ (ymed ) + πθτ (ymed )AG (ymed ; θτ ) s − ∥ϕ(ymed )∥2 πθτ (ymed ) − sπθτ (y⋆ ) i X − πθτ (z)2 AG (z; θτ )∥ϕ(z)∥2 z∈Ybad h ≤ πθτ (y⋆ ) πθτ (y⋆ )AG (y⋆ ; θτ ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθτ (y⋆ ) − sπθτ (ymed ) + πθτ (ymed )AG (ymed ; θτ ) s − ∥ϕ(ymed )∥2 πθτ (ymed ) − sπθτ (y⋆ ) i √ X πθτ (z)2 ∥ϕ(z)∥2 + (∆2 − γ) z∈Ybad h 2 ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥ − s)πθτ (y⋆ )AG (y⋆ ; θτ ) 1 − πθτ (y⋆ ) − ∥ϕ(ymed )∥2 πθτ (ymed )2 AG (ymed ; θτ ) i √ X + (∆2 − γ) πθτ (z)2 ∥ϕ(z)∥2 z∈Ybad h 2 ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥ − s)πθ0 (y⋆ )AG (y⋆ ; θ0 ) 1 − πθ0 (y⋆ ) − ∥ϕ(ymed )∥2 πθτ (ymed )2 AG (ymed ; θτ ) i √ + maxz∈Ybad ∥ϕ(z)∥2 (∆2 − γ)πθτ (Ybad )2
46
(25)
√ The first inequality is from −AG (z; θτ ) ≤ −AG (z; θt√γ ) ≤ ∆2 − γ; the second inequality is from s ≤ 0 and πθτ (ymed ) ≤ 1 − πθτ (y⋆ ); and the third Pinequality is from AG (y⋆ ; θτ ) ≤ AG (y⋆ ; θ0 ), πθτ (y⋆ ) ≤ πθ0 (y⋆ ), πθτ (ymed ) ≥ πθ0 (ymed ) and z∈Ybad πθτ (z)2 ≤ πθτ (Ybad )2 . We continue d bounding dt πθt (y⋆ ) t=τ as follows: d πθ (y⋆ ) dt t ht=τ ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θ0 ) 1 − πθ0 (y⋆ ) i √ − ∥ϕ(ymed )∥2 πθτ (ymed )2 AG (ymed ; θτ ) + maxz∈Ybad ∥ϕ(z)∥2 (∆2 − γ)πθτ (Ybad )2 h ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θ0 ) 1 − πθ0 (y⋆ ) i − (∥ϕ(ymed )∥2 /8)πθτ (ymed )2 AG (ymed ; θτ ) h √ i ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θ0 ) 1 − πθ0 (y⋆ ) − (∥ϕ(ymed )∥2 /8)πθτ (ymed )2 γ . (26) Here, the second inequality is from Equation (25) and πθ0 (ymed ) ≤ πθτ (ymed ), according to which √ (∆2 − γ)πθτ (Ybad )2 ≤ ∆2 πθτ (Ybad )2 7∥ϕ(ymed )∥2 πθ0 (ymed )2 AG (ymed ; θτ ) 8 maxz∈Ybad ∥ϕ(z)∥2 7∥ϕ(ymed )∥2 πθτ (ymed )2 AG (ymed ; θτ ) , ≤ 8 maxz∈Ybad ∥ϕ(z)∥2
≤
and the third inequality is from the fact that VG (θt ) is monotonically non-decreasing when maximizing it via gradient flow, so AG (ymed ; θτ ) ≥ AG (ymed ; θt√γ ). Now, by Lemma 6, since ∥ϕ(y⋆ )∥2 > 0 > s we get: ∥ϕ(ymed )∥2 rG (ymed )2/7 γ 13/14 40B 2 (∆1 + ∆2 ) √ ∥ϕ(ymed )∥2 γ · γ 3/7 ≤ 40B 2 AG (y⋆ ; θ0 ) √ 0.82 ∥ϕ(ymed )∥2 γ < 16B 2 AG (y⋆ ; θ0 ) √ ∥ϕ(ymed )∥2 πθ0 (ymed )2 γ ≤ . 8(∥ϕ(y⋆ )∥2 − s)AG (y⋆ ; θ0 )
πθ0 (y⋆ ) ≤
The first inequality is from Item 2 in Lemma 6; the second inequality is from rG (ymed ) ≤ 1 and AG (y⋆ ; θ0 ) ≤ ∆1 + ∆2 ; the third inequality is from γ 3/7 ≤ 1; and the last inequality is from πθ0 (ymed ) = 1 − πθ0 (y⋆ ) − πθ0 (Ybad ) ≥ 1 − γ 13/14 − 0.1rG (ymed ) ≥ 0.8 and ∥ϕ(y⋆ )∥2 − s ≤ 2B 2 . Plugging this upper bound on πθ0 (y⋆ ) into Equation (26), and using πθτ (ymed ) ≥ πθ0 (ymed ) together d with 1 − πθ0 (y⋆ ) < 1, yields dt πθt (y⋆ ) t=τ < 0. This contradicts the fact that τ is defined as the d d initial time at which dt πθt (y⋆ ) t=τ ≥ 0. Thus, it must be that dt πθt (y⋆ ) < 0 for all t ∈ [0, t√γ ]. Step 2: upper bound on πθt√γ (y⋆ ) and monotonicity in [t√γ , tγ ]. Next, we prove that πθt (y⋆ ) keeps decreasing and πθt (ymed ) keeps increasing for all t ∈ [t√γ , tγ ]. To do so, we first derive d d upper bounds on dt πθt (y⋆ ) and dt πθt (ymed ) for t ∈ [0, t√γ ] that allow quantifying how small the √ probability of y⋆ becomes until t γ . d Step 2.1: upper bound on dt πθt (y⋆ ) in [0, t√γ ]. In this step, we prove that for all t ∈ [0, t√γ ]:
d πθ (y⋆ ) ≤ −0.3(∥ϕ(ymed )∥2 − s) · πθt (y⋆ )πθt (ymed )2 AG (ymed ; θt ). dt t 47
d First, for t ∈ [0, t√γ ], based on the expression derived in Lemma 13 we upper bound dt πθt (y⋆ ) as follows:
d πθ (y⋆ ) dt t h
= πθt (y⋆ ) πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed )
+ πθt (ymed )AG (ymed ; θt ) s − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) i X − πθt (z)2 AG (z; θt )∥ϕ(z)∥2 z∈Ybad h ≤ πθt (y⋆ ) (∥ϕ(y⋆ )∥2 − s)πθt (y⋆ )AG (y⋆ ; θt ) 1 − πθt (y⋆ )
(27)
− (∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ) i X − πθt (z)2 AG (z; θt )∥ϕ(z)∥2 z∈Ybad h ≤ πθt (y⋆ ) (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θt ) 1 − πθ0 (y⋆ ) − (∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ) i X − πθt (z)2 AG (z; θt )∥ϕ(z)∥2 . z∈Ybad
The first inequality is from πθt (ymed ) ≤ 1 − πθt (y⋆ ) and the last inequality is by πθt (y⋆ ) ≤ πθ0 (y⋆ ) < 0.5, which implies πθt (y⋆ )(1 − πθt (y⋆ )) ≤ πθ0 (y⋆ )(1 − πθ0 (y⋆ )). Denote by G the term inside the square brackets in the last inequality above. We now prove that G ≤ −0.3(∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ), which is equivalent to G + 0.3(∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ) ≤ 0. Towards doing so, we perform the following sequence of computations: G + 0.3(∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ) = −0.7(∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ) X + (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θt ) 1 − πθ0 (y⋆ ) −
z∈Ybad
πθt (z)2 AG (z; θt )∥ϕ(z)∥2
≤ −0.7(∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ) X + (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θt ) − maxz∈Ybad ∥ϕ(z)∥2 πθt (z)2 AG (z; θt ) z∈Ybad
2
2
= −0.7(∥ϕ(ymed )∥ − s)πθt (ymed ) AG (ymed ; θt ) X + (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θt ) + maxz∈Ybad ∥ϕ(z)∥2 πθt (z)2 (VG (θt ) − rG (z)) z∈Ybad = −0.7(∥ϕ(ymed )∥2 − s)πθt (ymed )2 (1 − πθt (ymed ))rG (ymed ) − πθt (y⋆ )rG (y⋆ ) X − rG (ybad ) πθt (z) + (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θt ) z∈Ybad X maxz∈Ybad ∥ϕ(z)∥2 πθt (z)2 (1 − πθt (z))(−rG (z)) + πθt (y⋆ )rG (y⋆ ) + z∈Ybad X ′ ′ + πθt (ymed )rG (ymed ) + π (z )r (z ) θ G t z ′ ∈Ybad \{z} 2 2 = −0.7(∥ϕ(ymed )∥ − s)πθt (ymed ) πθt (Ybad )∆2 − πθt (y⋆ )∆1 + (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θt ) X + maxz∈Ybad ∥ϕ(z)∥2 πθt (z)2 πθt (ymed )∆2 + πθt (y⋆ )(∆1 + ∆2 ) , z∈Ybad
where the inequality is from 1 − πθ0 (y⋆ ) ≤ 1. 48
Continuing from the last expression above, we get: G + 0.3(∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ) ≤ −0.7(∥ϕ(ymed )∥2 − s)πθt (ymed )2 πθt (Ybad )∆2 − πθt (y⋆ )∆1
+ (∥ϕ(y⋆ )∥2 − s)πθ0 (y⋆ )AG (y⋆ ; θt ) + maxz∈Ybad ∥ϕ(z)∥2 πθt (Ybad )2 πθt (ymed )∆2 + πθt (y⋆ )(∆1 + ∆2 ) h ≤ πθt (Ybad ) − 0.7(∥ϕ(ymed )∥2 − s)πθt (ymed )2 ∆2 i + maxz∈Ybad ∥ϕ(z)∥2 πθt (Ybad )πθt (ymed )∆2 h + πθ0 (y⋆ ) 0.7(∥ϕ(ymed )∥2 − s)πθt (ymed )2 ∆1 + (∥ϕ(y⋆ )∥2 − s)AG (y⋆ ; θt ) i + maxz∈Ybad ∥ϕ(z)∥2 (∆1 + ∆2 )πθt (Ybad )2 h i ≤ πθt (Ybad )∆2 − 0.7(∥ϕ(ymed )∥2 − s)πθt (ymed )2 + ∥ϕ(ymed )∥2 πθt (Ybad )πθt (ymed ) {z } | I1
h + πθ0 (y⋆ ) 0.7(∥ϕ(ymed )∥2 − s)πθt (ymed )2 ∆1 + maxz∈Ybad ∥ϕ(z)∥2 πθt (Ybad )2 (∆1 + ∆2 ) i + (∥ϕ(y⋆ )∥2 − s)AG (y⋆ ; θt ) . P Here, the first inequality is due to z∈Ybad πθt (z)2 ≤ πθt (Ybad )2 ; the second inequality is from πθt (y⋆ ) ≤ πθ0 (y⋆ ); and the last inequality is by Assumption 4. We denote the second term in the last inequality by I2 , i.e., I2 is equal to πθ0 (y⋆ ) multiplied by the term inside the last square brackets. Next, we show that: I1 ≤ −0.2(∥ϕ(ymed )∥2 − s)πθt (Ybad )πθt (ymed )2 ∆2 , I2 ≤ 0.2(∥ϕ(ymed )∥2 − s)πθt (Ybad )πθt (ymed )2 ∆2 , from which it follows that G + 0.3(∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ) ≤ 0, and so G ≤ −0.3(∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ). Plugging this inequality into Equation (27) will d then yield the desired upper bound on dt πθt (y⋆ ). For I1 , it suffices to show that 0.5(∥ϕ(ymed )∥2 − s)πθt (ymed ) ≥ ∥ϕ(ymed )∥2 πθt (Ybad ). From Lemma 6, we have πθ0 (Ybad ) < 0.2 and γ < 0.05, so πθ0 (y⋆ ) ≤ γ 13/14 < 0.13 and πθ0 (ymed ) = 1 − πθ0 (y⋆ ) − πθ0 (Ybad ) > 0.67. This implies that πθt (Ybad ) ≤ πθ0 (Ybad ) < 0.4πθ0 (ymed ) ≤ 0.4πθt (ymed ). Since s ≤ 0, we further obtain: ∥ϕ(ymed )∥2 πθt (Ybad ) < 0.4πθt (ymed )∥ϕ(ymed )∥2 < 0.5(∥ϕ(ymed )∥2 − s)πθt (ymed ), which implies that I1 ≤ −0.2(∥ϕ(ymed )∥2 − s)πθt (Ybad )πθt (ymed )2 ∆2 . For I2 , we have: 0.7(∥ϕ(ymed )∥2 − s)πθt (ymed )2 ∆1 + maxz∈Ybad ∥ϕ(z)∥2 πθt (Ybad )2 (∆1 + ∆2 ) + (∥ϕ(y⋆ )∥2 − s)AG (y⋆ ; θt ) ≤ 0.7(∥ϕ(ymed )∥2 − s)∆1 + 0.1 maxz∈Ybad ∥ϕ(z)∥2 (∆1 + ∆2 ) + (∥ϕ(y⋆ )∥2 − s)AG (y⋆ ; θ0 ) ≤ 2(B 2 − s)(∆1 + ∆2 ), where the first inequality is from πθt (Ybad )2 ≤ πθ0 (Ybad )2 ≤ πθ0 (Ybad ) ≤ 0.1 and VG (θt ) ≥ VG (θ0 ), and the second inequality is from AG (y⋆ ; θ0 ) ≤ ∆1 + ∆2 . Meanwhile, as AG (ymed ; θt ) ≥ √ γ for t ∈ [0, t√γ ], we get: X AG (ymed ; θt ) = (1 − πθt (ymed ))rG (ymed ) − πθt (y⋆ )rG (y⋆ ) − πθt (z)rG (z) z∈Ybad
= −∆1 πθt (y⋆ ) + ∆2 πθt (Ybad ). 49
Thus, −∆1 πθt (y⋆ ) + ∆2 πθt (Ybad ) ≥
√
γ, and rearranging this inequality gives √ √ γ + ∆1 πθt (y⋆ ) γ πθt (Ybad ) ≥ ≥ . ∆2 ∆2
(28)
Here, the second inequality uses ∆1 πθt (y⋆ ) ≥ 0. By the bound on γ in Item 1 of Lemma 6, together with πθ0 (ymed ) > 0.67 (shown above when bounding I1 ), it holds that γ 3/7 ≤
21∥ϕ(ymed )∥2 ∥ϕ(ymed )∥2 πθ0 (ymed )2 ≤ . 1100B 2 (∆1 + ∆2 ) 10B 2 (∆1 + ∆2 )
Moreover, notice that since s < 0 and ∥ϕ(ymed )∥2 ≤ B 2 , we have ∥ϕ(ymed )∥2 − s ∥ϕ(ymed )∥2 ≤ . B2 B2 − s The last two inequalities thus give: πθ0 (y⋆ ) ≤ γ 13/14 =
√
γ · γ 3/7 ≤
√ (∥ϕ(ymed )∥2 − s) γπθ0 (ymed )2 , 10(B 2 − s)(∆1 + ∆2 )
which leads to: I2 ≤ 2(B 2 − s)(∆1 + ∆2 )πθ0 (y⋆ ) ≤ 0.2(∥ϕ(ymed )∥2 − s)πθt (Ybad )πθt (ymed )2 ∆2 . Combining the upper bounds on I1 and I2 , we have shown that I1 + I2 ≤ 0, and so G ≤ −0.3(∥ϕ(ymed )∥2 − s)πθt (ymed )2 AG (ymed ; θt ). Plugging this upper bound on G into Equation (27) establishes that for all t ∈ [0, t√γ ]: d πθ (y⋆ ) ≤ −0.3(∥ϕ(ymed )∥2 − s) · πθt (y⋆ )πθt (ymed )2 AG (ymed ; θt ), dt t as desired. d d πθt (ymed ) in [0, t√γ ]. In Step 1, we showed that dt πθt (ymed ) > 0 for Step 2.2: upper bound on dt t ∈ [0, t√γ ]. In this step, we prove that πθt (ymed ) does not grow too fast by establishing that:
d πθ (ymed ) ≤ 2.1(∥ϕ(ymed )∥2 − s/2)πθt (ymed )2 (1 − πθt (ymed ))AG (ymed ; θt ). dt t d πθt (ymed ) using the expression derived in Lemma 13: We begin upper bounding dt
d πθ (ymed ) dt t h
= πθt (ymed ) πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥2 − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) + πθt (y⋆ )AG (y⋆ ; θt ) s − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) i X − πθt (z)2 AG (z; θt )∥ϕ(z)∥2 z∈Ybad h ≤ πθt (ymed ) πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥2 − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) i X − πθt (z)2 AG (z; θt )∥ϕ(z)∥2 z∈Ybad h ≤ πθt (ymed ) πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥2 − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) i X + maxz∈Ybad ∥ϕ(z)∥2 (∆2 − AG (ymed ; θt )) πθt (z)2 z∈Ybad h 2 ≤ πθt (ymed ) πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥ − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) i + maxz∈Ybad ∥ϕ(z)∥2 (∆2 − AG (ymed ; θt ))πθt (Ybad )2 . 50
(29)
2
Here, the Pfirst inequality is from s − ∥ϕ(y⋆ )∥ πθt (y⋆ ) − sπθt (ymed ) ≤ 0 and the last inequality is from z∈Ybad πθt (z)2 ≤ πθt (Ybad )2 . Our next claim is that 1 − πθt (ymed ) ≥ 2πθt (y⋆ ) for all t ∈ [0, t√γ ]. Indeed, by Lemma 16, for t ∈ [0, t√γ ] it holds that: √ √ γ γ 1 − πθt (ymed ) ≥ ≥ . 1 + rG (ymed ) 2 Moreover, Items 2 and 3 in Lemma 6 give πθt (y⋆ ) ≤ πθ0 (y⋆ ) ≤ γ 13/14 . Under the conditions on γ in √ Lemma 6, it holds that γ ≥ 4γ 13/14 . Thus, 1 − πθt (ymed ) ≥ 2πθt (y⋆ ), from which we conclude: ∥ϕ(ymed )∥2 − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) ≤ (∥ϕ(ymed )∥2 − s/2)(1 − πθt (ymed )). Returning to Equation (29), this leads to: h d πθt (ymed ) ≤ πθt (ymed ) (∥ϕ(ymed )∥2 − s/2)πθt (ymed )(1 − πθt (ymed ))AG (ymed ; θt ) dt i + maxz∈Ybad ∥ϕ(z)∥2 (∆2 − AG (ymed ; θt ))πθt (Ybad )2 h ≤ πθt (ymed )(1 − πθt (ymed )) (∥ϕ(ymed )∥2 − s/2)πθt (ymed )AG (ymed ; θt ) i + maxz∈Ybad ∥ϕ(z)∥2 (∆2 − AG (ymed ; θt ))πθt (Ybad ) {z } |
(30)
G
where the second inequality is from πθt (Ybad ) ≤ 1−πθt (ymed ). We now show that 1.1(∥ϕ(ymed )∥2 − s/2)πθt (ymed )AG (ymed ; θt ) − G ≥ 0: 1.1(∥ϕ(ymed )∥2 − s/2)πθt (ymed )AG (ymed ; θt ) − G = 1.1(∥ϕ(ymed )∥2 − s/2)πθt (ymed )AG (ymed ; θt ) − maxz∈Ybad ∥ϕ(z)∥2 πθt (Ybad )(∆2 − AG (ymed ; θt )) ≥ 1.1∥ϕ(ymed )∥2 πθt (ymed ) rG (ymed ) − VG (θt ) − maxz∈Ybad ∥ϕ(z)∥2 πθt (Ybad ) VG (θt ) − rG (ybad ) = 1.1∥ϕ(ymed )∥2 πθt (ymed ) πθt (Ybad )∆2 − πθt (y⋆ )∆1 − maxz∈Ybad ∥ϕ(z)∥2 πθt (Ybad ) πθt (ymed )∆2 + πθt (y⋆ )(∆1 + ∆2 ) = πθt (Ybad ) 1.1∥ϕ(ymed )∥2 πθt (ymed )∆2 − maxz∈Ybad ∥ϕ(z)∥2 πθt (ymed )∆2 − πθt (y⋆ ) 1.1∥ϕ(ymed )∥2 πθt (ymed )∆1 + maxz∈Ybad ∥ϕ(z)∥2 πθt (Ybad )(∆1 + ∆2 )
≥ 0.1πθt (Ybad )πθt (ymed )∆2 ∥ϕ(ymed )∥2 {z } | I1
− πθt (y⋆ )∥ϕ(ymed )∥2 1.1πθt (ymed )∆1 + πθt (Ybad )(∆1 + ∆2 ) . | {z } I2
Here, the first inequality is from s ≤ 0 and the last inequality is from Assumption 4. We proceed by showing that I2 ≤ 0.1∥ϕ(ymed )∥2 πθt (ymed )πθt (Ybad )∆2 = I1 . From Lemma 6, we have πθt (Ybad ) ≤ πθ0 (Ybad ) < 0.4πθ0 (ymed ) ≤ 0.4πθt (ymed ). Therefore, 1.1πθt (ymed )∆1 + πθt (Ybad )(∆1 + ∆2 ) < 1.1πθt (ymed )∆1 + 0.4πθt (ymed )(∆1 + ∆2 ) ≤ 1.5πθt (ymed )(∆1 + ∆2 ). √
γ
As shown in Equation (28), πθt (Ybad ) ≥ ∆2 for t ∈ [0, t√γ ]. Furthermore, by Lemma 6 we know that 7/3 ∥ϕ(ymed )∥2 γ≤ , 50B 2 (∆1 + ∆2 ) and √ γ∥ϕ(ymed )∥2 πθ0 (y⋆ ) ≤ γ 13/14 ≤ . 50B 2 (∆1 + ∆2 ) 51
Thus, we arrive at: I2 ≤ 1.5B 2 πθt (ymed )πθ0 (y⋆ )(∆1 + ∆2 ) ≤ 0.03∥ϕ(ymed )∥2 πθt (ymed )πθt (Ybad )∆2 ≤ 0.1∥ϕ(ymed )∥2 πθt (ymed )πθt (Ybad )∆2 = I1 , which implies that G ≤ 1.1(∥ϕ(ymed )∥2 −s/2)πθt (ymed )AG (ymed ; θt ). Going back to Equation (30), d the desired upper bound on dt πθt (ymed ) follows: h i d πθt (ymed ) ≤ πθt (ymed )(1 − πθt (ymed )) (∥ϕ(ymed )∥2 − s/2)πθt (ymed )AG (ymed ; θt ) + G dt h ≤ πθt (ymed )(1 − πθt (ymed )) (∥ϕ(ymed )∥2 − s/2)πθt (ymed )AG (ymed ; θt )
i + 1.1(∥ϕ(ymed )∥2 − s/2)πθt (ymed )AG (ymed ; θt ) = 2.1(∥ϕ(ymed )∥2 − s/2)πθt (ymed )2 (1 − πθt (ymed ))AG (ymed ; θt ). Step 2.3: upper bound on πθt√γ (y⋆ ). To summarize, we showed that for all t ∈ [0, t√γ ]: d πθ (y⋆ ) ≤ −0.3(∥ϕ(ymed )∥2 − s) · πθt (y⋆ )πθt (ymed )2 AG (ymed ; θt ), dt t d πθ (ymed ) ≤ 2.1(∥ϕ(ymed )∥2 − s/2) · πθt (ymed )2 (1 − πθt (ymed ))AG (ymed ; θt ). dt t Since both sides of the first inequality are negative and both sides of the second inequality are positive, we can divide them to obtain: d ∥ϕ(ymed )∥2 − s πθt (y⋆ ) dt πθt (y⋆ ) ≤ − · d 2 7(∥ϕ(ymed )∥ − s/2) 1 − πθt (ymed ) dt πθt (ymed )
= −h(s) ·
πθt (y⋆ ) , 1 − πθt (ymed )
2
∥ϕ(ymed )∥ −s (Equation (18)). We can rewrite this inequality as: where recall that h(s) = 7(∥ϕ(y 2 med )∥ −s/2) d dt πθt (y⋆ )
πθt (y⋆ )
≤ h(s) ·
d d − dt πθt (ymed ) (1 − πθt (ymed )) = h(s) · dt , 1 − πθt (ymed ) 1 − πθt (ymed )
from which it follows that: d d ln πθt (y⋆ ) ≤ h(s) · ln(1 − πθt (ymed )). dt dt Integrating both sides over [0, t√γ ] gives: ln πθt (y⋆ )
t√γ 0
≤ h(s) · ln(1 − πθt (ymed ))
t√ γ
.
0
Therefore, πθt√γ (y⋆ ) πθ0 (y⋆ )
≤
1 − πθt√γ (ymed )
!h(s)
1 − πθ0 (ymed )
,
and hence πθt√γ (y⋆ ) ≤
πθ0 (y⋆ )(1 − πθt√γ (ymed ))h(s) (1 − πθ0 (ymed ))h(s) 52
.
(31)
√
Next, we prove that 1 − πθt√γ (ymed ) ≤ √
that 1 − πθt√γ (ymed ) >
γ+πθ0 (y⋆ ) rG (ymed ) . To see this, assume by way of contradiction
γ+πθ0 (y⋆ ) rG (ymed ) . Then, we would have:
VG (θt√γ ) = πθt√γ (ymed )rG (ymed ) + πθt√γ (y⋆ )rG (y⋆ ) + πθt√γ (Ybad )rG (ybad ) √ γ + πθ0 (y⋆ ) < 1− rG (ymed ) + πθ0 (y⋆ ) rG (ymed ) √ ≤ rG (ymed ) − γ, where ybad ∈ Ybad , and so rG (ybad ) ≤ 0. This contradicts the definition of t√γ as the initial time at √ which VG (θt ) ≥ rG (ymed ) − γ. Plugging the upper bound on 1 − πθt√γ (ymed ) into Equation (31) then gives: h(s) √ πθ0 (y⋆ ) γ + πθ0 (y⋆ ) πθt√γ (y⋆ ) ≤ h(s) rG (ymed )(1 − πθ0 (ymed )) √ πθ0 (y⋆ )(1.3 γ)h(s) ≤ (0.05rG (ymed )2 )h(s) h(s) √ 26 γ . = πθ0 (y⋆ ) · rG (ymed )2
(32)
where the second inequality is from 1 − πθ0 (ymed ) ≥ πθ0 (Ybad ) ≥ 0.05 · rG (ymed ) and πθ0 (y⋆ ) ≤ √ γ 13/14 ≤ 0.3 γ (see Lemma 6). From Equation (32) and Item 2 in Lemma 6 we then get: √ πθt√γ (y⋆ ) ≤ πθ0 (y⋆ ) · (26 γ/rG (ymed )2 )h(s) h(s) √ 26 γ ∥ϕ(ymed )∥2 γ 13/14 rG (ymed )2/7 · ≤ 40B 2 (∆1 + ∆2 ) rG (ymed )2 h(s) √ 26 γ 0.82 ∥ϕ(ymed )∥2 γ 13/14 rG (ymed )2/7 · < rG (ymed )2 16 · 261/7 B 2 (∆1 + ∆2 ) 13
≤
h(s)
C∥ϕ(ymed )∥2 πθ0 (ymed )2 γ 14 + 2 . 8(∥ϕ(y⋆ )∥2 − s)AG (y⋆ ; θ0 )
(33)
Here, the last inequality uses h(s) = 1/7 + 13α/7, and hence rG (ymed )2/7 (26/rG (ymed )2 )h(s) = 261/7 C, together with πθ0 (ymed ) ≥ 0.8, ∥ϕ(y⋆ )∥2 − s ≤ 2B 2 , and AG (y⋆ ; θ0 ) ≤ ∆1 + ∆2 . Step 2.4: completing the proof of monotonicity in [t√γ , tγ ] Finally, we conclude the proof of Proposition 5. We first show that t√γ < tγ , i.e., we prove that the interval [t√γ , tγ ] is non-empty. Recall that h(s) = 1/7 + 13α/7 and C = (26/rG (ymed )2 )13α/7 . Therefore, 13α/7 √ 26 γ Cγ 13/14+h(s)/2 √ = Cγ 3/7+h(s)/2 = γ . √ γ rG (ymed )2 Moreover, by Item 3 in Lemma 6, √ 2 γ ≤ πθ0 (Ybad ) ≤ 0.1rG (ymed ), ∆2 √ and since ∆2 = rG (ymed ) − rG (ybad ) ≤ rG (ymed ) + 1 ≤ 2, for ybad ∈ Ybad , we get γ ≤ 0.1rG (ymed ). Using also the fact that 13α/7 ∈ (0, 1/7) and rG (ymed ) ≤ 1, this yields 13α/7 Cγ 13/14+h(s)/2 2.6 ≤ 0.1rG (ymed ) √ γ rG (ymed ) = 0.1 · 2.613α/7 rG (ymed )1−13α/7 < 1. √
Hence, Cγ 13/14+h(s)/2 < γ. Thus, the threshold rG (ymed ) − Cγ 13/14+h(s)/2 is strictly larger than √ rG (ymed ) − γ, and by continuity of VG (θt ) together with the definitions of t√γ and tγ , it follows that t√γ < tγ . 53
Now, we can consider the optimization dynamics over [t√γ , tγ ] and show that πθt (y⋆ ) continues to decrease during this time interval. Assume by way of contradiction that there exists t′ ∈ [0, tγ ] such d d that dt πθt (y⋆ ) t=t′ ≥ 0. Let τ be the initial time in [0, tγ ] at which dt πθt (y⋆ ) ≥ 0, i.e.: d τ := min t ∈ [0, tγ ] : dt πθt (y⋆ ) ≥ 0 . d d By definition, dt πθt (y⋆ ) < 0, and so dt πθt (ymed ) > 0 (Lemma 7), for all t ∈ [0, τ ). We also d √ know that if τ exists, then τ ≥ t γ since dt πθt (y⋆ ) < 0 for all t ∈ [0, t√γ ]. This implies that πθτ (y⋆ ) ≤ πθt√γ (y⋆ ) and πθτ (ymed ) ≥ πθ0 (ymed ). d πθt (Ybad ) ≤ 0 for all t ∈ [0, τ ). We Now, by the same arguments made in Step 1, it follows that dt
continue by showing that πθτ (Ybad )2 ≤
7∥ϕ(ymed )∥2 πθ0 (ymed )2 AG (ymed ;θτ ) . To see this, notice that: 8 maxz∈Ybad ∥ϕ(z)∥2 ∆2
AG (ymed ; θτ ) = (1 − πθτ (ymed ))rG (ymed ) − πθτ (y⋆ )rG (y⋆ ) −
X z∈Ybad
πθτ (z)rG (z)
= −∆1 πθτ (y⋆ ) + ∆2 πθτ (Ybad ). Thus, πθτ (Ybad ) =
AG (ymed ; θτ ) + ∆1 πθτ (y⋆ ) 1.3AG (ymed ; θτ ) ≤ , ∆2 ∆2 2
2/7
)∥ rG (ymed ) γ where the inequality is from Item 2 in Lemma 6 since πθ0 (y⋆ ) ≤ ∥ϕ(ymed40B 2 (∆ +∆ ) 1 2
13/14
≤
rG (ymed )2/7 γ 13/14 0.3γ 13/14 ≤ ∆1 (26/r 2 1/7 , which combined with Equation (32) yields ∆1 πθτ (y⋆ ) ≤ 40(∆1 +∆2 ) G (ymed ) ) √ ∆1 πθt√γ (y⋆ ) ≤ ∆1 πθ0 (y⋆ ) · (26 γ/rG (ymed )2 )h(s) ≤ 0.3Cγ 13/14+h(s)/2 ≤ 0.3AG (ymed ; θτ ).
Squaring both sides of the inequality, we then have: 1.69AG (ymed ; θτ )2 ∆22 7∥ϕ(ymed )∥2 πθ0 (ymed )2 AG (ymed ; θτ ) 2 maxz∈Ybad ∥ϕ(z)∥2 AG (ymed ; θτ ) < · 8 maxz∈Ybad ∥ϕ(z)∥2 ∆2 ∥ϕ(ymed )∥2 ∆2 πθ0 (ymed )2 2 2 7∥ϕ(ymed )∥ πθ0 (ymed ) AG (ymed ; θτ ) 2AG (ymed ; θ0 ) ≤ · . 8 maxz∈Ybad ∥ϕ(z)∥2 ∆2 ∆2 πθ0 (ymed )2
πθτ (Ybad )2 ≤
The second inequality is from 1.69 × 8 < 2 × 7 and the last inequality is by maxz∈Ybad ∥ϕ(z)∥2 ≤ ∥ϕ(ymed )∥2 (Assumption 4) and AG (ymed ; θτ ) ≤ AG (ymed ; θ0 ). We now prove that the second term on the right-hand side of the inequality above is at most 1. To do so, we upper bound AG (ymed ; θ0 ): X AG (ymed ; θ0 ) = (1 − πθ0 (ymed ))rG (ymed ) − πθ0 (y⋆ )rG (y⋆ ) − πθ0 (z)rG (z) z∈Ybad
≤ (1 − πθ0 (ymed ))rG (ymed ) + πθ0 (Ybad ) ≤ 0.2rG (ymed ) + 0.1rG (ymed ) = 0.3rG (ymed ) 0.82 rG (ymed ) 2 ∆2 πθ0 (ymed )2 ≤ . 2 <
The first inequality is due to rG (z) ≥ −1 for all z ∈ Ybad ; the second inequality is from Lemma 6 since πθ0 (Ybad ) ≤ 0.1rG (ymed ) ≤ 0.1 and πθ0 (ymed ) ≥ 0.8; and the last inequality is from 2AG (ymed ;θ0 ) rG (ybad ) ≤ 0 and πθ0 (ymed ) ≥ 0.8. Therefore, ∆ 2 ≤ 1, which leads to 2 πθ (ymed ) 0
πθτ (Ybad )2 ≤
7∥ϕ(ymed )∥2 πθ0 (ymed )2 AG (ymed ; θτ ) . 8 maxz∈Ybad ∥ϕ(z)∥2 ∆2 54
(34)
d Let us now consider dt πθt (y⋆ ) t=τ and obtain a contradiction by showing that it is negative. We start with the following computations, based on the expression derived in Lemma 13:
h d = πθτ (y⋆ ) πθτ (y⋆ )AG (y⋆ ; θτ ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθτ (y⋆ ) − sπθτ (ymed ) πθt (y⋆ ) dt t=τ + πθτ (ymed )AG (ymed ; θτ ) s − ∥ϕ(ymed )∥2 πθτ (ymed ) − sπθτ (y⋆ ) i X − πθτ (z)2 AG (z; θτ )∥ϕ(z)∥2 z∈Ybad h ≤ πθτ (y⋆ ) πθτ (y⋆ )AG (y⋆ ; θτ ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθτ (y⋆ ) − sπθτ (ymed )
+ πθτ (ymed )AG (ymed ; θτ ) s − ∥ϕ(ymed )∥2 πθτ (ymed ) − sπθτ (y⋆ ) i h(s) X 13 + maxz∈Ybad ∥ϕ(z)∥2 ∆2 − γ 14 + 2 πθτ (z)2 z∈Ybad h 2 ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥ − s)πθτ (y⋆ )AG (y⋆ ; θτ ) 1 − πθτ (y⋆ )
− ∥ϕ(ymed )∥2 πθτ (ymed )2 AG (ymed ; θτ ) 13
h(s)
+ maxz∈Ybad ∥ϕ(z)∥2 (∆2 − γ 14 + 2 )
X z∈Ybad
13
i πθτ (z)2 .
h(s)
The first inequality is from −AG (z; θτ ) = ∆2 − AG (ymed ; θτ ) ≤ ∆2 − γ 14 + 2 for all z ∈ Ybad (this holds since τ ≤ tγ and C ≥ 1) and the second inequality is from s ≤ 0 and πθτ (ymed ) ≤ d 1 − πθτ (y⋆ ). We continue upper bounding dt πθt (y⋆ ) t=τ as follows:
h d ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥2 − s)πθt√γ (y⋆ )AG (y⋆ ; θ0 ) 1 − πθt√γ (y⋆ ) πθt (y⋆ ) dt t=τ − ∥ϕ(ymed )∥2 πθτ (ymed )2 AG (ymed ; θτ ) h(s) 13 + maxz∈Ybad ∥ϕ(z)∥2 ∆2 − γ 14 + 2 πθτ (Ybad )2 h ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥2 − s)πθt√γ (y⋆ )AG (y⋆ ; θ0 ) 1 − πθt√γ (y⋆ ) i − (∥ϕ(ymed )∥2 /8)πθτ (ymed )2 AG (ymed ; θτ ) h ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥2 − s)πθt√γ (y⋆ )AG (y⋆ ; θ0 ) 1 − πθt√γ (y⋆ ) i h(s) 13 − (C∥ϕ(ymed )∥2 /8)πθτ (ymed )2 γ 14 + 2 .
The first inequality is from AG (y⋆ ; θτ ) ≤ AG (y⋆ ; θ0 ), πθτ (y⋆ ) ≤ πθt√γ (y⋆ ), πθτ (ymed ) ≥ P πθt√γ (ymed ), and z∈Ybad πθτ (z)2 ≤ πθτ (Ybad )2 ; the second inequality is from
13
h(s)
maxz∈Ybad ∥ϕ(z)∥2 (∆2 − γ 14 + 2 )πθτ (Ybad )2 ≤ maxz∈Ybad ∥ϕ(z)∥2 ∆2 πθτ (Ybad )2 ≤
7∥ϕ(ymed )∥2 πθτ (ymed )2 AG (ymed ; θτ ) , 8
which is implied by Equation (34), and πθτ (ymed ) ≥ πθ0 (ymed ); and the last inequality is from h(s) 13 AG (ymed ; θτ ) ≥ AG (ymed ; θtγ ) = Cγ 14 + 2 . 55
Therefore, using the upper bound on πθt√γ (y⋆ ) from Equation (33): d πθ (y⋆ ) dt t ht=τ ≤ πθτ (y⋆ ) (∥ϕ(y⋆ )∥2 − s)πθt√γ (y⋆ )AG (y⋆ ; θ0 ) 1 − πθt√γ (y⋆ ) i h(s) 13 − (C∥ϕ(ymed )∥2 /8)πθτ (ymed )2 γ 14 + 2 h i h(s) h(s) 13 13 < πθτ (y⋆ ) (C∥ϕ(ymed )∥2 /8)πθ0 (ymed )2 γ 14 + 2 − (C∥ϕ(ymed )∥2 /8)πθτ (ymed )2 γ 14 + 2 h(s) 13 Cπθτ (y⋆ )∥ϕ(ymed )∥2 γ 14 + 2 πθ0 (ymed )2 − πθτ (ymed )2 8 ≤ 0,
=
where the last inequality is from πθτ (ymed ) ≥ πθ0 (ymed ). Thus, the strict inequality above implies d d dt πθt (y⋆ ) t=τ < 0, which contradicts the fact that τ is the initial time at which dt πθt (y⋆ ) t=τ ≥ 0. d Hence, it must be that dt πθt (y⋆ ) < 0 for all t ∈ [0, tγ ]. Additionally, by Lemma 7, we also have d dt πθt (ymed ) > 0 for all t ∈ [0, tγ ]. d d πθt (y⋆ ) < 0 and dt πθt (ymed ) > 0 for all t ∈ [0, tγ ]. Thus, To conclude, we showed that dt √ mint∈[0,tγ ] πθt (y⋆ ) = πθtγ (y⋆ ). Furthermore, since t γ < tγ , by Equation (33) it follows that 13
h(s)
C∥ϕ(ymed )∥2 πθ0 (ymed )2 γ 14 + 2 πθtγ (y⋆ ) < πθt√γ (y⋆ ) ≤ , 8(∥ϕ(y⋆ )∥2 − s)AG (y⋆ ; θ0 ) completing the proof of Proposition 5. Based on Proposition 5, we can now complete the proof of Case I. 13
h(s)
Proof of Theorem 3, Case I. By Proposition 5, as long as VG (θt ) ≤ rG (ymed ) − Cγ 14 + 2 , we know that πθt (y⋆ ) is monotonically decreasing and πθt (ymed ) is monotonically increasing. Recall h(s) 13 that tγ denotes the initial time at which VG (θt ) ≥ rG (ymed )−Cγ 14 + 2 . Since VG (θt ) is continuous h(s) 13 in t, at time tγ it must be that VG (θtγ ) = rG (ymed ) − Cγ 14 + 2 and tγ is the initial time at which this equality holds. We first prove that at tγ the policy is highly concentrated on ymed by upper bounding 1 − πθtγ (ymed ). This follows from 13
h(s)
rG (ymed ) − Cγ 14 + 2 = VG (θtγ ) = rG (ymed )πθtγ (ymed ) + πθtγ (y⋆ )rG (y⋆ ) +
X z∈Ybad
πθtγ (z)rG (z),
which implies: ! X h(s) 13 1 + Cγ 14 2 + πθtγ (y⋆ )rG (y⋆ ) + πθtγ (z)rG (z) 1 − πθtγ (ymed ) = rG (ymed ) z∈Ybad ! h(s) 13 h(s) 13 C∥ϕ(ymed )∥2 πθ0 (ymed )2 γ 14 + 2 1 + Cγ 14 2 + ≤ rG (ymed ) 8(∥ϕ(y⋆ )∥2 − s)AG (y⋆ ; θ0 ) 13
(35)
h(s)
Kγ 14 + 2 = . rG (ymed ) C∥ϕ(y
)∥2 π
(y
13
)2 γ 14 +
h(s) 2
med θ0 med The inequality is due to πθtγ (y⋆ ) ≤ from Proposition 5, together 8(∥ϕ(y⋆ )∥2 −s)AG (y⋆ ;θ0 ) P with rG (y⋆ ) ≤ 1 and z∈Ybad πθtγ (z)rG (z) ≤ 0, and the last equality is by the definition of K in Theorem 3.
56
Now, the fact that the policy assigns high probability to ymed at tγ allows us to lower bound t⋆ —the initial time at which VG (θt ) ≥ rG (y⋆ ) − ϵ. For any time T ≥ tγ , we have: Z T ∥θT − θtγ ∥ =
d θt dt dt tγ
Z T ≤ tγ
d θt dt dt
(36)
Z T ∥∇VG (θt )∥dt
= tγ
Z T ≤
2B(∆1 + ∆2 )(1 − πθt (ymed ))dt, tγ
where the last inequality is by Lemma 11. We proceed by upper bounding the rate at which 1 − πθt (ymed ) increases (i.e., the rate at which πθt (ymed ) decreases). From the chain rule we obtain: D d d E (1 − πθt (ymed )) = ∇(1 − πθt (ymed )), θt dt dt = ⟨∇(1 − πθt (ymed )), ∇VG (θt )⟩ D E X = ∇(πθt (y⋆ ) + πθt (z)), ∇VG (θt ) . z∈Ybad
For all y ∈ Y, the gradient of πθt (y) with respect to θt can be computed as follows: X ∇πθt (y) = πθt (y) ϕ(y) − πθt (z)ϕ(z) = πθt (y) ϕ(y) − ϕ̄θt , z∈Y
where ϕ̄θt :=
P
z∈Y πθt (z)ϕ(z). This yields:
X d (1 − πθt (ymed )) ≤ πθt (y) ϕ(y) − ϕ̄θt , ∇VG (θt ) y∈Y\{ymed } dt X ≤ πθt (y)∥ϕ(y) − ϕ̄θt ∥∥∇VG (θt )∥ y∈Y\{ymed } X ≤ maxy∈Y\{ymed } ∥ϕ(y) − ϕ̄θt ∥∥∇VG (θt )∥ ·
y∈Y\{ymed }
πθt (y)
≤ 4B 2 (∆1 + ∆2 )(1 − πθt (ymed ))2 . Here, the second inequality is due to the Cauchy-Schwarz inequality; the third inequality is due to ∥ϕ(z) − ϕ̄θt ∥ ≤ maxy∈Y\{ymed } ∥ϕ(y) − ϕ̄θt ∥ for all z ∈ Y \ {ymed }; and the last inequality is from maxy∈Y\{ymed } ∥ϕ(y) − ϕ̄θt ∥ ≤ 2B and Lemma 11. This implies that: d (1 − πθt (ymed )) d 1 = − dt ≥ −4B 2 (∆1 + ∆2 ). dt 1 − πθt (ymed ) (1 − πθt (ymed ))2 Integrating both sides from tγ to t, for any t ≥ tγ , leads to: 1 1 − ≥ −4B 2 (∆1 + ∆2 )(t − tγ ). 1 − πθt (ymed ) 1 − πθtγ (ymed ) Hence, for any t ≥ tγ , such that 4B 2 (∆1 + ∆2 )(1 − πθtγ (ymed ))(t − tγ ) < 1, it holds that 1 − πθt (ymed ) ≤
1 − πθtγ (ymed ) 1 − 4B 2 (∆
1 + ∆2 )(1 − πθtγ (ymed ))(t − tγ )
57
.
Thus, for any T ≥ tγ satisfying 4B 2 (∆1 + ∆2 )(1 − πθtγ (ymed ))(T − tγ ) < 1, we can plug this upper bound into Equation (36) to get: Z T ∥θT − θtγ ∥ ≤ 2B(∆1 + ∆2 ) (1 − πθt (ymed ))dt tγ
Z T −tγ = 2B(∆1 + ∆2 ) 0
1 ln = 2B
1 − πθtγ (ymed ) 1 − 4B 2 (∆
1 + ∆2 )(1 − πθtγ (ymed ))u
du
(37)
1 . 1 − 4B 2 (∆1 + ∆2 )(1 − πθtγ (ymed ))(T − tγ )
By Lemma 12, VG is B-Lipschitz with respect to θ, and so: 1 1 |VG (θT ) − VG (θtγ )| ≤ B∥θT − θtγ ∥ ≤ ln . 2 1 − 4B 2 (∆1 + ∆2 )(1 − πθtγ (ymed ))(T − tγ ) If t⋆ = ∞, then the desired lower bound holds trivially. Hence, assume t⋆ < ∞. By the definition of t⋆ and continuity of VG (θt ), VG (θt⋆ ) = rG (y⋆ ) − ϵ. In particular, t⋆ > tγ since VG (θt⋆ ) = rG (y⋆ ) − ϵ > rG (ymed ) > VG (θtγ ). Therefore, 13
h(s)
VG (θt⋆ ) − VG (θtγ ) = rG (y⋆ ) − ϵ − rG (ymed ) + Cγ 14 + 2 ≥ ∆1 − ϵ. If 4B 2 (∆1 + ∆2 )(1 − πθtγ (ymed ))(t⋆ − tγ ) ≥ 1, then directly t⋆ − tγ ≥
4B 2 (∆
1 − e−2(∆1 −ϵ) 1 ≥ , 2 4B (∆1 + ∆2 )(1 − πθtγ (ymed )) 1 + ∆2 )(1 − πθtγ (ymed ))
since 1 − e−2(∆1 −ϵ) ≤ 1. Otherwise, we apply the bound in Equation (37) to get: 1 1 ∆1 − ϵ ≤ VG (θt⋆ ) − VG (θtγ ) ≤ ln , 2 1 − 4B 2 (∆1 + ∆2 )(1 − πθtγ (ymed ))(t⋆ − tγ ) which implies t⋆ − t γ ≥
4B 2 (∆
1 − e−2(∆1 −ϵ) . 1 + ∆2 )(1 − πθtγ (ymed ))
Therefore, in either case: t⋆ ≥ t⋆ − tγ ≥ ≥
1 − e−2(∆1 −ϵ) 4B 2 (∆1 + ∆2 )(1 − πθtγ (ymed )) rG (ymed )(1 − e−2(∆1 −ϵ) ) 13
h(s)
4B 2 (∆1 + ∆2 )Kγ 14 + 2
rG (ymed )M ′14/13+α (1 − e−2(∆1 −ϵ) ) · πθ0 (y⋆ )−14/13−α 4B 2 (∆1 + ∆2 )K = Ω πθ0 (y⋆ )−14/13−α .
=
where the third inequality follows from Equation (35). This completes the proof for Case I of Theorem 3. B.5.2
Proof of Case II
Suppose that gradient flow is used to maximize the expected reward with respect to rP , which assigns ymed a low reward rP (ymed ) = miny∈Ybad rG (y). Similarly to the proof of Case II in Theorem 2, we show that in this case πθt (y⋆ ) increases already from t = 0 and that tno−med = O(πθ0 (y⋆ )−1 ), ⋆ no−med where recall that t⋆ is the initial time at which VG (θt ) ≥ rG (y⋆ ) − ϵ when maximizing rP . 58
Proof of Theorem 3, Case II. Notice that VG (θ) ≥ VP (θ) for all θ ∈ RD since rG (y) ≥ rP (y) for all y ∈ Y. Thus, when VP (θt ) ≥ rG (y⋆ ) − ϵ it also holds that VG (θt ) ≥ rG (y⋆ ) − ϵ. We can therefore focus on VP and consider the time it takes until it reaches a value of rG (y⋆ ) − ϵ. In particular, denote by tno−med the initial time at which VP (θt ) ≥ rG (y⋆ ) − ϵ, i.e.: P tno−med := min{t ≥ 0 : VP (θt ) ≥ rG (y⋆ ) − ϵ}. P The discussion above implies that tno−med ≤ tno−med , and so it suffices to upper bound tno−med . ⋆ P P rG (y⋆ )+1−ϵ For simplicity of notation, denote ρ := rG (y⋆ )+1 . Observe that when πθt (y⋆ ) ≥ ρ, it holds that VP (θt ) ≥ rG (y⋆ ) − ϵ. This is because: VP (θt ) ≥ πθt (y⋆ )rP (y⋆ ) − (1 − πθt (y⋆ )) = πθt (y⋆ )(rG (y⋆ ) + 1) − 1 ≥ rG (y⋆ ) − ϵ. Thus, πθt (y⋆ ) ≤ ρ for all t ∈ [0, tno−med ]. P We prove that T :=
1 (1 − ρ)2 AP (y⋆ ; θ0 )∥ϕ(y⋆ )∥2
1 1 − πθ0 (y⋆ ) ρ
≥ tno−med , P
from which the desired upper bound on tno−med immediately follows by tno−med ≤ tno−med and ⋆ ⋆ P substituting the value of ρ. Assume by way of contradiction that tno−med > T . Let us lower bound the rate at which πθt (y⋆ ) P increases over [0, tno−med ]. Notice first that VP (θ0 ) > rP (y) for all y ∈ Y \ {y⋆ } since all such y P have the same proxy reward value, which is lower than rP (y⋆ ) = rG (y⋆ ). Now, consider the time derivative of πθt (y⋆ ), starting from the expression derived in Lemma 13: d πθ (y⋆ ) dt t h
= πθt (y⋆ ) πθt (y⋆ )AP (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed )
+ πθt (ymed )AP (ymed ; θt ) s − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) i X − πθt (z)2 AP (z; θt )∥ϕ(z)∥2 z∈Ybad h ≥ πθt (y⋆ ) πθt (y⋆ )AP (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) − sπθt (ymed ) i + πθt (ymed )AP (ymed ; θt ) s − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) h ≥ πθt (y⋆ ) πθt (y⋆ )AP (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) i + πθt (ymed )AP (ymed ; θt ) s − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) ≥ πθt (y⋆ )2 ∥ϕ(y⋆ )∥2 AP (y⋆ ; θt ) 1 − πθt (y⋆ ) . Here, the first inequality is from AP (z; θt ) ≤ 0 for all z ∈ Ybad ; the second inequality is from s ≤ 0; and the last inequality is from AP (ymed ; θt ) ≤ AP (ymed ; θ0 ) ≤ 0 together with s − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) = s(1 − πθt (y⋆ )) − ∥ϕ(ymed )∥2 πθt (ymed ) ≤ 0, so the ymed term is non-negative. Next, we show that AP (y⋆ ; θt ) ≥ (1 − ρ)AP (y⋆ ; θ0 ). Observe that since rP (y⋆ ) = rG (y⋆ ) and VP (θ0 ) ≥ −1, rG (y⋆ ) − ϵ − VP (θ0 ) ϵ ϵ rG (y⋆ ) + 1 − ϵ =1− ≤1− = = ρ. rP (y⋆ ) − VP (θ0 ) rP (y⋆ ) − VP (θ0 ) rP (y⋆ ) + 1 rG (y⋆ ) + 1 Using this fact, we have that: rP (y⋆ ) − VP (θt ) VP (θt ) − VP (θ0 ) rG (y⋆ ) − ϵ − VP (θ0 ) AP (y⋆ ; θt ) = =1− ≥1− ≥ 1 − ρ. AP (y⋆ ; θ0 ) rP (y⋆ ) − VP (θ0 ) rP (y⋆ ) − VP (θ0 ) rP (y⋆ ) − VP (θ0 ) 59
Therefore, along with 1 − πθt (y⋆ ) ≥ 1 − ρ for t ∈ [0, tno−med ] (as proven above), this leads to: P d πθ (y⋆ ) ≥ (1 − ρ)2 AP (y⋆ ; θ0 )∥ϕ(y⋆ )∥2 πθt (y⋆ )2 . dt t Dividing both sides of the inequality above by πθt (y⋆ )2 , integrating from 0 to t, and rearranging terms results in the following lower bound: πθt (y⋆ ) ≥
πθ0 (y⋆ ) . 1 − (1 − ρ)2 AP (y⋆ ; θ0 )∥ϕ(y⋆ )∥2 πθ0 (y⋆ ) · t
At time t = T for T =
1 2 (1 − ρ) AP (y⋆ ; θ0 )∥ϕ(y⋆ )∥2
1 1 − , πθ0 (y⋆ ) ρ
this lower bound on πθt (y⋆ ) implies that πθt (y⋆ ) ≥ ρ, in contradiction to our assumption that tno−med > T . Thus, P tno−med ≤ tno−med ≤T ≤ ⋆ P
B.6
2 (rG (y⋆ ) + 1) −1 = O πθ0 (y⋆ )−1 . 2 · πθ0 (y⋆ ) 2 ϵ rP (y⋆ ) − VP (θ0 ) ∥ϕ(y⋆ )∥
Proof of Proposition 3
The core of the proof is to show that for all t ≥ 0: πθt (y⋆ ) ∆2 <ζ≤ . πθt (ybad ) ∆1
(38)
That is, the policy never assigns substantially higher probability to y⋆ relative to ybad . We first establish that this bound implies the desired conclusion. Towards this, we start by showing that VG (θ0 ) ≤ rG (ymed ). Indeed: X rG (ymed ) − VG (θ0 ) = rG (ymed ) − πθ0 (y⋆ )rG (y⋆ ) − πθ0 (ymed )rG (ymed ) − πθ0 (z)rG (z) z∈Ybad
= rG (ymed ) − πθ0 (y⋆ )rG (y⋆ ) − πθ0 (ymed )rG (ymed ) − πθ0 (Ybad )rG (ybad ) = (πθ0 (y⋆ ) + πθ0 (Ybad ))rG (ymed ) − πθ0 (y⋆ )rG (y⋆ ) − πθ0 (Ybad )rG (ybad ) = −∆1 πθ0 (y⋆ ) + ∆2 πθ0 (Ybad ) ≥ 0, (39) where the inequality is due to ∆1 πθ0 (y⋆ ) ≤ ∆2 πθ0 (Ybad ) (see the second assumption in the proposition statement). Next, we prove that ζ ≤ ∆2 /∆1 . Since VG (θ0 ) ≤ rG (ymed ) (Equation (39)), it holds that VG (θ0 ) − rG (ybad ) rG (ymed ) − rG (ybad ) ∆2 ≤ = . rG (y⋆ ) − VG (θ0 ) rG (y⋆ ) − rG (ymed ) ∆1
(40)
Furthermore, because θ0 = C(ϕ(ybad ) − ϕ(y⋆ )) for some C > 0 and ⟨ϕ(ybad ), ϕ(y⋆ )⟩ = 0, we 2 2 have that ⟨ϕ(y⋆ ), θ0 ⟩ = −C∥ϕ(y⋆ )∥ ≤ 0 ≤ C∥ϕ(ybad )∥ = ⟨ϕ(ybad ), θ0 ⟩. That is, the logit of y⋆ is lower than that of ybad at initialization, and so πθ0 (y⋆ ) ≤ πθ0 (ybad ). Together with the first assumption in the proposition statement, which gives πθ0 (ymed ) > πθ0 (ybad ), this implies ln πθ0 (ymed ) − ln πθ0 (ybad ) ≤ 1. ln πθ0 (ymed ) − ln πθ0 (y⋆ ) Combining Equations (40) and (41) with the definition of ζ, we then obtain ζ=
∆2 ln πθ0 (ymed ) − ln πθ0 (ybad ) VG (θ0 ) − rG (ybad ) · ≤ . ln πθ0 (ymed ) − ln πθ0 (y⋆ ) rG (y⋆ ) − VG (θ0 ) ∆1 60
(41)
Now, assuming Equation (38) holds for all t ≥ 0, we lower bound rG (y⋆ ) − VG (θt ) as follows: X rG (y⋆ ) − VG (θt ) = rG (y⋆ )(1 − πθt (y⋆ )) − πθt (ymed )rG (ymed ) − πθt (z)rG (z) z∈Ybad
≥ rG (y⋆ )(1 − πθt (y⋆ )) − πθt (ymed )rG (ymed ) − πθt (ybad )rG (ybad ) ≥ rG (y⋆ )(1 − πθt (y⋆ )) − (1 − πθt (y⋆ ) − πθt (ybad ))rG (ymed ) − πθt (ybad )rG (ybad ) = ∆1 (1 − πθt (y⋆ )) + ∆2 πθt (ybad ) > ∆1 (1 − πθt (y⋆ )) + ∆1 πθt (y⋆ ) = ∆1 . Here, the first inequality is from rG (z) ≤ 0 for all z ∈ Ybad ; the second inequality is from πθt (ymed ) ≤ 1 − πθt (y⋆ ) − πθt (ybad ); and the last inequality is from Equation (38). Thus, Equation (38) implies that for all t ≥ 0, VG (θt ) < rG (y⋆ ) − ∆1 = rG (ymed ). It remains to prove Equation (38). Let fy (θ) := ⟨ϕ(y), θ⟩ denote the logit of y ∈ Y under θ ∈ RD . At initialization, πθ0 (y⋆ ) = exp fy⋆ (θ0 ) − fybad (θ0 ) πθ0 (ybad ) = exp ⟨ϕ(y⋆ ) − ϕ(ybad ), θ0 ⟩ = exp − C∥ϕ(ybad ) − ϕ(y⋆ )∥2 < exp(ln(ζ)) = ζ. Both the third equality and the last inequality are due to the last assumption in the proposition state 2 ment, i.e., due to θ0 = C · (ϕ(ybad ) − ϕ(y⋆ )) for some C > max 0, − ln(ζ)/∥ϕ(ybad ) − ϕ(y⋆ )∥ . πθ (y⋆ )
Now, assume by way of contradiction that there exists a time t′ at which πθ t′(ybad ) ≥ ζ and let τ t′ denote the initial such time, i.e.: πθt (y⋆ ) ≥ζ . τ := min t ≥ 0 : πθt (ybad ) Then, for all t ∈ [0, τ ) it holds that πθt (y⋆ )/πθt (ybad ) < ζ. We now show that πθτ (y⋆ )/πθτ (ybad ) < ζ, in contradiction to the definition of τ as the initial time at which this ratio is at least ζ. We will do so by proving that πθt (y⋆ )/πθt (ybad ) is decreasing for all t ∈ [0, τ ), which, along with the fact that πθ0 (y⋆ )/πθ0 (ybad ) < ζ, implies that πθτ (y⋆ )/πθτ (ybad ) < ζ. Notice that, since πθt (y⋆ )/πθt (ybad ) = exp fy⋆ (θt ) − fybad (θt ) : πθt (y⋆ ) d d = sign (fy (θt ) − fybad (θt )) . sign dt πθt (ybad ) dt ⋆ d (fy⋆ (θt ) − fybad (θt )) < 0 for all t ∈ [0, τ ). It therefore suffices to show that dt
The time derivative of fy (θt ) for an output y ∈ Y is given by: X d d fy (θt ) = ϕ(y), dt θt = ⟨ϕ(y), ∇VG (θt )⟩ = πθt (z)AG (z; θt ) ⟨ϕ(y), ϕ(z)⟩ , z∈Y dt where the last equality is by Lemma 10. Thus, for all t ∈ [0, τ ) we have that: d fy⋆ (θt ) − fybad (θt ) dt X = ⟨ϕ(y⋆ ), ϕ(y)⟩ − ⟨ϕ(ybad ), ϕ(y)⟩ πθt (y)AG (y; θt ) y∈Y X = ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(y) − ϕ(ymed ) + ϕ(ymed )⟩ πθt (y)AG (y; θt ) y∈Y X = ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(y) − ϕ(ymed )⟩ πθt (y)AG (y; θt ) y∈Y X + ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(ymed )⟩ πθt (y)AG (y; θt ). y∈Y
61
Because X y∈Y
πθt (y)AG (y; θt ) =
X y∈Y
πθt (y)(rG (y) − VG (θt )) = VG (θt ) − VG (θt ) = 0,
we may write: d fy⋆ (θt ) − fybad (θt ) dt X = ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(y) − ϕ(ymed )⟩ πθt (y)AG (y; θt ) y∈Y
= ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(y⋆ ) − ϕ(ymed )⟩ πθt (y⋆ )AG (y⋆ ; θt ) + ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(ybad ) − ϕ(ymed )⟩ πθt (ybad )AG (ybad ; θt ) X + ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(z) − ϕ(ymed )⟩ πθt (z)AG (z; θt ). z∈Ybad \{ybad }
(42)
Notice that the contribution of outputs in Ybad \ {ybad } is non-positive. Indeed, for every z ∈ Ybad \ {ybad }, Assumption 4 implies ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(z) − ϕ(ymed )⟩ = − ⟨ϕ(y⋆ ), ϕ(ymed )⟩ > 0, and since all outputs in Ybad have the same minimal ground truth reward (Assumption 1), AG (z; θt ) < 0 for all z ∈ Ybad . Using Equation (42), we therefore get d fy⋆ (θt ) − fybad (θt ) dt ≤ ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(y⋆ ) − ϕ(ymed )⟩ πθt (y⋆ )AG (y⋆ ; θt ) + ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(ybad ) − ϕ(ymed )⟩ πθt (ybad )AG (ybad ; θt ). Next, using AG (y⋆ ; θt ) = rG (y⋆ ) − VG (θt ) and AG (ybad ; θt ) = rG (ybad ) − VG (θt ), the right-hand side of the equation above factors as d fy⋆ (θt ) − fybad (θt ) dt ≤ − ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(ybad ) − ϕ(ymed )⟩ πθt (ybad )(VG (θt ) − rG (ybad )) πθt (y⋆ ) rG (y⋆ ) − VG (θt ) ⟨ϕ(y⋆ ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ · − · · + 1 . (43) ⟨ϕ(ybad ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ πθt (ybad ) VG (θt ) − rG (ybad ) It remains to show that the right-hand side of Equation (43) is negative. By the first assumption in the proposition statement, πθ0 (ymed ) > πθ0 (ybad ). This implies that fymed (θ0 ) > fybad (θ0 ), or equivalently, ⟨ϕ(ymed ) − ϕ(ybad ), θ0 ⟩ > 0. Substituting θ0 = C(ϕ(ybad ) − ϕ(y⋆ )) and dividing by C > 0 gives ⟨ϕ(ybad ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ = ⟨ϕ(ybad ) − ϕ(y⋆ ), ϕ(ymed ) − ϕ(ybad )⟩ > 0. Furthermore, ⟨ϕ(y⋆ ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ = ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(y⋆ ) − ϕ(ybad )⟩ + ⟨ϕ(ybad ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ ≥ ⟨ϕ(ybad ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ > 0. These are exactly the numerator and denominator of the inner product terms appearing in ζ=
⟨ϕ(ybad ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ VG (θ0 ) − rG (ybad ) · , ⟨ϕ(y⋆ ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ rG (y⋆ ) − VG (θ0 )
and both are positive. Thus, for t ∈ [0, τ ), the bound πθt (y⋆ )/πθt (ybad ) < ζ (Equation (38)) implies ⟨ϕ(y⋆ ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ πθt (y⋆ ) VG (θ0 ) − rG (ybad ) · < . ⟨ϕ(ybad ) − ϕ(ymed ), ϕ(y⋆ ) − ϕ(ybad )⟩ πθt (ybad ) rG (y⋆ ) − VG (θ0 ) Since under gradient flow VG is monotonically non-decreasing, we have VG (θ0 ) ≤ VG (θt ), and so rG (y⋆ ) − VG (θt ) rG (y⋆ ) − VG (θ0 ) ≤ . VG (θt ) − rG (ybad ) VG (θ0 ) − rG (ybad ) 62
Combining the last two inequalities shows that the product inside the parentheses in Equation (43) is less than 1. Since ⟨ϕ(y⋆ ) − ϕ(ybad ), ϕ(ybad ) − ϕ(ymed )⟩ > 0, πθt (ybad ) > 0, and VG (θt ) > rG (ybad ), we conclude from Equation (43) that d fy⋆ (θt ) − fybad (θt ) < 0. (44) dt Equation (44) implies that fy⋆ (θt ) − fybad (θt ) is non-increasing, and so πθt (y⋆ )/πθt (ybad ) is nonincreasing, for all t ∈ [0, τ ). Thus, πθτ (y⋆ ) πθ0 (y⋆ ) ≤ < ζ, πθτ (ybad ) πθ0 (ybad ) in contradiction to our assumption that πθτ (y⋆ )/πθτ (ybad ) ≥ ζ. This establishes Equation (38), concluding the proof. B.7
Proof of Theorem 4
For convenience of notation, throughout the proof we denote s := ⟨ϕ(y⋆ ), ϕ(ymed )⟩. B.7.1
Proof of Case I
Suppose that gradient flow is used to maximize the expected reward with respect to rG . We begin by proving that πθt (Ybad ) is monotonically non-increasing (Lemma 8) and πθt (y⋆ ) is monotonically non-decreasing (Lemma 9) throughout optimization. We then upper bound t⋆ —the initial time at which VG (θt ) ≥ rG (y⋆ ) − ϵ—by lower bounding the growth rate of πθt (y⋆ ) over the interval [0, t⋆ ). d Lemma 8. At any time t ≥ 0, it holds that dt πθt (Ybad ) ≤ 0.
Proof. Suppose first that VG (θt ) ≤ rG (ymed ). Under Assumption 5, for every z ∈ Ybad , the feature vector ϕ(z) is orthogonal to the feature vectors of all other outputs. Therefore, by Lemma 13 we have for all z ∈ Ybad : d πθ (z) = πθt (z)2 AG (z; θt )∥ϕ(z)∥2 dt t X − πθt (z) πθt (z ′ )2 AG (z ′ ; θt )∥ϕ(z ′ )∥2 z ′ ∈Ybad + πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) + πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed ) . Summing over all z ∈ Ybad , we have: X d πθt (Ybad ) = πθt (z)2 AG (z; θt )∥ϕ(z)∥2 1 − πθt (Ybad ) z∈Ybad dt {z } | I1
− πθt (Ybad ) πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) + πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed ) , where we denote by I2 the second term on the right-hand side of the equation above (excluding the minus sign). Since all outputs in Ybad have the minimal ground truth reward, we have AG (z; θt ) ≤ 0 for all z ∈ Ybad . Together with AG (ymed ; θt ) ≥ 0 and AG (y⋆ ; θt ) ≥ 0, this gives I1 ≤ 0 and I2 ≥ 0. d This implies that dt πθt (Ybad ) ≤ 0 when VG (θt ) ≤ rG (ymed ). Now, consider the case where VG (θt ) > rG (ymed ). In this case, I1 ≤ 0 still holds. Furthermore, X VG (θt ) − rG (ymed ) = (πθt (ymed ) − 1)rG (ymed ) + πθt (y⋆ )rG (y⋆ ) + πθt (z)rG (z) z∈Ybad
≤ (πθt (ymed ) − 1)rG (ymed ) + πθt (y⋆ )rG (y⋆ ) ≤ −πθt (y⋆ )rG (ymed ) + πθt (y⋆ )rG (y⋆ ) = πθt (y⋆ )∆1 , (45) 63
where the first inequality is from rG (z) ≤ 0 for all z ∈ Ybad and the second inequality is because 1 − πθt (ymed ) ≥ πθt (y⋆ ) and rG (ymed ) > 0. Thus, |AG (ymed ; θt )| = VG (θt ) − rG (ymed ) ≤ πθt (y⋆ )∆1 . Moreover, X AG (y⋆ ; θt ) = rG (y⋆ ) − πθt (y⋆ )rG (y⋆ ) − πθt (ymed )rG (ymed ) − πθt (z)rG (z) z∈Ybad
(46)
≥ rG (y⋆ ) − πθt (y⋆ )rG (y⋆ ) − (1 − πθt (y⋆ ))rG (ymed ) = (1 − πθt (y⋆ ))∆1 . We can therefore lower bound I2 /πθt (Ybad ) as: πθt (ymed )AG (ymed ; θt ) ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) + πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed ) ≥ πθt (y⋆ )(1 − πθt (y⋆ ))∆1 ∥ϕ(y⋆ )∥2 πθt (y⋆ ) + sπθt (ymed ) − πθt (ymed )πθt (y⋆ )∆1 ∥ϕ(ymed )∥2 πθt (ymed ) + sπθt (y⋆ ) ≥ πθt (y⋆ )(1 − πθt (y⋆ ))∆1 (∥ϕ(y⋆ )∥2 − s)πθt (y⋆ ) + (s − ∥ϕ(ymed )∥2 )πθt (ymed )
≥ 0. Here, the first inequality is from |AG (ymed ; θt )| ≤ πθt (y⋆ )∆1 and AG (y⋆ ; θt ) ≥ (1 − πθt (y⋆ ))∆1 , along with the fact that the multipliers of these advantages are non-negative. The second inequality 2 is from 1 − πθt (y⋆ ) ≥ πθt (ymed ), combined with s > 0 and ∥ϕ(ymed )∥ ≥ 0. The last inequality 2 2 follows from the assumption in Theorem 4 that ∥ϕ(ymed )∥ < s < ∥ϕ(y⋆ )∥ . Thus, I2 ≥ 0, which d means dt πθt (Ybad ) ≤ 0. d Overall, we showed that dt πθt (Ybad ) ≤ 0 both when VG (θt ) ≤ rG (ymed ) and when VG (θt ) > rG (ymed ), and so πθt (Ybad ) is non-increasing for all t ≥ 0. d Lemma 9. At any time t ≥ 0, it holds that dt πθt (y⋆ ) ≥ 0.
Proof. Let ℓ(θ) := ⟨ϕ(y⋆ ), θ⟩−⟨ϕ(ymed ), θ⟩ = ⟨ϕ(y⋆ ) − ϕ(ymed ), θ⟩ be the logit difference between y⋆ and ymed under θ ∈ RD . We prove that ℓ(θt ) is non-decreasing for all t ≥ 0. We can write the time derivative of ℓ(θt ) as: d d ℓ(θt ) = ϕ(y⋆ ) − ϕ(ymed ), dt θt dt = ∥ϕ(y⋆ )∥2 πθt (y⋆ )AG (y⋆ ; θt ) + sπθt (ymed )AG (ymed ; θt ) − sπθt (y⋆ )AG (y⋆ ; θt ) − ∥ϕ(ymed )∥2 πθt (ymed )AG (ymed ; θt ) = (∥ϕ(y⋆ )∥2 − s)πθt (y⋆ )AG (y⋆ ; θt ) + (s − ∥ϕ(ymed )∥2 )πθt (ymed )AG (ymed ; θt ). If VG (θt ) ≤ rG (ymed ) (i.e., AG (ymed ; θt ) ≥ 0), since ∥ϕ(y⋆ )∥2 > s > ∥ϕ(ymed )∥2 and d AG (y⋆ ; θt ) ≥ 0, it follows directly from the expression above that dt ℓ(θt ) ≥ 0. On the other hand, if VG (θt ) > rG (ymed ), Equations (45) and (46) hold, as established in the proof of Lemma 8. d Thus, in this case it also holds that dt ℓ(θt ) ≥ 0 since: d ℓ(θt ) ≥ (∥ϕ(y⋆ )∥2 − s)πθt (y⋆ )(1 − πθt (y⋆ ))∆1 − (s − ∥ϕ(ymed )∥2 )πθt (ymed )πθt (y⋆ )∆1 dt ≥ πθt (y⋆ )(1 − πθt (y⋆ ))∆1 (∥ϕ(y⋆ )∥2 − 2s + ∥ϕ(ymed )∥2 ) ≥ 0, where the second inequality is from πθt (ymed ) ≤ 1 − πθt (y⋆ ), and the last inequality is from s ≤ ∥ϕ(y⋆ )∥∥ϕ(ymed )∥ ≤ (∥ϕ(y⋆ )∥2 + ∥ϕ(ymed )∥2 )/2. Thus, ℓ(θt ) is monotonically non-decreasing for all t ≥ 0. Now, notice that πθt (y⋆ ) = (1 − πθt (Ybad )) ·
64
πθt (y⋆ ) . πθt (y⋆ ) + πθt (ymed )
The factor 1 − πθt (Ybad ) is monotonically non-decreasing by Lemma 8, since πθt (Ybad ) is nonincreasing. Furthermore, the factor πθt (y⋆ )/(πθt (y⋆ ) + πθt (ymed )) is also non-decreasing. To see this, let rt := πθt (y⋆ )/πθt (ymed ). By the softmax parametrization of the policy, ℓ(θt ) = ln rt , and so rt = exp(ℓ(θt )). Since both ℓ(θt ) and the exponential function are non-decreasing, rt is non-decreasing. Furthermore, πθt (y⋆ ) rt = , πθt (y⋆ ) + πθt (ymed ) 1 + rt and the map r 7→ r/(1 + r) is non-decreasing for r ≥ 0. Thus, πθt (y⋆ )/(πθt (y⋆ ) + πθt (ymed )) is d non-decreasing in t. Overall, the above implies that πθt (y⋆ ) is non-decreasing, and so dt πθt (y⋆ ) ≥ 0, for all t ≥ 0. We now complete the proof of Case I. Proof of Theorem 4, Case I. Denote ρ :=
rG (ymed ) + ∆1 − ϵ + 1 , rG (y⋆ ) + 1
and notice that when πθt (y⋆ ) ≥ ρ it must be that VG (θt ) ≥ rG (ymed ) + ∆1 − ϵ = rG (y⋆ ) − ϵ since: VG (θt ) ≥ πθt (y⋆ )rG (y⋆ ) − (1 − πθt (y⋆ )) = πθt (y⋆ )(rG (y⋆ ) + 1) − 1 ≥ rG (y⋆ ) − ϵ, where the first inequality is from rG (y) ≥ −1 for all y ∈ Y and the second inequality follows from the definition of ρ. This implies that πθt (y⋆ ) < ρ for all t ∈ [0, t⋆ ). In particular, Lemma 15 implies that VG (θ0 ) < rG (ymed ) < rG (y⋆ ) − ϵ, and so πθ0 (y⋆ ) < ρ and t⋆ > 0. Next, we lower bound the time derivative of πθt (y⋆ ) for t ∈ [0, t⋆ ), based on the expression derived in Lemma 13: h d πθt (y⋆ ) = πθt (y⋆ ) πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 − sπθt (ymed ) − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) dt + πθt (ymed )AG (ymed ; θt ) s − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) i X − πθt (z)2 AG (z; θt )∥ϕ(z)∥2 z∈Ybad h ≥ πθt (y⋆ ) πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 − sπθt (ymed ) − ∥ϕ(y⋆ )∥2 πθt (y⋆ ) i + πθt (ymed )AG (ymed ; θt ) s − ∥ϕ(ymed )∥2 πθt (ymed ) − sπθt (y⋆ ) h = πθt (y⋆ ) πθt (y⋆ )AG (y⋆ ; θt ) ∥ϕ(y⋆ )∥2 (1 − πθt (y⋆ )) − sπθt (ymed ) i + πθt (ymed )AG (ymed ; θt ) s(1 − πθt (y⋆ )) − ∥ϕ(ymed )∥2 πθt (ymed ) h ≥ πθt (y⋆ )2 ∆1 ∥ϕ(y⋆ )∥2 (1 − πθt (y⋆ ))2 − 2sπθt (ymed )(1 − πθt (y⋆ )) i + ∥ϕ(ymed )∥2 πθt (ymed )2 . (47) Here, the first inequality is from AG (z; θt ) ≤ 0 for all z ∈ Ybad . The second inequality is from AG (y⋆ ; θt ) ≥ (1 − πθt (y⋆ ))∆1 (as in Equation (46)), AG (ymed ; θt ) ≥ −πθt (y⋆ )∆1 (which follows from Equation (45) when VG (θt ) > rG (ymed ) and is trivial otherwise), and the fact that ∥ϕ(y⋆ )∥2 (1 − πθt (y⋆ )) − sπθt (ymed ) and s(1 − πθt (y⋆ )) − ∥ϕ(ymed )∥2 πθt (ymed ) are non-negative, since πθt (ymed ) ≤ 1 − πθt (y⋆ ) and ∥ϕ(ymed )∥2 < s < ∥ϕ(y⋆ )∥2 . To simplify the bracketed term on the right-hand side of Equation (47), we can view the expression ∥ϕ(y⋆ )∥2 (1 − πθt (y⋆ ))2 − 2sπθt (ymed )(1 − πθt (y⋆ )) + ∥ϕ(ymed )∥2 πθt (ymed )2 65
as a function of πθt (ymed ) over the interval [0, 1−πθt (y⋆ )]. Its derivative with respect to πθt (ymed ) is −2s(1 − πθt (y⋆ )) + 2∥ϕ(ymed )∥2 πθt (ymed ), which is at most −2(s − ∥ϕ(ymed )∥2 )(1 − πθt (y⋆ )) ≤ 0, since πθt (ymed ) ≤ 1 − πθt (y⋆ ) and s > ∥ϕ(ymed )∥2 . Thus, the expression is minimized at πθt (ymed ) = 1 − πθt (y⋆ ). This implies that: d πθ (y⋆ ) ≥ πθt (y⋆ )2 (1 − πθt (y⋆ ))2 ∆1 (∥ϕ(y⋆ )∥2 + ∥ϕ(ymed )∥2 − 2s) dt t (48) ≥ (1 − ρ)2 ∆1 (∥ϕ(y⋆ )∥2 + ∥ϕ(ymed )∥2 − 2s)πθt (y⋆ )2 2
= (1 − ρ)2 ∆1 ∥ϕ(y⋆ ) − ϕ(ymed )∥ πθt (y⋆ )2 , where the second inequality is from πθt (y⋆ ) < ρ for all t ∈ [0, t⋆ ). Dividing both sides of Equation (48) by πθt (y⋆ )2 , integrating from 0 to t, and rearranging terms yields: πθ0 (y⋆ ) πθt (y⋆ ) ≥ . 2 2 1 − (1 − ρ) ∆1 ∥ϕ(y⋆ ) − ϕ(ymed )∥ πθ0 (y⋆ ) · t Since πθt (y⋆ ) < ρ for all t ∈ [0, t⋆ ), we get that for all such t: 1 1 1 t≤ − . 2 (1 − ρ)2 ∥ϕ(y⋆ ) − ϕ(ymed )∥ ∆1 πθ0 (y⋆ ) ρ Since this holds for all t ∈ [0, t⋆ ), we get t⋆ ≤
1
1
· . 2 (1 − ρ)2 ∥ϕ(y⋆ ) − ϕ(ymed )∥ ∆1 πθ0 (y⋆ )
Plugging in the definition of ρ gives t⋆ ≤
(rG (y⋆ ) + 1)
2
1
· , 2 ϵ2 ∥ϕ(y⋆ ) − ϕ(ymed )∥ ∆1 πθ0 (y⋆ )
which completes the proof of Case I. B.7.2
Proof of Case II
Suppose that gradient flow is used to maximize the expected reward with respect to rP , and that ⟨ϕ(y⋆ ), ϕ(ymed )⟩ is sufficiently high, in the sense that 2
⟨ϕ(y⋆ ), ϕ(ymed )⟩ >
2
πθ0 (y⋆ )AP (y⋆ ; θ0 )∥ϕ(y⋆ )∥ − πθ0 (ymed )AP (ymed ; θ0 )∥ϕ(ymed )∥ . πθ0 (y⋆ )AP (y⋆ ; θ0 ) − πθ0 (ymed )AP (ymed ; θ0 )
(49)
Proof of Theorem 4, Case II. We prove that, due to the high inner product between ϕ(y⋆ ) and ϕ(ymed ), the logit difference ⟨ϕ(y⋆ ), θt ⟩ − ⟨ϕ(ymed ), θt ⟩ is monotonically non-increasing with respect to t. Consequently, the ratio πθt (y⋆ )/πθt (ymed ) is bounded by its initial value for all t ≥ 0, and the policy never assigns y⋆ sufficiently high probability to reach an expected ground truth reward above the constant rG (y⋆ ) − (rG (y⋆ ) − rG (ymed ))πθ0 (ymed ). We first note that AP (ymed ; θ0 ) < 0 because ymed has a minimal proxy reward of rP (ymed ) ≤ miny∈Ybad rG (y), while y⋆ has strictly larger proxy reward and πθ0 has full support as a softmax policy. Moreover, AP (y⋆ ; θ0 ) > 0 since y⋆ has the unique maximal proxy reward. Now, let ℓ(θ) := ⟨ϕ(y⋆ ), θ⟩ − ⟨ϕ(ymed ), θ⟩ = ⟨ϕ(y⋆ ) − ϕ(ymed ), θ⟩ be the logit difference between y⋆ and ymed under θ ∈ RD . We prove that ℓ(θt ) is non-increasing for all t ≥ 0. We can write the time derivative of ℓ(θt ) as: d d ℓ(θt ) = ϕ(y⋆ ) − ϕ(ymed ), dt θt dt = ∥ϕ(y⋆ )∥2 πθt (y⋆ )AP (y⋆ ; θt ) + sπθt (ymed )AP (ymed ; θt ) − sπθt (y⋆ )AP (y⋆ ; θt ) − ∥ϕ(ymed )∥2 πθt (ymed )AP (ymed ; θt ) = (∥ϕ(y⋆ )∥2 − s)πθt (y⋆ )AP (y⋆ ; θt ) + (s − ∥ϕ(ymed )∥2 )πθt (ymed )AP (ymed ; θt ) ≤ (∥ϕ(y⋆ )∥2 − s)πθt (y⋆ )AP (y⋆ ; θ0 ) + (s − ∥ϕ(ymed )∥2 )πθt (ymed )AP (ymed ; θ0 ) = πθt (ymed ) (∥ϕ(y⋆ )∥2 − s)AP (y⋆ ; θ0 ) exp(ℓ(θt )) + (s − ∥ϕ(ymed )∥2 )AP (ymed ; θ0 ) . (50) 66
Here, the first inequality is from VP (θt ) being monotonically non-decreasing when maximizing it via gradient flow, together with the positivity of ∥ϕ(y⋆ )∥2 − s and s − ∥ϕ(ymed )∥2 , and the last equality is by the definition of ℓ(θt ) since πθt (y⋆ ) = πθt (ymed ) · exp(ℓ(θt )). Since AP (y⋆ ; θ0 ) > 0 > AP (ymed ; θ0 ) and ∥ϕ(ymed )∥2 < s < ∥ϕ(y⋆ )∥2 , the denominator and numerator of the right-hand side below are both positive. Therefore, when exp(ℓ(θt )) <
−(s − ∥ϕ(ymed )∥2 )AP (ymed ; θ0 ) , (∥ϕ(y⋆ )∥2 − s)AP (y⋆ ; θ0 )
(51)
d we get that dt ℓ(θt ) < 0.
Equation (49) implies Equation (51) at t = 0. Indeed, the denominator on the right-hand side of Equation (49) is positive because AP (y⋆ ; θ0 ) > 0 > AP (ymed ; θ0 ). Therefore, multiplying both sides of Equation (49) by this denominator and rearranging gives πθ0 (y⋆ )AP (y⋆ ; θ0 ) ∥ϕ(y⋆ )∥2 − s < −πθ0 (ymed )AP (ymed ; θ0 ) s − ∥ϕ(ymed )∥2 . Since πθ0 (ymed ) > 0, AP (y⋆ ; θ0 ) > 0, and ∥ϕ(y⋆ )∥2 − s > 0, we may divide both sides by πθ0 (ymed )AP (y⋆ ; θ0 ) ∥ϕ(y⋆ )∥2 − s to obtain
πθ0 (y⋆ ) −(s − ∥ϕ(ymed )∥2 )AP (ymed ; θ0 ) < . πθ0 (ymed ) (∥ϕ(y⋆ )∥2 − s)AP (y⋆ ; θ0 ) Finally, πθ0 (y⋆ )/πθ0 (ymed ) = exp(ℓ(θ0 )), so Equation (51) holds at t = 0. We prove that Equation (51) holds for all t ≥ 0 by contradiction. Assume that there exists a time t′ > 0 at which exp(ℓ(θt′ )) ≥
−(s − ∥ϕ(ymed )∥2 )AP (ymed ; θ0 ) , (∥ϕ(y⋆ )∥2 − s)AP (y⋆ ; θ0 )
and denote by τ the initial such time, i.e.: −(s − ∥ϕ(ymed )∥2 )AP (ymed ; θ0 ) τ := min t ≥ 0 : exp(ℓ(θt )) ≥ . (∥ϕ(y⋆ )∥2 − s)AP (y⋆ ; θ0 ) By the definition of τ , for all t ∈ [0, τ ) Equation (51) holds. Thus, for all t ∈ [0, τ ), by Equation (50) d dt ℓ(θt ) < 0, and so ℓ(θτ ) < ℓ(θ0 ). This implies that: exp(ℓ(θτ )) < exp(ℓ(θ0 )) <
−(s − ∥ϕ(ymed )∥2 )AP (ymed ; θ0 ) , (∥ϕ(y⋆ )∥2 − s)AP (y⋆ ; θ0 )
d in contradiction to the definition of τ . Thus, for all t ≥ 0, Equation (51) holds and dt ℓ(θt ) < 0.
Now, examining πθt (y⋆ ), we can see that: πθt (y⋆ ) πθt (ymed ) + πθt (y⋆ ) πθt (y⋆ )/πθt (ymed ) = 1 + πθt (y⋆ )/πθt (ymed ) πθ0 (y⋆ )/πθ0 (ymed ) ≤ 1 + πθ0 (y⋆ )/πθ0 (ymed ) πθ0 (y⋆ ) = , πθ0 (y⋆ ) + πθ0 (ymed )
πθt (y⋆ ) ≤
where the second inequality is due to πθt (y⋆ )/πθt (ymed ) = exp(ℓ(θt )) and the fact that ℓ(θt ) ≤ ℓ(θ0 ) for all t ≥ 0. Thus, πθ0 (ymed ) 1 − πθt (y⋆ ) ≥ . πθ0 (y⋆ ) + πθ0 (ymed ) Furthermore, since rG (z) < rG (ymed ), for all z ∈ Ybad , we know that VG (θt ) < πθt (y⋆ )rG (y⋆ ) + (1 − πθt (y⋆ ))rG (ymed ). 67
We may therefore conclude that for all t ≥ 0: rG (y⋆ ) − VG (θt ) > (1 − πθt (y⋆ ))rG (y⋆ ) − (1 − πθt (y⋆ ))rG (ymed ) = (rG (y⋆ ) − rG (ymed ))(1 − πθt (y⋆ )) (rG (y⋆ ) − rG (ymed ))πθ0 (ymed ) ≥ πθ0 (y⋆ ) + πθ0 (ymed ) ≥ (rG (y⋆ ) − rG (ymed ))πθ0 (ymed ), where the second inequality is from the lower bound on 1 − πθt (y⋆ ) above, and the last inequality is from πθ0 (y⋆ ) + πθ0 (ymed ) ≤ 1. B.8
Auxiliary Lemmas
In this appendix, we include auxiliary lemmas that are used throughout the proofs in Appendix B. Lemma 10. Let r : Y → [−1, 1]. For a linear softmax policy parameterized by θ ∈ RD , denote the expected reward with respect to r by V (θ) := Ey∼πθ [r(y)]. Then: X ∇V (θ) = πθ (y)A(y; θ) · ϕ(y) , y∈Y
where A(y; θ) := r(y) − V (θ) is the advantage of y ∈ Y under r and πθ . Proof. Since the probability of an output y ∈ Y under πθ is given by πθ (y) = P
exp(⟨ϕ(y), θ⟩) , z∈Y exp(⟨ϕ(z), θ⟩)
the gradient of πθ (y) with respect to θ can be written as: X ∇πθ (y) = πθ (y) ϕ(y) −
z∈Y
πθ (z) · ϕ(z) .
Hence: ∇V (θ) =
X y∈Y
r(y)∇πθ (y)
X πθ (y)r(y) πθ (z) · ϕ(z) y∈Y y∈Y z∈Y X X = πθ (y)r(y) · ϕ(y) − V (θ) πθ (z) · ϕ(z) y∈Y z∈Y X = πθ (y)A(y; θ) · ϕ(y). =
X
πθ (y)r(y) · ϕ(y) −
X
y∈Y
Lemma 11. For a linear softmax policy parameterized by θ ∈ RD and ground truth reward function rG upholding Assumption 1, it holds that: ∥∇VG (θ)∥ ≤ 2B(∆1 + ∆2 )(1 − πθ (ymed )) , where ∆1 := rG (y⋆ ) − rG (ymed ), ∆2 := rG (ymed ) − rG (ybad ) for some ybad ∈ Ybad , and B := maxy∈Y ∥ϕ(y)∥. Proof. For a linear softmax policy πθ , by Lemma 10 we have that: X ∇VG (θ) = πθ (y)AG (y; θ) · ϕ(y). y∈Y
Since all outputs in Ybad have the same reward of rG (ybad ), we can write VG (θ) = πθ (y⋆ )rG (y⋆ ) + πθ (ymed )rG (ymed ) + πθ (Ybad )rG (ybad ) = rG (ymed ) + πθ (y⋆ )∆1 − πθ (Ybad )∆2 . 68
Hence the three possible advantage values are: AG (y⋆ ; θ) = rG (y⋆ ) − VG (θ) = (1 − πθ (y⋆ ))∆1 + πθ (Ybad )∆2 , AG (ymed ; θ) = rG (ymed ) − VG (θ) = −πθ (y⋆ )∆1 + πθ (Ybad )∆2 , AG (z; θ) = rG (ybad ) − VG (θ) = −πθ (y⋆ )∆1 − (1 − πθ (Ybad ))∆2 ,
for all z ∈ Ybad .
Substituting these expressions into the gradient formula gives: ∇VG (θ) = πθ (y⋆ )AG (y⋆ ; θ)ϕ(y⋆ ) + πθ (ymed )AG (ymed ; θ)ϕ(ymed ) X + πθ (z)AG (z; θ)ϕ(z). z∈Ybad
Therefore, by the triangle inequality and the definition of B, X ∥∇VG (θ)∥ ≤ B πθ (y⋆ )AG (y⋆ ; θ) + πθ (ymed )|AG (ymed ; θ)| +
πθ (z)|AG (z; θ)| z∈Ybad = B πθ (y⋆ )AG (y⋆ ; θ) + πθ (ymed )|−πθ (y⋆ )∆1 + πθ (Ybad )∆2 | + B πθ (Ybad )(πθ (y⋆ )∆1 + (1 − πθ (Ybad ))∆2 ) .
Applying the triangle inequality, collecting the coefficients of ∆1 and ∆2 , and using πθ (y⋆ ) + πθ (ymed ) + πθ (Ybad ) = 1, we can further upper bound the above expression as: ∥∇VG (θ)∥ ≤ B πθ (y⋆ )((1 − πθ (y⋆ ))∆1 + πθ (Ybad )∆2 ) + πθ (ymed )(πθ (y⋆ )∆1 + πθ (Ybad )∆2 ) + B πθ (Ybad )(πθ (y⋆ )∆1 + (1 − πθ (Ybad ))∆2 ) = B 2πθ (y⋆ )(1 − πθ (y⋆ ))∆1 + 2πθ (Ybad )(1 − πθ (Ybad ))∆2 . Since 1 − πθ (y⋆ ) ≤ 1 and 1 − πθ (Ybad ) ≤ 1, we may conclude: ∥∇VG (θ)∥ ≤ 2B(πθ (y⋆ )∆1 + πθ (Ybad )∆2 ) ≤ 2B(∆1 + ∆2 )(πθ (y⋆ ) + πθ (Ybad )) = 2B(∆1 + ∆2 )(1 − πθ (ymed )). Lemma 12. For a linear softmax policy parameterized by θ ∈ RD and a ground truth reward function rG : Y → [−1, 1], let B := maxy∈Y ∥ϕ(y)∥. Then, the expected ground truth reward VG (θ) = Ey∼πθ [rG (y)] is B-Lipschitz with respect to the Euclidean norm, i.e., for any θ, θ′ ∈ RD : |VG (θ) − VG (θ′ )| ≤ B∥θ − θ′ ∥. Proof. By Lemma 10, for r = rG we have: X ∇VG (θ) = πθ (y)AG (y; θ) · ϕ(y). y∈Y P Since y∈Y πθ (y)AG (y; θ) = 0, we may rewrite this as: X ∇VG (θ) = πθ (y)AG (y; θ) · ϕ(y) − ϕ̄θ , y∈Y P where ϕ̄θ := z∈Y πθ (z)ϕ(z). Applying the triangle inequality followed by the Cauchy-Schwarz inequality gives: X ∥∇VG (θ)∥ ≤ πθ (y)|AG (y; θ)| ϕ(y) − ϕ̄θ y∈Y X 1/2 X 2 1/2 ≤ πθ (y)AG (y; θ)2 πθ (y) ϕ(y) − ϕ̄θ y∈Y y∈Y 2 1/2 1/2 2 = Vary∼πθ [rG (y)] Ey∼πθ ∥ϕ(y)∥ − ϕ̄θ ≤ B, where the last inequality follows from the fact that rG (y) ∈ [−1, 1] implies Vary∼πθ [rG (y)] ≤ 1, and 2 2 2 from Ey∼πθ ∥ϕ(y)∥ − ∥ϕ̄θ ∥ ≤ Ey∼πθ ∥ϕ(y)∥ ≤ B 2 . Therefore, by the mean value theorem, |VG (θ) − VG (θ′ )| ≤ supξ∈[θ,θ′ ] ∥∇VG (ξ)∥ ∥θ − θ′ ∥ ≤ B∥θ − θ′ ∥.
69
Lemma 13. Let r : Y → [−1, 1]. For a linear softmax policy parameterized by θ ∈ RD , denote the expected reward with respect to r by V (θ) := Ey∼πθ [r(y)]. Suppose that gradient flow is used to maximize the expected reward with respect to r (Equation (1) with V in place of VP ). Then, for any output y ∈ Y, D E X X d πθt (y) = πθt (y) ϕ(y) − πθt (z)ϕ(z), πθt (z)A(z; θt )ϕ(z) , z∈Y z∈Y dt where A(z; θt ) := r(z) − V (θt ) is the advantage of z ∈ Y under r and πθt . Moreover, if the feature vectors {ϕ(z)}z∈Y are orthonormal, then X d πθt (y) = πθt (y) πθt (y)A(y; θt ) − πθt (z)2 A(z; θt ) . z∈Y dt Proof. By the chain rule and the gradient flow dynamics, d d πθ (y) = ∇πθt (y), dt θt = ⟨∇πθt (y), ∇V (θt )⟩ . dt t Since πθ is a linear softmax policy, we have X ∇πθt (y) = πθt (y) ϕ(y) −
z∈Y
πθt (z)ϕ(z) .
Moreover, by Lemma 10, ∇V (θt ) =
X z∈Y
πθt (z)A(z; θt )ϕ(z).
Substituting these two expressions into the derivative identity above gives the first claim. If, in addition, the feature vectors are orthonormal, then ⟨ϕ(y), ϕ(z)⟩ = 1[y = z] for all y, z ∈ Y, and therefore D E X X X ϕ(y) − πθt (z)ϕ(z), πθt (u)A(u; θt )ϕ(u) = πθt (y)A(y; θt )− πθt (z)2 A(z; θt ). z∈Y
u∈Y
z∈Y
Plugging this back into the general expression yields the orthonormal-features formula. Lemma 14. Under Assumptions 1 and 2, it holds that VG (θ0 ) = Ey∼πθ0 [rG (y)] < rG (ymed ) − for γ := πθ0 (y⋆ )14/13 M −14/13 with M defined as in Assumption 2.
√
γ,
Proof. Fix some ybad ∈ Ybad . Under Assumptions 1 and 2, Lemma 4 establishes bounds√on γ, πθ0 (y⋆ ), and πθ0 (Ybad ). Based on these bounds, we can show that VG (θ0 ) < rG (ymed ) − γ as follows: X VG (θ0 ) = rG (ymed )πθ0 (ymed ) + rG (y⋆ )πθ0 (y⋆ ) + rG (z)πθ0 (z) z∈Ybad
≤ (1 − πθ0 (Ybad ))rG (ymed ) + γ
13/14
= rG (ymed ) − πθ0 (Ybad )∆2 + γ √ ≤ rG (ymed ) − 2 γ + γ 13/14 √ < rG (ymed ) − γ,
13/14
+ πθ0 (Ybad )rG (ybad )
The first inequality is from rG (y⋆ )√≤ 1 and πθ0 (y⋆ ) ≤ γ 13/14 (Item 2 in Lemma 4); the second inequality is from πθ0 (Ybad ) ≥ 2 γ/∆2 (Item 3 in Lemma 4); and the last inequality is due to √ γ > γ 13/14 for γ ≤ 0.01 (Item 1 in Lemma 4). √ Lemma 15. Under Assumptions 1 and 3, it holds that VG (θ0 ) = Ey∼πθ0 [rG (y)] < rG (ymed ) − γ, for γ := πθ0 (y⋆ )14/13 M ′−14/13 with M ′ defined as in Assumption 3. Proof. Fix some ybad ∈ Ybad . Under Assumptions 1 and 3, Lemma 6 establishes bounds on γ, πθ0 (y⋆ ), and πθ0 (Ybad ). The proof of this lemma is analogous to that of Lemma 14, using the bounds 70
provided by Lemma 6 in √ place of those from Lemma 4. Based on these bounds, we can show that VG (θ0 ) < rG (ymed ) − γ as follows: X VG (θ0 ) = rG (ymed )πθ0 (ymed ) + rG (y⋆ )πθ0 (y⋆ ) + rG (z)πθ0 (z) z∈Ybad
≤ (1 − πθ0 (Ybad ))rG (ymed ) + γ 13/14 + πθ0 (Ybad )rG (ybad ) = rG (ymed ) − πθ0 (Ybad )∆2 + γ 13/14 √ ≤ rG (ymed ) − 2 γ + γ 13/14 √ < rG (ymed ) − γ. The first inequality is from rG (y⋆ )√≤ 1 and πθ0 (y⋆ ) ≤ γ 13/14 (Item 2 in Lemma 6); the second inequality is from πθ0 (Ybad ) ≥ 2 γ/∆2 (Item 3 in Lemma 6); and the last inequality is due to √ γ > γ 13/14 for γ ≤ 0.01 (Item 1 in Lemma 6). Lemma 16. Under Assumption 1, for all γ > 0, if VG (θt ) ≤ rG (ymed ) − γ, then πθt (ymed ) ≤ 1 − 1+rGγ(ymed ) . Proof. We will proceed by contradiction. Suppose to the contrary that πθt (ymed ) > 1 − 1+rGγ(ymed ) . Then, X VG (θt ) = πθt (ymed )rG (ymed ) + πθt (y⋆ )rG (y⋆ ) + πθt (z)rG (z) z∈Ybad
γ rG (ymed ) − πθt (y⋆ ) − πθt (Ybad ) 1 + rG (ymed ) γ γ rG (ymed ) − > 1− 1 + rG (ymed ) 1 + rG (ymed ) = rG (ymed ) − γ. >
1−
Here, the first inequality is from our assumption on πθt (ymed P), rG (ymed ) > 0 (Assumption 1), and rG (y) ≥ −1 for all y ∈ Y; the second inequality is due to y∈Y\{ymed } πθt (y) < 1+rGγ(ymed ) also by assumption. This contradicts the premise that VG (θt ) ≤ rG (ymed ) − γ. Hence, πθt (ymed ) ≤ 1 − 1+rGγ(ymed ) , as desired. Lemma 17. For T ∈ (0, ∞], let g : [0, T ) → (0, ∞) be a continuously differentiable function satisfying d g(t) ≤ c · g(t)p dt for all t ∈ [0, T ), where c ̸= 0 and p > 1 are constants. Then it holds that g(t) ≤
g(0) 1 1 − (p − 1)c g(0)p−1 · t p−1
for all t ∈ [0, T ) such that the denominator is positive. Similarly, if d g(t) ≥ c · g(t)p , dt then it holds that g(t) ≥
g(0) 1 1 − (p − 1)c g(0)p−1 · t p−1
for all t ∈ [0, T ) such that the denominator is positive, i.e.: 1 , c > 0, (p − 1)c g(0)p−1 t < T and t < +∞, c < 0. 71
Proof. Fix t ∈ [0, T ). Since g(s) > 0 for all s ∈ [0, T ), we may divide the inequality g ′ (s) ≤ c g(s)p by g(s)p and integrate over [0, t] to obtain Z t ′ g (s) ds ≤ c · t. p g(s) 0 d Since ds g(s)−(p−1) = −(p − 1)g ′ (s)/g(s)p , we have that 1 g(t)−(p−1) − g(0)−(p−1) ≤ c · t. − p−1
Rearranging this inequality, we then get g(t)−(p−1) ≥ g(0)−(p−1) − (p − 1)c · t. Thus, g(0)
g(t) ≤
1 − (p − 1)c g(0)p−1 · t
1/(p−1) .
If c > 0, this requires t < 1/((p − 1)c g(0)p−1 ). When c < 0, the denominator is always positive, so the bound holds for all t ∈ [0, T ). The proof of the lower bound from g ′ (t) ≥ c · g(t)p is analogous.
C
Additional Experimental Results
C.1
Empirical Demonstration of Theoretical Results With Linear Softmax Policies
Figure 6 corroborates Figure 2 by presenting the results of an identical experiment, except that the policy is trained using REINFORCE [90], i.e., sample-based gradients of the expected proxy reward, instead of exact gradients. The results show that outputs with mediocre proxy reward can impede optimization regardless of whether one uses exact or sample-based gradients. C.2
Comparison of Ranking Accuracy Variants
Listed below are experimental results omitted from Section 4. • Figure 7 supplements Figure 3 by reporting the correlation with ground truth reward increase and regret of ranking accuracy variants, when measuring reward increase on test examples from the UltraFeedback dataset as opposed to examples from the policy gradient training set. The results are analogous to Figure 3 since, as can be seen in Tables 2, 4, 6, and 8, the reward increase on test examples is nearly identical to that on training examples. • Figure 8 supplements Figure 3 by considering ranking accuracy variants computed on RewardBench2 instead of on UltraFeedback examples from the policy gradient training set. Across language models, accuracy values computed on RewardBench2 are substantially less predictive of which reward model leads to better language model performance, as indicated by the negative correlation and high regret. This highlights the importance of evaluating reward models on prompts close in distribution to those used for policy gradient training. • Figure 9 supports the experiments of Figure 3 by showing that similar trends persist when using: (i) a different dataset for policy gradient training and reward model evaluation (WildChat-IFOn-Policy-8B [37] instead of UltraFeedback); or (ii) a different ground truth reward model (Skywork-Reward-V2-Llama-3.1-8B [43] instead of ArmoRM). • Figure 10 supports the experiments of Figure 3 by showing that similar trends persist when evaluating language model performance through win-rate against the initial language model, according to a frontier (GPT) judge model, instead of through ground truth reward increase. • Tables 2 to 15 include the numerical results based on which Figures 3, 7, 8, 9, and 10 were generated. Specifically, for each setting (i.e., language model, reward model, and dataset), they report the increase in ground truth reward due to policy gradient training (or win-rate against the initial language model), as well as reward model accuracy values. 72
C.3
When Should Partially Correct Outputs Be Rewarded?
Listed below are experimental results omitted from Section 5. • Figures 11 and 12 present analogous results to Figure 4 for the Llama-3.2-3B-Instruct and OLMo-2-1B-Instruct language models (instead of Qwen3-1.7B), respectively. • Figure 13 demonstrates that the probability of initially satisfying a constraint is not the sole factor determining its ease of learnability. In particular, it shows that when one constraint is initially satisfied with substantially higher probability than the other, it is possible for both constraints to be learned quickly under partial rewards. In such cases, under binary rewards (i.e., rewarding only fully correct outputs), the constraints can either also be learned quickly or fail to be learned, especially when the initial probability of satisfying both constraints is near-zero. • Table 16 specifies the IFBench constraint pairs used in each of the experiments corresponding to Figures 4, 11, 12, and 13.
D
Additional Implementation Details
In this appendix, we provide implementation details omitted from the main text and Appendix C. Code for reproducing our results, based on the PyTorch [60], Hugging Face TRL [91], and open-instruct [37, 57] libraries, can be found at https://github.com/princeton-pli/imperfect-rewards. D.1
Empirical Demonstration of Theoretical Results With Linear Softmax Policies
Output features. The feature vectors are of dimension D = 5 and the number of outputs is 5 in all experiments. For Figures 2 and 6, we use standard basis vectors as output features. Specifically, ϕ(y⋆ ) = (1, 0, 0, 0, 0), ϕ(ymed ) = (0, 1, 0, 0, 0), and the remaining three outputs have feature vectors (0, 0, 1, 0, 0), (0, 0, 0, 1, 0), and (0, 0, 0, 0, 1). For Figure 5, the feature vectors of outputs other than y⋆ and ymed are the same as in the experiments of Figures 2 and 6. The feature vectors of y⋆ and ymed are set as follows. • In the case of ⟨ϕ(y⋆ ), ϕ(ymed )⟩ = 0: ϕ(y⋆ ) = (3/2, 0, 0, 0, 0) • In the case of ⟨ϕ(y⋆ ), ϕ(ymed )⟩ < 0: ϕ(y⋆ ) = (3/2, 0, 0, 0, 0) • In the case of ⟨ϕ(y⋆ ), ϕ(ymed )⟩ > 0: ϕ(y⋆ ) = (3/2, 0, 0, 0, 0)
, ,
,
ϕ(ymed ) = (0, 1, 0, 0, 0) .
√ √ ϕ(ymed ) = (−1/ 2, 1/ 2, 0, 0, 0) . √ √ ϕ(ymed ) = (1/ 2, 1/ 2, 0, 0, 0) .
Note that the norm of ϕ(y⋆ ) is taken to be larger than that of ϕ(ymed ) to accord with the conditions of Theorem 4. We observed trends analogous to those in Figure 5 when setting the first coordinate of ϕ(y⋆ ) to 1 instead of 3/2, in which case all feature vectors have unit norm. Initial policy. For the experiments of Figures 2 and 6, we set the initial policy weights θ0 such that πθ0 (y⋆ ) is either 0.05, 0.1, or 0.15, πθ0 (ymed ) = 0.5, and the remaining probability mass is uniformly divided between the remaining three outputs. Since the feature vectors in these experiments are standard basis vectors, we can achieve this by setting the coordinate of θ0 associated with each output y ∈ Y to the logarithm of the desired probability. For the experiments of Figure 5, we set θ0 such that πθ0 (y⋆ ) = 0.05, πθ0 (ymed ) = 0.5, and the remaining probability mass is uniformly divided between the remaining three outputs. This is done by using a linear equations solver to find a θ0 satisfying ⟨ϕ(y), θ0 ⟩ = ln π(y) for all y ∈ Y, where π(y) is the desired initial probability of y. Policy gradient optimization. In the experiments of Figures 2 and 5, we train linear softmax policies by running gradient ascent over the expected proxy or ground truth rewards using exact gradients. In Figure 6, we use REINFORCE [90]. Namely, at each training step, we sample an output y from the current policy and update the parameters via θt+1 = θt + η · rP (y) · ∇θ ln πθt (y), where rP (y) · ∇θ ln πθt (y) is an unbiased estimate of ∇VP (θt ) and η > 0 is the learning rate. We set the learning rate to 0.1 in all experiments. Each figure caption specifies the reward assignments in the corresponding experiments. Hardware. Standard laptop CPU. 73
D.2
Comparison of Ranking Accuracy Variants
Language models. We consider as initial policies four language models that cover different model families and types (i.e., pretrained, supervised finetuned, and instruction-tuned): Llama-3.2-3BInstruct, Llama-3.2-1B-Instruct, OLMo-2-1B-SFT, and Qwen3-1.7B-Base. When processing inputs to the language models we use their default chat templates. Reward models. Table 1 lists the reward models used in the experiments of Section 4 and Appendix C.2. The reward models were chosen so that they cover a range of RewardBench2 scores [47]. When using Skywork-Reward-V2-Llama-3.1-8B as the ground truth reward model (Figure 9), we exclude it from the reward models used for training and include instead the ArmoRM [88] model. Table 1: Listed are the reward models used in the experiments of Section 4 and Appendix C.2. The RewardBench2 scores are taken from the official leaderboard [47]. The two models without a RewardBench2 score were chosen and positioned in the table based on their original RewardBench score [38]. Reward Model
RewardBench2 Score
Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Qwen3-8B Skywork-Reward-V2-Qwen3-4B Llama-3.1-8B-Instruct-RM-RB2 Skywork-Reward-V2-Qwen3-1.7B GRM-Llama3.2-3B-rewardmodel-ft Llama-3-OffsetBias-RM-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B RM-Mistral-7B internlm2-1_8b-reward llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B
84.1 78.4 75.5 72.8 68.2 – 64.8 64.4 61.2 59.6 39.0 – 30.6
Reward normalization. To ensure fair comparison across reward models, which can differ in the scale of rewards they produce, we normalize rewards in each batch of training by subtracting the mean and dividing by the standard deviation. Data. We take the binarized version of UltraFeedback8 [14] and filter out examples in which the prompt or one of the outputs exceeds 512 tokens. We then select a subset of 10000 examples from the training set (train_prefs split), relabel output preferences in the selected subset and the test set (test_prefs split) using the ground truth reward model (ArmoRM or Skywork-Reward-V2-Llama3.1-8B, depending on the experiment), and filter out examples where both outputs have the same ground truth reward. For the ArmoRM ground truth reward model, the resulting training and test sets contain 9808 and 1406 examples, respectively, while for the Skywork-Reward-V2-Llama-3.1-8B ground truth reward model, the resulting training and test sets contain 9930 and 1422 examples, respectively. For experiments on the WildChat-IF-On-Policy-8B dataset9 (Figure 9), which does not have a test set, we follow the same procedure using 3000 randomly selected examples from the dataset and split the examples remaining after filtering into a training set of 1969 examples and a test set of 985 examples. Policy gradient optimization. Our RLOO implementation is based on the RLOOTrainer class from the TRL framework, which uses the Adam optimizer. We set the learning rate to 1e-7, batch size to 32, KL regularization coefficient to 0.01, and the num_mini_batches hyperparameter to 1. Optimization is carried out for two full passes over the prompts in the training set, where for each prompt in a batch we sample four outputs. Generation hyperparameters. For both training and evaluation, we generate outputs from the policies (i.e., language models) using a temperature of 1 and a maximum output length of 512 tokens. Ground truth reward increase policy evaluation. We evaluate the ground truth reward achieved by a policy based on fixed (randomly selected) subsets of 500 prompts from the policy gradient training 8 9
https://huggingface.co/datasets/HuggingFaceH4/ultrafeedback_binarized https://huggingface.co/datasets/allenai/tulu-3-wildchat-if-on-policy-8b
74
and test sets. We sample 10 outputs from the policy for each prompt, compute their ground truth rewards, and then average the rewards across all outputs and prompts. As noted in Section 4, we mainly consider the ground truth reward increase on training examples as it directly reflects policy gradient optimization, which is the focus of our work, without conflating it with generalization. Nonetheless, as reported in Appendix C, we found the reward increase on test examples to be nearly identical. For each combination of reward model and initial policy, the reported ground truth reward increase is the mean across three separate runs with different random seeds. Win-rate policy evaluation. For Figure 10, instead of using a ground truth reward model for policy evaluation, we adopt the AlpacaEval framework [41] to compute win-rates against the initial policy according to a GPT judge model. The win-rates are computed for a single policy gradient run per combination of reward model and initial policy, based on the same 500 prompts used for evaluation via ground truth reward increase. We sample one output from the initial policy and one from the final policy, for each prompt, and ask the judge model to select the better output using the format provided by AlpacaEval. For all models, except Qwen3-1.7B-Base, gpt-4-1106-preview served as the judge model. For Qwen3-1.7B-Base we used gpt-4.1-2025-04-14 as the judge since the win-rates for this model were computed after the deprecation of gpt-4-1106-preview. Reward model evaluation. For each reward model and initial policy in Figures 3, 7, 9, and 10, we compute all ranking-accuracy variants (Acc, Acc-W, HAcc, and HAcc-W) on the same 500 training prompts used for policy evaluation, where output preferences are relabeled using the ground truth reward model. In Figure 8, accuracy values are computed on RewardBench2, after excluding examples from the “Ties” subset and relabeling preferences using the ground truth reward model. Lastly, to compute HAcc and HAcc-W, we estimate the mean proxy reward achieved by an initial policy for each prompt using 10 sampled outputs. Hardware. All experiments ran on Nvidia H100 GPUs with 80GB memory. For the Llama-3.21B-Instruct and OLMo-2-1B-SFT models, we used two GPUs per policy gradient run. For the Llama-3.2-3B-Instruct and Qwen3-1.7B-Base models, we used four GPUs per policy gradient run. Evaluations were carried out on a single GPU. D.3
When Should Partially Correct Outputs Be Rewarded?
Language models. We consider as initial policies three instruction-tuned language models: Qwen31.7B (thinking disabled), OLMo-2-1B-Instruct, and Llama-3.2-3B-Instruct. When processing inputs to the language models we use their default chat templates. Data. We take the binarized version of UltraFeedback and randomly sample 4000 prompts that do not exceed 512 tokens from the train_prefs split. Then, we construct instruction following datasets by appending to each prompt a pair of constraints from IFBench [62] using the following template: {original prompt} {constraint 1} {constraint 2} All prompts within a dataset share the same pair of constraints and the datasets differ in which constraints they contain. For each initial policy, we experimented with multiple pairs of constraints and report the results for pairs exhibiting representative behaviors. Reward design. We consider two reward schemes. The “partial rewards” scheme corresponds to giving a reward of 0.5 for each constraint satisfied. Thus, an output receives reward 0, 0.5, or 1 depending on whether it satisfies none, exactly one, or both constraints. The “binary (full-correctness) rewards” scheme corresponds to assigning a reward of 1 to fully correct outputs, satisfying both constraints, and 0 to all other outputs. Policy gradient optimization. We rely on the open-instruct framework and train in bfloat16 mixed precision using its default DAPO objective implementation [96]. Under our configuration, which adopts symmetric clipping (with ε = 0.2), disables dynamic sampling, sets non_stop_penalty to false, and does not use KL regularization, the objective reduces to the GRPO objective with token-level normalization. Each batch consists of 16 unique prompts. The number of sampled outputs per prompt varies across language models: 96 for Qwen3-1.7B, 64 for Llama-3.2-3B-Instruct, and 32 for OLMo-2-1B-Instruct. Optimization is carried out for four full passes over the training prompts using the Adam optimizer and a linear decay learning rate schedule with warmup ratio 0.03 and base learning rate 1e-6. We chose these relatively large numbers of outputs per prompt together with this learning rate schedule to promote stable training. Note that for the learning rate decay schedule we 75
set the number of epochs to eight, but ended up stopping early after four epochs (1000 training steps) since we observed that the probabilities of success tend to plateau by that point. Policy evaluation. The probability of satisfying a constraint (or both constraints jointly) is estimated at each training step by the fraction of outputs in a batch that satisfy the constraint (or both constraints). Generation hyperparameters. We generate outputs from the policies (i.e., language models) using a temperature of 1 and a maximum output length of 512 tokens. Hardware. Each experiment ran on two Nvidia H100 GPUs with 80GB memory. Effect of randomness on the success of binary (full-correctness) rewards. When the initial probability of satisfying both constraints is low, success under binary rewards can vary across runs since it depends on fully correct outputs being sampled often enough during training. This phenomenon persists even when fixing the random seed due to non-determinism in GPU computation. In particular, in the setting of Figure 4, binary rewards led to successful learning of both constraints in roughly 50% of our runs, whereas in the remaining runs the policy failed to learn either constraint. When the initial probability of satisfying both constraints is not low, we found the outcome under binary rewards to be more consistent across runs.
76
πθt (ymed )
Output Probability
πθ0 (y ) = 0.15
Output Probability
Case II: rP (ymed ) = −1
Case I: rP (ymed ) = 0.8
πθt (y )
πθt (other outputs)
πθ0 (y ) = 0.1
πθ0 (y ) = 0.05
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0
5000
10000
15000
20000
Training Step
0.0
0
5000
10000
15000
20000
Training Step
0.0
1.0
1.0
1.0
0.5
0.5
0.5
0.0
0
5000
10000
15000
20000
Training Step
0.0
0
5000
10000
15000
20000
Training Step
0.0
0
5000
0
5000
10000
15000
20000
10000
15000
20000
Training Step
Training Step
Figure 6: Attraction to outputs with mediocre proxy reward can impede policy gradient optimization (with sample-based gradients). This figure presents the results of an experiment identical to that of Figure 2, except that the policy is trained using REINFORCE [90], i.e., sample-based gradients of the expected proxy reward, instead of exact gradients. As when training with exact gradients of the expected proxy reward, outputs with mediocre proxy reward can impede optimization. See Appendix D for additional implementation details.
Spearman Correlation
0.8 0.6 0.39
0.4 0.2 0.0 0.2
Regret in Choosing Best Reward Model Acc Acc-W HAcc HAcc-W
0.08 0.10 -0.03 -0.04
0.31
0.21 0.05
0.01 -0.14
Llama-3B Instruct
Llama-1B Instruct
17.2%
16.9%
3.0%
c-W
24.7%
17.2%
16.9%
3.0%
60%
cc HA
0.0%
0.0%
21.3%
3.0%
40%
-W Acc
0.0%
0.0%
16.9%
3.0%
Llama-3B Instruct
Llama-1B Instruct
OLMo-1B SFT
Qwen-1.7B Base
Ac 0.08
-0.14 -0.29
0.4
24.7%
0.21 0.08
-0.25
OLMo-1B SFT
H
Qwen-1.7B Base
100%
Acc
80%
Regret
Correlation With Ground Truth Reward Increase 1.0
20% 0%
Figure 7: Harm-aware ranking accuracy variants are more predictive of which reward model leads to better language model performance. This figure supplements Figure 3 by reporting the same metrics, but based on the reward increase over test examples from the UltraFeedback dataset. The results are analogous to Figure 3, where reward increase is measured on UltraFeedback examples used for policy gradient training. Specifically, as can be seen in Tables 2, 4, 6, and 8, the reward increase on test examples is nearly identical to that on training examples. Additional implementation details are provided in Appendix D.
77
Spearman Correlation
Acc Acc-W HAcc HAcc-W
0.5 0.0
Regret in Choosing Best Reward Model
0.18 0.13 0.18 0.13
-0.09 -0.25
0.5
-0.36
-0.34
-0.40
-0.47
-0.36
1.0
Llama-3B Instruct
28.4%
23.9%
24.5%
6.4%
-W Acc
36.5%
20.7%
19.8%
1.9%
60%
cc HA
28.4%
23.9%
24.5%
6.4%
40%
W
36.5%
20.7%
19.8%
1.9%
Llama-3B Instruct
Llama-1B Instruct
OLMo-1B SFT
Qwen-1.7B Base
-0.47 -0.89
Llama-1B Instruct
-0.77
-0.76
ccHA
-0.92
OLMo-1B SFT
Qwen-1.7B Base
100%
Acc
80%
Regret
Correlation With Ground Truth Reward Increase
1.0
20% 0%
Figure 8: Ranking accuracy variants computed on prompts differing from those used for policy gradient training are less predictive of which reward model leads to better language model performance. This figure supplements Figure 3 by considering ranking accuracy variants computed on RewardBench2 instead of on UltraFeedback examples from the policy gradient training set. Across language models, accuracy values computed on RewardBench2 are substantially less predictive of which reward model leads to better language model performance, as indicated by the negative correlation and high regret. This highlights a pitfall of evaluating reward models based on prompts that differ from those used for training the language model. See Appendix D for additional implementation details.
Correlation With Ground Truth Reward Increase Spearman Correlation
0.8 0.6
Regret in Choosing Best Reward Model 100%
0.91 0.88 0.94
Acc Acc-W HAcc HAcc-W
0.75
Acc
15.7%
-W
11.9%
5.5%
60%
cc
11.9%
0.0%
40%
W
11.9%
0.0%
Llama-1B-Instruct Different Dataset
Llama-1B-Instruct Different Ground Truth
80%
Acc
0.4 0.2 0.0
-0.06
0.2
HA -0.14 -0.10
Acc
-0.26
0.4
Llama-1B-Instruct Different Dataset
H
Llama-1B-Instruct Different Ground Truth
5.5%
Regret
1.0
20% 0%
1.0 0.8 0.6 0.4 0.2 0.0 0.2 0.4
Correlation With Win-Rate Against Initial Policy Acc Acc-W HAcc HAcc-W
0.55 0.49
0.01 -0.10 -0.09
0.47 0.43 0.47 0.43
0.08
-0.08 -0.03
-0.04 -0.29
Llama-3B Instruct
Regret in Choosing Best Reward Model
Llama-1B Instruct
-0.20
-0.27
OLMo-1B SFT
Qwen-1.7B Base
100%
Acc
16.0%
6.2%
7.0%
0.3%
-W Acc cc HA
16.0%
6.2%
7.0%
0.3%
60%
4.0%
2.4%
6.3%
0.3%
40%
W ccHA
4.0%
2.4%
7.0%
0.3%
Llama-3B Instruct
Llama-1B Instruct
OLMo-1B Qwen-1.7B SFT Base
80%
Regret
Spearman Correlation
Figure 9: Harm-aware ranking accuracy variants are more predictive of which reward model leads to better language model performance. This figure supports the experiments of Figure 3 by showing the same trends persist when using: (i) a different dataset for policy gradient training and reward model evaluation (WildChat-IF-On-Policy-8B [37] instead of UltraFeedback); or (ii) a different ground truth reward model for evaluating language model performance and accuracy values (Skywork-Reward-V2-Llama-3.1-8B [43] instead of ArmoRM). Both sets of experiments were conducted with the Llama-3.2-1B-Instruct language model. See Appendix D for additional implementation details.
20% 0%
Figure 10: Harm-aware ranking accuracy variants are more predictive of which reward model leads to better language model performance. This figure supports Figure 3 by showing that similar trends persist when evaluating language model performance through win-rate against the initial language model, according to a frontier (GPT) judge model, instead of through ground truth reward increase. See Appendix D for additional implementation details.
78
Binary (Full-Correctness) Rewards
Partial Rewards
Gap in Learnability
Constraint Pair B:
No Gap in Learnability
Probability of Success
Constraint Pair A:
Probability of Success
(0.5 for each satisfied constraint)
(1 if both constraints satisfied, 0 otherwise)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
constraint 1 constraint 2 both
0.2 0.0
0
200
400
600
Training Step
800
1000
0.4 0.0
1.0
1.0
0.8
0.8
0.6
0.6
0.4
constraint 1 constraint 2 both
0.2 0.0
0
200
400
600
Training Step
800
1000
constraint 1 constraint 2 both
0.2 0
200
400
600
Training Step
0.4
1000
constraint 1 constraint 2 both
0.2 0.0
800
0
200
400
600
Training Step
800
1000
Figure 11: Rewarding partially correct outputs can impede policy gradient optimization. This figure presents the results of an experiment analogous to that of Figure 4, where the language model is Llama-3.2-3BInstruct instead of Qwen3-1.7B. Similarly to Figure 4, when the initial probability of satisfying one constraint is noticeably higher than the probability of satisfying the other, rewarding partial correctness can cause the policy to stall on satisfying only the easier constraint. In this case, rewarding only fully correct outputs can lead to faster learning of both constraints. However, when both constraints are initially satisfied with similar probability, both reward designs tend to work well. See Appendix D for further implementation details.
Binary (Full-Correctness) Rewards
Partial Rewards
Gap in Learnability
Constraint Pair B:
No Gap in Learnability
Probability of Success
Constraint Pair A:
Probability of Success
(0.5 for each satisfied constraint)
(1 if both constraints satisfied, 0 otherwise)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
constraint 1 constraint 2 both
0.2 0.0
0
200
400
600
Training Step
800
1000
0.4 0.0
1.0
1.0
0.8
0.8
0.6
0.6
0.4
constraint 1 constraint 2 both
0.2 0.0
0
200
400
600
Training Step
800
1000
constraint 1 constraint 2 both
0.2 0
200
400
600
Training Step
0.4
1000
constraint 1 constraint 2 both
0.2 0.0
800
0
200
400
600
Training Step
800
1000
Figure 12: Rewarding partially correct outputs can impede policy gradient optimization. This figure presents the results of an experiment analogous to that of Figure 4, where the language model is OLMo-2-1BInstruct instead of Qwen3-1.7B. Similarly to Figure 4, when the initial probability of satisfying one constraint is noticeably higher than the probability of satisfying the other, rewarding partial correctness can cause the policy to stall on satisfying only the easier constraint. In this case, rewarding only fully correct outputs can lead to faster learning of both constraints. However, when both constraints are initially satisfied with similar probability, both reward designs tend to work well. See Appendix D for further implementation details.
79
Binary (Full-Correctness) Rewards
Partial Rewards
Constraint Pair B:
Gap in Initial Probabilities but Only Partial Rewards Learn
Probability of Success
Constraint Pair A:
Gap in Initial Probabilities but No Gap in Learnability
Probability of Success
(0.5 for each satisfied constraint)
(1 if both constraints satisfied, 0 otherwise)
1.0
1.0
0.8
0.8
0.6
0.6
0.4
constraint 1 constraint 2 both
0.2 0.0
0
200
400
600
800
Training Step
1000
0.4 0.0
1.0
1.0
0.8
0.8
0.6
0.6
0.4
constraint 1 constraint 2 both
0.2 0.0
0
200
400
600
800
Training Step
1000
constraint 1 constraint 2 both
0.2 0
200
400
600
800
1000
600
800
1000
Training Step
constraint 1 constraint 2 both
0.4 0.2 0.0
0
200
400
Training Step
Figure 13: The probability of initially satisfying a constraint is not the sole factor determining its ease of learnability. This figure supplements Figure 4 by demonstrating that when one constraint is initially satisfied with substantially higher probability than the other, it is possible for both constraints to be learned at a similar rate under partial rewards. In such cases, under binary rewards, we find that the constraints can either also be learned relatively in tandem (top row) or fail to be learned (bottom row). The latter often occurs when the initial probability of satisfying both constraints is near-zero. These experiments were conducted with the Qwen3-1.7B language model and the same training procedure as in Figure 4, but used different constraint pairs.
Table 2: Numerical results underlying Figures 3 and 7 for the Llama-3.2-3B-Instruct language model. Specifically, reported is the ground truth reward increase (GT Increase) on train and test prompts (mean and standard deviation over three separate runs), along with accuracy values for all considered reward models. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the ArmoRM ground truth reward model. Language Model: Llama-3.2-3B-Instruct Dataset: UltraFeedback Reward Model (used for training)
GT Increase (train)
GT Increase (test)
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.374 ±0.011 0.220 ±0.018 0.268 ±0.005 0.278 ±0.007 0.273 ±0.014 0.329 ±0.013 0.290 ±0.016 0.238 ±0.010 0.286 ±0.001 0.277 ±0.002 0.272 ±0.004 0.268 ±0.008 0.281 ±0.002
0.369 ±0.002 0.223 ±0.015 0.270 ±0.004 0.278 ±0.003 0.293 ±0.008 0.334 ±0.006 0.293 ±0.008 0.243 ±0.007 0.292 ±0.005 0.281 ±0.007 0.276 ±0.005 0.274 ±0.008 0.285 ±0.004
0.850 0.828 0.810 0.864 0.792 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.842 0.802 0.791 0.850 0.765 0.764 0.794 0.739 0.779 0.766 0.793 0.809 0.800
0.930 0.892 0.880 0.926 0.856 0.878 0.890 0.890 0.920 0.904 0.906 0.912 0.924
0.925 0.859 0.853 0.914 0.842 0.867 0.847 0.874 0.883 0.879 0.872 0.885 0.894
80
Table 3: For the Llama-3.2-3B-Instruct language model, this table reports reward model accuracy values computed on examples from RewardBench2, with preference labels provided by the ArmoRM ground truth reward model. These results were used for producing Figure 8. Language Model: Llama-3.2-3B-Instruct Dataset: RewardBench2 Reward Model (used for training)
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.698 0.467 0.739 0.704 0.602 0.387 0.653 0.720 0.672 0.647 0.690 0.712 0.720
0.646 0.467 0.714 0.693 0.623 0.408 0.630 0.725 0.638 0.644 0.672 0.700 0.707
0.753 0.555 0.767 0.735 0.626 0.442 0.684 0.751 0.718 0.681 0.721 0.732 0.744
0.684 0.571 0.733 0.721 0.633 0.439 0.650 0.742 0.665 0.665 0.688 0.717 0.724
Table 4: Numerical results underlying Figures 3 and 7 for the Llama-3.2-1B-Instruct language model. Specifically, reported is the ground truth reward increase (GT Increase) on train and test prompts (mean and standard deviation over three separate runs), along with accuracy values for all considered reward models. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the ArmoRM ground truth reward model. Language Model: Llama-3.2-1B-Instruct Dataset: UltraFeedback Reward Model (used for training)
GT Increase (train)
GT Increase (test)
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.457 ±0.015 −1.945 ±0.100 0.348 ±0.010 0.376 ±0.006 0.421 ±0.012 0.403 ±0.008 0.402 ±0.011 0.362 ±0.005 0.390 ±0.004 0.384 ±0.008 0.390 ±0.010 0.383 ±0.010 0.389 ±0.014
0.465 ±0.014 −1.960 ±0.105 0.359 ±0.012 0.385 ±0.010 0.433 ±0.021 0.425 ±0.012 0.418 ±0.009 0.364 ±0.009 0.398 ±0.004 0.396 ±0.008 0.400 ±0.010 0.388 ±0.014 0.393 ±0.005
0.850 0.828 0.810 0.864 0.792 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.835 0.801 0.796 0.852 0.777 0.775 0.819 0.742 0.779 0.773 0.796 0.823 0.809
0.896 0.850 0.838 0.888 0.826 0.864 0.864 0.822 0.876 0.840 0.852 0.862 0.850
0.895 0.820 0.822 0.876 0.809 0.874 0.854 0.773 0.837 0.807 0.814 0.835 0.820
81
Table 5: For the Llama-3.2-1B-Instruct language model, this table reports reward model accuracy values computed on examples from RewardBench2, with preference labels provided by the ArmoRM ground truth reward model. These results were used for producing Figure 8. Language Model: Llama-3.2-1B-Instruct Dataset: RewardBench2 Reward Model (used for training)
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.696 0.467 0.739 0.704 0.602 0.387 0.653 0.720 0.672 0.647 0.690 0.712 0.720
0.642 0.462 0.711 0.693 0.621 0.417 0.631 0.727 0.634 0.649 0.682 0.707 0.716
0.714 0.523 0.746 0.710 0.612 0.427 0.663 0.729 0.693 0.662 0.698 0.718 0.726
0.654 0.530 0.716 0.699 0.623 0.445 0.637 0.730 0.646 0.657 0.685 0.709 0.719
Table 6: Numerical results underlying Figures 3 and 7 for the OLMo-2-1B-SFT language model. Specifically, reported is the ground truth reward increase (GT Increase) on train and test prompts (mean and standard deviation over three separate runs), along with accuracy values for all considered reward models. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the ArmoRM ground truth reward model. Language Model: OLMo-2-1B-SFT Dataset: UltraFeedback Reward Model (used for training)
GT Increase (train)
GT Increase (test)
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.218 ±0.011 0.281 ±0.006 0.212 ±0.004 0.236 ±0.009 0.260 ±0.006 0.275 ±0.004 0.266 ±0.006 0.213 ±0.002 0.234 ±0.006 0.238 ±0.001 0.241 ±0.001 0.225 ±0.009 0.226 ±0.008
0.198 ±0.014 0.251 ±0.011 0.191 ±0.006 0.209 ±0.009 0.234 ±0.002 0.242 ±0.006 0.233 ±0.004 0.192 ±0.007 0.212 ±0.004 0.216 ±0.005 0.219 ±0.009 0.206 ±0.002 0.204 ±0.006
0.850 0.828 0.810 0.864 0.792 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.827 0.793 0.778 0.856 0.759 0.760 0.798 0.748 0.763 0.757 0.784 0.807 0.794
0.880 0.854 0.828 0.878 0.814 0.814 0.846 0.818 0.826 0.824 0.840 0.856 0.846
0.861 0.809 0.789 0.863 0.782 0.800 0.809 0.769 0.776 0.771 0.794 0.814 0.802
82
Table 7: For the OLMo-2-1B-SFT language model, this table reports reward model accuracy values computed on examples from RewardBench2, with preference labels provided by the ArmoRM ground truth reward model. These results were used for producing Figure 8. Language Model: OLMo-2-1B-SFT Dataset: RewardBench2 Reward Model (used for training)
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.691 0.467 0.739 0.704 0.602 0.387 0.653 0.720 0.672 0.647 0.690 0.712 0.720
0.651 0.465 0.715 0.703 0.622 0.414 0.645 0.735 0.653 0.667 0.703 0.723 0.738
0.708 0.526 0.750 0.708 0.615 0.402 0.659 0.733 0.683 0.666 0.704 0.727 0.733
0.658 0.527 0.721 0.705 0.624 0.418 0.647 0.742 0.658 0.675 0.708 0.730 0.744
Table 8: Numerical results underlying Figures 3 and 7 for the Qwen3-1.7B-Base language model. Specifically, reported is the ground truth reward increase (GT Increase) on train and test prompts (mean and standard deviation over three separate runs), along with accuracy values for all considered reward models. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the ArmoRM ground truth reward model. Language Model: Qwen3-1.7B-Base Dataset: UltraFeedback Reward Model (used for training)
GT Increase (train)
GT Increase (test)
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
2.257 ±0.009 −0.969 ±0.038 2.113 ±0.007 2.187 ±0.050 2.206 ±0.006 2.011 ±0.074 2.207 ±0.006 2.179 ±0.008 2.213 ±0.004 2.200 ±0.007 2.203 ±0.009 2.201 ±0.008 2.214 ±0.008
2.264 ±0.006 −0.964 ±0.052 2.106 ±0.006 2.197 ±0.056 2.207 ±0.004 2.026 ±0.065 2.205 ±0.004 2.177 ±0.007 2.216 ±0.004 2.213 ±0.008 2.207 ±0.005 2.199 ±0.003 2.216 ±0.008
0.850 0.828 0.810 0.864 0.792 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.859 0.807 0.817 0.865 0.794 0.790 0.843 0.756 0.793 0.768 0.803 0.821 0.814
0.850 0.832 0.810 0.866 0.794 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.859 0.807 0.817 0.868 0.795 0.790 0.843 0.756 0.793 0.768 0.803 0.821 0.814
83
Table 9: For the Qwen3-1.7B-Base language model, this table reports reward model accuracy values computed on examples from RewardBench2, with preference labels provided by the ArmoRM ground truth reward model. These results were used for producing Figure 8. Language Model: Qwen3-1.7B-Base Dataset: RewardBench2 Reward Model (used for training)
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.691 0.467 0.739 0.704 0.602 0.387 0.653 0.720 0.672 0.647 0.690 0.712 0.720
0.648 0.470 0.714 0.697 0.652 0.421 0.652 0.729 0.655 0.669 0.700 0.723 0.732
0.691 0.469 0.739 0.705 0.602 0.387 0.653 0.720 0.672 0.647 0.690 0.712 0.720
0.648 0.470 0.714 0.697 0.652 0.421 0.652 0.729 0.655 0.669 0.700 0.723 0.732
Table 10: Numerical results underlying Figure 9 for the Llama-3.2-1B-Instruct language model trained on the WildChat-IF-On-Policy-8B dataset. Specifically, reported is the ground truth reward increase (GT Increase) on train and test prompts (mean and standard deviation over three separate runs), along with accuracy values for all considered reward models. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the ArmoRM ground truth reward model. Language Model: Llama-3.2-1B-Instruct Dataset: WildChat-IF-On-Policy-8B Reward Model (used for training)
GT Increase (train)
GT Increase (test)
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.361 ±0.033 −0.689 ±0.924 0.297 ±0.049 0.360 ±0.049 0.380 ±0.050 0.391 ±0.057 0.356 ±0.045 0.306 ±0.054 0.329 ±0.054 0.321 ±0.064 0.329 ±0.058 0.330 ±0.039 0.344 ±0.042
0.338 ±0.035 −0.693 ±0.915 0.277 ±0.049 0.329 ±0.035 0.350 ±0.041 0.369 ±0.051 0.324 ±0.041 0.278 ±0.048 0.318 ±0.041 0.302 ±0.051 0.299 ±0.044 0.297 ±0.035 0.315 ±0.036
0.838 0.740 0.848 0.846 0.782 0.738 0.838 0.856 0.854 0.818 0.844 0.876 0.874
0.841 0.732 0.826 0.824 0.777 0.735 0.827 0.840 0.845 0.832 0.836 0.864 0.869
0.852 0.762 0.858 0.850 0.792 0.776 0.848 0.868 0.872 0.836 0.850 0.878 0.880
0.847 0.748 0.832 0.826 0.779 0.773 0.834 0.842 0.863 0.850 0.838 0.864 0.870
84
Table 11: Numerical results underlying Figure 9 for the Llama-3.2-1B-Instruct language model. Specifically, reported is the ground truth reward increase (GT Increase) on train and test prompts (mean and standard deviation over three separate runs), along with accuracy values for all considered reward models. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the Skywork-Reward-V2-Llama-3.1-8B ground truth reward model. Language Model: Llama-3.2-1B-Instruct Dataset: UltraFeedback (Ground Truth: Skywork-Reward-V2-Llama-3.1-8B) Reward Model (used for training)
GT Increase (train)
GT Increase (test)
Acc
Acc-W
HAcc
HAcc-W
ArmoRM-Llama3-8B-v0-1 GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.420 ±0.105 0.466 ±0.084 −1.768 ±0.221 0.416 ±0.082 0.440 ±0.062 0.414 ±0.146 0.330 ±0.176 0.471 ±0.116 0.533 ±0.095 0.503 ±0.101 0.510 ±0.101 0.503 ±0.078 0.504 ±0.080
0.389 ±0.107 0.462 ±0.103 −1.824 ±0.247 0.406 ±0.078 0.417 ±0.072 0.397 ±0.152 0.338 ±0.178 0.453 ±0.119 0.522 ±0.106 0.498 ±0.101 0.489 ±0.097 0.479 ±0.086 0.488 ±0.078
0.810 0.808 0.774 0.862 0.826 0.810 0.740 0.824 0.878 0.852 0.856 0.894 0.896
0.779 0.805 0.721 0.804 0.814 0.767 0.703 0.801 0.886 0.850 0.864 0.881 0.893
0.834 0.842 0.802 0.878 0.852 0.826 0.832 0.848 0.920 0.898 0.904 0.912 0.908
0.811 0.837 0.745 0.822 0.836 0.776 0.785 0.816 0.914 0.890 0.907 0.899 0.908
Table 12: Numerical results underlying Figure 10 for the Llama-3.2-3B-Instruct language model. Specifically, for each reward model, the table reports the win-rate of the language model after policy gradient against the initial language model, as judged by a frontier GPT model, along with the accuracy values for the reward model. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the ArmoRM ground truth reward model. Language Model: Llama-3.2-3B-Instruct Dataset: UltraFeedback Reward Model (used for training)
Win-Rate
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.738 0.704 0.715 0.646 0.702 0.769 0.726 0.677 0.720 0.735 0.713 0.724 0.702
0.850 0.828 0.810 0.864 0.792 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.842 0.802 0.791 0.850 0.765 0.764 0.794 0.739 0.779 0.766 0.793 0.809 0.800
0.930 0.892 0.880 0.926 0.856 0.878 0.890 0.890 0.920 0.904 0.906 0.912 0.924
0.925 0.859 0.853 0.914 0.842 0.867 0.847 0.874 0.883 0.879 0.872 0.885 0.894
85
Table 13: Numerical results underlying Figure 10 for the Llama-3.2-1B-Instruct language model. Specifically, for each reward model, the table reports the win-rate of the language model after policy gradient against the initial language model, as judged by a frontier GPT model, along with the accuracy values for the reward model. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the ArmoRM ground truth reward model. Language Model: Llama-3.2-1B-Instruct Dataset: UltraFeedback Reward Model (used for training)
Win-Rate
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.737 0.009 0.701 0.708 0.722 0.733 0.755 0.695 0.741 0.725 0.704 0.712 0.716
0.850 0.828 0.810 0.864 0.792 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.835 0.801 0.796 0.852 0.777 0.775 0.819 0.742 0.779 0.773 0.796 0.823 0.809
0.896 0.850 0.838 0.888 0.826 0.864 0.864 0.822 0.876 0.840 0.852 0.862 0.850
0.895 0.820 0.822 0.876 0.809 0.874 0.854 0.773 0.837 0.807 0.814 0.835 0.820
Table 14: Numerical results underlying Figure 10 for the OLMo-2-1B-SFT language model. Specifically, for each reward model, the table reports the win-rate of the language model after policy gradient against the initial language model, as judged by a frontier GPT model, along with the accuracy values for the reward model. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the ArmoRM ground truth reward model. Language Model: OLMo-2-1B-SFT Dataset: UltraFeedback Reward Model (used for training)
Win-Rate
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.616 0.628 0.607 0.611 0.627 0.642 0.657 0.610 0.611 0.634 0.624 0.610 0.586
0.850 0.828 0.810 0.864 0.792 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.827 0.793 0.778 0.856 0.759 0.760 0.798 0.748 0.763 0.757 0.784 0.807 0.794
0.880 0.854 0.828 0.878 0.814 0.814 0.846 0.818 0.826 0.824 0.840 0.856 0.846
0.861 0.809 0.789 0.863 0.782 0.800 0.809 0.769 0.776 0.771 0.794 0.814 0.802
86
Table 15: Numerical results underlying Figure 10 for the Qwen3-1.7B-Base language model. Specifically, for each reward model, the table reports the win-rate of the language model after policy gradient against the initial language model, as judged by a frontier GPT model, along with the accuracy values for the reward model. We measure accuracy values on examples from the policy gradient training set, with preference labels provided by the ArmoRM ground truth reward model. Language Model: Qwen3-1.7B-Base Dataset: UltraFeedback Reward Model (used for training)
Win-Rate
Acc
Acc-W
HAcc
HAcc-W
GRM-Llama3.2-3B-rewardmodel-ft internlm2-1_8b-reward Llama-3.1-8B-Instruct-RM-RB2 Llama-3-OffsetBias-RM-8B llama-3-tulu-2-8b-uf-mean-rm RM-Gemma-2B RM-Mistral-7B Skywork-Reward-V2-Llama-3.1-8B Skywork-Reward-V2-Llama-3.2-1B Skywork-Reward-V2-Qwen3-0.6B Skywork-Reward-V2-Qwen3-1.7B Skywork-Reward-V2-Qwen3-4B Skywork-Reward-V2-Qwen3-8B
0.957 0.191 0.963 0.962 0.936 0.919 0.955 0.957 0.961 0.951 0.946 0.959 0.965
0.850 0.828 0.810 0.864 0.792 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.859 0.807 0.817 0.865 0.794 0.790 0.843 0.756 0.793 0.768 0.803 0.821 0.814
0.850 0.832 0.810 0.866 0.794 0.778 0.840 0.796 0.806 0.802 0.826 0.844 0.834
0.859 0.807 0.817 0.868 0.795 0.790 0.843 0.756 0.793 0.768 0.803 0.821 0.814
87
Table 16: This table specifies the IFBench constraint pairs used in each of the experiments corresponding to Figures 4, 11, 12, and 13. We include the constraint name according to the IFBench taxonomy (e.g., words:start_verb), followed by the instruction that is appended to the prompt. Language Model
Constraint Pair Pair A
Constraint 1 words:start_verb
words:alphabet
The response must start with a verb.
Each word must start with the next letter of the alphabet, looping back to ‘A’ after ‘Z’.
Qwen3-1.7B Figure 4 Pair B
Pair A
count:numbers
format:newline
Include exactly 13 numbers in the response.
Write each word on a new line.
count:pronouns
words:last_first
The response should include at least 5 pronouns.
The last word of each sentence must become the first word of the next sentence.
Llama-3.2-3B-Instruct Figure 11 Pair B
Pair A
words:start_verb
count:pronouns
The response must start with a verb.
The response should include at least 5 pronouns.
count:pronouns
sentence:increment
The response should include at least 5 pronouns.
Each sentence must contain exactly 5 more words than the previous one.
format:quotes
words:repeats
OLMo-2-1B-Instruct Figure 12 Pair B
Constraint 2
Include quotes within The response should not requotes within quotes, at peat any word more than 10 least 3 levels deep, alternat- times. ing between double quotes and single quotes. Pair A
Qwen3-1.7B Figure 13 Pair B
count:punctuation
count:person_names
Use every standard punctuation mark at least once, including semicolons, colons, and the interrobang (?!).
Mention at least 7 different person names in the response, from this list of person names: Emma, Liam, Sophia...
words:start_verb
custom:multiples
The response must start with a verb.
Count from 10 to 50 but only print multiples of 7.
88