RL2ML: Finite-Rollout Surrogate Objectives from Reinforcement Learning to Maximum Likelihood
arXiv:2605.30154v1 [cs.LG] 28 May 2026
Yifu Zheng University of Southern California [email protected]
Abstract Correctness-based Reinforcement Learning with Verifiable Rewards (RLVR) trains language models from binary feedback on sampled outputs, but the objective optimized in expectation and the stochastic update geometry induced by finite rollout groups are often conflated. This paper develops RL2ML, a family of finite-rollout surrogate objectives with a closed-form, exactly unbiased gradient estimator. The family continuously connects standard reinforcement learning, maximum-likelihood-like training, and beyond-maximum-likelihood objectives while preserving estimator-objective alignment under a fixed rollout budget. We introduce the grouplevel update scale to characterize how a rollout group is reweighted after its empirical success count is observed, revealing a subcritical-supercritical update-scale transition that is hidden by population-level objective notation alone. Building on this distinction, calibrated metric-gain analysis and exact variance decomposition show that the best choice of surrogate objective is determined neither by proximity to maximum likelihood nor by the population-level weight alone. Instead, it depends jointly on the evaluation metric, local sensitivity, and estimator variance. The remaining degree of freedom in the surrogate objective family can therefore be formulated as a one-dimensional optimization problem rather than treated as an unconstrained hyperparameter.
1
Introduction
For a learning task with binary verifiable feedback, let p := pθ (x) denote the probability that a model with parameter θ produces a correct output for prompt x. In fully differentiable supervised learning, maximizing log-likelihood objectives is a standard statistical principle and gives rise to classical losses such as those used in linear regression and logistic regression[1]. In the singleprompt abstraction considered here, the maximum-likelihood objective is JθML (x) = log p. RLVR can also be interpreted as changing the probability mass assigned to response trajectories, but it usually operates through sampled rollouts and binary verifier rewards rather than direct likelihood labels. This naturally raises the question studied by MaxRL[2], which shows that correctness-based RLVR optimizes only a first-order approximation to maximum likelihood and derives a compute-indexed surrogate objective by expanding maximum-likelihood objective. In training, the MaxRL objective gives larger weight to low-success prompts than ordinary reinforcement learning in a maximum-likelihood-like form. However, even though the MaxRL objective approaches the maximum-likelihood objective in the large-rollout limit, this does not automatically imply that it is the best objective for finitehorizon RLVR training. In a more realistic RL training setting, under finite rollout budgets, finite optimization horizons, and concrete validation metrics, we must ask how to allocate weight to low-success prompts more effectively, and how to design a surrogate objective on that basis. This is precisely the problem this paper attempts to address. This paper introduces RL2ML, a finite-rollout surrogate objective family that preserves the estimator-objective alignment of MaxRL while exposing a single continuous degree of freedom. For γ ≥ 0, the untruncated power-likelihood
1
gradient is
−γ
∇θ JγRL2ML (x) = pθ (x)
∇θ pθ (x).
The point γ = 0 recovers the ordinary reinforcement learning gradient, while γ = 1 recovers the maximum-likelihood gradient. γ < 1 form a subcritical regime relative to the ML boundary, whereas values above one form a supercritical regime that upweights low-success prompts beyond maximum likelihood. Along this path, RL2ML continuously connects ordinary RL, maximumlikelihood-style training, and beyond-maximum-likelihood low-success amplification. The contributions are fourfold. First, RL2ML defines a truncated power-likelihood surrogate objective and provides closed-form coefficients for an exactly unbiased estimator under a finite rollout budget. Second, the group-level update scale αK exposes a subcritical-supercritical boundary at γ = 1, clarifying how empirical success counts change the actual stochastic update. Third, calibrated local-gain analysis and exact variance decomposition show why supercritical weighting of the hardest prompts need not dominate in finite-horizon training. Fourth, the choice of γ is formulated as a one-dimensional outer optimization problem, under which γ = 1 is only an important reference point and a structural boundary, not a universal optimum. The rest of the paper is organized as follows. Section 2 introduces the setup, the MaxRL estimator, and the Bernstein representation of K-only estimators. Section 3 defines the RL2ML surrogate objective, derives the unbiased estimator, and analyzes its update-scale geometry. Section 4 develops the calibrated local-gain criterion and the variance-aware outer selection rule. Section 5 positions RL2ML relative to prior induced-objective, policy-gradient, and RLVR work, and Section 6 concludes. Appendix A gives proofs and derivations, Appendix B gives implementation details for fixed-γ estimation and γ ∗ selection, and Appendix C records the triad-family extension.
2
Preliminaries
2.1
Basic Definitions
Let the input space be X , the latent rollout space be Z, and the task distribution be x ∼ ρ. For a given prompt x, the model first samples a latent rollout z ∼ mθ (· | x) and then obtains the final answer through a deterministic decoding function y = f (z). Denote the correct answer by y ∗ (x). The corresponding correctness reward is defined as r(x, z) := If (z) = y ∗ (x), and the single-sample success probability is therefore pθ (x) := Ez∼mθ (·|x) [r(x, z)]. For a fixed prompt x, we write p := pθ (x) for brevity. Let the score function be S(x, z) := ∇θ log mθ (z | x). This score-function form is the basis of REINFORCE and policy-gradient methods[3, 4]. When P N rollouts are sampled independently, write ri := r(x, zi ), Si := S(x, zi ), and K := N i=1 ri . If K ≥ 1, define the average score over successful samples by S̄K (x) :=
N N 1 X 1 X ri Si = r(x, zi )∇θ log mθ (zi | x). K i=1 K i=1
We abbreviate S̄K (x) as S̄K . Define the success-conditioned first and second moments by µx := E[S | r = 1, x], and Σx := Cov(S | r = 1, x). Since ∇θ J ML (x) = Ez∼mθ (·|x) [∇θ log mθ (z | x) | f (z) = y ∗ (x)] by Theorem 1 of the MaxRL paper [2], µx = ∇θ log pθ (x), and ∇θ pθ (x) = pθ (x)µx . Thus, in correctness-based latent-generation models, the average score function of successful trajectories estimates the maximum-likelihood gradient direction. 2
2.2
MaxRL Estimator
For a single prompt x, the population gradients of ordinary reinforcement learning and maximum likelihood are ∇θ J RL (x) = ∇θ p, ∇θ J ML (x) = ∇θ log p =
1 ∇θ p. p
The corresponding population-level weights are wRL (p) = 1 and wML (p) = 1/p. MaxRL uses the expansion log p = −
∞ X (1 − p)k
k
k=1
,
which implies ∇θ log p =
∞ X 1 k=1
k
∇θ 1 − (1 − p)k =
∞ X 1 k=1
k
∇θ pass@k(x).
The truncated MaxRL objective is therefore JTMaxRL (x) =
T X 1 k=1
k
pass@k(x).
MaxRL uses the estimator 1 PN r S , i i MaxRL ĝN (x) = K i=1
0,
if K ≥ 1, if K = 0.
It is unbiased for the truncated MaxRL objective with T = N . This exact alignment between a finite-rollout objective and a finite-rollout gradient estimator is the structural property preserved by RL2ML.
2.3
Bernstein Representation of K-only Estimators
The closed-form estimator of RL2ML relies on a general fact: any estimator whose scalar coefficient depends on a rollout group only through the success count K realizes a Bernstein-polynomial population weight. Related Bernstein-type representations for binary-reward reasoning RL are also discussed in [5]. Theorem 2.1 (Bernstein representation of K-only estimators). Consider the estimator ĝf (x) = f (K)
N X
ri Si , where K =
i=1
N X
ri .
i=1
Let βm := N f (m + 1), for m = 0, . . . , N − 1, and denote the Bernstein basis by !
Bm,N −1 (p) :=
N −1 m p (1 − p)N −1−m . m
3
Then, for any fixed prompt x, E[ĝf (x) | x] =
N −1 X
!
βm Bm,N −1 (p) ∇θ p = wf (p)∇θ J RL (x),
m=0
P
−1 where wf (p) = N m=0 βm Bm,N −1 (p) is the population-level weighting function. Hence the population-level weight of any K-only estimator is a Bernstein polynomial of degree N − 1.
The proof is given in Appendix A.1. The theorem separates two notions that are often implicitly merged. The population-level weight wf (p) specifies the expected gradient of the surrogate objective, whereas the finite-rollout update observed by the optimizer is governed by the empirical success count. Define the group-level update scale αK :=
K βK−1 , for K ≥ 1. N
Then, conditional on K ≥ 1, ĝf (x) =
N βK−1 X ri Si = αK S̄K . N i=1
The direction S̄K is the average score function of successful trajectories; the scalar αK determines how strongly that direction is attenuated or amplified after the rollout group is observed. Moreover, conditional on K = k ≥ 1, E[ĝf (x) | x, K = k] = αk µx , Cov(ĝf (x) | x, K = k) =
αk2 Σx . k
Thus αK determines the sample-level update geometry more directly than the population weight wf (p) alone.
3
RL2ML
3.1
Single-Prompt RL2ML Surrogate Objective
3.1.1
RL2ML Objective Family
Define the RL2ML objective family by JγRL2ML (x) = ϕγ (pθ (x)), where ϕ′γ (p) = p−γ . We choose the antiderivative as
ϕγ (p) =
1−γ p −1 ,
γ ̸= 1,
γ = 1.
1−γ
log p,
Consequently, γ = 0 recovers the ordinary RL population gradient and γ = 1 recovers the ML population gradient. The untruncated population gradient is −γ
∇θ JγRL2ML (x) = pθ (x)
∇θ pθ (x).
Finite-rollout training does not estimate this untruncated target directly. Following the MaxRL principle, RL2ML instead defines a rollout-budget-aligned truncated surrogate objective. 4
3.1.2
Truncated RL2ML Objective
For rollout budget N , define RL2ML Jγ,N (x) :=
N X
(γ)k−1 pass@k(x), (k − 1)! k k=1
where (γ)k−1 is the rising factorial and (γ)0 = 1. Its associated population-level weight is RL2ML wγ,N (p) :=
N −1 X
(γ)m (1 − p)m . m! m=0
Proposition 3.1 (Gradient of the truncated RL2ML objective). For every fixed prompt x, RL2ML RL2ML ∇θ Jγ,N (x) = wγ,N (pθ (x))∇θ pθ (x). RL2ML (p) approaches the power weight The proof is given in Appendix A.2. When N is large, wγ,N −γ p on the interior of (0, 1]. For finite N , it is a truncated, rollout-budget-aligned approximation
whose gradient admits an exactly unbiased estimator.
3.2 3.2.1
Exactly Unbiased Gradient Estimator Closed-Form Coefficients
RL2ML (p) in the Bernstein basis: By Theorem 2.1, it suffices to express wγ,N
RL2ML wγ,N (p) =
N −1 X
(γ,N )
βj
Bj,N −1 (p).
j=0
The corresponding coefficients admit the closed form (γ,N )
βK−1 =
Γ(N + γ) Γ(K) , for K = 1, . . . , N. Γ(N ) Γ(K + γ)
Equivalently, the group-level update scale is (γ,N )
αK
=
K (γ,N ) Γ(N + γ) Γ(K + 1) β = . N K−1 Γ(N + 1) Γ(K + γ)
The derivation is given in Appendix A.3. Theorem 3.2 (Exactly unbiased estimator for the truncated RL2ML objective). Define (γ,N )
RL2ML ĝγ,N (x) = αK
(γ,N )
S̄K , where αK
=
Γ(N + γ) Γ(K + 1) . Γ(N + 1) Γ(K + γ)
Then RL2ML RL2ML E[ĝγ,N (x) | x] = ∇θ Jγ,N (x).
The proof is given in Appendix A.4. The theorem shows that RL2ML is not merely a heuristic reweighting rule: for each fixed γ and N , it defines a finite-rollout surrogate objective and an exactly unbiased estimator for that surrogate objective.
5
3.2.2
Control Variate
The estimator can be used directly as N 1 X (γ,N ) β ri Si , for K ≥ 1, N i=1 K−1
direct ĝγ,N (x) =
with zero update when K = 0. Since E[N −1 unbiasedness while reducing variance:
P
i Si | x] = 0, subtracting N
−1 P S preserves i i
N 1 X (γ,N ) g̃γ,N (x) = β ri − 1 Si , N i=1 K−1
The sequence-level advantage for the control-variate form is (γ,N )
Ai
(γ,N )
= βK−1 ri − 1.
For γ = 1, (1,N )
βK−1 =
N N (1,N ) , and Ai = ri − 1, K K
which recovers the MaxRL control-variate advantage. The direct form and the control-variate form estimate the same population gradient in expectation, but they differ in finite-sample variance and in how all-failure groups are treated.
3.3
Update-Scale Geometry Controlled by γ
The parameter γ controls both the population-level weight and the sample-level update scale. From the closed form, (γ,N )
αK+1
(γ,N ) αK
=
K +1 . K +γ
Thus γ = 1 is a structural boundary in the observed rollout-group geometry. Proposition 3.3 (Subcritical and supercritical update-scale regimes). For fixed N , the following statements hold: (0,N )
1. If γ = 0, then αK
= K/N , which is the ordinary RL.
(γ,N )
(γ,N )
2. If 0 < γ < 1, then αK is increasing in K and satisfies K/N < αK This is the subcritical regime. (1,N )
3. If γ = 1, then αK
< 1 for 1 ≤ K < N .
≡ 1, which is the MaxRL boundary.
(γ,N )
(γ,N )
4. If γ > 1, then αK is decreasing in K and satisfies αK supercritical regime.
> 1 for all K < N . This is the
The proof is given in Appendix A.5. The terms subcritical and supercritical refer to the update scale assigned to low-success rollout groups. Ordinary RL attenuates such groups in proportion to K/N ; subcritical RL2ML still attenuates them but less severely; MaxRL removes the successcount dependence of the successful-score average; and supercritical RL2ML amplifies rare-success groups beyond the MaxRL scale. Figure 1 reports the population-level weight and the sample-level update scale in the two panels.
6
Figure 1: Population-level weights and sample-level update scales for RL2ML. The left panel shows the population-level weight wγ,N (p), while the right panel shows the sample-level update scale αK for N = 32. Values γ < 1 form the subcritical regime, γ = 1 is the MaxRL boundary, and γ > 1 enters the supercritical low-K amplification regime.
4
Finite-Horizon Selection of γ
The preceding sections define a family of rollout-budget-aligned surrogate objectives and exactly unbiased finite-rollout estimators. The remaining question is how to select the member of this family used for training. Throughout this section, γ ∗ denotes the value of γ chosen by an outer selection criterion over a candidate interval Γ = [γmin , γmax ]. A static rule such as choosing the surrogate closest to maximum likelihood is therefore insufficient. Finite-horizon RLVR is evaluated after a finite number of optimizer steps, and different choices of gamma can induce different update magnitudes and different estimator noise. This section first constructs a calibrated metric-gain criterion, then incorporates a variance penalty, and finally defines γ ∗ as the solution of the resulting one-dimensional outer optimization problem. For notational simplicity, write RL2ML wγ,N (p) := wγ,N (p) =
4.1
N −1 X
(γ)m (1 − p)m . m! m=0
Why Effective Learning-Rate Calibration Is Necessary (γ,N )
(γ,N )
Changing γ changes the coefficients βK−1 , the group-level scale αK , and the population-level weight wγ,N (p) simultaneously. If all values of γ are compared under the same base learning rate, a larger observed improvement may simply reflect a larger effective step rather than a better surrogate objective, thereby conflating objective design with learning-rate scaling. Accordingly, to compare surrogate objectives rather than update magnitudes, we normalize the expected update direction for each γ and apply the same target update length c, so that all candidates are evaluated under the same effective parameter-update length. Let C be the prompt set used for local analysis, and let the validation metric be promptseparable: VC (θ) =
X
vx (px ), where px := pθ (x).
x∈C
The function vx encodes the marginal value assigned by the actual evaluation metric to the success probability of prompt x. For example, pass@1 gives vx (p) = p, pass@k gives vx (p) = 1 − (1 − p)k , and a smoothed log-success metric gives vx (p) = log(p + τ ). 7
For analytical clarity, we first consider a prompt-separable approximation in which the gradient contributions of different prompts are treated as orthogonal. Under this approximation, write θ = (θx )x∈C and define the expected RL2ML update direction Dγ,C (θ) =
X
wγ,N (px )∇θx px ,
x∈C
and define ℓx := ∥∇θx px ∥22 . Here ℓx measures how strongly the success probability of prompt x changes under its corresponding parameter block. Set
A(γ) :=
X
ax := vx′ (px )ℓx , and bx := ℓx , ax wγ,N (px ), and B(γ) :=
x∈C
X
bx wγ,N (px )2 .
(1)
x∈C
Here A(γ) is the first-order alignment with the validation metric, while B(γ) is the squared norm of the expected update direction. Given a target update length c > 0, we define θγ+ := θ + ηγ Dγ,C (θ), where ηγ :=
c c . =p ∥Dγ,C (θ)∥2 B(γ)
This gives ∥θγ+ − θ∥2 = c, so different values of γ are compared under the same parameter-update length. The calibrated first-order metric-gain criterion is A(γ) U (γ) := p = B(γ)
P ′ x∈C vx (px ) wγ,N (px ) ℓx q . P x∈C wγ,N (px )
(2)
2ℓ x
Proposition 4.1 (Calibrated metric gain under fixed update length). Under the prompt-separable approximation introduced above, the fixed-c update satisfies VC (θγ+ ) − VC (θ) = c U (γ) + O(c2 ). The derivation is given in Appendix A.6. Here O(c2 ) denotes a standard local Taylor remainder bounded by a constant times c2 as the target update length size c → 0. This proposition gives the first reason why γ cannot be selected only by proximity to maximum likelihood. Even if increasing γ gives more population-level weight to low-p prompts, the calibrated metric gain also depends jointly on the metric marginal value vx′ (px ), the population-level weight wγ,N (px ), and the prompt sensitivity coefficient ℓx . Therefore, prompts with large surrogate weight do not necessarily dominate the finite-horizon improvement: if they have weak prompt sensitivity or low marginal value under the evaluation metric, increasing γ can reduce the gain per unit effective update length. Corollary 4.2 (A sufficient condition for improving over the MaxRL). Assume A(1) > 0 and B(1) > 0. If A′ (1) B ′ (1) >2 , B(1) A(1) then U ′ (1) < 0. Hence there exists ε > 0 such that U (1 − δ) > U (1) for every 0 < δ < ε. The proof is given in Appendix A.6. This criterion formalizes when the MaxRL point can be locally improved by moving into the subcritical region under a given parameter state and evaluation metric. Figure 2 shows a stylized example in which the uncalibrated first-order metric gain A(γ) favors larger γ, while the calibrated first-order metric-gain criterion U (γ) can have an interior maximum. 8
Figure 2: Stylized finite-horizon selection curves. This example illustrates that the best γ is metric-dependent and calibration-dependent, rather than a static boundary point closest to 1/p.
4.2
Estimator Variance and the Outer Objective
Calibrated metric gain describes the expected first-order improvement. However, an update direction that has high expected gain can still be undesirable if its finite-rollout estimator is too noisy. γ ∗ estimator variance. Define (γ,N )
aK (γ) := αK
I{K ≥ 1}, and ĝγ,N (x) = aK (γ)S̄K ,
with the convention that aK (γ)2 /K := 0 when K = 0. Theorem 4.3 (Conditional variance decomposition). For any fixed prompt x and fixed γ, N , "
Cov(ĝγ,N (x) | x) = Var(aK (γ) | x)µx µ⊤ x +E
#
aK (γ)2 x Σx . K
Consequently, h
E ∥ĝγ,N (x) − E[ĝγ,N (x) | x]∥22 x
i
"
= Var(aK (γ) | x)∥µx ∥22 + E
#
aK (γ)2 x tr(Σx ). K
The proof is given in Appendix A.8. The first term is count variance caused by the random success count; the second is within-success variance caused by the randomness of successful trajectory score function. Both terms scale quadratically with aK (γ). In particular, the supercritical regime γ > 1 carries an explicit finite-rollout noise cost because it assigns larger update scales to low-K rollout groups. Aggregating over the calibration prompt set C gives the variance proxy R(γ): R(γ) :=
X
"
"
Var(aK (γ) | x)∥µx ∥22 + E
x∈C
9
#
#
aK (γ)2 x tr(Σx ) . K
(3)
Figure 3: Noise curves induced by the exact variance decomposition for N = 32 under the normalization ∥µx ∥2 = tr(Σx ) = 1. As γ increases, both count variance and within-success variance increase, with faster growth at smaller success probabilities. For theoretical evaluation, K | x ∼ Binom(N, px ) can be used in Eq. (3). In practical controllers, px and the norm factors are replaced by smoothed estimates and low-cost proxies. Figure 3 visualizes the variance growth implied by Theorem 4.3 in a normalized setting. After effective learning-rate calibration, the variance-aware selection problem is h
i
γλ∗ = arg max U (γ) − λvar R(γ)1/2 , where Γ = [γmin , γmax ] and λvar ≥ 0.
(4)
γ∈Γ
Equation (4) makes the outer choice explicit: U (γ) measures expected metric improvement per unit effective update scale, while R(γ)1/2 measures stochastic update noise. The coefficient λvar reflects the tolerance for estimator variance. The derivative, first-order condition, and a Newton update for the unpenalized part U (γ) are given once in Appendix A.7. This objective preserves the main conclusion of the paper. The selection of γ is not equivalent to choosing the population weight closest to 1/p, and it is not determined by static hard-prompt emphasis. It is a finite-horizon optimization problem governed by the evaluation metric V , the current prompt set C, local sensitivity ℓx , and estimator variance R.
5
Related Work
Induced objectives for binary-reward reasoning RL. Davis and Recht provide a broader induced-objective view of binary-reward LLM reinforcement learning, showing that several algorithms can be interpreted as optimizing monotone transforms of prompt-level success probability[5]. RL2ML can be viewed as a specialization of this perspective: it studies the canonical power-likelihood path with population derivative p−γ , connects this path to finite-rollout surrogate objectives, and derives a closed-form K-only unbiased estimator with a group-level update-scale transition at γ = 1. Unlike Davis and Recht, RL2ML specifically organizes this family as a continuous path from RL to ML and beyond ML, and derives the associated coefficients, subcritical-supercritical transition, and variance-aware selection rule for γ. MaxRL is more directly adjacent because it derives a finite-compute surrogate for maximum-likelihood-style 10
optimization in correctness-based RLVR[2]. RL2ML keeps the likelihood-style interpretation but does not assume that the objective closest to log p is always best under a finite training horizon. Instead, it separates the population-level weighting function from effective update scale and estimator variance, which makes maximum likelihood a reference point rather than a universal selection rule. Policy-gradient estimators and group-based LLM RL. Classical policy-gradient methods estimate gradients by likelihood-ratio scores, with REINFORCE and the policy-gradient theorem providing the standard foundations[3, 4]. PPO became a common optimizer for RLHF-style language-model post-training because it stabilizes policy updates through clipping and related trust-region heuristics[6, 7]. More recent LLM post-training work revisits simpler REINFORCE-style estimators and group-based relative advantages, including RLOO-style methods and GRPO[8, 9, 10]. RL2ML is orthogonal to the choice of outer optimizer: it specifies the finite-rollout surrogate objective and the success-count-dependent scalar coefficient, whereas PPO, RLOO, and GRPO specify how sampled sequence advantages are converted into policy updates.
6
Conclusion
This paper introduced RL2ML as a finite-rollout surrogate-objective framework for correctnessbased RLVR. Relative to prior induced-objective analyses and MaxRL, RL2ML preserves the estimator-objective alignment while exposing a continuous objective path from ordinary reinforcement learning, through the maximum-likelihood boundary, to supercritical objectives that reweight low-success prompts beyond maximum likelihood. A central contribution of this work is the separation between population-level weighting and sample-level update geometry. The Bernstein representation characterizes what a K-only estimator optimizes in expectation, but the group-level update scale αK determines how a realized rollout group is actually amplified or attenuated after its success count is observed. This distinction exposes the subcritical–supercritical transition: below the maximum-likelihood boundary, low-success groups remain attenuated relative to MaxRL; above it, they are amplified beyond the MaxRL scale. Thus, the effect of changing the surrogate objective is not fully captured by the static population-level weighting function alone. The finite-horizon analysis further shows that proximity to maximum likelihood is not a sufficient principle for objective selection. After effective learning-rate calibration, the relevant comparison is the metric gain per unit update scale, not the uncalibrated size of the expected gradient. The calibrated metric-gain criterion makes the dependence on the validation metric and local sensitivity explicit, while the exact variance decomposition shows that supercritical low-success amplification carries a quantifiable finite-rollout noise cost. Consequently, neither ordinary reinforcement learning nor direct maximum-likelihood approximation is generally optimal across metrics and prompt distributions. The resulting view treats the remaining degree of freedom in RL2ML as an outer-control variable rather than an unconstrained hyperparameter. In the oracle setting, it defines a onedimensional variance-aware optimization problem balancing calibrated metric gain against estimator noise. In practical Verl-style training, the same structure can be approximated from recent on-policy success counts or from a calibration shard, then implemented by updating the grouplevel coefficient table together with effective learning-rate calibration. This provides a principled route for selecting a finite-rollout training surrogate for a given metric and parameter state, while keeping the underlying RLVR rollout and verification pipeline unchanged.
11
Appendix A
Proofs and Derivations
A.1
Proof of Theorem 2.1
Restatement of Theorem 2.1.
For a fixed prompt x, consider
ĝf (x) = f (K)
N X
ri Si , where K =
i=1
N X
ri .
i=1
With βm = N f (m + 1) and Bm,N −1 (p) = Nm−1 pm (1 − p)N −1−m , N −1 X
E[ĝf (x) | x] =
!
βm Bm,N −1 (p) ∇θ p.
m=0
Proof.
By exchangeability, E[ĝf (x) | x] = N E[f (K)r1 S1 | x].
P
Let M−1 := j̸=1 rj . Since r1 S1 = 0 on r1 = 0, the event contributing to the expectation has K = 1 + M−1 . The random variable M−1 is conditionally independent of r1 S1 given x, so E[f (K)r1 S1 | x] = E[f (1 + M−1 ) | x]E[r1 S1 | x]. Furthermore, E[r1 S1 | x] =
X
mθ (z | x)r(x, z)∇θ log mθ (z | x)
z
=
X
r(x, z)∇θ mθ (z | x) = ∇θ p.
z
Since M−1 ∼ Binom(N − 1, p),
!
N −1 X
N −1 m p (1 − p)N −1−m . f (m + 1) E[f (1 + M−1 ) | x] = m m=0 Multiplying by N ∇θ p and using βm = N f (m + 1) gives the claimed Bernstein representation.
A.2
Proof of Proposition 3.1
Restatement of Proposition 3.1.
For
RL2ML Jγ,N (x) =
N X
(γ)k−1 pass@k(x), (k − 1)! k k=1
we have RL2ML RL2ML ∇θ Jγ,N (x) = wγ,N (pθ (x))∇θ pθ (x).
Proof.
For fixed x, write p = pθ (x). Since pass@k(x) = 1 − (1 − p)k , ∇θ pass@k(x) = k(1 − p)k−1 ∇θ p.
Substitution gives RL2ML ∇θ Jγ,N (x) =
=
N X
(γ)k−1 k(1 − p)k−1 ∇θ p (k − 1)! k k=1 N −1 X
!
(γ)m (1 − p)m ∇θ p. m! m=0
RL2ML (p). The term in parentheses is wγ,N
12
A.3
Bernstein Expansion and Closed-Form Coefficients
This subsection combines the Bernstein-basis conversion and the coefficient simplification. For m = 0, . . . , N − 1, (1 − p)
m
N −1−j N −1−m X m = Bj,N −1 (p). N −1 j=0 m
To verify the identity, expand the right-hand side and use N −1−j m
!
!
N −1 j
!
N −1 m
=
!
N −1−m . j
The remaining sum is the binomial expansion of (p + (1 − p))N −1−m after factoring out (1 − p)m . Using this identity in RL2ML wγ,N (p) =
N −1 X
(γ)m (1 − p)m m! m=0
and grouping terms by Bj,N −1 (p) yields (γ,N ) βj =
NX −1−j m=0
(γ)m N −1−j m . N −1 m! m
Set j = K − 1. Then (γ,N ) βK−1 =
Using
−K (γ)m N m . m! Nm−1 m=0
NX −K
A m m = (−A)m /((−1) m!) for nonnegative integer A gives
(γ,N )
βK−1 = 2 F1 −(N − K), γ; −(N − 1); 1 . By the Chu–Vandermonde identity[11, 12], 2 F1 (−n, b; c; 1) =
(c − b)n , (c)n
so with n = N − K, b = γ, and c = −(N − 1), (γ,N )
βK−1 =
(−(N − 1) − γ)N −K . (−(N − 1))N −K
The identity (−a)n = (−1)n Γ(a + 1)/Γ(a − n + 1) gives (γ,N )
βK−1 =
Γ(N + γ) Γ(K) . Γ(N ) Γ(K + γ)
(1,N )
At γ = 1, this reduces to βK−1 = N/K, recovering the MaxRL coefficient.
A.4
Proof of Theorem 3.2
Restatement of Theorem 3.2.
Define
(γ,N )
RL2ML ĝγ,N (x) = αK
(γ,N )
S̄K , where αK
=
Γ(N + γ) Γ(K + 1) . Γ(N + 1) Γ(K + γ)
Then RL2ML RL2ML E[ĝγ,N (x) | x] = ∇θ Jγ,N (x).
13
Proof.
For K ≥ 1, N (γ,N ) 1 X
RL2ML ĝγ,N (x) = αK
K i=1
(γ,N )
ri Si =
N βK−1 X ri Si . N i=1
This is a K-only estimator of the form in Theorem 2.1. Therefore,
N −1 X
RL2ML E[ĝγ,N (x) | x] =
(γ,N )
βj
Bj,N −1 (p) ∇θ p.
j=0 RL2ML (p), and Proposition 3.1 gives the By Appendix A.3, the coefficients are exactly those of wγ,N desired identity.
A.5
Proof of Proposition 3.3 (0,N )
(γ,N )
Restatement of Proposition 3.3. For fixed N , αK = K/N at γ = 0; αK is increasing (γ,N ) (1,N ) in K and satisfies K/N < αK < 1 for 0 < γ < 1 and 1 ≤ K < N ; αK ≡ 1 at the MaxRL (γ,N ) (γ,N ) boundary; and αK is decreasing in K and satisfies αK > 1 for γ > 1 and K < N . Proof.
(γ,N )
From the closed form of αK (γ,N )
αK+1
(γ,N ) αK
=
,
Γ(K + 2)Γ(K + γ) K +1 = . Γ(K + 1)Γ(K + 1 + γ) K +γ (γ,N )
(γ,N )
If 0 ≤ γ < 1, this ratio is larger than one and αN = 1, so αK is increasing in K and does (0,N ) (1,N ) not exceed one. The case γ = 0 gives αK = K/N . If γ = 1, the ratio is one and αK ≡ 1. If (γ,N ) (γ,N ) γ > 1, the ratio is smaller than one and αN = 1, so αK > 1 for K < N .
A.6
Proof of Proposition 4.1 and Corollary 4.2
Restatement of Proposition 4.1. Under the prompt-separable local model, the matchednorm update θγ+ = θ + ηγ Dγ,C (θ) with ηγ = c/∥Dγ,C (θ)∥2 satisfies VC (θγ+ ) − VC (θ) = c U (γ) + O(c2 ). Restatement of Corollary 4.2. If A(1) > 0, B(1) > 0, and B ′ (1)/B(1) > 2A′ (1)/A(1), then U ′ (1) < 0 and there exists ε > 0 such that U (1 − δ) > U (1) for every 0 < δ < ε. Proof.
Under the prompt-separable local model, ∇θx VC (θ) = vx′ (px )∇θx px .
Orthogonality of different prompt blocks gives ⟨∇VC (θ), Dγ,C (θ)⟩ =
X
vx′ (px )wγ,N (px )∥∇θx px ∥22 = A(γ),
x∈C
and ∥Dγ,C (θ)∥22 =
X
wγ,N (px )2 ∥∇θx px ∥22 = B(γ).
x∈C
A first-order Taylor expansion at θ gives VC (θ + ηγ Dγ,C ) − VC (θ) = ηγ A(γ) + O(ηγ2 B(γ)). 14
p
p
With ηγ = c/ B(γ), the leading term is cA(γ)/ B(γ) = cU (γ) and the remainder is O(c2 ). For the corollary, differentiate U (γ) = A(γ)B(γ)−1/2 : U ′ (γ) =
2A′ (γ)B(γ) − A(γ)B ′ (γ) . 2B(γ)3/2
Under the stated condition at γ = 1, the numerator is negative, so U ′ (1) < 0. Continuity implies U (1 − δ) > U (1) for all sufficiently small δ > 0.
A.7
Derivatives of the Calibrated Objective
For γ > 0,
∂γ (γ)m = (γ)m ψ(γ + m) − ψ(γ) , where ψ is the digamma function. Hence ẇγ,N (p) := ∂γ wγ,N (p) =
N −1 X
(γ)m ψ(γ + m) − ψ(γ) (1 − p)m . m! m=1
Let ψ1 be the trigamma function. Then ẅγ,N (p) := ∂γγ wγ,N (p) =
i 2 (γ)m h ψ(γ + m) − ψ(γ) + ψ1 (γ + m) − ψ1 (γ) (1 − p)m . m! m=1
N −1 X
From Eq. (1), A′ (γ) =
X
ax ẇγ,N (px ),
x∈C
A′′ (γ) =
X
ax ẅγ,N (px ),
x∈C
B ′ (γ) = 2
X
bx wγ,N (px )ẇγ,N (px ),
x∈C
B ′′ (γ) = 2
X
h
i
bx ẇγ,N (px )2 + wγ,N (px )ẅγ,N (px ) .
x∈C
By Eq. (2), U ′ (γ) =
2A′ (γ)B(γ) − A(γ)B ′ (γ) . 2B(γ)3/2
Stationary points of U satisfy F (γ) = 0, where F (γ) := 2A′ (γ)B(γ) − A(γ)B ′ (γ). A projected Newton step is
γ ← Π[γmin ,γmax ]
F (γ) . γ− ′ F (γ)
with F ′ (γ) = 2A′′ (γ)B(γ) + A′ (γ)B ′ (γ) − A(γ)B ′′ (γ). If γ = 0 is included in the candidate interval, it is usually evaluated as a boundary point, while Newton steps are run with γmin > 0. Since U need not be globally concave, practical selection should compare boundary points, a grid of candidates, and stationary points found from multiple initializations. 15
A.8
Proof of Theorem 4.3 (γ,N )
Restatement of Theorem 4.3.
For aK (γ) = αK
I{K ≥ 1} and ĝγ,N (x) = aK (γ)S̄K , "
Cov(ĝγ,N (x) | x) = Var(aK (γ) | x)µx µ⊤ x +E
#
aK (γ)2 x Σx . K
Proof. Conditional on K = k ≥ 1, the k successful samples are conditionally i.i.d. from the success-conditioned rollout distribution. Therefore, E[S̄K | x, K = k] = µx , and Cov(S̄K | x, K = k) =
Σx . k
Thus E[ĝγ,N | x, K = k] = ak (γ)µx , Cov(ĝγ,N | x, K = k) =
ak (γ)2 Σx . k
The total covariance identity gives Cov(ĝγ,N | x) = Cov(E[ĝγ,N | x, K] | x) + E[Cov(ĝγ,N | x, K) | x]. 2 The first term is Var(aK (γ) | x)µx µ⊤ x , and the second is E[aK (γ) /K | x]Σx . Taking the trace gives the squared-norm expression in Theorem 4.3.
B
Fixed-Gamma Estimation and Practical γ ∗ Selection in Verl
This appendix consolidates the implementation details that are not needed in the main theoretical development. The purpose is to specify where the fixed-γ RL2ML estimator enters a Verl-style on-policy RLVR loop and how the practical γ ∗ controller can collect success counts from recent rollouts or from a calibration shard.
B.1
Fixed-γ Advantage Construction
In Verl-style on-policy RLVR training[13, 14], RL2ML is implemented at the group-level advantageestimator layer. The reward function still returns binary correctness, and the rollout pipeline is (γ,N ) unchanged. For a fixed γ, the only additional object is the coefficient table {βK−1 }N K=1 indexed by the prompt-level success count. Algorithm 1 Fixed-γ RL2ML advantage construction in Verl (γ,N )
Require: Current model parameters θt , rollout budget N , fixed γ, coefficient table {βK−1 }N K=1 1: for each prompt group x in the current on-policy batch do 2: Sample N rollouts z1 , . . . , zN ∼ mθt (· | x) 3: Compute binary rewards ri = I{f (zi ) = y ∗ (x)} P 4: Compute Kx = N i=1 ri 5: if using the direct estimator then (γ,N ) 6: Set Ai = 0 for all i if Kx = 0; otherwise set Ai = βKx −1 ri 7: else (γ,N ) (γ,N ) 8: Use the score-baseline control variate and set Ai = βKx −1 ri − 1, with βKx −1 ri = 0 when Kx = 0 9: end if 10: Broadcast Ai to response tokens using the response mask 11: end for 12: Apply the standard on-policy policy loss with the constructed token-level advantages
16
Listing 1: RL2ML coefficient table and sequence-level advantages import numpy as np from scipy.special import gammaln
def precompute_beta_table(gamma: float, N: int) -> np.ndarray: """Return beta[K - 1] for K = 1,...,N.""" K = np.arange(1, N + 1, dtype=np.float64) log_beta = ( gammaln(N + gamma) - gammaln(N) + gammaln(K) - gammaln(K + gamma) ) return np.exp(log_beta)
def compute_group_advantages(group_rewards: np.ndarray, beta_table: np.ndarray, use_control_variate: bool = True) -> np.ndarray: """group_rewards has shape [num_prompts, N] and values in {0,1}.""" num_prompts, N = group_rewards.shape seq_adv = np.zeros_like(group_rewards, dtype=np.float32) for b in range(num_prompts): r = group_rewards[b].astype(np.float64) K = int(r.sum()) if use_control_variate: if K == 0: seq_adv[b, :] = -1.0 else: seq_adv[b, :] = beta_table[K - 1] * r - 1.0 else: if K == 0: seq_adv[b, :] = 0.0 else: seq_adv[b, :] = beta_table[K - 1] * r return seq_adv
def broadcast_to_tokens(seq_adv: np.ndarray, response_mask: np.ndarray) -> np.ndarray: """seq_adv: [B, N], response_mask: [B, N, T].""" return seq_adv[:, :, None] * response_mask
B.2
Collecting {Kx } from an On-Policy Batch or Calibration Shard
The controller requires prompt-level success counts under the current policy. For prompt x, this statistic is Kx :=
N X
rx,i ,
i=1
where the N responses are the rollout group generated for the same prompt. In a recent on-policy batch, Kx should be computed after the verifier has produced binary sequence-level rewards and before response-level examples are flattened irreversibly. If responses for the same prompt are contiguous, the reward vector can be reshaped to [−1, N ]. If the training recipe shuffles responses, a prompt identifier should be retained and used for grouping. A calibration shard uses the same statistic but collects it without optimizer steps. The current policy is rolled out on a fixed set of prompts, the verifier assigns binary rewards, and the resulting success counts are used only to evaluate the one-dimensional outer objective. This avoids comparing different training trajectories and keeps the controller tied to the current policy state. Listing 2: Collecting prompt-level success counts from Verl-style tensors import numpy as np from collections import defaultdict
def collect_K_contiguous(sequence_rewards: np.ndarray, N: int) -> np.ndarray:
17
"""Use when each prompt has exactly N contiguous responses.""" rewards = np.asarray(sequence_rewards, dtype=np.float64) assert rewards.ndim == 1 assert rewards.size % N == 0 group_rewards = rewards.reshape(-1, N) return group_rewards.sum(axis=1).astype(np.int64)
def collect_K_by_prompt_id(sequence_rewards: np.ndarray, prompt_ids: np.ndarray, N: int) -> np.ndarray: """Use when responses may not be contiguous but prompt ids are retained.""" bucket = defaultdict(list) for pid, reward in zip(prompt_ids, sequence_rewards): bucket[int(pid)].append(float(reward)) K_values = [] for pid, rewards in bucket.items(): if len(rewards) != N: raise ValueError(f"prompt {pid} has {len(rewards)} responses, expected {N}") K_values.append(int(np.sum(rewards))) return np.asarray(K_values, dtype=np.int64)
def collect_K_from_calibration_shard(policy, prompts, reward_fn, N: int) -> np.ndarray: """Sketch: run the current policy on a fixed shard without optimizer steps.""" K_values = [] for prompt in prompts: responses = policy.generate(prompt, n=N) rewards = np.asarray([reward_fn(prompt, y) for y in responses], dtype=np.float64) K_values.append(int(rewards.sum())) return np.asarray(K_values, dtype=np.int64)
B.3
Practical Determination of γ ∗ in Verl
Given success counts Kx , use a smoothed success-probability estimate p̂x :=
Kx + a , N +a+b
with small Beta-prior parameters such as a = b = 1. The metric marginal vx′ (p̂x ) is chosen from the validation metric: pass@1 gives 1, pass@k gives k(1 − p̂x )k−1 , and a smoothed log-success metric gives (p̂x + τ )−1 . The local sensitivity scalar ℓx can be approximated by p̂x (1 − p̂x ), by an exponential moving average of per-prompt policy-gradient norms, or by a task-specific reliability weight. The first choice is crude but stable and requires only Kx . The practical controller evaluates U (γ) and optionally the plug-in variance proxy R(γ) on a one-dimensional grid, refines the best candidates using a safeguarded line search, and applies the selected γ ∗ to construct the RL2ML coefficient table for the next training window. The effective learning rate should be calibrated using the token-level advantage RMS v u u 1 ARMSγ := t PR
Ti R X X
i=1 Ti i=1 t=1
(γ) 2
Ai,t
,
where R is the set of response sequences used for RMS estimation, Ti counts response tokens only, and εrms > 0 is a numerical stabilizer. Relative to a reference γref , base ηγbase = ηref
EMA[ARMSγref ] . EMA[ARMSγ ] + εrms
For offline comparison, the EMA can be replaced by the average on a calibration shard. For online adaptation, updating γ less frequently than the policy parameters and clipping the learning-rate multiplier reduce sensitivity to count noise.
18
Listing 3: Online gamma selection from recent success counts import numpy as np from scipy.special import psi, polygamma
def weight_and_derivs(gamma: float, N: int, p: np.ndarray): one_minus_p = 1.0 - p w = np.ones_like(p, dtype=np.float64) dw = np.zeros_like(p, dtype=np.float64) ddw = np.zeros_like(p, dtype=np.float64) coeff = 1.0 for m in range(1, N): coeff *= (gamma + m - 1.0) / m delta_psi = psi(gamma + m) - psi(gamma) delta_tri = polygamma(1, gamma + m) - polygamma(1, gamma) basis = one_minus_p ** m w += coeff * basis dw += coeff * delta_psi * basis ddw += coeff * (delta_psi**2 + delta_tri) * basis return w, dw, ddw
def gamma_objective(gamma, N, p_hat, ell_hat, vprime, eps=1e-8): w, dw, ddw = weight_and_derivs(gamma, N, p_hat) A = np.sum(vprime * w * ell_hat) B = np.sum((w**2) * ell_hat) + eps Ap = np.sum(vprime * dw * ell_hat) Bp = 2.0 * np.sum(w * dw * ell_hat) App = np.sum(vprime * ddw * ell_hat) Bpp = 2.0 * np.sum((dw**2 + w * ddw) * ell_hat) U = A / np.sqrt(B) F = 2.0 * Ap * B - A * Bp Fp = 2.0 * App * B + Ap * Bp - A * Bpp return U, F, Fp
def metric_marginal(p_hat: np.ndarray, mode: str = "pass1", pass_k: int = 1, tau: float = 0.05) -> np.ndarray: if mode == "pass1": return np.ones_like(p_hat) if mode == "passk": return pass_k * (1.0 - p_hat) ** (pass_k - 1) if mode == "logp": return 1.0 / (p_hat + tau) raise ValueError(f"unknown metric mode: {mode}")
def select_gamma_from_counts(K_counts: np.ndarray, N: int, gamma_init: float = 0.8, gamma_min: float = 1e-3, gamma_max: float = 1.5, a: float = 1.0, b: float = 1.0, metric_mode: str = "pass1", pass_k: int = 1, iters: int = 8) -> float: p_hat = (K_counts.astype(np.float64) + a) / (N + a + b) ell_hat = p_hat * (1.0 - p_hat) vprime = metric_marginal(p_hat, mode=metric_mode, pass_k=pass_k) grid = np.linspace(gamma_min, gamma_max, 41) values = [gamma_objective(g, N, p_hat, ell_hat, vprime)[0] for g in grid] gamma = float(grid[int(np.argmax(values))]) gamma = float(np.clip(0.5 * gamma + 0.5 * gamma_init, gamma_min, gamma_max)) for _ in range(iters): _, F, Fp = gamma_objective(gamma, N, p_hat, ell_hat, vprime) if abs(Fp) < 1e-10: break
19
step = F / Fp candidate = float(np.clip(gamma - step, gamma_min, gamma_max)) if not np.isfinite(candidate): break gamma = candidate return gamma
C
Triad Family, the Role of M , and Frontier Theory
The main text fixes M = N to keep RL2ML a one-parameter finite-rollout family. A more general triad family introduces a truncation order M ≤ N in addition to γ and N . This extension is useful for understanding fidelity–stability tradeoffs, but it is not needed for the main estimator or for the one-dimensional γ selection rule.
C.1
General Triad Family
Define Jγ,M (x) =
M X
(γ)k−1 pass@k(x), for M ≤ N. (k − 1)! k k=1
The corresponding exactly unbiased estimator has the form (γ,M,N )
ĝγ,M,N (x) = αK
S̄K ,
where K (γ,M,N ) αK = N
min(M −1,N −K)
X
m=0
−K (γ)m N m . m! Nm−1
Thus M changes the tail depth of the finite-compute surrogate objective, not the basic estimator class.
C.2
Why M Moves to the Boundary Without a Penalty
For fixed γ > 0, wγ,M +1 (p) − wγ,M (p) =
(γ)M (1 − p)M > 0. M!
Therefore, if the outer criterion rewards larger static weights without any stability or optimization cost, the optimal M is pushed to the boundary M = N . Values M < N become meaningful only when fidelity is traded against stability, tail amplification, or finite-horizon optimization constraints.
C.3
Frontier Theory
The infinite-order target is p−γ , and the truncation tail is Rγ,M (p) := p−γ − wγ,M (p) =
∞ X (γ)m m=M
m!
(1 − p)m .
For large m, (γ)m mγ−1 ∼ , and (1 − p)m ≈ e−pm . m! Γ(γ) 20
This gives the approximation Rγ,M (p) ≈ Q(γ, pM ), p−γ where Q is the regularized upper incomplete gamma function. Requiring relative truncation error at most δ for p ≥ pmin suggests &
'
Q−1 (γ, δ) . Mneed (γ; pmin , δ) := pmin A low-K amplification cap Amax gives (γ,M,N )
Mcap (γ; Amax , N ) := max{M ≤ N : α1
≤ Amax }.
Using the approximation (γ,M,N )
α1
≈
Mγ , Γ(γ + 1)N
one obtains Mcap (γ; Amax , N ) ≈
j
Γ(γ + 1)Amax N
1/γ k
.
The feasible window is therefore Mneed (γ; pmin , δ) ≤ M ≤ Mcap (γ; Amax , N ), with M ≤ N. Figure 4 shows a representative frontier for N = 32.
Figure 4: A representative triad-family frontier for N = 32. The fidelity requirement determines Mneed , while the low-K stability requirement determines Mcap . The truncation order M is useful as an independent degree of freedom only when the feasible interval is nonempty.
References [1] Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. MIT Press, 2012. 21
[2] Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, Daman Arora, Yiding Jiang, Jeff Schneider, Ruslan Salakhutdinov, Haiwen Feng, and Andrea Zanette. Maximum likelihood reinforcement learning. arXiv preprint arXiv:2602.02710, 2026. [3] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3–4):229–256, 1992. [4] Richard S. Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems, volume 12, pages 1057–1063, 1999. [5] Damek Davis and Benjamin Recht. What is the objective of reasoning with reinforcement learning? arXiv preprint arXiv:2510.13651, 2025. [6] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [7] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744, 2022. [8] Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-style optimization for learning from human feedback in LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, pages 12248–12267. Association for Computational Linguistics, 2024. [9] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [10] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [11] W. N. Bailey. Generalized Hypergeometric Series, volume 32 of Cambridge Tracts in Mathematics and Mathematical Physics. Cambridge University Press, 1935. [12] Wolfram Koepf. Hypergeometric Summation: An Algorithmic Approach to Summation and Special Function Identities. Vieweg, 1998. [13] verl contributors. verl: Volcano engine reinforcement learning for LLMs. https://github. com/verl-project/verl, 2025. Accessed 2026-05-28. [14] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025.
22