arXiv CSopen access

How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum

2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

machine learning, deep learning, neural networks

arXiv:2604.25907v1 [cs.LG] 28 Apr 2026

How Fast Should a Model Commit to Supervision? Training Reasoning Models on the Tsallis Loss Continuum Chu-Cheng Lin Eugene Ie Google {kitsing,eugeneie}@google.com

Abstract Adapting reasoning models to new tasks during post-training with only output-level supervision stalls under reinforcement learning from verifiable rewards (RLVR) when the initial success probability p0 is small. Using the Tsallis q-logarithm, we define a loss family JQ that interpolates between RLVR (at q=0, the exploitation pole) and the log-marginal-likelihood over latent trajectories (at q=1, the densityestimation pole). All members share the same per-example gradient direction, differing only by a scalar amplification Pθ−q that reweights each instance independently of the learning rate. This amplification is the mechanism that addresses cold-start stalling: under gradient flow, the exploitation pole requires Ω(1/p0 ) time to escape cold start, while the density-estimation pole escapes in Θ log(1/p0 ) ; intermediate q trades escape speed against noise memorization. Because Pθ is intractable, we derive two Monte Carlo estimators from the two factorizations of the gradient: Gradient-Amplified RL (GARL) samples from the prior and amplifies the RL gradient, and Posterior-Attenuated Fine-Tuning (PAFT) importance-resamples from the posterior and runs standard SFT. Both have bias O q/M Pθq+1 ; GARL has lower variance, PAFT has semantically coherent gradients. On FinQA, HotPotQA, and MuSiQue, GARL at q=0.75 substantially mitigates cold-start stalling, escaping cold start where GRPO fails entirely. In warm start, GARL at low q dominates FinQA where training is stable; on HotPotQA and MuSiQue, GARL destabilizes during training, and PAFT at q=0.75 provides stable gradients (best overall on HotPotQA at 47.9 maj@16, +14.4 over GRPO).

Introduction

Language models reason most effectively when they generate latent computational trajectories — chains of thought, proof sketches, search traces — before producing an answer [Lin et al., 2021, Merrill and Sabharwal, 2024]. Reinforcement learning from verifiable rewards (RLVR) [DeepSeekAI, 2025, Shao et al., 2024] is commonly used to learn such reasoning models, where the latent rationales are action sequences for reaching correct answers. With supervision only at the output level, RLVR can be prohibitively slow at cold start, when the initial model is too unaligned to make progress. Rao–Blackwellized rewards [Zhou et al., 2026] ensure non-zero reward (and thus non-zero gradients) for all trajectories, but as we show, this reduces gradient variance without addressing the escape-speed bottleneck. Even when RLVR succeeds, it is mode-seeking, and the reasoning capability boundary can narrow as training proceeds [Yue et al., 2025], limiting sample diversity and self-consistency decoding [Wang et al., 2023]. Instruction engineering supplies enough structure for SFT and RL to progress [Ouyang et al., 2022, DeepSeek-AI, 2025, Chu et al., 2025], but the recipe depends on task-specific prompts, and naive SFT on weak annotations risks memorizing label errors Preprint.

Gradient: ∇θ ℓq = Pθ−q ∇θ ℓ0 = Pθ1−q ∇θ ℓ1

(Proposition 4.1)

q: commitment to supervision ←− resolves noise

resolves ambiguity −→

1−q

Loss: ℓq = − logq (Pθ ) = (1−Pθ

q=0

0.25

0.5

Exploitation pole Loss: ℓ0 = 1 − Pθ (bounded) mode-seeking minimizer Gradient: ∇ℓ0 = −∇Pθ recovers REINFORCE Noise-robust; cold start Ω(p−1 0 ) (Theorem 5.1)

)/(1−q)

(Equation (3))

0.75

q=1

Density-estimation pole Loss: ℓ1 = − log Pθ (unbounded) mode-covering minimizer (proper) Gradient: ∇ℓ1 = −∇ log Pθ gradient of log-marginal-likelihood Memorizes noise; cold start Θ(log p−1 0 ) (Theorem 5.2)

Figure 1: The JQ loss family is a continuum between exploitation (q = 0) and density estimation (q = 1) losses (poles at either end of the axis below); correspondingly, commitment is the induced gradient amplification (Pθ−q ; top arrow). High q resolves ambiguity (fast cold-start escape) but also memorizes noise; low q resolves noise (robust filtering) but cannot escape cold start. p0 denotes initial success probability; convergence results assume bounded score (Section 5).

[Zhang and Sabuncu, 2018]. The two failure modes — cold-start stagnation and noise memorization — pull in opposite directions, and a unifying theoretical account has been lacking. We provide such an account, built around a per-instance gradient amplification that directly addresses the cold-start stalling problem. Let Pθ = pθ (y∗ | x∗ ) denote the model’s conditional success probability. We show that exploitation and density estimation behaviors arise as two endpoints (or poles) of a one-parameter loss continuum JQ derived from this quantity, under the Tsallis qlogarithm [Tsallis, 1988]: the exploitation pole J0 = E [1 − Pθ ] (maximization of expected accuracy, equivalent to RLVR under exact-match reward1 ) and the density-estimation pole J1 = E [− log Pθ ] (maximization of log-marginal-likelihood over latent trajectories). All members share the same per-example gradient direction, differing only by a scalar amplification Pθ−q (Figure 1): q, which we denote as commitment, amplifies the pull on low-Pθ (unfamiliar) examples relative to high-Pθ (familiar) ones. Since the learning rate sets one global step size for all examples, no global learning rate can exactly reproduce this per-instance reweighting. This amplification is precisely what is absent from RLVR’s success-probability dynamics, and is the mechanism that addresses cold-start stalling. Commitment is thus the training-time analog of the inference-time exploration-exploitation tradeoff studied in RL [Lee et al., 2018, Nachum et al., 2018]: low q concentrates on what the model already knows, high q pushes toward unfamiliar supervision. High commitment (q → 1) resolves ambiguity — escaping cold start in Θ(log(1/p0 )) time (Theorem 5.2) — but memorizes noise, since the model fits the training distribution exactly, including errors [Zhang and Sabuncu, 2018]. Low commitment (q → 0) resolves noise — the bounded loss and escort tempering filter corrupted labels (Proposition C.2) — but escape slows to Ω(1/p0 ) (Theorem 5.1). Intermediate q balances this tradeoff between ambiguity resolution and noise resistance. Because Pθ is intractable, practical optimization requires Monte Carlo estimation. The gradient admits two factorizations — through the RL and FT endpoints (Figure 2) — each of which extends a classical estimator at its endpoint to the full continuum. The two resulting methods are complementary: one uses all M sampled rationales but mixes in contributions that may contradict the answer; the other approximately samples from the posterior over rationales that agree with the answer and runs standard fine-tuning on them, trading statistical efficiency for semantically coherent gradients. Both have the same bias; the choice is dictated by the training regime. 1 Following Zhou et al. [2026], our framework assumes exact-match supervision R(ŷ, y∗ ) = I(ŷ = y∗ ), which enables the Rao–Blackwellized reward wm = pθ (y∗ | x∗ , z). Gold input-output pairs are readily available in supervised settings, while task-specific reward functions are not. Extending to general rewards is an interesting direction but beyond our scope.

Gradient identity (Proposition 4.1): ∇θ ℓq = Pθ−q ∇θ ℓ0 = Pθ1−q ∇θ ℓ1 via ℓ0 amplify

via ℓ1 attenuate

GARL (Section 6.1, RL view)

PAFT (Section 6.2, FT view)

Sample from prior pθ (z | x∗ ) Scale ∇θ ℓ0 by (w̄M )−q All M samples; has pathwise term

Approx. sample posterior pθ (z | x∗ , y∗ ) Scale ∇θ ℓ1 by (w̄M )1−q K coherent samples; pure SFT

same bias, different variance and dynamics

Figure 2: Two estimators from one gradient identity. The JQ gradient factors through either the RL endpoint ∇θ ℓ0 (yielding GARL) or the FT endpoint ∇θ ℓ1 (yielding PAFT). Both have the same bias O(q/M Pθq+1 ). GARL conditionally Rao–Blackwellizes PAFT (with respect to the resampling randomness): it has lower variance but mixes bad rationales into the gradient, while PAFT excludes them via posterior sampling at the cost of resampling noise. GARL recovers RB-REINFORCE (q=0) and IWAE (q=1); PAFT recovers EM (q=1). The choice is regime-dependent: GARL at large q for cold-start escape; in warm start, GARL at low q when training is stable, PAFT at q=0.75 when GARL collapses (Section 7). Overview of contributions. Figures 1 and 2 visualize the loss continuum and the gradient duality; our contributions follow from the Pθ−q amplification factor (Proposition 4.1). 1. The JQ loss family (Sections 3 to 5). JQ interpolates between a bounded, noise-robust loss at q = 0 and an unbounded, mode-covering loss at q = 1, with minimizers given 1/q by the escort distribution θj∗ ∝ αj (Theorem 3.1) — a training-time analog of inference temperature — and a dispersion penalty that encourages uniform success across training examples (Proposition B.1). All members share the same gradient direction, differing only by Pθ−q , which controls cold-start escape speed: the exploitation pole cannot escape faster than Ω(1/p0 ) (Theorem 5.1), while the density-estimation pole escapes in Θ(log(1/p0 )) (Theorem 5.2). 2. Two gradient estimators: GARL and PAFT (Section 6). The gradient admits two factorizations — via the RL endpoint (Pθ−q ∇θ ℓ0 ) and the FT endpoint (Pθ1−q ∇θ ℓ1 ) — each yielding a practical Monte Carlo estimator. Gradient-Amplified RL (GARL) samples trajectories from the prior and amplifies the RL gradient, generalizing RB-REINFORCE [q=0; Zhou et al., 2026] and the IWAE gradient estimator [q=1; Burda et al., 2015]. PosteriorAttenuated Fine-Tuning (PAFT) approximately samples from the posterior pθ (z | x∗ , y∗ ) over rationales that agree with the answer and runs standard fine-tuning on them, generalizing the E-step of EM [q=1; Dempster et al., 1977, Phan et al., 2023]. Both have the same bias O(q/M Pθq+1 ); GARL has lower variance, PAFT produces semantically coherent gradients. GARL is essential at cold start (posterior sampling yields no trajectories); in warm start, GARL at low q works when training is stable (FinQA), but destabilizes on HotPotQA and MuSiQue. PAFT does not collapse on any benchmark we tested, at the cost of slower per-step learning (Section 7). 3. Empirical validation (Section 7). On three reasoning benchmarks (FinQA, HotPotQA, MuSiQue) with strict (exact-match) training rewards, GARL at intermediate q escapes cold start where GRPO fails entirely. At warm start, the best stable method at each benchmark improves maj@16 over GRPO by +6.6 to +14.4 points: GARL at q=0.25 leads on FinQA (38.7 vs. 26.9) where training is stable; PAFT at q=0.75 is best on HotPotQA (47.9 vs. 33.5) where GARL collapses at all tested q, and on MuSiQue (22.4 vs. 15.8) where GARL’s higher peak does not survive training.

Setup and Background

We consider supervised conditional generation with latent reasoning trajectories. Let Θ ⊆ Rd be the parameter space of an autoregressive language model pθ with alphabet Σ. Inputs come from a 3

task distribution we do not model. We train on a supervised dataset D of input-output pairs (x∗ , y∗ ), where x∗ ∈ X ⊆ Σ∗ and y∗ ∈ Y ⊆ Σ∗ . Generative story. Given input x, the model samples an unannotated latent rationale z ∈ Z ⊆ Σ∗ from pθ (· | x), then generates an output ŷ ∼ pθ (· | x, z). This P defines the joint pθ (z, y | x) = pθ (z | x) pθ (y | x, z) and the induced marginal pθ (y | x) = z∈Z pθ (z, y | x).2 The latent z may represent a chain of thought [Wei et al., 2022], proof trace, search trajectory, program, or other internal computational object. We treat z as an operational latent: with supervision only at the output level, the latent trajectory mediates the output distribution. Success probability and endpoint losses. For each supervised example (x∗ , y∗ ), the central quantity is the success probability Pθ ≜ pθ (y∗ | x∗ ). From this we define two endpoint losses: the exploitation loss J0 (θ) ≜ E(x∗ ,y∗ )∼D [1 − Pθ ] and the density-estimation loss J1 (θ) ≜ E(x∗ ,y∗ )∼D [− log Pθ ]. Both are minimized at 0 when Pθ = 1, but transform Pθ into optimization signal differently. Under exact-match supervision (R(ŷ, y∗ ) = I(ŷ = y∗ )), J0 equals 1 minus the expected reward (Proposition A.1), so minimizing J0 is equivalent to maximizing expected reward.3 The JQ family.

We interpolate using the Tsallis q-logarithm [Tsallis, 1988]: logq (u) =

u1−q − 1 , 1−q

0 < u ≤ 1,

(1)

with log1 (u) ≜ limq→1 logq (u) = log u. We define the loss family JQ (θ, q) =

(x∗ ,y∗ )∼D

[− logq (Pθ )],

(2)

or equivalently JQ (θ, q) =

E∗ ∗

(x ,y )∼D

[− logq

pθ (z, y∗ | x∗ ) ].

z∈Z

It recovers the endpoints: JQ (θ, 0) = J0 (θ) and JQ (θ, 1) = J1 (θ).

Loss Landscape of the JQ Continuum

For a fixed supervised example (x∗ , y∗ ), define the per-example q-loss ℓq (θ; x∗ , y∗ ) ≜ − logq Pθ =

1 − Pθ1−q , 1−q

(3)

so that JQ (θ, q) = E(x∗ ,y∗ )∼D [ℓq (θ; x∗ , y∗ )]. At q = 0 this gives ℓ0 = 1 − Pθ (bounded in [0, 1]); at q = 1 it gives ℓ1 = − log Pθ (unbounded as Pθ → 0). The parameter q shapes the loss landscape in four ways: • Dataset-level coverage: q > 0 penalizes non-uniform success across training examples (dispersion penalty). 1/q

• Prediction-level coverage: the minimizer is the escort distribution θj∗ ∝ αj , interpolating from mode-seeking (q → 0) to mode-covering (q = 1). • Propriety: q = 1 is the unique strictly proper scoring rule in the family; q < 1 introduces controlled mode-seeking bias. • Robustness: at q < 1 the loss is bounded and the escort tempering concentrates the minimizer away from corrupted labels; at q = 1 the model fits noise exactly. We develop the first two below; formal statements for all four are in Section B. 2 Since Z ⊆ Σ∗ is at most countable, we write finite sums over Z; results extend to countably infinite Z under dominated convergence. 3 Full proofs of all proof sketches are in the appendix, organized by section.

3.1

Dataset-level coverage: the dispersion penalty

Let P̄ ≜ E(x∗ ,y∗ )∼D [Pθ ] denote the mean success probability. The exploitation loss J0 = 1 − P̄ depends only on P̄ and is indifferent to how success is distributed across examples. For q > 0, − logq is strictly convex, so Jensen’s inequality gives JQ ≥ − logq (P̄ ): the loss penalizes non-uniform success. To second order, the excess loss scales as 2q P̄ −q−1 VarD (Pθ ), with the penalty coefficient monotonically increasing in q. 3.2

Prediction-level coverage: the escort minimizer

At the prediction level, q controls whether the model’s output distribution matches the data or ∗ concentrates on the mode. Consider a categorical model with a single input Px , outputs {v1 , . . . , vN }, ∗ model pθ (vj | x ) = θj ∈ ∆N , and empirical frequencies αj > 0 with j αj = 1. β

The escort distribution [Beck and Schögl, 1993] of order β of a distribution α is αj / β = 1/q gives the data distribution tempered at temperature q.

β k αk

; setting

Theorem 3.1. P [Minimizers of JQ in the categorical model] For q ∈ (0, 1], the unique minimizer of JQ (θ, q) = j αj (− logq θj ) over ∆N is the escort distribution of order 1/q: 1/q αj ∗ θj (q) = PN , 1/q k=1 αk

j = 1, . . . , N.

(4)

For q = 0, the objective is linear and minimized at any vertex ej with j ∈ argmaxk αk . Proof sketch. For q > 0, strict convexity ensures uniqueness. Lagrange multipliers give αj θj−q = µ 1/q

for all j, yielding θj ∝ αj . The escort distribution interpolates continuously from full coverage (q = 1: θ∗ = α) to pure mode seeking (q → 0: θ∗ concentrates on the most frequent output). In particular, q = 1 is the unique strictly proper scoring rule in the JQ family (Corollary B.3).

Gradient Geometry of JQ

All members of JQ share the same per-example gradient direction. The gradient factors through either the RL endpoint ∇θ ℓ0 or the FT endpoint ∇θ ℓ1 , motivating the two Monte Carlo estimators of Section 6. Proposition 4.1 (Gradient geometry and dual factorization). For any fixed supervised example (x∗ , y∗ ) with Pθ > 0 and any q ∈ [0, 1], ∇θ ℓq (θ; x∗ , y∗ ) = Pθ−q ∇θ ℓ0 (θ; x∗ , y∗ ) = Pθ1−q ∇θ ℓ1 (θ; x∗ , y∗ ). |{z} | {z }

(5)

attenuate

amplify

d Proof. By the chain rule and du logq (u) = u−q : ∇θ ℓq = −Pθ−q ∇θ Pθ = Pθ−q ∇θ ℓ0 . Since ∇θ ℓ0 = −∇θ Pθ = Pθ ∇θ ℓ1 , the second equality follows.

The scalar rescales either the RL endpoint gradient (by Pθ−q ∈ [1, ∞), amplification) or the FT endpoint gradient (by Pθ1−q ∈ [0, 1], attenuation). Setting q = 0 recovers ∇ℓ0 (no amplification); q = 1 recovers ∇ℓ1 (no attenuation). The scalar Pθ−q controls both cold-start escape speed (ṗ = p2−q ∥s∥2 , yielding the Ω(1/p0 ) versus Θ(log(1/p0 )) separation of Section 5) and finite-sample estimator bias (larger q increases the O(q/(M Pθq+1 )) bias of Section 6). Each factorization motivates a Monte Carlo estimator: the RL factorization yields GARL (prior sampling with amplification; Section 6.1), the FT factorization yields PAFT (posterior sampling with attenuation; Section 6.2). 5

Commitment Dynamics under Gradient Flow

Under gradient flow, escape from a cold start (p0 = Pθ(0) ≪ 1) takes Ω(1/p0 ) time at the exploitation pole (q=0) but only Θ(log(1/p0 )) at the density-estimation pole (q=1). This exponential separation in 1/p0 is governed by the amplification factor Pθ−q and the dynamics ṗ = p2−q ∥s(θ)∥2 . Our analysis is stylized: it tracks single-example success probability under continuous-time gradient flow, isolating the role of the amplification factor rather than fully modeling multi-example LM optimization. 5.1

Dynamics of the success probability

We study gradient flow, the continuous-time limit of gradient descent, in which parameters evolve as θ̇ = −∇θ ℓ(θ) [Su et al., 2016]. This removes step-size effects and yields closed-form rates that capture the qualitative behavior of discrete optimization. The results below require no convexity: ṗ ≥ 0 always (Equation (6)), so p is monotone along the flow. Fix a single example’s q-loss ℓq (θ) = − logq (Pθ ), θ ∈ Rd . Let p(t) ≜ Pθ(t) denote the success probability along the flow, with time derivative ṗ ≜ dp/dt. We combine ṗ(t) = ∇θ Pθ(t) · θ̇(t) (chain −q rule), and θ̇(t) = −∇θ ℓq (θ(t)) = Pθ(t) ∇θ Pθ(t) (the second equality uses Proposition 4.1, which gives ∇θ ℓq = −Pθ−q ∇θ Pθ ). Substituting and writing ∇θ Pθ(t) = Pθ(t) ∇θ log Pθ(t) with score s(θ) ≜ ∇θ log Pθ , −q −q ṗ = ∇θ Pθ(t) · θ̇(t) = ∇θ Pθ(t) · Pθ(t) ∇θ Pθ(t) = Pθ(t) ∥∇θ Pθ(t) ∥2 = p2−q ∥s(θ(t))∥2 . (6) The entire effect of q on convergence speed is captured by the exponent 2 − q on p; the factor ∥s(θ)∥2 depends on the architecture but not on q. 5.2

Cold-start escape rates

Let p0 ≜ p(0) ≪ 1. With ∥s∥ approximately constant, Equation (6) implies that the escape time to Rδ a target δ is T ∼ p0 u−(2−q) du, and the exponent 2 − q controls its growth as p0 → 0: at q = 0 the integrand is u−2 and T diverges as 1/p0 ; at q = 1 the integrand is u−1 and T diverges only as log(1/p0 ) (equivalently, p(t) = p0 et under ṗ = p). We formalize this separation in two results.4 The first requires only an upper bound on the score norm and establishes that the exploitation pole is provably slow. The second adds a lower bound and shows the density-estimation pole is provably fast, giving tight Θ(·) rates across the continuum. Theorem 5.1. [Exploitation is provably slow] Let θ ∈ Rd parameterize any differentiable model. Consider gradient flow on ℓq (θ) = − logq (Pθ ), starting from p0 = Pθ(0) ∈ (0, 1/2] with fixed target δ ∈ (0, 1/2]. Suppose only that ∥s(θ(t))∥ ≤ C throughout the trajectory. Then as p0 → 0: ! −(1−q) p0 Tq (p0 , δ) = Ω for q ∈ [0, 1), 1−q 1 T1 (p0 , δ) = Ω log . p0 In particular, the exploitation pole cannot escape cold start faster than Ω(1/p0 ). Proof sketch. From ṗ = p2−q ∥s∥2 ≤ C 2 p2−q , the success probability grows no faster than C 2 p2−q . Rδ −(1−q) Integrating: Tq ≥ C12 p0 u−(2−q) du, which evaluates to Ω(p0 /(1−q)). The upper bound ∥s∥ ≤ C holds for any autoregressive softmax model P with bounded parameterto-logit Jacobian: the per-trajectory score ∇θ log p(z, y∗ | x∗ ) = t (eyt − pt )⊤ ∇θ zt combines bounded softmax residuals with the Jacobian ∇θ zt , and s is a posterior expectation of these, so ∥s∥ is bounded whenever the weights are bounded and activations Lipschitz. No matter how favorable the 4 The p2−q factor in Equation (6) is assumption-free; converting it to escape times needs bounds on ∥s∥: an upper bound

for Ω(·) on time (Theorem 5.1), and additionally a lower bound for matching Θ(·) rates (Theorem 5.2).

architecture, the exploitation pole requires escape time at least linear in 1/p0 — a prediction Section 7 confirms: q = 0 fails to escape cold start in practice. Theorem 5.2. [Tight cold-start escape rates] Under the same setup as Theorem 5.1, suppose additionally that ∥s(θ(t))∥ ≥ c > 0 throughout the trajectory. Then: 1. General q ∈ [0, 1): −(1−q)

p Tq (p0 , δ) = Θ 0 1−q

! as p0 → 0.

2. Density-estimation pole (q = 1):

1 T1 (p0 , δ) = Θ log p0

as p0 → 0.

3. Speedup ratio: for any q < q ′ with q ′ ≤ 1, Tq (p0 , δ) →∞ Tq′ (p0 , δ)

as p0 → 0.

Proof sketch. The lower bound on ∥s∥ gives ṗ ≥ c2 p2−q , yielding the matching upper bound Rδ Tq ≤ c12 p0 u−(2−q) du. Combined with Theorem 5.1, this gives the Θ(·) bounds. Robustness of the separation. The upper bound ∥s∥ ≤ C alone is enough for Theorem 5.1’s Ω(·) time bound; the additional lower bound ∥s∥ ≥ c is used only to promote this to the matching Θ(·) in Theorem 5.2. The q-dependent separation itself comes from the assumption-free factor p2−q in Equation (6), so the ordering across poles survives even where ∥s∥ ≥ c fails — at a critical point, for instance, every q stalls equally. Section C.1 works out exact escape times for a sigmoid model. Why momentum-based optimization cannot substitute for q. The parameter q controls perinstance commitment: how much to prioritize hard instances relative to easy ones. This is orthogonal to the global step size set by the learning rate. Momentum-based adaptive optimizers such as Adam [Kingma and Ba, 2014] adjust per-parameter step sizes aggregated across examples, but cannot compensate for per-example reweighting. The scalars Pθ−q (for GARL) and Pθ1−q (for PAFT) are thus preserved under both minibatch SGD and Adam, and the cold-start separation persists in practice. Noise fitting is symmetric. The same machinery gives a dual result for label noise: for example, in the binary categorical model with symmetric label-flip rate ϵ, the time to grow noise contamination p̃ = 1 − pθ (c | x∗ ) to target level η scales as Tqnoise (η) = Θ(η q+1 /((q + 1)ϵ)), with speedup ′ ratio Tq /Tq′ = Θ(η −(q −q) ) for q < q ′ (Proposition C.2). The speedup ratio matches the cold-start ′ −(q −q) speedup Θ(p0 ) exactly in form: the same amplification Pθ−q accelerates commitment to clean and corrupted supervision alike, with matching exponents in p0 and η. High commitment thus compresses both timescales — the time to resolve ambiguity and the time to memorize noise. SFT-then-RL asymmetry. The cold-start escape and noise fitting results explain the familiar SFT-then-RL pipeline [Ouyang et al., 2022, DeepSeek-AI, 2025, Chu et al., 2025]. SFT on annotated (input, CoT, answer) triples is the q = 1 pole with a degenerate proposal (marginalization collapses onto the supervised CoT), so it escapes in Θ(log(1/p0 )) via Pθ−1 amplification; RL (q = 0) pays the full Θ(1/p0 ) cost. Switching to RL after SFT then halts commitment to noisy annotations: q = 1 memorizes noise fastest (T1noise = Θ(η 2 /ϵ)) while q = 0 does not memorize at all (limq→0+ Tqnoise (η) = ∞ for any η > 0; Proposition C.2). The JQ continuum replaces this hard switch with a smooth interpolation.

Gradient Estimators for JQ

P The marginal Pθ = z∈Z pθ (z, y∗ | x∗ ) in ∇θ JQ is intractable, so we estimate the gradient by Monte Carlo. The dual factorization (Proposition 4.1) yields two natural estimators: 7

• GARL (Section 6.1): sample from the prior pθ (z | x∗ ), estimate ∇θ ℓ0 and Pθ from the same samples, amplify by (w̄M )−q . • PAFT (Section 6.2): approximately sample from the posterior pθ (z | x∗ , y∗ ), estimate ∇θ ℓ1 via teacher forcing, attenuate by (w̄M )1−q . Drop-in compute cost. Both estimators are drop-in replacements for RB-REINFORCE/RLOO at the same rollout budget. GARL replaces the scalar 1 in RB-RLOO with (w̄M )−q , reusing the M prior samples and per-token log-probabilities RB-RLOO already computes [Zhou et al., 2026]; the only added work is the scalar (w̄M )−q and the leave-one-out baseline in Equation (12), both O(M ) in compute. PAFT adds one categorical resample over the M prior weights, followed by teacher forcing on K resampled trajectories whose tokens have already been generated. Neither requires forward passes beyond what RL training already does. In our experiments (Section 7), GRPO, GARL, and PAFT all use M = 32 rollouts per prompt at training time. 6.1 6.1.1

GARL: Gradient-Amplified RL A plug-in Monte Carlo estimator

Fix a supervised example (x∗ , y∗ ) and draw M i.i.d. latent trajectories z(1) , . . . , z(M ) ∼ pθ (· | x∗ ). Define the per-sample likelihood weight and gradient contribution: wm ≜ pθ (y∗ | x∗ , z(m) ), gm ≜ −wm ∇θ log pθ (z(m) , y∗ | x∗ ), P P 1 1 with empirical means w̄M ≜ M m wm and ḡM ≜ M m gm . By the log-trick, X ∇θ pθ (z, y∗ | x∗ ) = −∇θ Pθ = ∇θ ℓ0 . E[w̄M ] = Pθ , E[ḡM ] = −

(7)

(8)

Plugging these into the RL factorization of Proposition 4.1 yields the plug-in estimator ∗ ∗ [ ∇ θ ℓq (q, θ; x , y , M ) ≜

ḡM . (w̄M )q

(9)

The dataset-level estimator of ∇θ JQ averages Equation (9) over a minibatch: GARL amplifies the RL gradient ḡM by the plug-in estimate (w̄M )−q of Pθ−q . At the endpoints, GARL recovers RBREINFORCE [q=0; Zhou et al., 2026] and the IWAE gradient estimator [q=1; Burda et al., 2015]; see Section D.2. Effective reward. The effective reward wm/(w̄M )q has a maximum value of M q , and varies along with q; we divide by M q to normalize it to [0, 1] (Section D). We use the maximum effective reward across samples to monitor training dynamics (Figure 3). The 1/M q factor in Algorithms 1 and 2 is an implementation choice equivalent to a q-dependent learning-rate rescaling; the mathematical estimators of Equations (12) and (14) target ∇θ ℓq directly without it. 6.1.2

Consistency and finite-sample bias

Equation (9) is a ratio estimator: it reuses the same samples in numerator and denominator, so it is biased at finite M even though w̄M and ḡM are individually unbiased.5 Theorem 6.1. [Consistency and bias expansion] Fix a supervised example (x∗ , y∗ ) and assume: 1. Pθ > 0; 2. E[∥gm ∥2 ] < ∞; 3. wm ≥ ϵ a.s. for some ϵ > 0. 5 Assumptions 1–2 of Theorem 6.1 are standard regularity. Assumption 3 holds for autoregressive softmax models: Q ∗ wm = T t=1 pθ (yt | ·) is a finite product of softmax probabilities bounded below by a logit-dependent ϵ0 > 0, so wm ≥ ϵT 0 . This is non-vacuous for short outputs but tightens to vanishingly small values for long generations; the bias expansion should accordingly be read as identifying the qualitative direction of finite-sample degradation rather than a uniform numerical bound in the long-sequence regime.

Then for any fixed q ∈ [0, 1], a.s. ∗ ∗ ∗ ∗ [ ∇ θ ℓq (q, θ; x , y , M ) −−−−→ ∇θ ℓq (θ, q; x , y ).

(10)

M →∞

Moreover, for fixed Pθ > 0 and q ∈ [0, 1], the bias satisfies q M Pθq+1

h i [ E ∇ θ ℓq − ∇θ ℓq = O

! as M → ∞.

(11)

At q = 0 the factor q in the numerator makes the bias vanish exactly for all M : the estimator reduces to the unbiased sample mean ḡM (Equation (8)). The explicit leading-order coefficient is in Section D. Proof sketch. Assumption 1 ensures continuity of f (a, b) = b a−q at (Pθ , ∇ℓ0 ); consistency then follows from the SLLN. For the bias, since f is linear in b, write f (w̄M , ḡM ) = ḡM · h(w̄M ) where h(a) = a−q . Expanding h around Pθ and separating ḡM = µg + (ḡM − µg ) yields the O(1/M ) bias from Var(wm ) and Cov(gm , wm ). The remainder is O(M −2 ): on the high-probability event {w̄M ≥ Pθ /2} (exponential concentration via wm ∈ [0, 1]), the derivatives of the scalar function −q h are bounded and Assumption 2 controls the higher-order terms; Assumption 3 gives w̄M ≤ ϵ−q −q −cM everywhere, making the complementary event’s contribution O(ϵ e ). The O(1/M ) rate is standard for ratio estimators; the JQ -specific feature is the joint dependence on q −(q+1) and Pθ . The bias grows with q (vanishing at q = 0) and explodes as Pθ → 0 with Pθ scaling: the same amplification that enables fast cold-start escape (Theorems 5.1 and 5.2) degrades estimator quality. This predicts intermediate q outperforms both endpoints in practice — confirmed in Section 7. The expansion is a fixed-Pθ , large-M asymptotic; in the cold-start regime where Pθ is small and M is bounded by compute, it identifies the direction of finite-sample degradation rather than providing a uniform bound. 6.1.3

Variance reduction for GARL

The GARL estimator (9) decomposes into a score-function term (for the sampled z) and a pathwise term (for the fixed y∗ ); only the score-function term admits baselines. Following Kool et al. [2019], P we center the score-function coefficient with a leave-one-out control variate using w̄¬m ≜ M1−1 j̸=m wj , yielding the RLOO estimator (derivation in Section D.1):

[ ∇ θ ℓq

RLOO

" # M 1 X wm ∇θ wm 1−q (m) ∗ = − − (w̄¬m ) ·∇θ log pθ (z |x )− . M m=1 (w̄M )q (w̄M )q | {z }

(12)

centered weight

[ Proposition 6.2 (RLOO bias preservation). Under the assumptions of Theorem 6.1, E[∇ θ ℓq plug-in [ E[∇ ], so the RLOO estimator inherits the bias of Equation (11). θ ℓq

RLOO

Algorithm 1 summarizes the complete estimator. At q = 0 it recovers the Rao–Blackwellized RLOO estimator [Zhou et al., 2026]; at q = 1 the centered weight becomes wm /w̄M − 1, a self-normalizing baseline (details in Section D.1). 6.2

PAFT: Posterior-Attenuated Fine-Tuning

GARL estimates the JQ gradient via the RL factorization: sample rationales from the prior pθ (z | x∗ ), then amplify by Pθ−q — sometimes massively, especially on hard instances. The FT factorization (Equation (5)) suggests an alternative: instead of sampling from the prior and amplifying, (approximately) sample from the posterior pθ (z | x∗ , y∗ ) — where rationales already agree with the answer — and attenuate by Pθ1−q ∈ [0, 1]. 9

Algorithm 1 GARL: per-example JQ gradient with RLOO control variate Require: Example (x∗ , y∗ ), interpolation parameter q ∈ [0, 1], number of latent samples M 1: Sample latent trajectories z(1) , . . . , z(M ) ∼ pθ (· | x∗ ) 2: for m = 1, . . . , M do 3: wm ← pθ (y∗ | x∗ , z(m) ) ▷ likelihood weight 4: ∇θ wm ← ∇θ pθ (y∗ | x∗ , z(m) ) ▷ pathwise gradient of output likelihood 5: end for PM 1 6: w̄M ← M ▷ batch mean (estimates Pθ ) m=1 wm 7: for m = 1, . . . , M do P 8: w̄¬m ← M1−1 j̸=m wj ▷ leave-one-out mean 9: cm ← (w̄wMm)q − (w̄¬m )1−q ▷ centered weight (RLOO baseline) θ wm ĝm ← −cm ∇θ log pθ (z(m) | x∗ ) − ∇ ▷ score-function + pathwise terms (w̄M )q 11: end for PM 1 ▷ per-example gradient estimate, normalized by M q 12: return ĝ ← M1q · M m=1 ĝm

10:

6.2.1

Posterior form of the gradient

Expanding ∇θ ℓ1 = −∇θ log Pθ as a posterior expectation: ∇θ ℓq = −Pθ1−q ·

z∼pθ (z|x∗ ,y∗ )

[∇θ log pθ (z, y∗ | x∗ )].

(13)

Each sample gradient is standard SFT (teacher forcing) on a semantically coherent (input, rationale, answer) triple: the rationale is posterior-weighted toward agreement with y∗ . Approximate posterior sampling. Equation (13) requires samples from the posterior pθ (z | x∗ , y∗ ), which is intractable for autoregressive models where z precedes y. The framework permits many approximate posterior samplers (learned proposals, MCMC, infilling models); here we use importance resampling [IR; Rubin and Rubin, 1988] because it reuses GARL’s prior sample pool and wm weights with minimal additional compute: resample K trajectories with replacement, with probability proportional to wm . IR guarantees exactly K resampled trajectories regardless of how small the individual wm values are. The PAFT estimator. estimate is

Let z(1) , . . . , z(K) denote the resampled trajectories. The PAFT gradient

ˆ PAFT = −(w̄M )1−q · 1 ∇ K

K X

∇θ log pθ (z(k) , y∗ | x∗ ).

(14)

k=1

At q = 1, the instance weight vanishes (Pθ1−1 = 1) and PAFT recovers the E-step of EM [Dempster et al., 1977, Phan et al., 2023]; see Section D.2 for all endpoint reductions. 6.2.2

Bias and variance

1−q Conditional on the prior pool {(z(m) , wm )}M m=1 , (w̄M ) P is deterministic and the IR average P (m) ∗ ∗ of fm = ∇θ log pθ (z , y | x ) has conditional mean m (wm/ j wj ) fm = −ḡM/w̄M (using gm = −wm fm ). Hence ˆ PAFT | pool] = −(w̄M )1−q · −ḡM/w̄M = ḡM = ∇ ˆ GARL . E[∇ (w̄M )q

ˆ PAFT ] = E[∇ ˆ GARL ] at every finite M , so Taking outer expectations gives an exact identity E[∇ q+1 q PAFT has the same O( /M Pθ ) bias as GARL (Theorem 6.1) even though the plug-in (w̄M )1−q is individually biased (Jensen). The plug-in bias is exactly canceled by the covariance between (w̄M )1−q and the IR average: the Rao–Blackwellization identity fixes the total, so components cannot be analyzed in isolation. By the law of total variance, GARL has strictly lower variance than PAFT (Propositions D.3 and D.4). Yet PAFT can produce better training dynamics. GARL’s lower variance comes from mixing bad rationales into the gradient with small weights; PAFT excludes them before the gradient is formed. 10

Algorithm 2 PAFT: per-example JQ gradient via importance resampling Require: Example (x∗ , y∗ ), interpolation parameter q ∈ [0, 1], prior samples M , resampled trajectories K 1: Sample latent trajectories z(1) , . . . , z(M ) ∼ pθ (· | x∗ ) 2: for m = 1, . . . , M do 3: wm ← pθ (y∗ | x∗ , z(m) ) ▷ likelihood weight (same as GARL) 4: end for PM 1 5: w̄M ← M ▷ batch m=1 wm P P mean (estimates Pθ ) 6: Resample indices r1 , . . . , rK ∼ Categorical(w1 / j wj , . . . , wM / j wj ) (w̄M )1−q PK (rk ) ∗ ∗ 7: ĝ ← − M q K ,y | x ) ▷ attenuated SFT on coherent rationales, k=1 ∇θ log pθ (z normalized by M q 8: return ĝ ▷ per-example gradient estimate, normalized by M q

The resampling noise is structured — preserving the semantic coherence of the FT endpoint — so PAFT is more stable at warm start despite its higher variance (Section 7). Unlike GARL’s rewardtimes-score structure, the PAFT gradient is a plain posterior expectation of the complete-data score, with no reward coefficient to center; variance reduction comes from the posterior sampling itself, which excludes bad rationales before they reach the gradient.

Empirical Validation

We validate the theoretical predictions and empirical effectiveness of GARL and PAFT on subsets of three reasoning datasets — FinQA [Chen et al., 2021], HotPotQA [Yang et al., 2018], and MuSiQue [Trivedi et al., 2022] — using post-trained Qwen 3 0.6B [Yang et al., 2025] under both cold-start and warm-start conditions. 7.1

Experimental setup

Warm-start scenario. Task inputs are natural-language prompts with standard task descriptions and answer-formatting instructions. The un-adapted model can occasionally produce correct answers (Section 7.3), so reward is not sparse. Cold-start scenario. We use linearized problem inputs and outputs as (x∗ , y∗ ) pairs with no task description and no answer-formatting instructions. The model must discover both how to solve the problem and how to format the answer; initial success probability Pθ is very low. Datasets. We sample subsets from the official splits of FinQA, HotPotQA, and MuSiQue. FinQA: train n = 6145, validation n = 872, test n = 1132. HotPotQA: train n = 9067, validation n = 342, test n = 343. MuSiQue: train n = 9985, validation n = 579, test n = 445. Exact-match training rewards are computed against the gold answer string; evaluation uses substring match (Section 7). Methods and compute budget. All methods — GRPO, GARL, and PAFT — use the same rollout budget of M = 32 latent trajectories per prompt during training and 16 samples per prompt at evaluation, so head-to-head comparisons are fair in compute. GARL (Algorithm 1) uses the RLOO variance reduction (Equation (12)); PAFT (Algorithm 2) importance-resamples K = M trajectories from the same pool. We evaluate fixed values of q ∈ {0, 0.25, 0.5, 0.75, 1.0}. Rationale budget. We enforce a per-rationale token budget by forcing the model to decode the thinking-end token (</think> for our Qwen experiments) once the allocated thinking budget is exhausted [Muennighoff et al., 2025]. The budget varies by dataset: FinQA uses 4k − 128 tokens, HotPotQA 3k − 128, and MuSiQue 2k − 128, where the 128-token offset reserves space for the answer. Evaluation. Training uses exact-match rewards (Section 2). Evaluation uses relaxed substring match — ŷ is correct if y∗ appears as a substring of ŷ — so rationale tokens around the answer do not penalize the model. We report pass@1 (p@1; single-sample accuracy), pass@k (p@k; best-of-k, 11

Table 1: Cold-start GARL results on FinQA. GARL with q ≤ 0.5 fails entirely; only q ≥ 0.75 escapes. Method

p@1

p@16

m@16

GRPO q = 0 (RB-RLOO) q = 0.25 q = 0.5 q = 0.75 q=1

0 0 0 0 30.5 21.9

0 0 0 0 61.1 58.7

0 0 0 0 38.3 33.5

Table 2: Cold-start GARL (no prompts) vs. warm-start GRPO (prompted). All methods use exactmatch rewards during training. In our setting, GARL at q=0.75 matches or exceeds GRPO on every metric across all three benchmarks (a confounded comparison: see body discussion). Dataset

Method

p@1

p@16

m@16

FinQA

GRPO q=0.75 q=1

18.9 30.5 21.9

48.4 61.1 58.7

26.9 38.3 33.5

HotPotQA

GRPO q=0.75 q=1

29.1 53.5 48.7

55.1 74.1 75.5

33.5 57.2 56.6

MuSiQue

GRPO q=0.75 q=1

13.6 26.8 21.6

37.3 58.9 58.1

15.8 34.8 32.5

rewards coverage), and maj@k (m@k; majority vote over k samples [Wang et al., 2023], rewards diverse correct trajectories). Reported test numbers are taken from the checkpoint with highest validation [email protected] 7.2

Cold-start results: the escape-time separation

Cold start tests whether commitment speed, controlled by Pθ−q , determines escape from a sparsereward regime (Theorem 5.2). Cold-start escape requires large q. GRPO, Rao–Blackwellized RLOO (q = 0), and all q ≤ 0.5 fail entirely — zero accuracy across all metrics. Rao–Blackwellization replaces the binary reward I(ŷ = y∗ ) with its conditional expectation wm = pθ (y∗ | x∗ , z(m) ) [Zhou et al., 2026], reducing variance but not helping escape: the underlying gradient remains ∇θ ℓ0 = −∇θ Pθ with dynamics ṗ = p2 ∥s∥2 and no amplification (Pθ−q = 1 at q = 0; Figure 3). These results suggest that in our setting the cold-start bottleneck is primarily gradient amplification rather than gradient variance. The −(1−q) sharp transition at q = 0.75 matches Theorem 5.1: the lower bound Ω(p0 ) grows rapidly as q decreases, so for a fixed training budget there is a critical q below which escape fails. The density-estimation pole escapes but overshoots. q = 1 successfully escapes cold start on all three benchmarks (Table 2), confirming Theorem 5.2. However, q = 0.75 achieves higher pass@1 and maj@16 on every benchmark. The pass@16 picture is more nuanced: q = 1 achieves higher pass@16 than q = 0.75 on HotPotQA (75.5 vs. 74.1), consistent with its stronger mode-covering behavior producing more diverse reasoning paths. But this diversity does not translate to higher maj@16, because the trajectories are trained with noisier gradients. This is exactly the escape-vs-bias tradeoff predicted by Theorem 6.1: q = 1’s stronger amplification enables faster escape but produces noisier gradient estimates, while q = 0.75 strikes a better balance. Cold-start GARL is competitive with prompted GRPO. Table 2 compares cold-start GARL at q ∈ {0.75, 1} (no task-specific prompts) against warm-start GRPO (with prompts) across all three benchmarks; all methods use exact-match training rewards. In our setting, GARL at q = 0.75 matches 6 All numbers in this paper are single-seed; seed variance and bootstrap confidence intervals are left to revision.

Figure 3: Cold-start training dynamics on FinQA: maximum amplified advantage cm /M q vs. training step, where cm = wm /(w̄M )q − (w̄¬m )1−q is the centered weight from Equation (12) (normalized to [0, 1] by dividing by M q ; cf. effective-reward bound in Section D). q = 1 escapes immediately, q = 0.75 escapes sharply around step 35, and q ≤ 0.5 remain flat — qualitatively consistent with the −(1−q) −(1−q) predicted ordering (Θ(log(1/p0 )) at q = 1, Θ(p0 ) for q < 1, with Ω(p0 ) exceeding the training budget at small q). We do not claim measured slopes validate the asymptotic rates. Despite its fast escape, q = 1 achieves lower test accuracy than q = 0.75 (Table 1), consistent with the O(q/M Pθq+1 ) ratio-estimator bias of Theorem 6.1 degrading gradient quality. or exceeds prompted GRPO on every metric across all three benchmarks (+11.6 p@1 on FinQA, +24.4 on HotPotQA, +13.2 on MuSiQue), and GARL at q = 1 exceeds GRPO on coverage metrics (p@16, m@16) while underperforming on p@1. We treat this as a hypothesis-generating observation rather than evidence that prompts are unnecessary: cold- and warm-start runs differ in more than prompts (input formatting, output constraints, target distribution), and isolating the prompt factor requires a controlled ablation we leave to future work. 7.3

Warm-start results across three benchmarks

Warm start tests whether GARL and PAFT still help when Pθ is not negligible and standard RL already makes progress. Table 3 reports warm-start maj@16 across all three benchmarks. Cold-start without instructions beats warm-start with them. The base model with task-specific prompts but no training performs weakly (12.6 / 22.2 / 8.9 maj@16; first row of Table 3), confirming that these tasks require adaptation. Every trained method in Tables 2 and 3 improves over this base. More striking: cold-start GARL at q = 0.75 without any task-specific prompts matches or beats 13

Table 3: Warm-start maj@16 across three benchmarks (exact-match training rewards; evaluation uses substring match). Base = un-adapted Qwen 3 0.6B evaluated with the same prompted inputs as the trained methods. GARL at q = 0 recovers RB-RLOO [Zhou et al., 2026]. GARL entries for MuSiQue and HotPotQA are peak-before-collapse (validation accuracy collapses to zero before end of training; see Section 7.3); only FinQA GARL and all PAFT entries are steady-state. Best steady-state result per benchmark in bold: GARL at q=0.25 on FinQA, PAFT at q=0.75 on HotPotQA and MuSiQue. The best stable method beats GRPO by +6.6 to +14.4 points. Method

FinQA

HotPotQA

MuSiQue

Base (no training, prompted) GRPO

12.6 26.9

22.2 33.5

8.9 15.8

GARL (q = 0, RB-RLOO) GARL (q = 0.25) GARL (q = 0.75)

38.3 38.7 37.6

21.6 22.9 46.8

9.1 24.3 19.7

PAFT (q = 0.25) PAFT (q = 0.75)

26.6 28.6

47.0 47.9

9.0 22.4

the best stable warm-start maj@16 on every benchmark — FinQA 38.3 vs. 38.7 (tie with warm-start GARL at q = 0.25), HotPotQA 57.2 vs. 47.9 (+9.3), MuSiQue 34.8 vs. 22.4 (+12.4). The swing from base-with-prompts to cold-start GARL is +25.7 to +35.0 points with no prompt engineering whatsoever. Instructions and answer-formatting supervision are not merely unnecessary under strong commitment; one interpretation is that the added prompt structure may constrain the learned policy toward narrower reasoning patterns. A controlled ablation isolating the prompt factor from other cold-start/warm-start differences is left to future work. With high-q amplification, the model discovers task structure directly from input-output pairs. Rao–Blackwellized rewards alone are insufficient. GARL at q = 0 recovers the Rao– Blackwellized REINFORCE estimator of Zhou et al. [2026] with leave-one-out baseline (RB-RLOO). It beats GRPO on FinQA (+11.4 m@16) but underperforms on HotPotQA (−11.9) and MuSiQue (−6.7): replacing the binary reward with wm = pθ (y∗ | x∗ , z) does not generalize across warmstart tasks. Raising q lifts peak accuracy on the unstable benchmarks (GARL q = 0.75: HotPotQA 21.6 → 46.8 peak; MuSiQue 9.1 → 19.7 peak; FinQA is roughly flat across q ∈ [0, 0.75]), but peaks do not survive training on HotPotQA or MuSiQue (next paragraphs). Low q wins on FinQA. On FinQA, GARL is stable throughout training at all tested q, so the cost of high q — estimator bias O(q/M Pθq+1 ) (Theorem 6.1) and noise memorization (Proposition C.2), both driven by Pθ−q — outweighs its amplification benefit, and lower-bias estimators extract more signal per step. GARL at q = 0.25 posts the best FinQA maj@16 (38.7, +11.8 over GRPO). On MuSiQue and HotPotQA, GARL’s warm-start training collapses (next paragraphs), so this low-q advantage cannot be realized without training-dynamics instability. GARL destabilizes on HotPotQA; PAFT is stable. GARL destabilizes on HotPotQA warm-start at every tested q: validation accuracy peaks early and collapses to zero before the end of training (q = 0.2: peak 41.1 at step 100, zero by step 150; q = 0.25: peak 22.9 at step 50, zero by step 100; q = 0.75: peak 46.8 at step 50, zero by step 100). HotPotQA exhibits broader instability — GRPO also degrades, peaking at ∼37.4 around step 100 and declining steadily to ∼5.0 by end of training — but GARL’s collapse is qualitatively different: a sharp drop to literal zero rather than a gradual decline. PAFT shows neither pattern, reaching 47.9 maj@16 (+14.4 over GRPO) and remaining stable. Figure 4 compares GARL and PAFT validation curves at matched q = 0.25. We do not have a verified mechanism for the GARL-specific zero-collapse: candidate explanations include pathwise-term corruption (the GARL gradient updates pθ (y∗ | x∗ , z) on every sampled z, including incoherent ones, while PAFT only updates on resampled coherent rationales) and HotPotQA-specific overfitting (also visible in GRPO), and disentangling them would require a pathwise-zeroed ablation and additional diagnostics. The practical implication holds regardless: PAFT is the stable choice on benchmarks where GARL collapses. 14

Figure 4: Warm-start validation maj@16 on HotPotQA at q = 0.25: GARL peaks at step 50 (30.6) and collapses to zero by step 100; PAFT remains stable throughout training and reaches 53.6. At fixed q, the contrast isolates the estimator (prior-sampled, all-M vs. posterior-resampled).

PAFT at low q is slow, not collapsed. PAFT at q = 0.25 underperforms on MuSiQue (9.0 vs. 15.8), but validation accuracy is still rising at the end of training rather than dropping: the attenuation factor Pθ1−q = Pθ0.75 heavily down-weights hard instances, so learning is slow but not unstable. This differs qualitatively from GARL’s warm-start collapse on HotPotQA and MuSiQue (validation drops to zero). The GARL-vs-PAFT trade-off is therefore speed vs. stability: PAFT gives up gradient signal per step but avoids the destabilization observed in GARL on HotPotQA and MuSiQue. Raising q to 0.75 recovers speed for PAFT without compromising stability, delivering best-overall HotPotQA (47.9) and the honest MuSiQue recommendation (22.4 steady-state vs. GARL’s 24.3 peak-before-collapse). 7.4

Discussion

GARL and PAFT trade speed against stability. Across regimes: cold start requires GARL’s amplification (PAFT is undefined when Pθ ≈ 0); warm start admits both. Within warm start: GARL delivers higher per-step signal but destabilizes during training on HotPotQA and MuSiQue (collapse to zero), where HotPotQA also exhibits broader instability visible in GRPO’s gradual decline. PAFT does not collapse on any benchmark tested, at the cost of lower per-step signal (Pθ1−q attenuation plus posterior-resampling variance). On these benchmarks, the practical decision is stable-vs-not rather than high-q-vs-low-q: use GARL at low q where it is stable (FinQA); use PAFT at q ≥ 0.75 where GARL collapses (HotPotQA, MuSiQue). The mechanism behind GARL’s zero-collapse is unverified; pathwise-term corruption and dataset-specific overfitting are both candidates that future ablations could disentangle. Practical recommendation: GARL at large q for cold-start escape; in warm start, use GARL at low q if training is stable (FinQA), and PAFT at q ≥ 0.75 otherwise (HotPotQA, MuSiQue). PAFT also acts as an automatic curriculum: early on, only the easiest rationales pass the importance resampling 15

filter; as Pθ grows, more rationales become coherent enough to be selected, broadening the training distribution without an explicit schedule.

Related Work

q-logarithmic losses. The Tsallis q-logarithm originates in non-extensive statistical mechanics [Tsallis, 1988]. Ferrari and Yang [2010] introduced the maximum Lq -likelihood estimator (MLqE), which replaces log with logq in the log-likelihood and is equivalent to reweighting the score by f (X; θ)1−q . For sample-size-dependent qn → 1, MLqE is asymptotically normal around θ0 ; for fixed q < 1, finite-sample MSE can fall below MLE’s at the cost of bias toward a surrogate parameter θ0 /q. The PAFT gradient Equation (13) is the marginal-likelihood analog of this weighted score. Extending the q-log to deep classification, Zhang and Sabuncu [2018] proposed generalized cross-entropy for noisy labels (the same loss family under a different parameterization), observing that bounded loss at q < 1 prevents gradient domination by mislabeled samples. Our escort minimizer analysis 1/q (Theorem 3.1) gives a precise mechanism: the tempering α̃j concentrates the minimizer on the clean mode. Concurrently, Wang et al. [2026] apply the deformed-log family at the token level for SFT, deriving a gate-times-error gradient structure; their token-level gate pα is the single-token specialization of our example-level Pθ−q , but their p is an exact softmax probability whereas our Pθ is an intractable marginal over latent trajectories. Training-time exploration-exploitation. Tsallis entropy has been used as a policy regularizer in RL [Lee et al., 2018, Nachum et al., 2018], providing inference-time exploration through sparse action distributions. Our use of the Tsallis q-logarithm in the loss function provides a different kind 1/q of control: training-time exploration-exploitation. The escort minimizer θj∗ ∝ αj (Theorem 3.1) is a training-time analog of inference temperature that permanently shapes what the model learns, and the Pθ−q factor automatically explores more on instances the model finds surprising — a per-instance effect not achievable by tuning the learning rate or inference temperature alone. Information-theoretic context. Escort distributions were studied by Beck and Schögl [1993]. Rényi variational inference [Li and Turner, 2016] provides a complementary continuum that tightens the ELBO toward exact log-marginal-likelihood; our JQ family approaches the same target from the exploitation side, with − log Pθ as their shared meeting point. The RL-as-inference connection [Levine, 2018, Norouzi et al., 2016, Guu et al., 2017] views MLE and RL as distinct frameworks; our contribution is embedding them as endpoints of a single continuously parameterized family. Latent-variable training for reasoning. On the RL side, RLVR and GRPO [DeepSeek-AI, 2025, Shao et al., 2024] optimize expected reward with policy gradients. On the latent-variable side, STaR [Zelikman et al., 2022] bootstraps reasoning by generating and filtering rationales, while TRICE [Phan et al., 2023] maximizes marginal log-likelihood via MCMC-EM. Our framework subsumes these as endpoints: RLVR corresponds to q = 0 and marginal log-likelihood training to q = 1. PAFT at q = 1 recovers the EM E-step underlying TRICE, and STaR’s rejection-sampling strategy can be viewed as a hard-acceptance variant of PAFT’s importance resampling (Section D.2). Concurrent RL-to-ML interpolations. MaxRL [Tajwar et al., 2026] defines another RL-to-ML continuum by truncating the Maclaurin expansion of log p at order T . Their estimator is unbiased for the truncated objective J (T ) (itself a biased approximation of log Pθ ), while GARL targets the true q-loss with O(1/M ) estimator bias (Theorem 6.1). A key distinction is cold-start behavior: the MaxRL estimator is exactly zero when no sample succeeds (K=0), while GARL always has nonzero gradient since wm > 0. MaxRL and PAFT share the principle of training on successful trajectories; in the limit T → ∞ and q = 1, both average posterior-sampled gradients, differing only in hard (MaxRL) vs. soft (PAFT) acceptance. Gradient estimators for marginal likelihoods. The IWAE estimator [Burda et al., 2015] that GARL recovers at q=1 has a well-known failure mode: Rainforth et al. [2018] showed that as M grows, the signal-to-noise ratio of the inference-network gradient shrinks, motivating doubly reparameterized variants [Roeder et al., 2017, Tucker et al., 2019]. Our bias expansion O(q/M Pθq+1 ) exposes a related phenomenon along the JQ continuum: the same amplification that enables cold-start 16

escape degrades estimator quality, and intermediate q balances the two — a prediction confirmed in Section 7. Rao–Blackwellization and verifier-free training. Zhou et al. [2026] propose VeriFree, which uses pθ (y∗ | x∗ , z) directly as the reward signal. This is the RB-REINFORCE estimator that GARL recovers at q = 0 (Section 6.1). While Rao–Blackwellization reduces gradient variance, our experimental results in Section 7 show it does not address the cold-start escape bottleneck: the gradient remains ∇θ ℓ0 = −∇θ Pθ regardless of Rao–Blackwellization, and the dynamics ṗ = p2 ∥s∥2 receive no amplification at q = 0 (Figure 3). Both GARL and PAFT are verifier-free throughout the JQ continuum. RLVR capability boundaries and reward hacking. Yue et al. [2025] showed that RLVR improves sampling efficiency but rarely elicits new reasoning patterns, and the capability boundary narrows during training. Our framework gives a direct mechanism: this narrowing is the mode-seeking behavior predicted by the escort distribution at q = 0 (Corollary B.2). Related, sustained GRPO training with exact-match rewards often collapses via reward hacking, where models exploit verifier formatting rather than reasoning. The JQ continuum exposes q as a principled control for mode concentration, and PAFT as an empirically more stable alternative to GARL during warm-start training (Section 7).

Conclusion and Future Work

We introduced a Tsallis loss continuum JQ that unifies RLVR-style exploitation and marginallikelihood training via a single parameter q controlling commitment to unfamiliar supervision. The per-instance amplification Pθ−q is the mechanism that addresses the cold-start stalling problem: GARL at large q escapes cold start where GRPO fails, and the Ω(1/p0 ) lower bound for RLVR-style training is bypassed by moving q away from the exploitation pole. The gradient admits a dual factorization through the RL and FT endpoints (Proposition 4.1), yielding two complementary estimators: GARL (prior-sampling amplification) and PAFT (posterior-sampling attenuation). High commitment (q → 1) resolves ambiguity but memorizes noise; low commitment (q → 0) resolves noise but cannot escape cold start. Within warm start, GARL destabilizes on HotPotQA and MuSiQue while PAFT remains stable across all benchmarks tested. Limitations and future work. Experiments in this work use a single model scale (Qwen 3 0.6B), three benchmarks, and fixed values of q. The cold-start escape theorems and bias expansion are scale-agnostic, but the GARL collapse / PAFT stability finding has been verified only at this scale; replication at bigger model scales is important. Our convergence analysis is stylized: single-example, gradient flow, bounded score (Theorems 5.1 and 5.2). Our framework assumes exact-match supervision (Section 2); extension to general rewards is open.

References Christian Beck and Friedrich Schögl. Thermodynamics of Chaotic Systems: An Introduction. Cambridge Nonlinear Science Series. Cambridge University Press, 1993. Yuri Burda, Roger Baker Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. volume abs/1509.00519, 2015. URL https://api.semanticscholar.org/CorpusID: 11383178. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. FinQA: A dataset of numerical reasoning over financial data. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3697–3711, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021. emnlp-main.300. URL https://aclanthology.org/2021.emnlp-main.300/. 17

Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. 2025. URL https://openreview.net/forum?id=dYur3yabMj. DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. 2025. URL https://arxiv.org/abs/2501.12948. AP Dempster, NM Laird, and DB Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), pages 1–38, 1977. Davide Ferrari and Yuhong Yang. Maximum Lq -likelihood estimation. The Annals of Statistics, 38 (2):753–783, 2010. Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Regina Barzilay and Min-Yen Kan, editors, Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1051–1062, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-1097. URL https://aclanthology.org/ P17-1097/. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. volume abs/1412.6980, 2014. URL https://api.semanticscholar.org/CorpusID:6628106. Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free! 2019. URL https://openreview.net/forum?id=r1lgTGL5DE. Kyungjae Lee, Sungjoon Choi, and Songhwai Oh. Sparse markov decision processes with causal sparse tsallis entropy regularization for reinforcement learning. IEEE Robotics and Automation Letters, 3(3):1466–1473, 2018. doi: 10.1109/LRA.2018.2800085. Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. ArXiv, abs/1805.00909, 2018. URL https://api.semanticscholar.org/CorpusID: 19077536. Yingzhen Li and Richard E Turner. Rényi divergence variational inference. In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016. URL https://proceedings.neurips.cc/paper_ files/paper/2016/file/7750ca3559e5b8e1f44210283368fc16-Paper.pdf. Chu-Cheng Lin, Aaron Jaech, Xin Li, Matthew R. Gormley, and Jason Eisner. Limitations of autoregressive models and their alternatives. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5147–5173, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.naacl-main. 405. URL https://aclanthology.org/2021.naacl-main.405/. William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview.net/forum?id=NjNGlPh8Wh. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL https://arxiv.org/abs/2501.19393. Ofir Nachum, Yinlam Chow, and Mohammad Ghavamzadeh. Path consistency learning in tsallis entropy regularized mdps. ArXiv, abs/1802.03501, 2018. URL https://api.semanticscholar. org/CorpusID:3653343. Mohammad Norouzi, Samy Bengio, Zhifeng Chen, Navdeep Jaitly, Mike Schuster, Yonghui Wu, and Dale Schuurmans. Reward augmented maximum likelihood for neural structured prediction. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS’16, page 1731–1739, Red Hook, NY, USA, 2016. Curran Associates Inc. ISBN 9781510838819. 18

Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022. URL https://api.semanticscholar.org/CorpusID:246426909. Du Phan, Matthew D. Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A. Saurous. Training chain-of-thought via latentvariable inference. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2023. Curran Associates Inc. Tom Rainforth, Adam R. Kosiorek, Tuan Anh Le, Chris J. Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. In International Conference on Machine Learning (ICML), pages 4277–4285, 2018. Geoffrey Roeder, Yuhuai Wu, and David K. Duvenaud. Sticking the landing: Simple, lower-variance gradient estimators for variational inference. In Advances in Neural Information Processing Systems (NeurIPS), 2017. Db Rubin and Db Rubin. Using the sir algorithm to simulate posterior distributions. 1988. URL https://api.semanticscholar.org/CorpusID:115305396. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y.K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Weijie Su, Stephen Boyd, and Emmanuel J. Candès. A differential equation for modeling nesterov’s accelerated gradient method: theory and insights. J. Mach. Learn. Res., 17(1):5312–5354, January 2016. ISSN 1532-4435. Fahim Tajwar, Guanning Zeng, Yueer Zhou, Yuda Song, Daman Arora, Yiding Jiang, Jeff Schneider, Ruslan Salakhutdinov, Haiwen Feng, and Andrea Zanette. Maximum likelihood reinforcement learning. 2026. URL https://arxiv.org/abs/2602.02710. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. MuSiQue: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 2022. Constantino Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of Statistical Physics, 52:479–487, 1988. URL https://api.semanticscholar.org/CorpusID: 16385640. George Tucker, Dieterich Lawson, Shixiang Gu, and Chris J. Maddison. Doubly reparameterized gradient estimators for Monte Carlo objectives. In International Conference on Learning Representations (ICLR), 2019. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw. Zecheng Wang, Deyuan Liu, Chunshan Li, Yupeng Zhang, Zhengyun Zhao, Dianhui Chu, Bingning Wang, and Dianbo Sui. Gradients must earn their influence: Unifying sft with generalized entropic objectives, 2026. URL https://arxiv.org/abs/2602.11424. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3–4):229–256, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992696. URL https://doi.org/10.1007/BF00992696. 19

An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model? In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. URL https://openreview.net/forum?id=4OsgYD7em5. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: self-taught reasoner bootstrapping reasoning with reasoning. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088. Zhilu Zhang and Mert R. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 8792–8802, Red Hook, NY, USA, 2018. Curran Associates Inc. Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. In The Fourteenth International Conference on Learning Representations, 2026. URL https://openreview.net/ forum?id=nnwvwge40d.

Proofs for Section 2: Setup and Background

Proposition A.1 (RLVR connection). Under the conditional model of Section 2 and exact-match reward R(ŷ, y∗ ) = I(ŷ = y∗ ), the expected reward equals ED [Pθ ]; consequently J0 (θ) = 1 − ED [Pθ ], and minimizing J0 is equivalent to maximizing expected reward. Proof. For a fixed example (x∗ , y∗ ), [R(ŷ, y∗ )]

z∼pθ (·|x∗ ), ŷ∼pθ (·|x∗ ,z)

pθ (z | x∗ )

z∈Z, y∈Y

z∈Z

= Pθ . Taking an expectation over training examples from D, we have E

(x∗ ,y∗ )∼D z∼pθ (·|x∗ ), ŷ∼pθ (·|x∗ ,z)

[R(ŷ, y∗ )] =

(x∗ ,y∗ )∼D

[Pθ ].

Proofs for Section 3: Loss Landscape

Proposition B.1 (Dispersion penalty). For q > 0, JQ (θ, q) ≥ − logq (P̄ ), with equality if and only if Pθ is constant across all examples in D. 1−q

Proof. For q > 0, the function hq (u) = − logq (u) = 1−u 1−q h′′q (u) = q u−q−1 > 0. Applying Jensen’s inequality: JQ (θ, q) =

(x∗ ,y∗ )∼D

≥ hq

is strictly convex on (0, 1], since

[hq (Pθ )]

(x∗ ,y∗ )∼D

[Pθ ] = − logq (P̄ ),

with equality iff Pθ is constant across all examples. Theorem 3.1. P [Minimizers of JQ in the categorical model] For q ∈ (0, 1], the unique minimizer of JQ (θ, q) = j αj (− logq θj ) over ∆N is the escort distribution of order 1/q: 1/q αj ∗ θj (q) = PN , 1/q k=1 αk

j = 1, . . . , N.

(4)

For q = 0, the objective is linear and minimized at any vertex ej with j ∈ argmaxk αk . Proof. Case q ∈ (0, 1]. Since hq is strictly convex for q > 0, the objective is strictly convex on the interior of ∆N , and the minimizer is unique. Since all αj > 0, the minimizer lies in the interior (any boundary point has infinite loss for q P = 1 and suboptimal loss for q < 1), so we can use Lagrange multipliers for the equality constraint j θj = 1: −αj θj−q − λ = 0

=⇒

αj θj−q = µ

for all j,

P 1/q P where µ ≜ −λ > 0. Solving: θj = (αj /µ)1/q . The constraint j θj = 1 yields µ1/q = k αk , giving Equation (4). P Case q = 0. The objective JQ (θ, 0) = 1 − j αj θj is linear, minimized at any vertex ej with j ∈ argmaxk αk . Corollary B.2 (Endpoint behavior and monotone sharpening). Under the categorical model: 1. Density-estimation pole (q = 1): θj∗ (1) = αj . The model exactly recovers the data distribution. 2. Exploitation pole (q → 0+ ): assuming a unique mode j ∗ = argmaxk αk , θj∗ (q) → I(j = j ∗ ). The model concentrates all mass on the most frequent output. 3. Monotone sharpening: for 0 < q ′ < q ≤ 1 and αj > αk , θj∗ (q ′ )/θk∗ (q ′ ) > θj∗ (q)/θk∗ (q). Proof. Part (1): 1/q = 1. Part (2): (αj /αj ∗ )1/q → 0 for j ̸= j ∗ . Part (3): θj∗ /θk∗ = (αj /αk )1/q , increasing in 1/q. Corollary B.3 (Propriety). The Tsallis q-logarithmic scoring rule is strictly proper if and only if q = 1. 1/q

Proof. By Theorem 3.1, the maximizer of Ey∼α [logq (θy )] is θj∗ ∝ αj , which equals α iff q = 1. For q ∈ (0, 1) the true distribution α is not even a maximizer (the rule is not proper at all), let alone the unique one. The robustness counterpart under label noise — both static (where the escort minimizer concentrates) and dynamic (how fast the model gets there) — is deferred to Section C.5, after the gradient-flow machinery of Section 5.

Proofs for Section 5: Commitment Dynamics under Gradient Flow

C.1

Warm-up: exact analysis on the sigmoid model

Before proving the general results, we work through the scalar sigmoid model P (θ) = σ(θ) = (1 + e−θ )−1 as a warm-up. This model admits exact closed-form escape times that validate the Θ(·) bounds in Theorem 5.2. Under gradient flow on ℓq (θ) = − logq (σ(θ)), the parameter evolves as θ̇ = P (θ)−q P ′ (θ). Since P ′ (θ) = P (θ)(1 − P (θ)), the chain rule gives: ṗ = [P ′ (θ)]2 P (θ)−q = p2−q (1 − p)2 . This is a special case of the general dynamics (Equation (6)) with score norm ∥s(θ)∥2 = (1 − p)2 , which satisfies ∥s∥2 ∈ [(1 − δ)2 , 1] on p ∈ [p0 , δ] — confirming the bounded score assumption. The separable ODE gives the exact escape time: Z δ Tq (p0 , δ) =

du . 2−q (1 − u)2 u p0

(15)

We evaluate this integral using a dominant/remainder decomposition. Write (1 − u)−2 = 1 + r(u) 2u−u2 where r(u) = (1−u) 2 . On u ∈ [0, δ] with δ ≤ 1/2, we have 0 ≤ r(u) ≤ 8u. Substituting and distributing: Z δ Z δ du r(u) Tq (p0 , δ) = + du . 2−q 2−q u u p0 p0 | {z } | {z } dominant

remainder

−(1−q)

−δ −(1−q)

−(1−q)

Case q ∈ (0, 1). The dominant integral evaluates to 0 = 01−q (1 + o(1)). The 1−q R R q−1 8δ q −(2−q) remainder satisfies 0 ≤ r(u) u du ≤ 8 u du = q , a constant. So the remainder is −(1−q)

negligible and Tq =

p0 1−q

(1 + o(1)).

Case q = 0. The dominant integral gives p10 (1+o(1)). The remainder is O(log(1/p0 )), still negligible compared to 1/p0 . So T0 = p10 (1 + o(1)). R Case q = 1. The dominant integral is log(1/p0 ) + log δ. The remainder satisfies r(u) u−1 du ≤ 8(δ − p0 ) = O(1). So T1 = log(1/p0 )(1 + o(1)). Note that the sigmoid model yields exact 1 + o(1) asymptotics (not just Θ(·)) because ∥s∥2 = (1 − p)2 → 1 as p → 0, so the score norm converges to a known constant. This is stronger than the general theorem, which only assumes bounded score norms. C.2

Proof of Theorem 5.1: Exploitation is provably slow

Theorem 5.1. [Exploitation is provably slow] Let θ ∈ Rd parameterize any differentiable model. Consider gradient flow on ℓq (θ) = − logq (Pθ ), starting from p0 = Pθ(0) ∈ (0, 1/2] with fixed target δ ∈ (0, 1/2]. Suppose only that ∥s(θ(t))∥ ≤ C throughout the trajectory. Then as p0 → 0: ! −(1−q) p0 Tq (p0 , δ) = Ω for q ∈ [0, 1), 1−q 1 T1 (p0 , δ) = Ω log . p0 In particular, the exploitation pole cannot escape cold start faster than Ω(1/p0 ). Proof. From Equation (6), ṗ = p2−q ∥s(θ)∥2 ≤ C 2 p2−q . By the ODE comparison principle (since u 7→ u2−q is nondecreasing on (0, 1]), p(t) ≤ p∗ (t) where p∗ solves ṗ∗ = C 2 (p∗ )2−q with p∗ (0) = p0 . So p reaches δ no sooner than p∗ : Z δ 1 du Tq ≥ 2 . C p0 u2−q −(1−q)

For q ∈ [0, 1), the integral evaluates to

−(1−q)

−δ −(1−q) 1−q

p0 1−q

(1 + o(1)), giving Tq =

−(1−q) Ω(p0 /(1 − q)).

For q = 1, the integral is log(δ/p0 ) = log(1/p0 )(1 + o(1)), giving T1 = Ω(log(1/p0 )). C.3

Proof of Theorem 5.2: Tight cold-start escape rates

Theorem 5.2. [Tight cold-start escape rates] Under the same setup as Theorem 5.1, suppose additionally that ∥s(θ(t))∥ ≥ c > 0 throughout the trajectory. Then: 1. General q ∈ [0, 1): −(1−q)

p Tq (p0 , δ) = Θ 0 1−q

! as p0 → 0.

2. Density-estimation pole (q = 1): 1 T1 (p0 , δ) = Θ log p0

as p0 → 0.

3. Speedup ratio: for any q < q ′ with q ′ ≤ 1, Tq (p0 , δ) →∞ Tq′ (p0 , δ) 23

as p0 → 0.

Proof. The lower bound on time (Ω) follows from Theorem 5.1. For the upper bound, the additional assumption ∥s∥ ≥ c > 0 gives ṗ ≥ c2 p2−q ; by the ODE comparison principle, p(t) ≥ p∗ (t) where p∗ solves ṗ∗ = c2 (p∗ )2−q , so p reaches δ no later than p∗ : Z 1 δ du Tq ≤ 2 . c p0 u2−q −(1−q)

This integral evaluates to

p0 1−q

(1 + o(1)) for q ∈ [0, 1) and log(1/p0 )(1 + o(1)) for q = 1. −(1−q)

Combined with the lower bound, Tq = Θ(p0 Speedup ratio. For q < q ′ < 1: Tq /Tq′ −(1−q) Θ(p0 / log(1/p0 )) → ∞. C.4

/(1 − q)) for q < 1 and T1 = Θ(log(1/p0 )).

−(q ′ −q) = Θ(p0 ) → ∞. For q < 1 and q ′ = 1: Tq /T1 =

Near-optimality convergence (supplementary result)

Proposition C.1 (Near-optimality convergence is q-independent). Suppose that near optimality, ∥s(θ)∥2 depends on θ only through Pθ (i.e., ∥s(θ)∥2 = h(Pθ ) for some function h). Then for ϵ0 ≪ 1 and ϵ1 < ϵ0 , the time to improve from Pθ = 1 − ϵ0 to Pθ = 1 − ϵ1 satisfies Tq (1 − ϵ0 , 1 − ϵ1 ) = Tq′ (1 − ϵ0 , 1 − ϵ1 ) 1 + O(ϵ0 ) for all q, q ′ ∈ [0, 1]. That is, the convergence time is the same for all members of the JQ family up to a correction that vanishes as ϵ0 → 0. Proof. Write ϵ = 1 − p with ϵ ≪ 1. From Equation (6), ϵ̇ = −(1 − ϵ)2−q ∥s(θ)∥2 < 0. Since ϵ decreases over time, the convergence time from ϵ0 to ϵ1 is: Z ϵ0 dϵ Tq = . 2−q ∥s(θ)∥2 (1 − ϵ) ϵ1 ′

For any q, q ′ ∈ [0, 1], the integrands of Tq and Tq′ differ by the factor (1 − ϵ)q−q . We bound this factor on ϵ ∈ [ϵ1 , ϵ0 ] with ϵ0 ≪ 1. Using the Taylor expansion log(1 − ϵ) = −ϵ − ϵ2 /2 − · · · : ′

log(1 − ϵ)q−q = (q − q ′ ) log(1 − ϵ) 2 = (q − q ′ ) −ϵ − ϵ2 − · · · . Since |q − q ′ | ≤ 1: ′

log(1 − ϵ)q−q ≤ ϵ + ϵ2 + · · · = O(ϵ). ′

Exponentiating and using ex = 1+x+O(x2 ) = 1+O(ϵ) for x = O(ϵ), we get (1−ϵ)q−q = 1+O(ϵ). Since ϵ ≤ ϵ0 on [ϵ1 , ϵ0 ], the integrands of Tq and Tq′ differ by a multiplicative 1 + O(ϵ0 ) factor, giving Tq /Tq′ = 1 + O(ϵ0 ). C.5

Noise-fitting rate under symmetric label noise

The cold-start escape rates (Theorems 5.1 and 5.2) measure how fast the model commits to correct supervision under the JQ amplification Pθ−q . The symmetric question is how fast the model commits to incorrect supervision: the same amplification drives both, giving the following dynamical formulation of robustness under label noise. Noise-contamination setup. We work with a two-label categorical model, chosen to expose the mechanism in the simplest possible setting. For a single input x∗ , the model predicts one of two labels {c, k} with probabilities pθ (c | x∗ ) = p and pθ (k | x∗ ) = 1 − p, where p is a differentiable function of θ ∈ Rd . The target label is corrupted: with probability 1 − ϵ it equals the clean value c, and with probability ϵ ∈ (0, 1/2) it flips to the noise value k, giving α̃ = (1 − ϵ, ϵ). The restriction to two labels is cosmetic: in the N -label categorical model with symmetric noise α̃ = (1 − ϵ)α + ϵ · Unif, conditioning on the two-subset {j ∗ , k} containing the clean mode j ∗ and any fixed wrong label k reduces to this binary setting. 24

Let p(t) = pθ (c | x∗ ) denote the clean-mode probability under gradient flow on JQ (θ) = Ey∼α̃ [ℓq (pθ (y | x∗ ))], and let p̃(t) = 1 − p(t) denote the noise contamination. As in Section C, we assume bounded score: c∗ ≤ ∥s(θ(t))∥ ≤ C where s ≜ ∇θ log p is the score of the clean-mode probability (the analog of ∇θ log Pθ in Section 5). The escort asymptote. Differentiating J(p) = (1 − ϵ)ℓq (p) + ϵℓq (1 − p) gives J ′ (p) = −(1 − ϵ)p−q + ϵp̃−q . Gradient flow on a scalar parameterization of p yields p̃˙ = −ṗ = [ϵp̃−q − (1 − ϵ)(1 − p̃)−q ] p2 ∥s∥2 .

(16)

For q > 0, the dynamics have a unique stable equilibrium at p̃∗ (q) ≜ (ϵ/(1 − ϵ))1/q (1 + o(1))

as ϵ → 0,

(17)

′

obtained by solving J (p) = 0. This equilibrium coincides with the static escort minimizer from Theorem 3.1 applied to α̃: at q = 1, p̃∗ (1) = ϵ (the model fits observed noise exactly); as q → 0, p̃∗ (q) → 0 (the model concentrates on the clean mode, paralleling Corollary B.2). The escort is both where JQ is minimized (static) and where gradient flow converges (dynamic). The noise-to-clean ratio ϵp̃−q /[(1 − ϵ)(1 − p̃)−q ] is monotone decreasing in p̃ on (0, 1): it diverges as p̃ → 0 (noise term dominates near the clean mode), equals 1 at p̃ = p̃∗ (q) (equilibrium), and vanishes as p̃ → 1. So for p̃ ≪ p̃∗ (q) — the regime of small noise contamination — the noise term in Equation (16) dominates by an arbitrarily large factor. This drives the asymptotic scaling. Proposition C.2 (Noise-fitting rate). Fix q ∈ (0, 1]. Under the setup above, starting from p̃(0) = 0+ , the time Tqnoise (η) to reach noise contamination level p̃(Tqnoise ) = η satisfies, for η below the stable equilibrium (i.e. η ≪ p̃∗ (q); in particular as η → 0): q+1 η Tqnoise (η) = Θ . (18) (q + 1) ϵ ′

The speedup ratio for 0 < q < q ′ ≤ 1 diverges: Tqnoise (η)/Tqnoise (η) = Θ(η −(q −q) ) → ∞ ′ 0 as η → 0. At q = 0, adopting the convention p̃ ≡ 1, the dynamics Equation (16) reduce to p̃˙ = −(1 − 2ϵ) p2 ∥s∥2 < 0 everywhere (for ϵ < 1/2), so p̃ decreases from p̃(0) = 0+ and never reaches any η > 0: T0noise (η) = ∞. Proof. By the noise-to-clean monotonicity established above, for any K > 1 there exists p̃K (q) = K −1/q p̃∗ (q)(1 + o(1)) such that for p̃ ≤ p̃K , the noise term in Equation (16) exceeds K times the clean term. Combined with p = 1 − p̃ → 1 as p̃ → 0: 1 p̃˙ ∈ (1 − K ) ϵ c2∗ p̃−q (1 + o(1)), ϵ C 2 p̃−q . Fix any K > 1 (e.g., K = 2). Separating variables, p̃q dp̃ = Θ(ϵ) dt integrates to p̃q+1 /(q + 1) = Θ(ϵ t), giving Tqnoise (η) = Θ(η q+1 /((q + 1)ϵ)) for all η ≤ p̃K (q); taking η → 0 removes the ′ ′ constraint on K. For the speedup ratio, Tq /Tq′ = [η q+1 /(q + 1)]/[η q +1 /(q ′ + 1)] = Θ(η −(q −q) ), which diverges as η → 0 for q < q ′ . −(1−q)

Structural parallel with cold-start escape. Theorem 5.2 gives Tqescape (p0 ) = Θ(p0

/(1 −

′

−(q −q) q)) for q < 1 with speedup ratio Tq /Tq′ = Θ(p0 ). Proposition C.2 gives Tqnoise (η) = ′ Θ(η q+1 /((q + 1)ϵ)) with matching speedup ratio Θ(η −(q −q) ). The exponents in p0 (cold start) and

η (noise) differ by a constant shift, but the q-dependence of the speedup ratio is identical in form: the same Pθ−q amplification accelerates commitment to all supervision, clean or corrupted. Static mode-seeking (Corollary B.2) is recovered as the t → ∞ limit of Equation (16): p̃(t) → p̃∗ (q) → 0 as q → 0.

Proofs for Section 6: Monte Carlo Estimators

Theorem 6.1. [Consistency and bias expansion] Fix a supervised example (x∗ , y∗ ) and assume: 1. Pθ > 0; 25

2. E[∥gm ∥2 ] < ∞; 3. wm ≥ ϵ a.s. for some ϵ > 0. Then for any fixed q ∈ [0, 1], a.s. ∗ ∗ ∗ ∗ [ ∇ θ ℓq (q, θ; x , y , M ) −−−−→ ∇θ ℓq (θ, q; x , y ). M →∞

(10)

Moreover, for fixed Pθ > 0 and q ∈ [0, 1], the bias satisfies q M Pθq+1

h i [ E ∇ θ ℓq − ∇θ ℓq = O

! as M → ∞.

(11)

Proof. We write µw ≜ E[wm ] = Pθ ,

µg ≜ E[gm ] = ∇θ ℓ0 (θ; x∗ , y∗ ).

Define the smooth map f (a, b) ≜ b a−q , for a > 0. Then ∗ ∗ [ ∇ θ ℓq (q, θ; x , y , M ) = f (w̄M , ḡM ),

while the target gradient is ∇θ ℓq (θ, q; x∗ , y∗ ) = f (µw , µg ) = µg µ−q w . The almost sure convergence in Equation (10) follows from the Strong Law of Large Numbers, since w̄M → µw and ḡM → µg almost surely, and since f is continuous at (µw , µg ) because µw = Pθ > 0. For the bias expansion, we exploit the linearity of f in its second argument: f (a, b) = b a−q , so f (w̄M , ḡM ) = ḡM · h(w̄M ) = µg h(w̄M ) + (ḡM − µg ) h(w̄M ), {z } | {z } | first piece

second piece

where h(a) ≜ a−q is a scalar function whose derivatives h(k) (a) = (−q)(−q − 1) · · · (−q − k + 1) a−(q+k) depend only on a. First piece. Expand h(w̄M ) to third order around µw , with h′ (a) = −qa−q−1 , h′′ (a) = q(q + 1)a−q−2 , h′′′ (a) = −q(q + 1)(q + 2)a−q−3 : h(w̄M ) = h(µw ) + h′ (µw )(w̄M − µw ) + 12 h′′ (µw )(w̄M − µw )2 | {z } | {z } | {z } −q

E [·]=µw

E [·]=0

E [·]=

q(q+1) −q−2 Var(wm ) 2M µw

(1)

+ 16 h′′′ (µw )(w̄M − µw )3 + RM . | {z } |{z} E [·]=O(M −2 ) via κ3 /M 2

4th-order

Therefore: µg E[h(w̄M )] = µg µ−q w +

q(q + 1) µg µ−q−2 Var(wm ) w 2M (1)

+ O(M −2 ) + µg E[RM ]. 26

Second piece. The factor (ḡM − µg ) = Op (M −1/2 ), so a second-order expansion of h(w̄M ) suffices. Multiplying (ḡM − µg ) by each term of the expansion and taking expectations: E[(ḡM − µg ) h(w̄M )] = h(µw ) E[ḡM − µg ] + h′ (µw ) E[(ḡM − µg )(w̄M − µw )] | {z } | {z } =0

q =− M µ−q−1 Cov(gm ,wm ) w

+ 12 h′′ (µw ) E[(ḡM − µg )(w̄M − µw )2 ] + {z } | =O(M −2 ) via i.i.d. expansion

(2)

E[RM ] | {z }

3rd-order remainder

P −3

For the cross moment, expand E [(ḡM −µg )(w̄M −µw )2 ] = M i,j,k E [(gi −µg )(wj −µw )(wk − µw )]. By independence, the only nonzero index pattern is i = j = k (all others vanish because E [gi − µg ] = 0 or E [wj − µw ] = 0). The M surviving terms give E [(gm − µg )(wm − µw )2 ]/M 2 = O(M −2 ), since |(wm − µw )2 | ≤ 1 and E [∥gm ∥] < ∞ (Assumption 2). The remainder has the form (2) RM = (ḡM − µg ) · O(|w̄M − µw |3 ). Combining.

Adding the two pieces and substituting µw = Pθ , µg = ∇θ ℓ0 , ∇θ ℓ1 = ∇θ ℓ0/Pθ : h i ∗ ∗ ∗ ∗ [ E ∇ θ ℓq (q, θ; x , y , M ) = ∇θ ℓq (θ, q; x , y ) q + M Pθq+1 q+1 · ∇θ ℓ1 (θ; x∗ , y∗ ) Var(wm ) − Cov(gm , wm ) 2 + E[RM ], (1)

(2)

where RM = µg RM + RM . Remainder bound.

Write E [RM ] = E [RM · 1A ] + E [RM · 1Ac ] where A = {w̄M ≥ Pθ /2}.

On A. The derivatives of h are bounded on {a ≥ Pθ /2}: |h(k) (a)| ≤ Ck . (1)

(1)

For RM (the fourth-order scalar remainder), the integral form gives |RM | ≤ C4 |w̄M − µw |4 on A. (1) Since wm ∈ [0,1], E [|w̄M − µw |4 ] = O(M −2 ), so E [|RM | · 1A ] = O(M −2 ). (2)

For RM = (ḡM − µg ) · O(|w̄M − µw |3 ) on A (the third-order remainder from the secp p (2) ond piece), Cauchy–Schwarz gives E [|RM | · 1A ] ≤ C3 E [∥ḡM − µg ∥2 ] E [(w̄M − µw )6 ] = O(M −1/2 ) O(M −3/2 ) = O(M −2 ), using Assumption 2 and the boundedness of wm . On Ac . Assumption 3 gives w̄M ≥ ϵ > 0, so |h(w̄M )| ≤ ϵ−q everywhere and |f (w̄M , ḡM )| ≤ ϵ−q ∥ḡM ∥. Therefore |RM | ≤ |f (w̄M , ḡM )| + |TM | ≤ C ϵ−q (1 + ∥ḡM ∥), where TM collects the (bounded) Taylor terms. Again by Cauchy–Schwarz, q p −q E[|RM | · 1Ac ] ≤ C ϵ E[(1 + ∥ḡM ∥)2 ] P (Ac ). The first factor is O(1) by Assumption 2. For the second, since wm ∈ [0, 1] are i.i.d. with mean Pθ , Hoeffding’s inequality with t = Pθ /2 gives P (Ac ) = P (w̄M − Pθ ≤ −Pθ /2) ≤ exp(−M Pθ2 /2). Thus E [|RM | · 1Ac ] decays faster than any polynomial in M . Combining: E [RM ] = O(M −2 ), yielding Equation (11). D.1

RLOO control variate derivation

We derive the RLOO estimator (12) from the plug-in estimator (9). Using the chain rule, gm from (7) decomposes into a score-function term and a pathwise term: gm = − wm ∇θ log pθ (z(m) | x∗ ) − ∇θ wm . 27

(19)

Substituting into the plug-in estimator isolates the score-function component: [ ∇ θ ℓq

plug-in

M 1 X −wm ∇θ wm (m) ∗ ∇ log p (z . | x ) − θ θ M m=1 (w̄M )q (w̄M )q

(20)

Since E[∇θ log pθ (z(m) | x∗ )] = 0, we can subtract any baseline from the score-function coefficient −wm /(w̄M )q without changing the expected value, provided the baseline does not depend on z(m) . P We use a leave-one-out approximation. Let w̄¬m = M1−1 j̸=m wj . Replacing wm with w̄¬m in the coefficient, the batch mean collapses to w̄¬m , giving a surrogate coefficient of −(w̄¬m )1−q . Subtracting this baseline yields the RLOO estimator (12). Endpoint recovery. At q = 0, the centered weight evaluates to wm − w̄¬m , and the score-function term becomes −(wm − w̄¬m ) ∇θ log pθ (z(m) | x∗ ), exactly recovering the REINFORCE leaveone-out (RLOO) estimator standard in RLVR. At q = 1, the centered weight is wm /w̄M − 1; since PM m=1 (wm /w̄M − 1) = 0, this acts as a self-normalizing baseline that strictly centers the importance weights across the batch. Proposition D.1 (RLOO bias preservation, restated). Under the assumptions of Theorem 6.1, the RLOO estimator (12) satisfies the same bias expansion as the plug-in estimator (9). Proof. The RLOO estimator (12) differs from the plug-in estimator (20) by subtracting (w̄¬m )1−q from the score-function coefficient wm /(w̄M )q for each sample m. Denoting sm = ∇θ log pθ (z(m) | x∗ ), the difference in expectations is M

∆=

1 X 1−q sm ]. E[(w̄¬m ) M m=1

P Since w̄¬m = M1−1 j̸=m wj is a function of {z(j) }j̸=m only, and sm = ∇θ log pθ (z(m) | x∗ ) is a function of z(m) only, the independence of the i.i.d. samples gives 1−q sm ] = E[(w̄¬m )1−q ] · E[sm ] = 0, E[(w̄¬m ) | {z } =0

where E [sm ] = Ez∼pθ [∇θ log pθ (z | x∗ )] = 0 is the standard score-function identity. Therefore ∆ = 0 and the two estimators have identical expectations for every M . D.2

Endpoint recovery

Proposition D.2 (Endpoint recovery for GARL and PAFT). Fix a supervised example (x∗ , y∗ ) with Pθ > 0. 1. GARL at q = 0 recovers Rao–Blackwellized REINFORCE [Williams, 1992, Zhou et al., 2026]: M 1 X [ −wm ∇θ log pθ (z(m) , y∗ | x∗ ) , ∇ ℓ = ḡ = θ q q=0 M M m=1

which is unbiased for ∇θ ℓ0 by Equation (8). Each gm marginalizes out the output y given z(m) analytically via wm = pθ (y∗ | x∗ , z(m) ), rather than relying on a sampled output and binary reward. 2. GARL at q = 1 recovers the IWAE gradient estimator [Burda et al., 2015], a self-normalized importance sampling (SNIS) estimator for ∇θ log Pθ : P log pθ (z(m) , y∗ | x∗ )) ḡM m wm (−∇θP [ ∇ ℓ = = . θ q q=1 w̄M m wm 28

3. PAFT at q = 0 reduces to posterior-resampled SFT scaled by Pθ : 1 ˆ PAFT ∇ = −w̄M · q=0 K

K X

∇θ log pθ (z(rk ) , y∗ | x∗ ).

k=1

The factor w̄M ≈ Pθ downweights hard instances so aggressively that this endpoint is overly conservative in practice. Unlike the other three endpoints, it does not correspond to a standard method. 4. PAFT at q = 1 recovers the E-step of EM [Dempster et al., 1977] / TRICE [Phan et al., 2023]: 1 ˆ PAFT ∇ =− q=1 K

K X

∇θ log pθ (z(rk ) , y∗ | x∗ ).

k=1

1−1

The instance weight (w̄M ) = 1 vanishes: all instances contribute equally, and the gradient is uniform SFT on approximate posterior samples. Proof. Each case follows by substituting q = 0 or q = 1 into the GARL estimator (9) or PAFT estimator (14) and simplifying (w̄M )0 = 1. D.3

PAFT bias and variance

Proposition D.3 (PAFT has the same bias as GARL). Under the assumptions of Theorem 6.1, ˆ PAFT ] = E[∇ ˆ GARL ] for all M . In particular, the PAFT estimator has the same O(q/M Pθq+1 ) bias E[∇ as in Equation (11). 1−q Proof. Conditional on the prior samples pool = {(z(m) , wm )}M is determinm=1 , the factor (w̄M ) istic. The importance-resampled average satisfies # " M K X w 1 X (rk ) P m f (z(m) ) = µ̂SNIS , E f (z ) pool = K j wj m=1 k=1

where f (z) = ∇θ log pθ (z, y∗ | x∗ ). Therefore ˆ PAFT | pool] = −(w̄M )1−q · µ̂SNIS E[∇ P wm fm = −(w̄M )1−q · m M w̄M 1 1 X · = (−wm fm ) (w̄M )q M m =

ḡM ˆ GARL . =∇ (w̄M )q

ˆ PAFT ] = E[∇ ˆ GARL ]. Taking outer expectations by the tower property: E[∇ Proposition D.4 (GARL has strictly lower variance than PAFT). Under the same setup, ˆ PAFT ) ≥ Var(∇ ˆ GARL ), with equality only when Var(∇ ˆ PAFT | pool) = 0 almost surely. Var(∇ ˆ PAFT | pool] = ∇ ˆ GARL . The law of total variance gives Proof. By Proposition D.3, E[∇ ˆ PAFT ) = Var E[∇ ˆ PAFT | pool] + E Var(∇ ˆ PAFT | pool) Var(∇ ˆ GARL ) + E Var(∇ ˆ PAFT | pool) , = Var(∇ | {z } ≥0

ˆ PAFT | pool) = 0 a.s. This holds when, for each pool realization, all with equality iff Var(∇ resampled trajectories produce the same gradient — e.g., when a single trajectory dominates the importance weights. In the non-degenerate case, the inequality is strict.

Record · ID 141466 · SHA-256 7a03c6f112af1f5e
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.