arXiv CSopen access

Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning

Kernelized Advantage Estimation: From Nonparametric Statistics to LLM Reasoning Shijin Gong1,∗

Kai Ye2,∗

Jin Zhu3

arXiv:2604.28005v1 [cs.LG] 30 Apr 2026

Hongyi Zhou5,†

Xinyu Zhang4,1

Chengchun Shi2

Abstract Recent advances in large language models (LLMs) have increasingly relied on reinforcement learning (RL) to improve their reasoning capabilities. Three approaches have been widely adopted: (i) Proximal policy optimization and advantage actor-critic rely on a deep neural network to estimate the value function of the learning policy in order to reduce the variance of the policy gradient. However, estimating and maintaining such a value network incurs substantial computational and memory overhead. (ii) Group relative policy optimization (GRPO) avoids training a value network by approximating the value function using sample averages. However, GRPO samples a large number of reasoning traces per prompt to achieve accurate value function approximation, making it computationally expensive. (iii) REINFORCE-type algorithms sample only a single reasoning trajectory per prompt, which reduces computational cost but suffers from poor sample efficiency. In this work, we focus on a practical, resource-constrained setting in which only a small number of reasoning traces can be sampled per prompt, while low-variance gradient estimation remains essential for high-quality policy learning. To address this challenge, we bring classical nonparametric statistical methods, which are both computationally and statistically efficient, to LLM reasoning. We employ kernel smoothing as a concrete example for value function estimation and the subsequent policy optimization. Numerical and theoretical results demonstrate that our proposal achieves accurate value and gradient estimation, leading to improved policy optimization.

1

Introduction

Large language models (LLMs) have achieved remarkable success across a wide range of tasks. From early developments in large-scale pretraining, which aims to model the distribution of human language for next-token prediction, to later improvements through task-specific post-training (Ouyang et al., 2022), the evolution of LLMs has progressively shifted from merely “speaking well” to “problem solving”. The emergence of large reasoning models has further accelerated this trend, demonstrating human-level or superhuman performance in complex tasks such as mathematical problem solving and code generation (Jaech et al., 2024). As detailed in Section 1.1, progress in ∗

Equal contribution. Corresponding author. 1 School of Management, University of Science and Technology of China; Email: [email protected]. 2 Department of Statistics, London School of Economics and Political Science; Emails: [email protected], [email protected]. 3 School of Mathematics, University of Birmingham; Email: [email protected]. 4 Academy of Mathematics and Systems Science, Chinese Academy of Sciences; Email: [email protected]. 5 Department of Mathematics, Tsinghua University; Email: [email protected]. †

1

reasoning has evolved from early prompting-based approaches (Wei et al., 2022), which explicitly encourage models to generate intermediate reasoning steps without additional training, to more recent large-scale reinforcement learning (RL)–based methods that directly retrain the model by exploring various reasoning trajectories to optimize its outcome (Lambert et al., 2024). Such RL-based reasoning algorithms are mostly policy gradient algorithms (Sutton and Barto, 2018, Chapter 13). These methods, with REINFORCE (Williams, 1992) as a prototype, are stochastic gradient ascent algorithms (Robbins and Monro, 1951) in nature that update model parameters by estimating the gradient of the expected outcome under the current policy. In practice, this gradient is approximated by sampling reasoning trajectories from the policy. When tackling complex problems, the model becomes highly uncertain about how to reason, leading to large variability across sampled trajectories. This variability, in turn, introduces substantial noise into the gradient estimation. Consequently, a central objective in policy gradient RL is to obtain sample efficient gradient estimates. There are two dominant approaches to reducing the variance of gradient estimation. The first type of approaches, including proximal policy optimization (PPO, Schulman et al., 2017) and advantage actor-critic (A2C, Mnih et al., 2016), reduces variance by modeling a value function parameterized by a separate deep neural network in addition to the LLM itself, to serve as a baseline. While sample efficient, these methods require training and storing the deep value network, which incurs substantial computational cost and memory overhead for reasoning tasks. The second type of approaches, represented by group relative policy optimization (GRPO, Shao et al., 2024) and its variants such as GSPO (Zheng et al., 2025a), Dr. GRPO (Liu et al., 2025e) and GPG (Chu et al., 2025), eliminates the value network completely. However, these approaches require generating a sufficiently large number of reasoning trajectories per prompt to accurately approximate the value function, which can be computationally expensive in practice. This paper considers a resource-constrained setting in which training a separate value network is computationally infeasible, rendering PPO- or A2C-type algorithms impractical. While GRPOtype algorithms remain applicable, only a limited number of reasoning trajectories can be sampled per prompt. As a result, the variance of the resulting gradient estimator can be substantial, which in turn lowers the quality of the learned policy (Greensmith et al., 2004). Such settings are common in universities, small research labs, and public sectors with limited computational resources. To address this challenge, we take a different perspective. Instead of relying on deep value networks or large-scale sampling, we draw inspiration from classical nonparametric statistical methods, which are doubly efficient, both computationally and statistically. More specifically, we leverage nonparametric statistics to enhance value function estimation, employ kernel smoothing (Nadaraya, 1964) as a concrete proposal, and incorporate the resulting estimates into RL-based policy optimization. Most notably, the proposed methodology enjoys an “oracle” property (see Figure 1 for an illustration): its performance on test data is comparable to that of an oracle algorithm that knows the oracle value function in advance and uses it for gradient estimation. This is particularly remarkable because, under limited computational budgets, we cannot afford to train complex neural networks for value estimation, nor can we sample a large number of trajectories to approximate it. Nevertheless, our method achieves performance comparable to that of an oracle with full access to the true value function. We validate our proposal both theoretically and empirically. Theoretically, we establish the oracle property of our method and demonstrate its advantages over GRPO- and REINFORCE-type algorithms along three dimensions: the mean squared error (MSE) of the value function estimator (Theorem 1 & Corollary 1), the MSE of the gradient estimator (Theorem 2 & Corollary 2), and the performance of the learned policy (Theorem 3 & Corollary 3). Empirically, we conduct extensive 2

Figure 1: Expected rewards of one-shot GRPO (Wang et al., 2025b), the oracle algorithm, and our method (denoted as KAE) on training (left) and testing (right) datasets in the one-shot regime where the training data consists of a single observation. One-shot GRPO applies the standard GRPO algorithm directly to this regime. Shaded areas represent confidence intervals.

experiments to validate these advantages over both GRPO- and REINFORCE-type algorithms in value estimation (Table 1), gradient estimation (Table 2) and policy optimization (Tables 3 and 4, Figure 4). Additionally, ablation studies confirm that these gains stem from improved value function estimation (Figures 4 and S.1). Finally, sensitivity analysis shows that our value estimator’s MSE is not overly sensitive to the choice of kernel bandwidth and kernel function (Figure 3).

1.1

Related Work

Our work sits at the intersection of artificial intelligence (AI) and statistics. On the AI side, it is closely related to the rapidly growing literature on LLM reasoning. On the statistics side, it connects to classical work on nonparametric estimation and modern work on RL. LLM reasoning. LLM reasoning methods can be broadly grouped into three categories: (1) prompting-based approaches; (2) inference-time1 approaches that enhance reasoning through search; and (3) training-time approaches via alignment or RL. Early work falls into the first two categories. In particular, prompting-based methods such as chain-of-thought prompting guide LLMs to produce step-by-step reasoning in a manner similar to humans (Wei et al., 2022). This is often achieved by simple, magical prompts such as “Let us think step by step.” Under such instructions, the model generates a chain of thought that decomposes a complex problem into a sequence of intermediate subproblems and produces the reasoning process before arriving at a final answer. Instead of generating a single reasoning trajectory, inference-time approaches explore multiple candidate reasoning paths and select the best one as the final output (Zhang et al., 2026; Zhu et al., 2026). More recent work has increasingly focused on training-time approaches, such as reinforcement learning from verifiable rewards (RLVR), which fine-tune model parameters using RL to directly enhance LLM reasoning capabilities (see, e.g., Lambert et al., 2024; Hu et al., 2025; Dai et al., 2025; Huang et al., 2026). These approaches are most closely related to our proposal. They rely on policy gradient algorithms such as REINFORCE, A2C and PPO, which estimate the gradient of the expected return of the policy using sampled trajectories and update the model parameters via stochastic gradient ascent. However, REINFORCE is well known to suffer from high variance in its gradient estimates. PPO and A2C mitigate this limitation by using the advantage function, 1 Here, “inference-time” refers to the stage at which a trained model is applied without retraining. This differs from the notion of statistical inference, which studies e.g., uncertainty quantification for an estimator.

3

defined as the difference between the return and the value function of the policy, in place of the return when constructing the gradient estimator. These methods estimate the value function via a deep neural network. However, as mentioned earlier, maintaining and updating such a value network can be computationally intensive. A major breakthrough in this line of work is GRPO, the post-training algorithm underlying large reasoning models such as DeepSeekMath (Shao et al., 2024) and DeepSeek-R1 (Guo et al., 2025), as well as a number of open-source LLMs (Liu et al., 2025d; Yang et al., 2025). Its main idea is closely related to that of Kool et al. (2019): for each prompt, multiple reasoning trajectories are sampled, leading to multiple rewards, which are averaged to approximate the value function for variance reduction. In contrast to PPO and A2C, GRPO eliminates the need for learning a separate value network, which facilitates the computation while maintaining statistically efficient policy optimization. GRPO has inspired a large number of follow-up methods (e.g., Hao et al., 2025; Li et al., 2025a; Lin et al., 2025; Xiong et al., 2025; Yan et al., 2026; Zhao et al., 2025; Zheng et al., 2025b; Li et al., 2026a,b). Our work is most closely related to those that focus on improving the statistical or sample efficiency of value and advantage estimators. Among those available, Zeng et al. (2025) and Han et al. (2026) propose shrinkage estimators that replace GRPO’s per-prompt reward average with a combination of per-prompt and cross-prompt averages. The intuition behind these methods is closely related to the James–Stein estimator (James et al., 1961) in classical statistics. Unlike these approaches, which borrow information across prompts within a single training iteration to improve estimation accuracy, our proposal borrows information across training iterations for the same prompt (see Section 3). Wang et al. (2025a) and Xu and Ding (2025) adopt similar ideas and employ Kalman filtering or Bayesian methods to smooth rewards across training iterations. Our proposal differs from these methods in three respects. Methodologically, we employ kernel smoothing rather than Kalman filtering, and we additionally design the prompt sampling schedule to further improve value and advantage estimation accuracy (see Section 5). Theoretically, we establish learning guarantees for the value estimator, the gradient estimator, and the resulting policy. These theoretical results are largely absent from the aforementioned work. Nonparametric statistics and RL. Nonparametric statistical methods estimate regression functions without imposing restrictive parametric assumptions. Classical approaches include kernel smoothing, local polynomial regression (Stone, 1977), and sieve estimators (Grenander, 1981). Statistically, these estimators can achieve the optimal convergence rates established by Stone (1982); see, e.g., Fan et al. (1997); Huang (1998); Chen (2007). Computationally, these algorithms are much more efficient to implement compared to deep neural networks. This makes them well suited for value function estimation in LLM reasoning under resource constraints. In this work, we employ kernel smoothing as a concrete example, although the framework naturally extends to other nonparametric estimators. More recently, there has been growing interest in developing RL algorithms in the statistics literature. These methods can be broadly categorized into three classes: (i) approaches designed for learning optimal dynamic treatment regimes without imposing Markov assumptions on the data generating process (see, e.g., Chakraborty and Moodie, 2013; Laber et al., 2014; Kosorok and Laber, 2019; Tsiatis et al., 2019; Ge et al., 2025; Gazi et al., 2026, for reviews); (ii) approaches developed for Markov decision processes (MDPs) under the Markov assumption (e.g., Ertefaie and Strawderman, 2018; Luckett et al., 2020; Liao et al., 2022; Wang et al., 2023; Chen et al., 2024; Li et al., 2024; Shi et al., 2024b; Zhou et al., 2024b,a; Bian et al., 2025; Chai et al., 2025; Jin et al., 2025; Li et al., 2025b; Miao et al., 2025; Liu et al., 2025c; Zhong et al., 2025); and (iii) approaches tailored to RL from human or AI feedback (e.g., Lee et al., 2024; Liu et al., 2025a,b; Lu et al., 2025; Xiao et al., 2025; Cho and Sun, 2026; Liu et al., 2026; Xia et al., 2026). Our work is related to those 4

methods based on A-learning, which focus on estimating advantage (contrast, or blip) functions for optimal policy learning (Murphy, 2003; Robins, 2004; Lu et al., 2013; Shi et al., 2018; Liang and Yu, 2023; Shi et al., 2024a). However, we study a fundamentally different application in LLM reasoning, leading to substantially different methodologies.

2

Preliminaries: RLVR for LLM Reasoning

We adopt a contextual bandit framework (see, e.g., Lai and Robbins, 1985) to formulate the RLVR problem in LLM reasoning. At each time step, the LLM receives a user query, referred to as a prompt X. To address this query, the LLM generates a reasoning trajectory along with a final answer. Together, the reasoning trajectory and the answer form a completion Y . Both X and Y are represented as sequences of tokens via tokenization, where each token (e.g., word, subword, or punctuation) is mapped to a unique integer according to a vocabulary that collects all possible tokens. The pair (X, Y ) is then evaluated by a verifiable reward function r, yielding a scalar reward Z = r(X, Y ). In mathematics, each problem typically has a unique correct solution, and the reward can be defined as 1 if the generated answer matches the ground truth and 0 otherwise. Similarly, in coding tasks, LLM-generated programs can be executed to verify whether they pass the corresponding test cases. Leading LLMs, such as GPT, Gemini, and Claude, are all autoregressive models. From an RL perspective, they can be viewed as policy networks, denoted by πθ , parameterized via the Transformer architecture. Specifically, given an input token sequence x of arbitrary length, πθ (•|x) defines a probability mass function over the vocabulary, representing the distribution of the next token. A completion is generated autoregressively: the model first takes X as input and produces the first token Y1 , then conditions on the concatenated sequence (X, Y1 ) to generate Y2 , and proceeds iteratively until an end-of-sequence token YT is produced. This yields the full completion Y = (Y1 , . . . , YT )⊤ . The objective of RLVR is to identify the optimal parameter θ∗ that maximizes the expected reward: θ∗ = arg max J(θ) where J(θ) := Eπθ (Z),

θ∈Θ π θ where the expectation E is taken with respect to the distribution over completions induced by

the policy πθ . As discussed in Section 1.1, existing RLVR algorithms are policy gradient methods. They are motivated by the observation that the gradient of J(θ) can be expressed as   ∇θ J(θ) = Eπθ Z∇θ log πθ (Y |X) , that is, as the expectation of the product of the reward Z and P the score function ∇θ log πθ (Y |X) (referred to as the policy score in RL), where log πθ (Y |X) = Tt=1 log πθ (Yt |X, Y1 , · · · , Yt−1 ). This representation gives birth to the following stochastic gradient ascent (SGA) algorithm: at each (b) is generated for each iteration, a minibatch of prompts {X (b) }B b=1 is sampled, a completion Y (b) B prompt, rewards {Z }b=1 are obtained via the verifiable reward function, and the corresponding policy scores are computed. The gradient is then estimated by averaging the products of rewards and policy scores across the sampled batch, denoted by gb(θ), and the policy parameter is updated via θ ← θ + ηb g (θ) for some learning rate parameter η. When confined to the classical MDP setting, such an algorithm is known as REINFORCE. However, REINFORCE’s gradient estimators are well known to suffer from high variance, which lowers the quality of the learned policy. Three major approaches have been developed in the literature to address this limitation:

5

Figure 2: Illustrations of a generic algorithm that unifies A2C, REINFORCE- and GRPO-type algorithms. 1. The first approach is A2C, which introduces a critic function C(X) to serve as a baseline and replaces the reward Z with an advantage function A = Z − C(X) in constructing the policy gradient estimator gb(θ). Its main idea is that ∇θ log πθ (Y |X) is a score function, and thus multiplying it by any C(X) yields a vector with zero expectation. Consequently, subtracting C(X) does not bias the gradient estimator. However, by choosing an appropriate baseline, its variance can be substantially reduced. Under an uncorrelatedness condition introduced in Section 4, the optimal baseline that minimizes the variance is the value function V πθ (X) (Greensmith et al., 2004). This motivates A2C, which maintains a separate value network Vb to approximate the value function and uses the plug-in estimator Z − Vb (X) as the advantage for constructing the policy gradient. 2. The second approach is REINFORCE++ (Hu et al., 2025), which replaces the critic with a simple baseline, given by the average reward across all prompts at the current training step, i.e., P (b) , to eliminate the need for the value network and facilitate the computation. Z Z̄ = B −1 B b=1 3. The third approach is GRPO-type algorithms, which sample a group of completions {Y (b,g) }G g=1 for each prompt X (b) to eliminate the need for a value network. Specifically, for each completion Y (b,g) , the corresponding reward is computed as Z (b,g) = r(X (b) , Y (b,g) ), and the within-group P G average Z̄ (b) = G−1 g=1 Z (b,g) is used as a proxy for the value function. This leads to an advantage function of the form A(b,g) = Z (b,g) − Z̄ (b) , yielding a variance-reduced policy gradient estimator without requiring a learned value network. Figure 2 unifies the three approaches by introducing a generic baseline for advantage function estimation and illustrates the resulting gradient estimator. When a single completion is sampled per prompt and the baseline is set to the empirical average of rewards across all prompts, the method reduces to REINFORCE++. When the baseline is given by a learned value network, it recovers A2C. Finally, when a group of completions are sampled and the baseline is set to be the empirical group mean, it corresponds to GRPO-style algorithms. Despite their statistical efficiency, both A2C and GRPO incur substantial computational costs. A2C is computationally intensive due to the requirement to maintain and update a separate value network. In contrast, GRPO avoids a learned critic but relies on sampling multiple completions 6

per prompt to obtain a sufficiently accurate gradient estimator; in practice, this group size can be large (e.g., G = 64 in the DeepSeekMath paper). REINFORCE++, on the other hand, reduces computational cost by sampling only a single completion per prompt. However, its baseline aggregates rewards across all prompts and is a biased estimator of the value function for a specific prompt. Consequently, it lowers the statistical efficiency. In summary, achieving both statistical and computational efficiency in gradient estimation remains challenging. This is precisely where nonparametric statistical methods can play a central role in achieving both types of efficiency, and it motivates the proposed methodology.

3

Kernelized Advantage Estimation

We first present the motivation of the proposed algorithm. We focus on a resource-constrained setting where the number of completions that can be sampled per prompt is limited. As a result, GRPO’s value estimator, constructed via averaging rewards over multiple completions, suffers from large variance. To address this limitation, we propose to borrow information across different training iterations to improve the estimation accuracy of the value function, and consequently, the advantage function. The motivation for our algorithm is best illustrated in the one-shot regime, where the training data consists of a single prompt x. Surprisingly, recent work suggests that LLMs can still generalize well even in such regimes (Wang et al., 2025b). One possible explanation is that, although there is only one data example, it consists of many tokens, so the effective sample size is not necessarily one. We do not pursue this phenomenon further here; instead, we use this setting to illustrate our main idea. In the one-shot regime, because the same prompt x appears repeatedly over training, rewards collected at previous iterations also contain information about the current value function. Therefore, rather than estimating V πθi (x) solely from the current batch, we may borrow information from past rewards. Of course, historical rewards cannot be pooled naively. Since the policy parameter θi π evolves over training, its value function V πθi (x) is generally different from V θj (x) for j < i. Thus, older rewards should contribute less to the current estimate than more recent rewards. This leads naturally to a one-dimensional nonparametric regression problem: We treat the training iteration index i as the predictor and the corresponding value function V πθi (x) as the target. The goal is to estimate V πθi (x) by combining rewards observed at both the current iteration and previous iterations. In this way, we connect value and advantage estimation in LLM reasoning to classical nonparametric regression, allowing us to leverage classical nonparametric statistical methods to borrow information over time. In this paper, we adopt kernel smoothing as a concrete nonparametric method to implement this idea. For simplicity, suppose that at each training iteration i we sample R ∞a single completion and let Zi denote its reward. Let K(·) be a kernel function that satisfies 0 K(u)du = 1. We estimate the current value function by a kernel-weighted average of these rewards,   i−1 1 X i−j b Vi (x) = K Zj , ih ih j=0

where h > 0 is a bandwidth parameter. Due to the use of kernel smoothing for value estimation, and the resulting construction of the advantage estimator, we refer to our method as kernelized advantage estimation (KAE). In more general regimes, prompts used in LLM posttraining are often drawn from a fixed set (e.g., a collection of mathematical questions), denoted by X = {x1 , x2 , · · · , xm }. Consequently, the same prompts may be encountered multiple times 7

Algorithm 1 Kernelized Advantage Estimation (KAE). Prompt set X , initial parameter θ0 ∈ Θ, learning rates {ηi }n−1 i=0 , batch size B, perprompt group size G, kernel function K(·), bandwidth h, and sets of historical rewards Hi (x) = ∅ for any i < n and x ∈ X . 2: for i = 0, 1, 2, . . . , n − 1 do 3: Sample a minibatch of prompts {X (b) }B b=1 from X . 4: for b = 1, . . . , B do (b) 5: Generate a group of completions {Y (b,g) }G g=1 ∼ πθi (•|X ). 6: Compute rewards Z (b,g) = r(X (b) , Y (b,g) ) for g = 1, . . . , G. 7: if Hi (X (b) ) = ∅ then P (g) 8: Set Vbi (X (b) ) = (G − 1)−1 k̸=g Z (b,k) for g = 1, · · · , G. 9: else 1: Input:

10: (g) Vbi (X (b) ) =

11: 12: 13: 14:

h 1 Mi (X (b) )

X

K

(Ij ,Zj )∈Hi (X (b) )

i − I  j

ih

Zj +

X

i K(0)Z (b,k) ,

k̸=g

where Mi (X (b) ) denotes the normalizing constant h|Hi (X (b) )| + (G − 1)K(0). end if (g) For each g = 1, . . . , G, define the advantage estimate A(b,g) = Z (b,g) − Vbi (X (b) ). end for Estimate the policy gradient: B

G

1 X X (b,g) gbKAE (θi ) = A ∇θ log πθi (Y (b,g) |X (b) ). BG b=1 g=1

15: 16:

Update the policy parameter θi+1 ← θi + ηi gbKAE (θi ). Update the sets of historical rewards: Hi+1 (X (b) ) ← Hi (X (b) ) ∪ {(i, Z (b,g) ) : g = 1, . . . , G},

b = 1, . . . , B.

17: end for 18: Output: πθn .

during training, and past rewards remain informative when estimating the value function under the current prompt. We summarize the complete procedure in Algorithm 1 and highlight some main steps below. Consistent with Section 2, we sample a minibatch of B prompts {X (b) }B b=1 at each training iteration (Line 3), generate G completions {Y (b,g) }G for each prompt (Line 5), and obtain their g=1 (b,g) G rewards {Z }g=1 . On Lines 7 and 8, the value function is estimated in a leave-one-out manner similar to Kool et al. (2019). Specifically, to construct the advantage estimator A(b,g) for the gth completion, we use all other rewards in the current group (excluding the gth), together with historical rewards associated with the same prompt. Although the gth reward could also be incorporated, doing so would only change the advantage estimator by a multiplicative constant asymptotically. Given these advantage estimates, the policy gradient is computed on Line 12 by averaging the product of the advantage and the policy score over all prompt–completion pairs. Finally, on Line 13, the model parameters are updated via stochastic gradient ascent using the learning rate ηi . 8

4

Theoretical Results

In this section, we establish the statistical properties of KAE. We begin with a high-level summary of our theoretical findings. As mentioned earlier, our theories are organized along three dimensions: (i) value estimation; (ii) gradient estimation and (iii) policy optimization. Their connections highlight the motivation behind KAE. Summary. KAE is designed to improve value estimation. We first establish an MSE bound for the proposed value estimator (Theorem 1) and show that, under an appropriate choice of kernel bandwidth, it achieves Stone’s optimal rate of convergence (Corollary 1). We next show that improvements in value estimation directly translate into improved gradient evaluation: the MSE of our gradient estimator differs from that of a corresponding oracle algorithm, one that is identical to Algorithm 1 except that it uses the true value function to construct the advantage and the resulting policy gradient, only by an extra term proportional to the MSE of the value estimator (Theorem 2). Finally, we show that improved gradient estimation further translates into better policy optimization. In particular, the upper bound on the suboptimality gap of the learned policy, defined as the difference in expected value between the optimal policy and the learned policy, depends directly on the MSE of the gradient estimator (Theorem 3). Taken together, these results establish the oracle property of KAE and imply its advantages over REINFORCE- and GRPO-type algorithms along all three dimensions (Corollaries 1 – 3). Next, we impose the following technical conditions. Assumption 1 (I.i.d. sampled prompts). The minibatches of prompts {X (b) }B b=1 are sampled i.i.d. across training iterations. At each iteration, a minibatch is drawn uniformly at random from X without replacement, and independently of previously sampled minibatches. Assumption 2 (Bounded reward). Z is almost surely bounded. Assumption 3 (Kernel on [0, 1], bounded, and of order s for some R1 R 1function). K is supported integer s > 0 such that 0 K(u)du = 1 and 0 uj K(u)du = 0 for any j = 1, · · · , s − 1. Assumption 4 (Smoothness). For any x ∈ X , V πθ (x) is p-times continuously differentiable, for some integer p ≥ 2, with uniformly bounded derivatives up to order p. The i.i.d. assumption in Assumption 1 is imposed for simplicity. In practice, prompts may be sampled dependently across iterations, as in our implementation (see Section 5). Assumption 2 is mild. In LLM reasoning, rewards are bounded, for example, as a binary indicator of the correctness of the model’s output, potentially with an additional format reward that encourages the output to follow a prescribed form. Conditions similar to Assumption 3 are standard in kernel smoothing. When R 1 j s > 1, the kernel must take negative values on its support to satisfy the moment conditions 0 u K(u)du = 0. The smoothness condition (Assumption 4) is also standard in kernel smoothing (Wand and Jones, 1994). The following theorem upper bounds the bias and variance of the KAE value estimator. Theorem 1 (Bias and variance of value estimator). Under Assumptions 1 – 4, for any x that is sampled at the ith iteration, 1  1  (g) (g) (g) Bias(Vbi (x)) := E[Vbi (x)] − V πθi (x) = O(hp∧s ) + O , Var(Vbi (x)) = O , ih Ni (x)h where p ∧ s = min(p, s), and Ni (x) is the sample size for estimating V πθi (x), given by G times the number of past occurrences of x plus G − 1 current samples (due to leave-one-out). The bias and variance are computed conditional on x being sampled at iteration i and on the past sampling history, under which Ni (x) is treated as fixed. 9

Our results in Theorem 1 are generally consistent with classical kernel smoothing theory, with the exception of an additional bias term of order O((ih)−1 ). This term arises from approximating a continuous kernel integral by a discrete sum over the iteration index. Nevertheless, this term is of the same order as the variance under the resource-constrained setting where Ni (x) ≤ Gi = O(i), and its square is therefore of higher order. Corollary 1 (Optimality of value estimator). Suppose s ≥ p and the kernel bandwidth h is proportional to [Ni (x)]−1/(2p+1) . Under the conditions in Theorem 1 and in the resource-constrained setting where G is fixed, KAE’s value estimator achieves Stone’s optimal convergence rate, i.e., (g) MSE(Vbi (x)) = O([Ni (x)]−2p/(2p+1) ). In contrast, neither the GRPO nor the REINFORCE++ value estimator is consistent in this resource-constrained setting. Corollary 1 establishes that KAE’s value estimator achieves the optimal (Stone’s minimax) rate of convergence. In contrast, the value estimators of GRPO and REINFORCE++ are both inconsistent in this resource-constrained setting. It thus formally verifies the advantage of KAE over both baseline algorithms in value estimation. We next analyze KAE’s gradient estimator gbKAE (θ); refer to its definition in Algorithm 1. We begin with the following condition on the policy score. Assumption 5 (Policy score). E∥∇θ log πθ (Y (b,g) |X (b) )∥2 is uniformly bounded for any θ ∈ Θ. Additionally, Z (b,g) is uncorrelated with ∥∇θ log πθ (Y (b,g) |X (b) )∥2 given X (b) . Assumption 5 is mild. Its first part is weaker than requiring uniform boundedness of the policy score, as it only assumes the boundedness of its second moment. Its second part guarantees that the value function serves as the optimal baseline for minimizing the variance of the policy gradient estimator (Greensmith et al., 2004). It justifies PPO and A2C, which use a value network to estimate the value function, as well as the proposed nonparametric statistical approach. Theorem 2 (MSE of gradient estimator). Let gboracle (θ) denote the oracle gradient estimator, with the oracle value function used to construct the advantage. Under Assumptions 1 – 5, we have !  m  h2(p∧s) MSE(b gKAE (θi )) = MSE(b goracle (θi )) + O +O . BG ihB 2 G2 Theorem 2 upper bounds the difference in MSE between KAE and the oracle gradient estimator. In the resource-constrained setting, the MSE of the oracle gradient estimator is of order O(B −1 ) (Zhou et al., 2026, Proposition 3). The two extra error terms on the right-hand-side are proportional to the squared bias and variance of KAE’s value estimator, respectively, each scaled by a factor of order O(B −1 ). This formally verifies that improved value estimation directly leads to improved gradient estimation. Next, we consider an asymptotic regime as the number of training iterations i → ∞. The following oracle property is immediate. Corollary 2 (Oracle property of gradient estimator). Under the same conditions in Theorem 2, if the kernel bandwidth satisfies h → 0 and ih → ∞ as i → ∞, then the MSE of KAE’s gradient estimator is asymptotically equivalent to that of the oracle estimator. Moreover, it is smaller than MSEs of the GRPO and REINFORCE++ gradient estimators. The first part of Corollary 2 establishes the asymptotic equivalence between KAE’s gradient estimator and the oracle estimator. The second part highlights KAE’s advantage over both baseline algorithms in gradient evaluation. 10

Finally, we analyze the suboptimality gap of KAE’s learned policy. For any policy π, define its suboptimality gap as ∆(π) = supθ∈Θ Eπθ (Z) − Eπ (Z). By definition, a smaller ∆(π) corresponds to a larger expected return, and hence a better policy. We impose the following assumptions for policy optimization. Assumption 6 (Polyak-Lojasiewicz (PL) condition). There exists some constant µ > 0 such that ∥∇θ J(θ)∥2 ≥ 2µ∆(πθ ) for any θ ∈ Θ. Assumption 7 (Learning rate). The sequence of learning rates {ηi }i follows a 1/i schedule, where ηi = β/i for some β > (2µ)−1 . Assumption 6 is frequently imposed for establishing convergence guarantees in nonconcave optimization, including in the analysis of deep learning models (Karimi et al., 2016). It allows the objective function J(θ) to be nonconcave. To illustrate, suppose θ can be decomposed into (θ1⊤ , θ2⊤ )⊤ , and J(θ) depends only on θ1 . In this case, J(θ) is not strictly concave, as its Hessian with respect to θ2 is zero. Nevertheless, the inequality in Assumption 6 can still hold, provided that J(θ) is concave in θ1 . The 1/i learning rate schedule in Assumption 7 is motivated by stochastic gradient algorithms, under which the parameter estimator achieves optimal convergence rates and admits a tractable limiting distribution (Zhang, 2016). Theorem 3 (Suboptimality gap of the policy). Suppose Assumptions 1 – 7 hold. Then for any 1 ≤ n0 ≤ n and any small 0 < ε < 2µβ − 1, we have  n 1+ε c 1 0 3 E[∆(πθn )] ≤ c1 1+ε + c2 n0 sup MSE(b gKAE (θk )), (1) + n n n k≥n0 where c1 , c2 , c3 are positive constants depending only on (β, µ, ε). Theorem 3 upper bounds the suboptimality gap of KAE’s learned policy. Its proof is more involved than those for standard stochastic gradient methods, where the observations used to estimate the gradient are independent of the current parameter, as in GRPO and REINFORCE++. In KAE, this independence assumption is violated due to the use of past rewards in constructing the current value estimate. Nevertheless, we develop a more refined analysis to obtain a similar convergence guarantee. Moreover, Theorem 3 shows that the MSE of KAE’s gradient estimator directly affects the suboptimality upper bound. This reinforces our motivation to improve value estimation, which in turn improves gradient evaluation and ultimately policy optimization. The following oracle property then establishes the asymptotic equivalence of the suboptimality upper bounds between KAE and the oracle algorithm, and shows that KAE achieves a suboptimality upper bound no larger than those of the two baseline algorithms. Corollary 3 (Oracle property of the policy). Suppose the assumptions in Theorem 3 hold. If the bandwidth satisfies h → 0, n0 h → ∞ as n → ∞, then the suboptimality upper bound of KAE in (1) is asymptotically equivalent to that achieved by the oracle gradient estimator. Moreover, it is no larger than those achieved by GRPO and REINFORCE++.

5

Experiments

In this section, we conduct extensive experiments to evaluate the effectiveness of KAE against GRPO- and REINFORCE-type algorithms, as well as its robustness to the choice of bandwidth parameter and kernel function. We begin with a summary of our findings. Consistent with our theories, KAE is evaluated along the following three dimensions: 11

1. Value estimation (Section 5.1): Across three training steps and two benchmark datasets, KAE achieves a 60%–70% reduction in the MSE of the value estimator compared to GRPO, and an over 90% reduction compared to REINFORCE++ (Table 1). Moreover, its MSE remains robust to the choice of bandwidth and kernel function (Figure 3). 2. Gradient evaluation (Section 5.2): The improvements in value estimation translate directly into more accurate gradient estimation. Specifically, across the same experimental settings, KAE achieves a 5%–9% reduction in the MSE of the gradient estimator compared to GRPO, and a 32%–65% reduction compared to REINFORCE++ (Table 2). 3. Policy optimization (Section 5.3): The gains in value and gradient estimation further translate into improved policy optimization. Compared to GRPO, Dr. GRPO and GPG, KAE achieves the best performance in most scenarios, with an average improvement of 5% on MATH (Table 3), and 11.8% on DAPO (Table 4). In some cases, the improvement reaches up to 79.9%. It also stabilizes training and achieves improvements of up to 15% over REINFORCE-type algorithms (Figure 4). Finally, ablation studies confirm that these gains stem from improved value function estimation (Figures 4 and S.1). Implementation details. Before detailing these results, we first describe several techniques employed in our implementation. Our Algorithm 1 is general and allows for any prompt sampling schedule in Line 3. To simplify our theoretical analysis, in Section 4, we assume that minibatches of prompts are sampled i.i.d. across training iterations (Assumption 1). Under this schedule, however, rewards associated with the same prompt may be too far apart in time to provide useful information for smoothing. To enable KAE to borrow information more effectively across training steps, we adopt the following sampling schedule: we partition all prompts into several minibatches, sample one minibatch without replacement at each training iteration, and then reuse the same minibatch for J consecutive training steps. Additionally, we consider three benchmark reasoning datasets: GSM8K, MATH, and DAPO. The parameter J is set to 10 for GSM8K and MATH, and to 8 for DAPO. In our main experiments, we use the triangular kernel K(x) = max(1 − x, 0). We also consider the exponential kernel K(x) = ρ|x| for some 0 < ρ < 1 when evaluating the sensitivity of KAE. Finally, we use Nadaraya-Watson estimators to construct the value estimator that involve normalization to improve the performance when the sample size is small.

5.1

Value estimation

Experimental setup. We first compare KAE’s value estimator with GRPO’s group mean estimator and REINFORCE++’s across-prompt reward average. For this evaluation, we consider two benchmark datasets: GSM8K and MATH. Following the literature, we post-train a smaller Qwen2.5-1.5B-Instruct model on the simpler GSM8K dataset and a larger Qwen2.5-Math-7B model on the more challenging MATH dataset. For each dataset–model combination, we first apply the proposed prompt sampling scheme to fix the set of prompts used throughout training. We then post-train the model using KAE to obtain a sequence of parameter estimates {θi }i across training steps. We aim to estimate the value function V πθi at three training steps: i = 10, 90, and 170. Evaluation. Since the prompts are fixed, the MSE of the value estimators is computed with respect to the variability from the sampled completions. We approximate the ground-truth value function via Monte Carlo (MC) sampling by generating a large number of completions at each selected training step, and treat the resulting MC estimates as ground truth. The MSEs of the GRPO and REINFORCE++ value estimators are straightforward to evaluate. At each selected training step, we generate G completions per prompt and use these samples to 12

Table 1: MSE (×10−3 ; lower is better) of value estimators computed by KAE, GRPO and REINFORCE++ across three training steps and two datasets. For each training step, the MSE is first evaluated at the prompt level and then aggregated across prompts. The last two rows report the percentage reduction in MSE achieved by KAE compared to REINFORCE++ and GRPO. MATH (Qwen2.5-Math-7B) GSM8K (Qwen2.5-1.5B) Algorithm

Step 10

Step 90

Step 170

Step 10

Step 90

Step 170

REINFORCE++ GRPO KAE

78.8 19.4 5.80

89.2 8.62 2.67

71.9 5.07 1.45

73.2 15.5 4.13

30.7 5.65 1.77

30.9 2.50 0.91

Reduction vs. REINFORCE++ Reduction vs. GRPO

92.6% 70.1%

97.0% 69.0%

98.0% 71.4%

94.4% 73.3%

94.2% 68.6%

97.1% 63.6%

compute the corresponding value estimators. We then repeat this procedure multiple times to estimate their MSEs. To ensure a fair comparison, we apply the same leave-one-out procedure to both GRPO and REINFORCE++ when estimating the value. Additionally, although REINFORCE++ is designed to use only a single completion per prompt, we adapt the algorithm by using all G completions to compute its average reward so that all methods utilize the same amount of data for value estimation. However, evaluating the MSE of KAE’s value estimator is more challenging, since the estimator’s variability arises not only from sampling completions at the current training step, but also from those at previous steps. Ideally, this would require sampling past completions conditional on the current parameter θi , which is intractable. To address this, we note that due to our use of a triangular kernel, the value estimator depends only on a finite number of previous steps (specifically, 4 steps) due to truncation. We therefore approximate the distribution of past completions using πθi−1 , · · · , πθi−4 . This approximation is reasonable given the small learning rate (10−6 ) used in our implementation, under which the model parameters change only minimally over a few training steps. Results. Table 1 reports the MSEs. We make two observations. First, KAE consistently achieves the smallest MSE in all cases, reducing GRPO’s MSE by 60%–70% and REINFORCE++’s by over 90%. This highlights the effectiveness of leveraging historical observations to improve value function estimation. Moreover, these reductions are consistent across datasets and training steps, indicating that these gains are not specific to particular datasets or training steps. Second, the MSE of all value estimators decreases noticeably on GSM8K at Steps 90 and 170. This suggests that the model’s response has become more certain at these steps, resulting in substantially smaller variance compared to Step 10. Sensitivity analysis. We further conduct a sensitivity analysis on the MATH dataset to investigate the MSE of the proposed value estimator under different choices of bandwidth and kernel function. Figure 3 visualizes the MSEs at the three training steps. The results show that, for both kernel functions, triangular and exponential, and across all training steps and a wide range of bandwidth values, the MSE of KAE remains much smaller than that of GRPO and substantially smaller than that of REINFORCE++. This demonstrates the robustness of KAE with respect to both kernel bandwidth and kernel function.

5.2

Gradient evaluation

Using the same datasets, MATH and GSM8K, we next compare KAE’s gradient estimators at the three training steps (10, 90, and 170) with those computed by GRPO and REINFORCE++.

13

Figure 3: MSE of KAE’s value estimator on the MATH dataset across three training steps under varying kernel bandwidths. The left and right panels visualize the MSEs under the triangular and exponential kernels, respectively. Horizontal lines denote the MSEs of REINFORCE++ and GRPO, which are independent of bandwidth and kernel function.

Table 2: MSE (×104 ) of gradient estimators computed by KAE, GRPO and REINFORCE++ across three training steps and two datasets. The remaining details are the same as in Table 1. GSM8K (Qwen2.5-1.5B) MATH (Qwen2.5-Math-7B) Method

Step 10

Step 90

Step 170

Step 10

Step 90

Step 170

REINFORCE++ GRPO KAE

11.98 7.62 7.00

10.61 6.63 6.15

15.60 6.31 5.97

14.75 10.77 9.94

3.88 1.59 1.46

3.41 1.29 1.18

Reduction vs. REINFORCE++ Reduction vs. GRPO

41.57% 8.14%

42.04% 7.24%

61.73% 5.39%

32.61% 7.71%

62.37% 8.18%

65.40% 8.53%

We similarly apply MC to evaluate the true gradients at these training steps. We then plug in each algorithm’s estimated value function to construct the corresponding advantage function and gradient estimator, and repeat this procedure multiple times to evaluate the gradient’s MSE. For the MATH dataset, the model contains 7B parameters, making it memory intensive to store the full gradient vector. We therefore restrict attention to the subvector corresponding to the parameters in the final decoder block and the output layer when computing the MSE. Table 2 summarizes the results. KAE again achieves the smallest MSE in all cases, with reductions ranging from 32.6%–65.4% compared to REINFORCE++ and 5.4%–8.5% compared to GRPO. The gains are consistent across datasets and training steps, highlighting the effectiveness of KAE for gradient evaluation.

5.3

Policy optimization

Experimental setup. Finally, we demonstrate that more accurate value and gradient estimation indeed improve policy learning. To reflect our focus on the resource-constrained setting, we consider three group sizes G ∈ {1, 4, 8}. We evaluate different algorithms using three benchmark training datasets (GSM8K, MATH, and DAPO) by post-training three base models: Qwen2.5-1.5B-Instruct, 14

Table 3: Accuracy of the Qwen2.5-Math-7B model post-trained on MATH (levels 3–5, 5k+). Results are reported at temperature 0.6 after 200 training steps with group size G = 8. The highest accuracy in each column is shown in bold, and the second-highest is underlined. Method

AIME24

AIME25

AMC

MATH

Minerva

Olympiad

Avg

Dr. GRPO GPG GRPO KAE

0.2719 0.3052 0.2844 0.2969

0.0875 0.0896 0.0802 0.0938

0.5715 0.5776 0.5858 0.5806

0.7580 0.7660 0.7540 0.7720

0.3309 0.3125 0.3162 0.3493

0.3748 0.3689 0.3852 0.3970

0.3991 0.4033 0.4010 0.4149

Table 4: Accuracy of the Qwen2.5-Math-7B model post-trained on DAPO (17k). Results are reported at temperature 0.6 after 500 training steps with group size G = 4. The highest accuracy in each column is shown in bold, and the second-highest is underlined. Method

AIME24

AIME25

AMC

MATH

Minerva

Olympiad

Avg

GRPO Dr. GRPO GPG KAE

0.2833 0.2375 0.2823 0.2698

0.1188 0.0990 0.1396 0.1781

0.6261 0.6408 0.6205 0.6453

0.7960 0.7920 0.8140 0.8100

0.3309 0.2757 0.3309 0.3456

0.4178 0.4104 0.4222 0.4237

0.4288 0.4092 0.4349 0.4454

Qwen2.5-Math-1.5B and Qwen2.5-Math-7B. More specifically, we consider two settings: single-stream and multi-stream. In the single-stream setting, we set G = 1 and compare KAE against REINFORCE, since GRPO-type algorithms are not applicable when only a single completion is sampled per prompt. We consider two datasets in this setting, GSM8K and MATH. Since GSM8K is a simpler benchmark, we post-train the generic Qwen2.5-1.5B-Instruct model on this dataset. For MATH, we post-train the more task-specific Qwen2.5-Math-1.5B model. In the multi-stream setting, we choose G ∈ {4, 8} and compare KAE against GRPO-type algorithms, including the vanilla GRPO, Dr. GRPO, and GPG. We again consider two training datasets, DAPO and MATH, and post-train the larger Qwen2.5-Math-7B model on both datasets. Multi-stream setting. Following standard practice, after training on DAPO and MATH, we evaluate the post-trained models on a set of reasoning benchmarks covering both in-distribution and out-of-distribution datasets: AIME24, AIME25, AMC, MATH, Minerva, and Olympiad. For each benchmark, we report the accuracy measure (the percentage of questions answered correctly) for the post-trained model, together with the average accuracy across all benchmarks. The sampling temperature is fixed at 0.6 during evaluation. Tables 3 and 4 report the accuracy results for the Qwen2.5-Math-7B model post-trained on MATH and DAPO, respectively. Across both training datasets, KAE achieves the highest accuracy in 5 out of 7 reasoning benchmarks, including the final average accuracy. On the remaining benchmarks, it typically attains the second-highest accuracy. Specifically, when trained on MATH, KAE improves over GRPO-type baselines by 0.8% to 17.0%, while when trained on DAPO, the improvement ranges from 0.4% to 79.9%. More importantly, these gains are observed across multiple benchmark datasets and multiple GRPO-type baselines, demonstrating the consistent improvements KAE delivers in policy optimization. Single-stream setting. Since GSM8K is a simpler dataset, the model’s generated reasoning trajectories are much shorter than those for MATH, making training on GSM8K considerably less computationally demanding. We therefore repeat our experiments on GSM8K five times to improve the reproducibility of our results, whereas existing work typically reports only a single 15

Figure 4: Test accuracy of models post-trained with standard REINFORCE (blue), KAE (red), and a REINFORCE variant using the proposed prompt sampling scheme, on GSM8K (left) and MATH (right) across different training steps. Shaded areas represent the standard error of the accuracy curves, aggregated over five training replications.

run. In contrast, since training on MATH is substantially more expensive, we report results from a single run in that setting. Figure 4 visualizes the test accuracy of models post-trained with REINFORCE and the proposed KAE on both datasets. For GSM8K, where training is repeated five times, we additionally report the standard error estimated across these replications. It can be seen from Figure 4 that, on both datasets, the accuracy of REINFORCE begins to decline after reaching its peak. We conjecture that this is due to the high variance of its gradient estimates, which makes training unstable. By contrast, the accuracy of KAE increases steadily throughout training. At the final training iteration, KAE improves over REINFORCE by 14.9% on GSM8K and 6.6% on MATH. These results show that more accurate value and gradient estimation indeed stabilize policy learning. Ablation study. Our implemented KAE differs from GRPO- and REINFORCE-type algorithms in two ways: (i) it adopts a prompt sampling schedule that reuses each minibatch of prompts for multiple consecutive training steps, and (ii) it exploits historical rewards to improve value and advantage estimation. Our results above show that, when combined, these two components enable KAE to achieve better policy performance than the baseline methods. To isolate the contribution of the second component, we conduct an ablation study comparing KAE against variants of GRPOand REINFORCE-type algorithms that adopt the proposed prompt sampling schedule, denoted (GRPO/REINFORCE + schedule). In this comparison, the primary difference is the value and advantage function estimation. Figure 4 visualizes the test accuracy of models trained by this variant of REINFORCE using GSM8K and MATH. It can be seen that the proposed prompt sampling schedule alone does not help much to stabilize training. The test accuracy still drops after reaching its peak across both datasets. Figure S.1 compares the test accuracy of the schedule-matched GRPO variant with that of KAE on MATH. After roughly the first 30 training steps, KAE consistently achieves higher test accuracy than the GRPO variant. Taken together, these results indicate that more accurate value and advantage estimation translates into meaningful gains in downstream policy optimization.

6

Conclusion

We introduce KAE, a kernel smoothing method for value and advantage function estimation in LLM reasoning, which can be seamlessly integrated into the subsequent policy optimization. Theoretically, we establish the oracle property of KAE and its superiority over GRPO-type algorithms 16

in resource-constrained settings across three aspects: value estimation (Corollary 1), gradient evaluation (Corollary 2) and policy learning (Corollary 3). Our empirical experiments further reinforce these theoretical findings: KAE achieves competitive performance against both GRPO- and REINFORCE-type algorithms (Tables 1 – 4, Figure 4), attains the oracle property in one-shot regimes (Figure 1), remains robust to the kernel bandwidth and kernel function (Figure 3), and its gains indeed come from the use of nonparametric methods for value and advantage estimation (Figures 4 and S.1).

References Zeyu Bian, Chengchun Shi, Zhengling Qi, and Lan Wang. Off-policy evaluation in doubly inhomogeneous environments. Journal of the American Statistical Association, 120(550):1102–1114, 2025. Jinhang Chai, Elynn Chen, and Jianqing Fan. Deep transfer q-learning for offline non-stationary reinforcement learning. arXiv preprint arXiv:2501.04870, 2025. Bibhas Chakraborty and Erica E Moodie. Statistical methods for dynamic treatment regimes. Springer-Verlag. doi, 10(978-1):4–1, 2013. Elynn Y Chen, Rui Song, and Michael I Jordan. Reinforcement learning in latent heterogeneous environments. Journal of the American Statistical Association, 119(548):3113–3126, 2024. Xiaohong Chen. Large sample sieve estimation of semi-nonparametric models. Handbook of econometrics, 6:5549–5632, 2007. Young Hyun Cho and Will Wei Sun. Privacy-preserving reinforcement learning from human feedback via decoupled reward modeling. arXiv preprint arXiv:2603.22563, 2026. Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546, 2025. Runpeng Dai, Linfeng Song, Haolin Liu, Zhenwen Liang, Dian Yu, Haitao Mi, Zhaopeng Tu, Rui Liu, Tong Zheng, Hongtu Zhu, et al. Cde: Curiosity-driven exploration for efficient reinforcement learning in large language models. arXiv preprint arXiv:2509.09675, 2025. Ashkan Ertefaie and Robert L Strawderman. Constructing dynamic treatment regimes over indefinite time horizons. Biometrika, 105(4):963–977, 2018. Jianqing Fan, Theo Gasser, Irène Gijbels, Michael Brockmann, and Joachim Engel. Local polynomial regression: Optimal kernels and asymptotic minimax efficiency. Annals of the Institute of Statistical Mathematics, 49(1):79–99, 1997. Asim H Gazi, Yongyi Guo, Daiqi Gao, Ziping Xu, Kelly W Zhang, and Susan A Murphy. Statistical reinforcement learning in the real world: A survey of challenges and future directions. arXiv preprint arXiv:2601.15353, 2026. Lin Ge, Hengrui Cai, Runzhe Wan, Yang Xu, and Rui Song. A review of causal decision making. arXiv preprint arXiv:2502.16156, 2025.

17

Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(Nov):1471– 1530, 2004. Ulf Grenander. Abstract Inference. Wiley, 1981. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638, 2025. Kevin Han, Yuhang Zhou, Mingze Gao, Gedi Zhou, Serena Li, Abhishek Kumar, Xiangjun Fan, Weiwei Li, and Lizhu Zhang. Ebpo: Empirical bayes shrinkage for stabilizing group-relative policy optimization. arXiv preprint arXiv:2602.05165, 2026. Yaru Hao, Li Dong, Xun Wu, Shaohan Huang, Zewen Chi, and Furu Wei. On-policy rl with optimal reward baseline. arXiv preprint arXiv:2505.23585, 2025. Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: Stabilizing critic-free policy optimization with global advantage normalization. arXiv preprint arXiv:2501.03262, 2025. Jianhua Z Huang. Projection estimation in multiple regression with application to functional anova models. The annals of statistics, 26(1):242–272, 1998. Yu Huang, Zixin Wen, Yuejie Chi, Yuting Wei, Aarti Singh, Yingbin Liang, and Yuxin Chen. On the learning dynamics of rlvr at the edge of competence. arXiv preprint arXiv:2602.14872, 2026. Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai o1 system card. arXiv preprint arXiv:2412.16720, 2024. William James, Charles Stein, et al. Estimation with quadratic loss. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1, pages 361–379. University of California Press, 1961. Ying Jin, Zhimei Ren, Zhuoran Yang, and Zhaoran Wang. Policy learning “without” overlap: Pessimism and generalized empirical bernstein’s inequality. The Annals of Statistics, 53(4): 1483–1512, 2025. Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximalgradient methods under the polyak-lojasiewicz condition. In Joint European conference on machine learning and knowledge discovery in databases, pages 795–811. Springer, 2016. Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! In ICLR 2019 Workshop on Deep Reinforcement Learning Meets Structured Prediction, 2019. Michael R Kosorok and Eric B Laber. Precision medicine. Annual review of statistics and its application, 6(1):263–286, 2019. Eric B Laber, Daniel J Lizotte, Min Qian, William E Pelham, and Susan A Murphy. Dynamic treatment regimes: Technical challenges and applications. Electronic journal of statistics, 8(1): 1225, 2014. Tze Leung Lai and Herbert Robbins. Asymptotically efficient adaptive allocation rules. Advances in applied mathematics, 6(1):4–22, 1985. 18

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. Tulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411.15124, 2024. Seong Jin Lee, Will Wei Sun, and Yufeng Liu. Low-rank contextual reinforcement learning from heterogeneous human feedback. arXiv preprint arXiv:2412.19436, 2024. Gang Li, Ming Lin, Tomer Galanti, Zhengzhong Tu, and Tianbao Yang. DisCO: Reinforcing large reasoning models with discriminative constrained optimization. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026a. Gen Li, Laixi Shi, Yuxin Chen, Yuejie Chi, and Yuting Wei. Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics, 52(1):233–260, 2024. Siheng Li, Zhanhui Zhou, Wai Lam, Chao Yang, and Chaochao Lu. Repo: Replay-enhanced policy optimization. arXiv preprint arXiv:2506.09340, 2025a. Yu Li, Tian Lan, and Zhengling Qi. When right meets wrong: Bilateral context conditioning with reward-confidence correction for grpo. arXiv preprint arXiv:2603.13134, 2026b. Yuhan Li, Eugene Han, Yifan Hu, Zhengling Qi, Yifan Cui, and Ruoqing Zhu. Reinforcement learning with continuous actions under unmeasured confounding. Journal of the American Statistical Association, To appear, 2025b. Muxuan Liang and Menggang Yu. Relative contrast estimation and inference for treatment recommendation. Biometrics, 79(4):2920–2932, 2023. Peng Liao, Zhengling Qi, Runzhe Wan, Predrag Klasnja, and Susan A Murphy. Batch policy learning in average reward markov decision processes. Annals of statistics, 50(6):3364, 2022. Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization-based reasoning models. arXiv preprint arXiv:2503.22342, 2025. Kaizhao Liu, Qi Long, Zhekun Shi, Weijie J Su, and Jiancong Xiao. Statistical impossibility and possibility of aligning llms with human preferences: From condorcet paradox to nash equilibrium. arXiv preprint arXiv:2503.10990, 2025a. Pangpang Liu, Junwei Lu, and Will Wei Sun. Uncertainty quantification for large language model reward learning under heterogeneous human feedback. arXiv preprint arXiv:2512.03208, 2025b. Pangpang Liu, Chengchun Shi, and Will Wei Sun. Reinforcement learning from human feedback: A statistical perspective. arXiv preprint arXiv:2604.02507, 2026. Weidong Liu, Jiyuan Tu, Xi Chen, and Yichen Zhang. Online estimation and inference for robust policy evaluation in reinforcement learning. The Annals of Statistics, 53(5):2128–2152, 2025c. Zhaowei Liu, Xin Guo, Zhi Yang, Fangqi Lou, Lingfeng Zeng, Jinyi Niu, Mengping Li, Qi Qi, Zhiqiang Liu, Yiyang Han, et al. Fin-r1: A large language model for financial reasoning through reinforcement learning. arXiv preprint arXiv:2503.16252, 2025d. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. In Second Conference on Language Modeling, 2025e. 19

Nan Lu, Ethan X Fang, and Junwei Lu. Contextual online uncertainty-aware preference learning for human feedback. arXiv preprint arXiv:2504.19342, 2025. Wenbin Lu, Hao Helen Zhang, and Donglin Zeng. Variable selection for optimal treatment decision. Statistical methods in medical research, 22(5):493–504, 2013. Daniel J Luckett, Eric B Laber, Anna R Kahkoska, David M Maahs, Elizabeth Mayer-Davis, and Michael R Kosorok. Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Association, 115(530):692, 2020. Rui Miao, Babak Shahbaba, and Annie Qu. Reinforcement learning for individual optimal policy from heterogeneous data. Annals of statistics, 53(4):1513, 2025. Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, pages 1928–1937. PMLR, 2016. Susan A Murphy. Optimal dynamic treatment regimes. Journal of the Royal Statistical Society Series B: Statistical Methodology, 65(2):331–355, 2003. Elizbar A Nadaraya. On estimating regression. Theory of Probability & Its Applications, 9(1): 141–142, 1964. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730– 27744, 2022. Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathematical statistics, pages 400–407, 1951. James M Robins. Optimal structural nested models for optimal sequential decisions. In Proceedings of the Second Seattle Symposium in Biostatistics: analysis of correlated data, pages 189–326. Springer, 2004. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Chengchun Shi, Alin Fan, Rui Song, and Wenbin Lu. High-dimensional a-learning for optimal dynamic treatment regimes. Annals of statistics, 46(3):925, 2018. Chengchun Shi, Shikai Luo, Yuan Le, Hongtu Zhu, and Rui Song. Statistically efficient advantage learning for offline reinforcement learning in infinite horizons. Journal of the American Statistical Association, 119(545):232–245, 2024a. Chengchun Shi, Zhengling Qi, Jianin g Wang, and Fan Zhou. Value enhancement of reinforcement learning via efficient and robust trust region optimization. Journal of the American Statistical Association, 119(547):2011–2025, 2024b.

20

Charles J Stone. Consistent nonparametric regression. The annals of statistics, pages 595–620, 1977. Charles J Stone. Optimal global rates of convergence for nonparametric regression. The annals of statistics, pages 1040–1053, 1982. Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, 2 edition, 2018. Anastasios A Tsiatis, Marie Davidian, Shannon T Holloway, and Eric B Laber. Dynamic treatment regimes: Statistical methods for precision medicine. Chapman and Hall/CRC, 2019. Matt P Wand and M Chris Jones. Kernel smoothing. CRC press, 1994. Hu Wang, Congbo Ma, Ian Reid, and Mohammad Yaqub. Kalman filter enhanced grpo for reinforcement learning-based language model reasoning. arXiv preprint arXiv:2505.07527, 2025a. Jiayi Wang, Zhengling Qi, and Raymond KW Wong. Projected state-action balancing weights for offline reinforcement learning. The Annals of Statistics, 51(4):1639–1665, 2023. Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.20571, 2025b. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022. Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992. Xintao Xia, Zhiqiu Xia, Linjun Zhang, and Zhanrui Cai. A statistical framework for alignment with biased ai feedback. arXiv preprint arXiv:2602.08259, 2026. Jiancong Xiao, Ziniu Li, Xingyu Xie, Emily Getzen, Cong Fang, Qi Long, and Weijie Su. On the algorithmic bias of aligning large language models with RLHF: Preference collapse and matching regularization. Journal of the American Statistical Association, 120(552):2154–2164, 2025. Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to llm reasoning: from rejection sampling to reinforce. arXiv preprint arXiv:2504.11343, 2025. Zhongwen Xu and Zihan Ding. Single-stream policy optimization. arXiv preprint arXiv:2509.13232, 2025. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. Learning to reason under off-policy guidance. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.

21

Guanning Zeng, Zhaoyi Zhou, Daman Arora, and Andrea Zanette. Shrinking the variance: Shrinkage baselines for reinforcement learning with verifiable rewards. arXiv preprint arXiv:2511.03710, 2025. Li-Xin Zhang. Central limit theorems of a recursive stochastic algorithm with applications to adaptive designs. The Annals of Applied Probability, pages 3630–3658, 2016. Yuanhe Zhang, Ilja Kuzborskij, Jason D. Lee, Chenlei Leng, and Fanghui Liu. DAG-math: Graphof-thought guided mathematical reasoning in LLMs. In The Fourteenth International Conference on Learning Representations, 2026. Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, et al. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673, 2025. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025a. Tong Zheng, Hongming Zhang, Wenhao Yu, Xiaoyang Wang, Runpeng Dai, Rui Liu, Huiwen Bao, Chengsong Huang, Heng Huang, and Dong Yu. Parallel-r1: Towards parallel thinking via reinforcement learning. arXiv preprint arXiv:2509.07980, 2025b. Han Zhong, Xun Deng, Ethan X. Fang, Zhuoran Yang, Zhaoran Wang, and Runze Li. Risk-sensitive deep rl: Variance-constrained actor-critic provably finds globally optimal policy. Journal of the American Statistical Association, To appear, 2025. Doudou Zhou, Yufeng Zhang, Aaron Sonabend-W, Zhaoran Wang, Junwei Lu, and Tianxi Cai. Federated offline reinforcement learning. Journal of the American Statistical Association, 119 (548):3152–3163, 2024a. Hongyi Zhou, Kai Ye, Erhan Xu, Jin Zhu, Ying Yang, Shijin Gong, and Chengchun Shi. Demystifying group relative policy optimization: Its policy gradient is a u-statistic. arXiv preprint arXiv:2603.01162, 2026. Wenzhuo Zhou, Ruoqing Zhu, and Annie Qu. Estimating optimal infinite horizon dynamic treatment regimes via pt-learning. Journal of the American Statistical Association, 119(545):625–638, 2024b. Tong Zhu, Baiting Chen, Jin Zhou, Hua Zhou, Sriram Sankararaman, and Xiaowu Dai. Align: Aligned delegation with performance guarantees for multi-agent llm reasoning. arXiv preprint arXiv:2602.00127, 2026.

22