arXiv CS open access

Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards

Yuante Li et al. · arxiv_cs

arXiv CS · Papers

Conceptio Open Archive

Analyze & explore related research in Conceptio

Connect this record directly into your research workflow or AI agent. Explore citations across open repositories, query full text, and export structured proof bundles.

Explore in Search →Connect via MCP / API ↗

Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards Yuante Li1 * , Junwei Quan2 * , Kai Qiu1 , Kate Zhang1 , Yicheng Tao1 1 Carnegie Mellon University, 2 University of Toronto # yuantel@cs.cmu.edu, junwei.quan@mail.utoronto.ca

Progress clock (clip on)

Abstract

160

Samples (k)

arXiv:2610.18853v1 [cs.LG] 13 Oct 2026

Reinforcement learning with verifiable rewards (RLVR) relies on costly model-generated responses, motivating larger rollout batches and square-root learning-rate scaling. Reported to approximately preserve progress per sample with one optimizer step per rollout, this rule is less understood in practical multi-step training. We develop a two-clock framework that separates the samples needed to make progress from those consumed before collapse. Under a local quadratic model, the square-root rule preserves first-order drift per sample but not progress: larger batches take fewer, longer updates and lose more to curvature, yielding a learning-ratedependent critical batch beyond which sample efficiency degrades. A drift-threshold hypothesis predicts a complementary failure clock whose sample count remains approximately invariant along the square-root rule. Controlled GRPO experiments on Qwen2.5-Math-1.5B across a 16× batch range support both predictions. Ratio clipping delays failure, especially for larger batches, but the progress gap persists without it. Widening the clip trades stability for a transient speed gain. The progress gap reappears on another dataset, a general-purpose checkpoint, and a 7B model. Finally, we use the two-clock framework to derive a simple multistep RLVR schedule: start below the critical batch and switch to a larger batch before failure. Our code is available at: two-clocks-rlvr.

to 15% acc. to 20% acc.

li pste

Failure clock (clip off)

d ite

failure law invariant

10 0.5

1.5

0.5

1.5

Rollout batch (k responses) Rollout batch (k responses)

Figure 1: The two clocks under the square-root rule with µ=8. Color marks the rollout batch, from 512 in blue to 8,192 in dark red. Left, with the clip: samples to reach 15% and 20% accuracy, with the critical-batch curve (gray) between perfect invariance (dotted) and the step-limited extreme (dashed). Right: samples to failure without the clip, against the failure-law band (gray), with an open marker for the run that never failed. Progress slows with the batch, while failure stays on the failure law.

amounts of policy-generated data can be scored without human preference labels, but training remains expensive because this data must be regenerated as the policy changes: every policy update requires fresh rollouts, and generation dominates the cost. Practitioners therefore enlarge the rollout batch, the number of responses generated by one policy before it is updated, and rely on the square-root rule to transfer the learning rate across batch sizes, scaling it with the square root of the batch (Malladi et al., 2022; Hilton et al., 2022). For Group Relative Policy Optimization (GRPO) (Shao et al., 2024) with one optimizer step per rollout, the rule approximately works: rollout batches from 512 to 8K responses follow learning curves within about one accuracy point of each other when plotted against the number of samples consumed (Li et al., 2026).

Introduction

Reinforcement learning with verifiable rewards (RLVR) (Lambert et al., 2025; Guo et al., 2025) has become a widely used post-training approach for language models, spanning mathematical reasoning (Hubert et al., 2025; Team et al., 2025), code (Wei et al., 2025; Luo et al., 2025a), and broader domains scored against reference answers (Su et al., 2025). Its appeal is that large

However, practical RLVR training rarely operates in this one-step regime. To amortize generation, practical RLVR recipes split each rollout into mini-batches and take one optimizer step on each.

* Equal contribution.

DAPO, for example, uses 16 optimizer steps per rollout (Yu et al., 2025). From the second step onward, the current policy has moved away from the policy that generated the data, so the token-level importance ratios depart from one and the PPO-style ratio clip begins to suppress gradient contributions from tokens whose ratios move beyond the clipping range. In our multi-step GRPO experiments, three phenomena appear together under the square-root rule: ➀ Progress slows with batch size. Larger rollout batches learn more slowly per sample. ➁ Failure keeps time in samples. Without the clip, larger batches collapse after the same number of samples as smaller ones, despite taking far fewer optimizer steps. ➂ Clipping helps larger batches most. Although clipping is often viewed as a bottleneck on learning (Qi et al., 2026; Yang et al., 2026a), its stabilizing effect is strongest for the larger batches. Figure 1 visualizes this split into two clocks: progress becomes increasingly sampleinefficient with larger rollout batches, while failure stays locked to the number of samples consumed. Prior work studies these effects separately: batchsize invariance (Hilton et al., 2022; Li et al., 2026), critical-batch behavior with the learning rate tuned at every batch (McCandlish et al., 2018), and clipping at a fixed batch size. None explains why all three emerge together under the same batchdependent learning-rate rule. We explain these phenomena with a two-clock framework of batch scaling. Its central observation is that the square-root rule preserves the policy’s drift per sample, the persistent displacement it accumulates, but not the progress that drift produces. A larger rollout batch takes fewer optimizer steps on the same samples, and the square-root rule makes each step longer. Under a local quadratic model, these fewer, longer steps accumulate the same drift per sample but lose more progress to curvature. This gives a progress clock, which counts samples to a target accuracy and slows beyond a learning-rate-dependent critical batch. If collapse occurs when accumulated drift reaches a threshold, the same scaling rule gives a failure clock that is invariant in samples. The progress clock follows from the local quadratic model, and the failure clock rests on a drift-threshold hypothesis that we test. We test the two-clock model with controlled GRPO experiments that vary rollout batch size, learning rate, and clipping over a sixteenfold range

of rollout batches. The progress clock departs from sample invariance beyond a critical batch: with eight steps per rollout, as the learning rate rises tenfold, the sample-cost exponent climbs from 0.15 to 0.44. Unlike the tuned-learning-rate picture (McCandlish et al., 2018), the critical batch therefore shrinks with the learning rate assigned by the square-root rule and lies well below the gradient noise scale. The failure clock, in contrast, does not move along the rule: without the clip, a 4,096 batch fails after 56 to 80 optimizer steps and a 512 batch after 648, yet both fail after 30 to 50k samples. Restoring the clip delays collapse twofold for the small batch and fourfold for the large one, and the progress gap is present with or without it. The progress gap reappears on a second dataset, a second checkpoint, and a 7B model. The separation between the two clocks further yields a simple batch schedule: start below the critical batch for sample-efficient progress, then switch to a larger batch when throughput or stability makes it preferable. The ramp approaches the best fixed batch while generating three quarters of its samples at the large batch. To summarize, our contributions are as follows. ➀ We introduce a two-clock framework for multistep GRPO, in which a local model derives a learning-rate-dependent critical batch for progress, while a drift-threshold hypothesis predicts failure at a fixed number of samples along the square-root rule. ➁ We test these predictions across batch size, learning rate, clipping, model scale, and dataset, with failure-law fits that favor drift scaling over noisedriven alternatives and clip and replay controls showing that the clip delays failure while the progress gap persists without it. ➂ We derive a practical batch schedule for multistep RLVR from the two clocks: start below the critical batch and switch to a larger batch before failure. In our experiments, this schedule approaches the best fixed-batch performance.

Preliminaries

GRPO with a ratio clip. For each prompt, the policy samples G responses and receives rule rewards ri ∈ {0, 1}, which GRPO normalizes within the group into advantages Ai = (ri − r̄)/(σr + ϵ) (Shao et al., 2024). For token yt , let ρt = πθ (yt | ·)/πθold (yt | ·) be the importance ratio between the current policy and the policy that generated the 2

Cov ξ = Σ/b, and σ 2 = tr Σ. One optimizer step is ∆θ = −η u(ĝ), and in the main text we use a scalar Adam-type normalizer

response. We use the DAPO objective, which av erages min ρt At , clip(ρt , 1−εlow , 1+εhigh )At over tokens with (εlow , εhigh ) = (0.2, 0.28) (Yu et al., 2025) and a dual clip at c=10 on negative advantages (Ye et al., 2020). A token whose ratio leaves this range in the direction favored by its advantage is clipped, so its contribution to the policy gradient is zero. At the first optimizer step after a rollout, every ratio equals one, so the clip can act only from the second step onward.

u(ĝ) = ĝ/ν,

(1)

treating the per-coordinate version in App. A.5. We make two assumptions for the progress clock: (A1) Noise-dominated normalization. The optimizer batch lies well √ below the gradient noise scale, so ν ≈ σ/ b. Our optimizer batches are 60 to 2000 times below the noise scale measured in App. E.1.

Definition 1 (Batches and the square-root rule). A rollout of P prompts with G responses each contains Br = P G responses, the rollout batch. We split these into µ mini-batches and take one optimizer step on each, so each step sees the optimizer batch b =√Br /µ. At fixed µ, the square-root √ rule sets η ∝ Br , equivalently η ∝ b. As in common practice (Li et al., 2026), Adam’s other hyperparameters stay fixed.

(A2) Local quadratic loss. Over one step the loss changes by ⟨ḡ, ∆θ⟩ + 12 ∆θ⊤ H∆θ, with H the local Hessian. For progress we treat the mini-batches within a rollout as fresh and ignore policy staleness, whose effect on importance ratios and clipping we model separately below.

Sample efficiency. Because generation dominates cost, we compare runs at equal numbers of consumed responses N . Let JBr (N ) be the validation accuracy after N responses and Nτ (Br ) the first N at which it reaches a target τ .

What the square-root rule fixes. Under the square-root rule, a larger optimizer batch takes longer steps but fewer of them, and the two effects cancel in the displacement accumulated per √ sample. Under (A1) the normalizer shrinks √ as 1/ b while the rule raises the learning rate as b, so writing √ η = κ b the mean step is

Definition 2 (Sample cost). The square-root rule is sample-invariant if JBr (N ) does not depend on Br . The sample-cost exponent α = d log Nτ /d log Br is 0 under perfect invariance and 1 when doubling the rollout batch doubles the samples needed.

E[∆θ] = −

Definition 3 (Failure point). The failure point Nfail of a run is the number of consumed responses at which it first meets a failure criterion fixed in advance (§4). A run that does not fail within its training budget is right-censored: the budget is a lower bound on its failure point.

ν 2 = E∥ĝ∥2 ,

κb ḡ, σ

(2)

while a fixed sample budget contains only 1/b as many steps. The persistent displacement per consumed response, which we call the drift per sample, d=

Method: A Two-Clock Model of Batch Scaling

κ ∥ḡ∥ , σ

(3)

is therefore independent of the batch (App. A.1). Here κ, the learning rate per square root of the optimizer batch, is what the rule holds fixed.

We now formalize the two-clock picture. We model one optimizer step on a locally quadratic loss and ask which per-sample quantities the square-root rule preserves. The model first yields a progress law from curvature, then adds a drift-threshold hypothesis for failure, and finally places the ratio clip on the drift accumulated within a rollout. Figure 2 summarizes the picture. Detailed proofs and extensions are provided in App. A.

The progress clock. The square-root rule preserves the drift per sample, but equal movement need not produce equal progress on a curved loss. Proposition 1 (Progress per sample). Under (A1)– √ (A2), with η = κ b, the expected loss decrease per consumed response is r(b) = r0 −Ccurv b,

Setting. Let ĝ = ḡ + ξ be the policy-gradient estimate from an optimizer batch of b responses, where ḡ is the population gradient, E ξ = 0,

Ccurv =

where r0 does not depend on b. 3

κ2 ḡ ⊤ H ḡ , (4) 2σ 2

Multi-step GRPO 𝜼 ∝ Fixed sample budget

𝑩𝒓 (fixed 𝜇)

(A) Progress clock start

small large

𝐌𝐚𝐭𝐜𝐡𝐞𝐝 𝐟𝐢𝐫𝐬𝐭−𝐨𝐫𝐝𝐞𝐫 𝐝𝐫𝐢𝐟𝐭 𝐩𝐞𝐫 𝐬𝐚𝐦𝐩𝐥𝐞 Under the local quadratic model

(B) Within-rollout clipping

Curved loss valley (schematic)

𝒍𝒐𝒈(𝟏 + 𝝐𝒉𝒊𝒈𝒉 )

Longer steps lose more progress to curvature

Small batch

ρ=1 0

target τ

θ2

θ1

At <0

𝒍𝒐𝒈(𝟏 − 𝝐𝒍𝒐𝒘 )

Large batch

Samples to reach target τ

small

extra samples

step

less added drift

Nτ

large

≈1.7 Nτ

Clip off

Above B*: larger batches need more samples to reach τ

(D) Batch ramp

Clipped tokens (hatched) At >0

Start small: B r < B *

Clip on

Samples N

Switch before failure

Drift threshold (hypothesis) similar failure samples

large batch: longer delay

Then large + ratio clipping more protection

Progress first, stability later

Figure 2: Schematic of the two-clock model, with small (blue) and large (red) batches at the same first-order drift per sample. (A) The large batch covers this drift in fewer, longer steps that lose more progress √ to curvature, so it needs more samples to reach the target τ . (B) Token log-ratios spread with each step, about 8 times faster for the large batch, and the clip cuts tokens outside the band. (C) Unclipped, both fail at similar sample counts, and the clip delays failure, most for the large batch. (D) A batch ramp thus starts below B ∗ and switches before failure.

The derivation and the form of r0 are given in App. A.1. Progress per sample falls linearly with the batch because a larger batch covers the same drift in fewer, longer steps, and those steps lose more progress to curvature. Reaching a target loss decrease ∆Lτ requires Nτ = ∆Lτ /r(b) responses, and to first order in b, with b = Br /µ,

Proposition 2 (Failure law). Under (A1)–(A3) with a constant learning rate, a run fails after √ Dσ b , C= , (7) Nfail = C η ∥ḡ∥

Br Nτ (Br ) = Nτmin 1 + ∗ , B 2σ∥ḡ∥2 B∗ = µ − B noise , κ ḡ ⊤ H ḡ

The derivation is given √ in App. A.2. Under the square-root rule, η = κ b, so Eq. (7) reduces to

responses. A noise-driven failure mechanism would instead give Nfail ∝ b/η 2 .

(5) (6)

Nfail =

with Bnoise = tr(HΣ)/ḡ ⊤ H ḡ. Unlike the critical batch of McCandlish et al. (2018), which equals Bnoise when the learning rate is tuned at every batch, B ∗ carries a term that falls as 1/κ because the rule assigns the learning rate. B ∗ therefore decreases as the learning rate grows and can lie far below the gradient noise scale (App. A.1). Beyond the small-batch regime we use Eq. (5) as a fitting form.

C , κ

(8)

which is independent of the batch. Where the clip acts. Within a rollout, the importance ratio records how far the current policy has moved from the policy that generated the token. To first order, after k optimizer steps X log ρt ≈ ⟨∆θj , st ⟩, st = ∇θ log πθ (yt ). j<k

(9) Because the square-root rule gives larger batches longer steps, the spread of log-ratios over tokens, the ratiop spread, grows faster within a rollout, by a factor BrL /BrS at matched k (App. A.3). The clip, which zeroes the contribution of tokens whose ratio crosses its bound in the direction favored by their advantage, therefore acts more strongly on large batches. Since every ratio resets at the next

The failure clock. For failure we add one hypothesis, which §5.2 tests. (A3) Drift threshold. P A run fails when its accumulated drift, ∥ j E ∆θj ∥, reaches a threshold D over a stretch in which the direction of the mean gradient changes slowly.

rollout, the clip limits how much drift each rollout adds rather than undoing accumulated drift, and it does not generate the curvature term behind the progress gap.

cell

Predictions. The model gives three testable predictions, corresponding to the three phenomena of the introduction.

64 192 512 1024

512 1536 4096 8192

8 8 8 8

b η 64 192 512 1024

Table 1: Configurations. P is the number of prompts per rollout, Br the responses per rollout, µ the optimizer steps per rollout, and b the responses per optimizer step. We name runs by their rollout batch, S, M, L or XL, with µ=8 unless stated otherwise, and the µ=4 and µ=16 runs are trained without the clip.

Validation accuracy

P1. Progress slows beyond a critical batch. Sample efficiency degrades with the batch as the curvature term becomes important, more so at higher learning rates, and even where the clip cannot act. P2. Failure keeps time in samples. Without the clip, the square-root rule preserves the failure point in samples, and lowering the learning rate delays failure linearly, as the drift law predicts, not quadratically, as a noise-driven 1/η 2 law would.

0.25 0.20 0.15 0.10

Rollout batch 512 4,096 1,536 8,192

0.05 0

P3. The clip buys stability rather than sustained speed. Larger batches are clipped more, so the clip should delay their failure most without removing the progress gap, and widening it should give this protection up.

ηS √ √3 η S 8 ηS 4 ηS √ S / L 64 / 512 512 / 4096 1 512 / 4096 ηS / 8 ηS S 64 512 4 / 16 128 / 32 ηS S M L XL

100

Samples (thousands of responses)

Figure 3: Learning curves of four rollout batches under the square-root rule with ηS =5×10−6 and µ=8. Color marks the rollout batch, and bands span two seeds where available. The 1,536 batch closely tracks the 512 batch, while larger batches require progressively more samples.

Experimental Setup

We train Qwen2.5-Math-1.5B (Yang et al., 2024) with GRPO in verl (Sheng et al., 2025) on DAPOMath-17k (Yu et al., 2025), using G=8, temperature 1, and responses up to 3072 tokens. We use AdamW (Loshchilov and Hutter, 2019) with a short linear warm-up followed by a constant learning rate. To isolate batch-size, learning-rate, and clipping effects, we disable KL penalty, entropy bonus, dynamic sampling, and gradient-norm clipping, and test gradient-norm clipping separately. Each run has a budget of 98,304 responses. We evaluate every 8,192 responses on 512 held-out problems, averaging four samples per problem. Additional details are in App. B.

Failure criterion. A run fails at the first point at which its accuracy falls below half of the best accuracy observed so far. If this never happens, we use the first sustained gradient-norm excursion: the first rollout whose gradient norm exceeds four times its early median and is followed by at least three of the next eight rollouts above the same threshold. Runs meeting neither condition within the training budget are right-censored. Controls. We compare matched runs with clipping removed or widened, and replay frozen checkpoints on identical data with and without clipping. We repeat key comparisons on DeepScaleR-Preview (Luo et al., 2025b), Qwen2.51.5B-Instruct (Yang et al., 2025), and Qwen2.5Math-7B (Yang et al., 2024). We use two seeds where available.

Batches and learning rates. We use rollout batches of 512, 1536, 4096, and 8192 responses (S, M, L, and XL in Table 1), spanning a 16× range. Unless stated otherwise, each rollout is split into µ=8 optimizer steps. Learning rates follow the square-root rule from ηS ∈ {10−6 , 5×10−6 , 10−5 }. At 10−6 , clipping is almost never active, and we call this regime conservative. At the two larger rates, clipping becomes regularly active, and we call these regimes active.

Experimental Results

5.1

Progress slows above a critical batch (P1)

P1 predicts that larger batches need more samples once the rollout batch exceeds a critical scale, even 5

ηS

1 1 8 8 8

10−6 0.190 5×10−6 0.272 10−6 0.251 5×10−6 0.229/0.048¶ −5 10 0.199/0.013‡

8D 5×10−6 8I 2×10−6 87B 2×10−6

0.305 0.082 0.400♮

DAUC N.15 L/S

0.171 0.255 0.240 0.254 0.228

0.022 0.020 0.016 0.039 0.022

1.43 1.23 1.40 1.72 2.34

0.17 0.13 0.15 0.28 0.44

0.347 0.076 0.375♮

0.029 0.019 0.057

2.04♭ 6.2♭ 2.00♭

0.37 0.67 0.36

fore does not preserve sample efficiency. 5.2

Failure keeps time in samples (P2)

P2 predicts that without clipping, failure is governed by consumed samples rather than optimizer steps and does not move along the square-root rule. Failure aligns in samples, not steps. Removing the clip exposes this clock (Fig. 4). At the gradientnorm onset, the 4,096 batch fails after only 7–10 rollouts, or 56–80 optimizer steps, while the 512 batch lasts 81 rollouts, or 648 steps. Yet both fail after 30–50k responses.

Table 2: Small versus large batch. J is the final accuracy (seed mean), DAUC the mean distance between the learning curves (seed noise 0.005–0.009), N.15 L/S the ratio of the samples needed to reach 15% accuracy, and α the pooled sample-cost exponent. Entries marked ¶ and ‡ list both seeds, the second of which fails late. Rows marked D use DeepScaleR, I Qwen2.5-1.5BInstruct and 7B Qwen2.5-Math-7B with one seed, and values marked ♮ are at 49k samples and ♭ at a modelspecific target.

The square-root rule preserves the failure point. Across 21 unclipped runs, we vary the optimizer batch by 32× and the learning rate by nearly 6× and fit Nfail ∝ bβ η −γ (Table 4). Along the squareroot rule the failure point does not move with the batch: β − γ/2 = −0.08 over the failed runs and 0.11 with the censored runs included, and both 90% intervals contain zero, which rules out the batchblind prediction of −1/2. The fitted γ = 0.64– 0.72 rules out the random-walk prediction of 2, and of the three candidate forms only the drift form survives a likelihood-ratio test against the free fit (p ≈ 0.3, against p < 10−4 for the other two). Fig. 5 plots each run against the law, and fourteen of the eighteen failing runs lie within a factor of 1.5 of it.

when clipping is inactive. In the active regimes, roughly 10−3 of tokens are clipped, compared with fewer than 2×10−4 in the conservative regime. The gap appears without the clip. At µ=1, where the ratio clip cannot act, Table 2 shows that the large batch still needs 20–45% more samples than the small batch to reach the same accuracy. The gap also appears at µ=8 in the conservative regime and, as Fig. 4 shows, persists when the clip is removed. The progress gap therefore does not require the clip.

The failure pattern favors accumulated drift. Failure depends on the learning rate far more weakly than the quadratic law of a noise-driven random walk, and at the highest learning rates it coincides with a jump of about four orders of magnitude in the KL divergence between the updated and rollout policies. The failure point is also not an artifact of disabling gradient-norm clipping: with a norm cap of 1.0, the small batch fails after 36k samples, within the 31–46k range of its seeds.

The penalty appears beyond a critical batch. Across four rollout batches, increasing the batch from 512 to 1536 costs little, while further increases become progressively less sample-efficient (Fig. 3). With µ=8, this transition occurs at a smaller batch as the learning rate increases. The sample-cost exponent rises from 0.15 to 0.44 across the three learning-rate regimes, while the critical batch implied by Eq. (5) falls from about 9k to 2k responses (Table 2, Fig. 10, App. A.6). Below the critical batch the square-root rule is nearly sample-invariant, while above it the sample cost increasingly approaches the step-limited regime. At the active rates the large batch still ends higher in Table 2, because only the small batch decays or fails late (§5.3).

Robustness. Failure also keeps time in samples on the 7B model, with a model-specific constant. On Qwen2.5-Math-7B, the two batches fail 6× apart in rollouts but only 1.3× apart in samples. Along the square-root rule, larger batches thus fail after the same number of consumed samples despite taking far fewer optimizer steps.

Robustness. A positive sample-cost gap also appears on DeepScaleR-Preview, Qwen2.5-1.5BInstruct, and Qwen2.5-Math-7B (Table 2, App. F). The direction of the effect holds in every setting. Above the critical batch, the square-root rule there-

5.3

The clip buys stability rather than sustained speed (P3)

P3 predicts that the clip, acting on the drift, delays failure, more for the large batch, and that widen6

104

0.20

Gradient norm

Validation accuracy

0.25

0.15 0.10 0.05 0.00

103

Rollout batch 512 1,536 4,096 clip on clip off failure point

102 101 100 10−1

100

Samples (thousands of responses)

100

Samples (thousands of responses)

512 1,536 4,096 8,192 μ=4

102

Failure point (k samples)

Observed failure point (k samples)

Figure 4: Clip on and off for three rollout batches with ηS =10−5 and µ=8. Color marks the rollout batch and line style the clip (solid on, dashed off), with bands over clipped seeds. Left: accuracy with crosses at failure. Right: gradient norm with dotted lines at failure. Unclipped runs fail at similar sample counts despite unequal step counts. μ=8 μ = 16 second seed no failure

101

160 80 40 20 10

512, clip off 512, clip on 4,096, clip off

0.5 101

4,096, clip on failure law

Learning rate η (×10−5)

102

Predicted failure point C√ b /η (k samples)

Figure 6: Failure point against learning rate for the 512 and 4,096 batches with µ=8, with and without the clip. Color marks the rollout batch, filled circles runs without the clip, open squares runs with it, arrows runs that never failed, and lines the failure law, Eq. (7). Every clipped run fails later than its unclipped counterpart, and the large batch gains most.

Figure 5: Observed failure points of unclipped runs against the failure law, Eq. (7). Color marks the rollout batch, marker shape µ, and a black edge the second seed, and open markers with arrows are right-censored. The gray band spans a factor of 1.5 around equality and holds 14 of the 18 failing runs.

ratio spread of the large batch predicts.

ing it gives up this protection without closing the progress gap.

Widening the clip trades stability for a transient gain. Widening the clip reduces the clipped fraction and speeds early learning, but the gain is not sustained: performance later decays and one of two seeds collapses (App. E.2). Allowing ratios to move farther therefore gives up much of the large batch’s stability without recovering sustained sample efficiency. Overall, the clip trades a transient speed-up for stability, a trade that favors large batches.

The clip delays failure, most for the large batch. With the clip on, the small batch still fails, but later, by a factor of two (Fig. 6). The large batch, which clips several times more tokens per step, never fails within our training budget at its scaled rates. Pushed to twice its rate so that it does fail, it lasts four times longer with the clip than without, while the small batch with a similar unclipped failure point gains only a factor of two. As the model predicts, the stabilizing effect is largest for the batch on which clipping acts most strongly. A practical side effect: since clipped runs always outlive the unclipped law, Eq. (7) is a conservative estimate of when a clipped run can fail.

Implications

The two clocks give a simple batch-selection rule. Its one free choice, the switch point, is forgiving over a broad range.

What the clip removes. In replays of frozen checkpoints, only about 0.5% of the large batch’s tokens are clipped, yet their gradient has more than half the norm of the full gradient, while the share is much smaller for the small batch (App. A.4, App. E.1). Clipping therefore removes a large, batch-dependent part of the update, as the wider

Below the critical batch, use the largest efficient batch. Since progress changes little below B ∗ while larger batches are no less stable, the model favors the largest batch that remains below the critical regime. At our learning rates this is 1,536 responses: it matches the small batch’s sample ef7

Validation accuracy

ing samples. With the learning rate re-tuned at every batch, the critical batch of McCandlish et al. (2018) equals the gradient noise scale and does not depend on the learning rate. Under square-root scaling, by contrast, the learning rate is coupled to batch size, leading to a different critical batch that depends on the learning-rate scale. In LLM RL, larger batches have been found to reach higher final accuracy (Yang et al., 2026b; Khatri et al., 2026) while learning more slowly (Khatri et al., 2026). Our two-clock view connects these observations by separating the batch scale governing progress from the sample scale governing failure.

0.25 0.20 0.15 0.10 0.05

fixed 512 fixed 1,536 fixed 4,096

switch

fixed 8,192 ramp 512 → 4,096 ramp 512 → 8,192

100

Samples (thousands of responses)

Figure 7: Batch ramp with ηS =5×10−6 and µ=8. Each ramp uses batch 512 for 24.6k samples (dotted line), then switches to 4,096 or 8,192 (two seeds each), and thin lines are fixed batches. Both ramps lie well above the fixed large batches, near the best fixed batch.

Training collapse. Batch scaling also affects training stability. Suk and Duan (2025) derive a step-size threshold above which RLVR collapses, while Song et al. (2026) find that collapse time in asynchronous GRPO scales roughly as 1/η, the fixed-batch case of Eq. (7), and attribute it to directional drift. We add the batch dimension: along the square-root rule, failure is approximately invariant in consumed samples.

ficiency to within seed noise, does not fail, and achieves the best final accuracy among the fixed batches. Above the critical batch, ramp the batch. When a larger batch is needed for throughput, train first with a small batch and switch, before its predicted failure point, to a larger and more stable batch. We reset the optimizer state and repeat the one-rollout warm-up at the switch. The resulting ramps lie well above the fixed large-batch curves and track the best fixed batch, while generating three quarters of their samples with the large batch (Fig. 7). App. G finds that switches between 25k and 74k samples perform similarly, and the failure law bounds how late the switch can come.

Ratio clipping. Training stability in GRPO is closely tied to ratio clipping. Trust-region methods replace or soften the hard clip (Qi et al., 2026; Dwyer et al., 2025; Yang et al., 2026a; Jia et al., 2026), while Tian et al. (2026) relax it to enable stale-rollout reuse and Khatri et al. (2026) show that the clip threshold changes RL scaling behavior. In contrast to work that treats clipping as a bottleneck (Qi et al., 2026; Yang et al., 2026a), we find that it mainly buys stability, at a transient cost in speed.

Related Work

Batch-size scaling and invariance. Batch size is a basic scaling variable in optimization, affecting both computational efficiency and training dynamics. The linear rule for SGD (Smith et al., 2018), which also motivates growing the batch during training, and the square-root rule for Adam (Malladi et al., 2022) keep training trajectories comparable across batch sizes. Hilton et al. (2022) make PPO batch-size invariant by decoupling the clip’s proximal policy from the behavior policy, given a sufficiently small optimizer batch, while Li et al. (2026) show that square-root scaling keeps GRPO learning curves aligned with one optimizer step per rollout over a bounded range of batch sizes. In contrast, we study the multi-step regime, where this batch-size invariance no longer holds and drift preservation does not imply progress preservation.

Conclusion

Scaling rules are often interpreted as promises of invariance. For multi-step GRPO, the squareroot rule instead preserves first-order drift per sample, not progress per sample, yielding two clocks: progress slows beyond a learning-rate-dependent critical batch, while a drift-threshold hypothesis predicts an approximately sample-invariant failure point. Our experiments support these predictions and show that ratio clipping buys stability, especially for larger batches, while the progress gap persists without it. Together, the two clocks suggest a simple strategy for multi-step RLVR: train below the critical batch for efficient progress, then switch to a larger batch before failure when throughput demands it.

Critical batch. Beyond a critical batch, larger batches stop saving optimizer steps and start cost8

Limitations

clipping policy optimization. arXiv:2609.00444.

We broaden our evaluation across multiple checkpoints, two datasets, and a 7B model, but compute and resource constraints still limit our coverage to one model family and one task domain. Our theory is intentionally local, and direct measurement of the curvature term is computationally expensive at this scale, so we evaluate it through its testable predictions. Failure points also exhibit intrinsic variation across seeds. Finally, we measure sample efficiency in responses, while token, compute, and wall-clock efficiency additionally depend on generation lengths, hardware, and system implementation.

arXiv preprint

Devvrit Khatri, Lovish Madaan, Rishabh Tiwari, Rachit Bansal, Venkata Sai Surya Subramanyam Duvvuri, Manzil Zaheer, Inderjit Dhillon, David Brandfonbrener, and Rishabh Agarwal. 2026. The art of scaling reinforcement learning compute for LLMs. In International Conference on Learning Representations, volume 2026, pages 72438–72467. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23, page 611–626, New York, NY, USA. Association for Computing Machinery.

Ethical Considerations

Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James Validad Miranda, Alisa Liu, Nouha Dziri, Xinxi Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Christopher Wilhelm, Luca Soldaini, and 4 others. 2025. Tulu 3: Pushing frontiers in open language model post-training. In Second Conference on Language Modeling.

This work analyzes the training dynamics of RLVR on public math datasets with open-weight models and releases no new model or dataset. Its main societal relevance is environmental: our batch schedule aims to reduce the number of generated samples a large batch needs to reach a given accuracy, although our own study required many training runs (App. B reports the hardware). We see no additional risks beyond those of LLM post-training in general.

Ziniu Li, Jinbo Wang, Guanhua Huang, Feiyuan Zhang, Pengbo Li, and Alex Chen. 2026. When do larger batches help scale LLM reinforcement learning? arXiv preprint arXiv:2608.29296. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.

References Madeleine Dwyer, Adam Sobey, and Adriane Chapman. 2025. It’s not you, it’s clipping: A soft trustregion via probability smoothing for LLM RL. arXiv preprint arXiv:2509.21282.

Michael Luo, Sijun Tan, Roy Huang, Ameen Patel, Alpay Ariyak, Qingyang Wu, Xiaoxiang Shi, Rachel Xin, Colin Cai, Maurice Weber, Ce Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025a. DeepCoder: A fully open-source 14B coder at O3-mini level. Together AI Blog.

Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprint arXiv:2501.12948.

Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. 2025b. DeepScaleR: Surpassing O1-Preview with a 1.5B model by scaling RL. https://pretty-radio -b75.notion.site/DeepScaleR-Surpassing-O 1-Preview-with-a-1-5B-Model-by-Scaling-R L-19681902c1468005bed8ca303013a4e2. Notion Blog.

Jacob Hilton, Karl Cobbe, and John Schulman. 2022. Batch size-invariance for policy optimization. In Advances in Neural Information Processing Systems, volume 35, pages 17086–17098. Thomas Hubert, Rishi Mehta, Laurent Sartran, Miklós Z. Horváth, Goran Žužić, Eric Wieser, Aja Huang, Julian Schrittwieser, Yannick Schroecker, Hussain Masoom, Ottavia Bertolli, Tom Zahavy, Amol Mandhane, Jessica Yung, Iuliya Beloshapka, Borja Ibarz, Vivek Veeriah, Lei Yu, Oliver Nash, and 19 others. 2025. Olympiad-level formal mathematical reasoning with reinforcement learning. Nature.

Sadhika Malladi, Kaifeng Lyu, Abhishek Panigrahi, and Sanjeev Arora. 2022. On the SDEs and scaling rules for adaptive gradient algorithms. In Advances in Neural Information Processing Systems, volume 35, pages 7697–7711. Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAI Dota Team. 2018. An empirical model of large-batch training. arXiv preprint arXiv:1812.06162.

Sheng Jia, Xiao Wang, Shiva Prasad Kasiviswanathan, and Rein Houthooft. 2026. Group adaptive

Penghui Qi, Xiangxin Zhou, Zichen Liu, Tianyu Pang, Chao Du, Min Lin, and Wee Sun Lee. 2026. Rethinking the trust region in LLM reinforcement learning. In Forty-third International Conference on Machine Learning.

An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. 2024. Qwen2.5Math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.

Shuo Yang, Jinda Lu, Chiyu Ma, Kexin Huang, Haoming Meng, Qihui Zhang, Yuyang Liu, Bolin Ding, Guoyin Wang, Li Yuan, et al. 2026a. Clipping bottleneck: Stabilizing RLVR via stochastic recovery of near-boundary signals. arXiv preprint arXiv:2605.22703.

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. HybridFlow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems, EuroSys ’25, page 1279–1297, New York, NY, USA. Association for Computing Machinery.

Zhicheng Yang, Zhijiang Guo, Yinya Huang, Yongxin Wang, Dongchun Xie, Hanhui Li, Yiwei Wang, Xiaodan Liang, and Jing Tang. 2026b. Depth-breadth synergy in RLVR: Unlocking LLM reasoning gains with adaptive exploration. In Forty-third International Conference on Machine Learning.

Samuel L. Smith, Pieter-Jan Kindermans, and Quoc V. Le. 2018. Don’t decay the learning rate, increase the batch size. In International Conference on Learning Representations.

Deheng Ye, Zhao Liu, Mingfei Sun, Bei Shi, Peilin Zhao, Hao Wu, Hongsheng Yu, Shaojie Yang, Xipeng Wu, Qingwei Guo, et al. 2020. Mastering complex control in MOBA games with deep reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 6672–6679.

Jingwei Song, Haofeng Xu, Jie Xiao, Chengke Bao, Jingwei Shi, Pengbin Feng, Yuhang Han, Weixun Wang, Eric Yang, and Tianyu Shi. 2026. Scaling laws for collapse in asynchronous GRPO. arXiv preprint arXiv:2607.01083.

Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. 2025. DAPO: An open-source LLM reinforcement learning system at scale. In Advances in Neural Information Processing Systems, volume 38, pages 113222–113244.

Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. 2025. Crossing the reward bridge: Expanding RL with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829. Joe Suk and Yaqi Duan. 2025. On the optimization dynamics of RLVR: Gradient gap and step size thresholds. arXiv preprint arXiv:2510.08539. Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. 2025. Kimi k1.5: Scaling reinforcement learning with LLMs. arXiv preprint arXiv:2501.12599. Minghao Tian, Yunfei Xie, and Chen Wei. 2026. How off-policy can GRPO be? Mu-GRPO for efficient LLM reinforcement learning. arXiv preprint arXiv:2605.17570. Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida Wang. 2025. SWE-RL: Advancing LLM reasoning via reinforcement learning on open software evolution. In Advances in Neural Information Processing Systems, volume 38, pages 78500–78525. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115.

√ √ With η = κ b we have η/ b = κ and η 2 /b = κ2 , which yields Eq. (4) with

Theory and Derivations

This appendix gives the derivations behind §3 in full: the progress law (A.1), the failure law (A.2), the ratio spread (A.3), the share of the clipped tokens (A.4), Adam’s per-coordinate normalization and larger batches (A.5), and the critical batches our measurements imply (A.6). Throughout, b = Br /µ is the optimizer batch, ĝ = ḡ + ξ the gradient estimate from b responses with E ξ = 0 and Cov ξ = Σ/b, σ 2 = tr Σ, and H the local Hessian. A.1

r0 =

From the rate to the critical batch. Write c = Ccurv for the slope in Eq. (4). If r is roughly constant over the stretch of training in which the loss falls by ∆Lτ , the samples needed are

The normalized step. The scalar normalizer is ν 2 = E∥ĝ∥2 . Since ξ has mean zero, σ2 b σ2 = 1+ , ν = ∥ḡ∥ + b b Bsimple 2

cb ∆Lτ ∆Lτ 1+ +O (c b/r0 )2 . = r0 − c b r0 r0 (18) Keeping the first-order term and writing b = Br /µ gives Eq. (5) with Nτmin = ∆Lτ /r0 and Nτ =

(10)

2 with Bsimple = σ 2 /∥ḡ∥ √ . Under (A1), b ≪ Bsimple , so ν ≈ σ/ b to leading order in b/Bsimple , and the step ∆θ = −η ĝ/ν becomes √ η b (ḡ + ξ). (11) ∆θ ≈ − σ

B∗ = µ

(12)

Substituting Eq. (11) and taking the expectation over ξ term by term: the linear term gives √ η b E ⟨ḡ, ∆θ⟩ = − ∥ḡ∥2 , (13) σ

α=

(19)

d log Nτ Br Br /B ∗ = . (20) = ∗ d log Br 1 + Br /B Br + B ∗

Contrast with a learning rate tuned at every batch. For SGD with the learning rate ϵ re-tuned at every batch (McCandlish et al., 2018), the same 2 expansiongives E ∆L = −ϵ∥ḡ∥2 + ϵ2 ḡ ⊤ H ḡ + tr(HΣ)/b . Minimizing over ϵ gives the best decrease per step,

because E ξ = 0. The quadratic term gives η2b ⊤ ⊤ ḡ H ḡ + E ξ Hξ σ2 η2 = 2 b ḡ ⊤ H ḡ + tr(HΣ) , (14) σ

E ∆θ⊤ H∆θ =

because the cross terms vanish and E ξ ⊤ Hξ = tr(H Cov ξ) = tr(HΣ)/b. Together, √ η b η2 E ∆L = − ∥ḡ∥2 + 2 b ḡ ⊤ H ḡ +tr(HΣ) . σ 2σ (15)

−E ∆L∗ =

∥ḡ∥4 1 · , ⊤ 2 ḡ H ḡ 1 + Bnoise /b

(21)

with Bnoise = tr(HΣ)/ḡ ⊤ H ḡ, the noise scale weighted by curvature. The samples per unit of progress are then proportional to b (1+Bnoise /b) = b + Bnoise , so the critical batch equals Bnoise and does not depend on the learning rate. Under the square-root rule the learning rate is not tuned but assigned, and Eq. (19) equals µ times 2σ∥ḡ∥2 /(κ ḡ ⊤ H ḡ) − Bnoise : the first term falls as 1/κ, which is why B ∗ moves with the learning rate.

Proof of Proposition 1. One step consumes b responses, so the expected loss decrease per consumed response is r = −E ∆L/b. Dividing Eq. (15) by b gives η ∥ḡ∥2 η 2 b ḡ ⊤ H ḡ + tr(HΣ) r=√ − . b 2σ 2 b σ

2σ∥ḡ∥2 r0 tr(HΣ) =µ − , c κ ḡ ⊤ H ḡ ḡ ⊤ H ḡ

which is Eq. (6). The expansion requires r0 > 0, that is κ < 2σ∥ḡ∥2 /tr(HΣ), and c b ≪ r0 , that is Br ≪ B ∗ . The local sample-cost exponent of Definition 2 follows by differentiating the logarithm of Eq. (5):

Expected loss change. Under (A2) the loss change over one step is ∆L = ⟨ḡ, ∆θ⟩ + 12 ∆θ⊤ H∆θ.

(17)

The first two terms do not depend on b, and only the curvature of the mean step, b ḡ ⊤ H ḡ, grows with the batch. □

Progress per sample

κ ∥ḡ∥2 κ2 tr(HΣ) . − σ 2σ 2

(16) 11

Noise per sample. The noise part of one step in √ Eq. (11) is −(η b/σ) ξ, with covariance η2 Σ η2b Σ · = Σ = κ2 b 2 . 2 2 σ b σ σ

the rule therefore separates the drift law from the batch-blind law, and the learning-rate exponent γ separates it from the noise law, which is how the fits in Table 4 tell the three apart.

(22)

Corollary 3 (Two clocks). Under (A1)–(A3) and the square-root rule at fixed µ, the failure point of Eq. (7) does not depend on Br , while the samples to a target, Nτ = ∆Lτ /r(Br /µ), increase with Br .

Fresh mini-batches give independent noise, so over the N/b steps of a run consuming N responses the covariances add to (N/b) κ2 b Σ/σ 2 = N κ2 Σ/σ 2 . The noise covariance accumulated per sample, κ2 Σ/σ 2 , is therefore independent of the batch, as is the drift per sample. A.2

Proof. Eq. (25) gives Nfail = C/κ, and κ is fixed by the rule. By Eq. (4), r decreases linearly in b = Br /µ whenever ḡ ⊤ H ḡ > 0, so Nτ = ∆Lτ /r increases with Br .

Failure law

Proof of Proposition 2. √ By Eq. (11), the mean of step j is E ∆θj = −(η b/σ) ḡj , where ḡj is the mean gradient at step j. Over the stretch assumed in (A3) the direction of ḡj changes slowly, so the means add almost collinearly, and after k steps √ X η b E ∆θj ≈ k ∥ḡ∥. (23) σ

On (A3). The threshold assumption is the model’s one empirical hypothesis, and the failurelaw fit of §5.2 tests the exponents it predicts. The threshold concerns movement along the slowly varying mean-gradient direction, where drift accumulates linearly in the samples consumed, whereas noise, spread over many directions, contributes only a random-walk component. It predicts γ = 1 and β = 12 , the noise-driven alternative γ = 2 and β = 1, and a batch-blind drift γ = 1 and β = 0. The fit excludes the second, since Table 4 gives γ ≈ 0.7, and pins the combination β − γ/2 that the scaling rule probes near zero, which (A3) predicts and the third alternative does not.

j<k

The run fails when this reaches D, after Dσ kfail = √ η b ∥ḡ∥

(24)

steps. Each step consumes b responses, so √ √ Dσ b b Nfail = kfail b = · =C . (25) ∥ḡ∥ η η √ Under the square-root rule η = κ b, so Nfail = C/κ. □

Clip-active steps. For the same N samples a run takes N/b optimizer steps, of which the fraction (µ − 1)/µ with k ≥ 1 can clip: 1344 for S against 168 for L at 98k. If failure required accumulating some number of clip-active steps, the small batch would reach it after eight times fewer samples. The unclipped runs rule this out: at the gradient-norm onset, L fails after 7 rollouts, or 56 optimizer steps (29k samples), and S after 81 rollouts, or 648 optimizer steps (41k), a 1.4× difference in samples against 12× in steps. The underlying instability counts samples, as (A3) assumes, and the different clipped failure points must come from the clip’s protection differing between the batches (Proposition 5).

Alternative mechanisms. If failure were instead driven by the noise part of the steps, the relevant displacement would be the sum of independent noise terms. By Eq. (22) each has total variance tr(η 2 Σ/σ 2 ) = η 2 , so after √ k steps the root-meansquare displacement is k η. It reaches D after k ≈ D2 /η 2 steps, or noise Nfail ≈

b D2 η2

(26)

responses. A batch-blind mechanism, in which failure depends only on the learning rate, gives Nfail ∝ 1/η. Writing each as Nfail ∝ bβ η −γ , the drift law has (β, γ) = ( 12 , 1), the noise law (1, 2) and the batch-blind √ law (0, 1). Along the squareroot rule η ∝ b, so Nfail ∝ bβ−γ/2 : the drift and noise laws both predict β − γ/2 = 0, while the batch-blind law predicts − 12 . Invariance along

A.3

Ratio spread

The log-ratio as accumulated movement. Let θ0 be the policy that generated a rollout and θk the policy after k optimizer steps on it. To first order in the steps, log πθk (yt ) − log πθ0 (yt ) ≈ ⟨θk − θ0 , st ⟩, 12

with the score st = ∇θ log πθ0 (yt ), so X log ρt (k) ≈ ⟨∆θj , st ⟩.

Proposition 5 (Share of the clipped tokens). Let g and g clip be the gradients of the unclipped and clipped objectives on the same mini-batch, q = ∥g clip ∥/∥g∥ and dclip = 1 − cos(g clip , g). The clipped tokens contribute gC = g − g clip , and

(27)

j<k

Define the score spread σs = sup∥v∥=1 stdt ⟨v, st ⟩, the largest standard deviation over tokens of the score projected onto a unit direction.

∥gC ∥ = ∥g∥

q 1 + q 2 − 2q(1 − dclip ) q ≥ dclip (2 − dclip ).

Lemma 4 (Ratio spread). To first order, the spread of log ρ over tokens after k steps is at most k η σs , and this bound is reached when the steps are aligned. When the√steps are uncorrelated, the spread is of order k η σs in root-mean-square. In either case it is proportional to η, so under the square-root p rule at matched k the large batch’s spread is BrL /BrS times the small batch’s.

Proof. The clipped objective’s gradient omits exactly the contributions of the clipped tokens, so g = g clip + gC . By the law of cosines, ∥gC ∥2 = ∥g∥2 + ∥g clip ∥2 − 2 ∥g∥ ∥g clip ∥ cos(g clip , g),

Proof. Under (A1) the normalized update u = √ b (ḡ + ξ)/σ has E∥u∥2 = 1 + b/Bsimple ≈ 1, so each step has norm close to η. Each term of Eq. (27) therefore has spread over tokens stdt ⟨∆θj , st ⟩ ≤ ∥∆θj ∥ σs ≈ η σs . The standard deviation of a sum is at most the sum of the standard deviations, which gives k η σs , with equality when the k terms are perfectly correlated across tokens. If the terms √ are uncorrelated, their variances add, giving k η σs in root-mean-square. For the batch dependence, √ note that under (A1) the noise part of u, b ξ/σ, has covariance Σ/σ 2 , independent of b, so the step directions have the same statistics at every batch and the spread scales exactly with p η. At fixed µ the square-root rule sets ηL /ηS = BrL /BrS .

(29)

and dividing by ∥g∥2 gives the equality. For the bound, minimize f (q) = 1+q 2 −2q(1−dclip ) over q ≥ 0. If dclip ≤ 1, the minimum is at q = 1−dclip with value 1 − (1 − dclip )2 = dclip (2 − dclip ). If dclip > 1, the minimum is at q = 0 with value 1 ≥ dclip (2 − dclip ). The bound uses the angle alone and holds whatever the norm ratio. Measured share. Both q and dclip are measured in every replay, so Eq. (28) is exact per step. In the active regime the gradient of the tokens the clip removes, 0.5% of the mini-batch for L, has 50– 70% of the norm of the full gradient at every k ≥ 2 (mean 0.50–0.55 over k, and up to 0.9–1.7 at the earliest checkpoint, step 2, where the clipped part is larger than the full gradient). For S, the 0.05% of clipped tokens give 10–20% (mean 0.13–0.15), rising to 0.42 with a step at 1.0 in the pre-collapse checkpoint of seed 1. In the conservative regime the share is ≤ 0.14 at every step, the level set by the few clipped tokens’ sampling noise. Clipping in the active regime therefore does not trim a tail: it removes a dominant component of the update, and a component that is dominant in the large batch at every step of the rollout.

Measured ratio spread. The replay measures the spread of Lemma 4 at matched k on the final active checkpoints used in §5.3: the 99th percentile of | log ρ| is 1.05, 1.9, 3.1, 2.7, 2.7, 2.8, 2.8 times larger for L than for S at k = 1,√. . . , 7. From k=3 on the ratio is the predicted 8 ≈ 2.8. At k ≤ 2 both batches sit near the 0.09–0.10 floor that the conservative runs show at every k. Fig. 12 shows that the clipped fraction rises with k for both batches, and its ratio between the batches grows from 1.1–1.6 at k=1 to 2–4 from k=3 on, around the spread ratio. A.4

(28)

A.5

Share of the clipped tokens

Per-coordinate normalization and larger batches

Per-coordinate normalization. Adam divides √ coordinate i by vi , with vi = E ĝi2 = ḡi2 + Σii /b. If (A1) holds coordinate-wise, b ≪ Σii /ḡi2 , then vi ≈ Σii /b and √ ∆θ ≈ −η b P (ḡ + ξ), P = diag(Σii )−1/2 . (30)

Our replay accumulates the squared norms of the clipped and unclipped gradients and their inner product over parameter shards, without storing either gradient. The following identity turns these three numbers into the share of the update carried by the clipped tokens. 13

Repeating the calculation of A.1 with this step gives r(b) = κ ḡ ⊤ P ḡ − −b

κ2 2

Design choices. We disable gradient-norm clipping because a fixed norm cap acts differently across batch sizes and therefore interferes with the square-root rule, and we test it separately in §5.2. In the sustained-excursion rule, when fewer than eight rollouts remain in a run, two of the remaining rollouts above the threshold suffice. The failure criterion checks accuracy first because some runs collapse quietly, with accuracy falling while the gradient norm stays at its baseline, and App. D lists these runs.

tr(P HP Σ)

κ2 ⊤ ḡ P HP ḡ, 2

(31)

which has the structure of Eq. (4), with ḡ ⊤ P ḡ in place of ∥ḡ∥2 /σ, tr(P HP Σ) in place of tr(HΣ)/σ 2 and ḡ ⊤ P HP ḡ in place of ḡ ⊤ H ḡ/σ 2 . The drift per sample is κ∥P ḡ∥, again independent of b, and the failure law keeps the form of Eq. (7) with C = D/∥P ḡ∥. Momentum averages consecutive steps, which changes the constants but not the dependence on b, as long as the gradient changes slowly over the averaging window. Remark 6 (Outside (A1)). By Eq. (10), the firstorder progress per sample is κ ∥ḡ∥2 b −1/2 η ∥ḡ∥2 = 1+ . bν σ Bsimple

prompts per rollout P 64 (S), 192 (M), 512 (L), 1024 (XL) responses per prompt G 8 optimizer steps per rollout µ 8 (main), 1, 4 and 16 (controls), mini-batch of P/µ prompts learning rate ηS ∈ {10−6 , 5×10−6 , 10−5 } and the clip-off sweeps of App. D, other batches by the square-root rule schedule linear warm-up over 8192 responses, then constant optimizer AdamW, β = (0.9, 0.999), weight decay 0.1 gradient-norm clip disabled (106 ), 1.0 in one control ratio clip εlow = 0.2, εhigh = 0.28, dual clip c = 10 KL / entropy terms none loss aggregation token mean over the mini-batch advantage GRPO group normalization, ϵ = 10−6 sampling temperature 1.0, top-p 1.0, top-k off (training and evaluation) lengths prompt ≤ 1024, response ≤ 3072 tokens, truncated responses kept without penalty micro-batch 4 sequences, fixed data DAPO-Math-17k (main), DeepScaleRPreview (second dataset), less than one epoch 512 held-out prompts, mean@4, every 8192 validation responses precision bf16 training and rollouts, fp32 master weights and gradient reduction parallelism 4-GPU FSDP with parameter and optimizer offload, one vLLM engine per GPU software verl (Sheng et al., 2025) 0.9.1, vLLM (Kwon et al., 2023) 0.19

(32)

For b ≪ Bsimple this is the batch-independent first term of √ Eq. (4). When b approaches Bsimple it falls as 1/ b, so even without curvature the samplecost exponent rises toward α = 21 , and once the curvature term dominates, toward α = 1. Our optimizer batches are 60 to 2000 times below the measured Bsimple , so this regime does not explain our gaps. A.6

Implied critical batch

Remark 7 (Implied critical batch). For a family with a small batch BS and a large batch 8BS , let R = Nτ (8BS )/Nτ (BS ) = 8α be the measured ratio of samples to the same target. Under Eq. (5), R=

1 + 8BS /B ∗ , 1 + BS /B ∗

Table 3: Hyperparameters shared by all runs.

Budgets. Unless stated otherwise, runs consume 98,304 responses, and selected failure probes use longer budgets.

(33)

and solving for B ∗ gives (8 − R) BS B∗ = , R−1

Experimental Details

Choosing the learning rates. We first ran 24rollout probes of the small batch over η ∈ {10−6 , 5×10−6 , 10−5 , 2×10−5 }. Clipping was essentially inactive at 10−6 , became regularly active at 5×10−6 and 10−5 , and was stronger at 2×10−5 . We therefore use 10−6 as the conservative rate and the next two as the active rates. Separate ladders are used for µ=1, Qwen2.5-1.5B-Instruct, and Qwen2.5-Math-7B, and App. F describes the Instruct ladder.

(34)

with BS = 512 responses per rollout. At µ=8, our exponents give B ∗ ≈ 9.3k, 4.0k (a direct fit over four batches gives 2.2–4.2k) and 1.9k at ηS =10−6 , 5×10−6 and 10−5 . The critical batch thus falls as the learning rate grows, in the direction Eq. (6) predicts. All three values lie well below Bsimple : as Eq. (6) shows, under the square-root rule the critical batch carries a term that scales as 1/κ and is not set by the noise scale alone.

Exact ratios at the first step. Dynamic token packing makes the rollout and the recomputed log14

the clip. One appears anyway. With µ=1, where the clip cannot act at all, the large batch lags the small one by 0.022 and 0.021 in the two seed pairs and needs 35–50% more samples to reach 15% accuracy (left panel of Fig. 11), about four times the DAUC =0.005 that separates two seeds of the same configuration. At µ=8 the lag is similar (DAUC =0.016, with 30k against 22k samples to 15% accuracy).

probabilities use differently shaped micro-batches, and under bf16 this produces spurious deviations from ρ = 1 at k=0, with the 99th percentile of | log ρ| reaching 0.03–0.12. Fixed micro-batches of four sequences in identical order make the ratio exactly one. All runs use this setting except the seed-0 small-batch runs in the conservative regime, where this noise stays inside the clip range: no token is clipped at k=0 in that regime, as App. C confirms. Hardware and compute. The 1.5B experiments used four NVIDIA RTX A6000 GPUs per run, while the 7B experiments used four H100 GPUs. In total, the experiments used roughly 2,000 GPUhours, including preliminary runs and the replay analyses.

Additional Results for the Progress Clock

Validation accuracy

μ = 1 (clip cannot act)

μ=8

0.25

This appendix gives the evidence behind §5.1. In Table 2, α is the slope of log Nτ against log Br , pooled over the accuracy targets each pair reaches and, for the 5×10−6 family, over the middle batch as well.

0.20 0.15 0.10

Rollout batch 512 4,096

0.05 0

100 0

100

Samples (thousands of responses)

Learning-rate dependence. At µ=8 the small batch’s lead over the large batch grows from the conservative to the active rates, as Fig. 10 shows, the dependence that the sample-cost exponents of Table 2 summarize. At the highest rate the lead closes late in training, when the small batch reaches its failure point inside the budget.

Validation accuracy

ηS = 10−6

ηS = 5×10−6

Figure 11: The 512 and 4,096 batches in the conservative regime, ηS =10−6 , with µ=1 (left, two seeds) and µ=8 (right, one seed). Color marks the rollout batch, and bands span the two seeds. The larger batch lags in both, although less than 2×10−4 of its tokens are clipped.

ηS = 10−5

0.2

0.1

0.0

Rollout batch 512 4,096

100 0

100

Samples (thousands of responses)

Figure 10: The 512 and 4,096 batches at three learningrate levels with µ=8. √ Color marks the rollout batch, the larger batch uses 8 times the smaller batch’s rate ηS , and bands span two seeds where available. The small batch’s lead grows with the rate, and at the highest rate it closes late, when the small batch starts to fail.

Clipped fraction by step. Lemma 4 predicts that clipping builds up over the steps of a rollout, and faster for the large batch. Figure 12 shows both. The clipped fraction rises with k at every learning rate, and the large batch’s excess grows from 1.1–1.6 times the small batch’s at k=1 to about 2–4 times from k=3 on, so the difference between batches accumulates within a rollout rather than being present from the first step.

Conservative regime. At ηS =10−6 fewer than 2×10−4 of tokens are clipped, and none at k=0, so any gap that appears here cannot come from 15

μ = 1, ηS = 10−6

μ = 8, ηS = 10−6

μ = 8, ηS = 5×10−6

μ = 8, ηS = 10−5

512 vs 4,096

0.1

Accuracy gap

μ = 1, ηS = 5×10−6

0.0 −0.1 −0.2 0

100 0

100

Samples (thousands of responses)

Figure 8: Accuracy gap between the 512 and 4,096 batches, J512 (N ) − J4096 (N ), for all five pairs. The two left panels have µ=1, where the clip cannot act, and the three right panels µ=8. Bands cover all seed pairings, and the gray band marks ±1 accuracy point, the reference band that prior one-step scaling results use to judge approximate invariance (Li et al., 2026). At µ=8 the conservative gap is an early lead of 2–4 points that decays into the band, the 5×10−6 gap reaches 8 points and stays above the band until the small batch decays late, and the 10−5 gap peaks at 9 points early and turns negative when the small batch fails. μ = 1, ηS = 10−6

μ = 1, ηS = 5×10−6

μ = 8, ηS = 10−6

μ = 8, ηS = 5×10−6

Samples to target (k)

100

μ = 8, ηS = 10−5 to 10% to 15% to 20% to 25%

50 20 10 5 512

4,096 512

Rollout batch

4,096 512

Rollout batch

4,096 512

Rollout batch

4,096 512

Rollout batch

4,096

Rollout batch

Clipped token fraction

Figure 9: Samples needed to first reach each accuracy target against the rollout batch, on log–log axes, for every pair. The slope of each line is the sample-cost exponent α of Table 2, a missing point means that the run never reaches the target within 98k samples, and the family with ηS =5×10−6 , µ=8 and all four batches is in Fig. 1 (left). The samples needed rise with the batch at every target.

The gap over training. Table 2 condenses each pair of learning curves into a few numbers, and Figs. 8 and 9 show what lies behind them. With µ=1, where the clip cannot act, the gap persists throughout training. With µ=8 it opens early and widens with the learning rate, and at the active rates it closes or reverses only when the small batch begins to fail late. The samples needed rise with the batch at every accuracy target, so the cost does not hinge on the 15% threshold of Table 2.

10−3

10−4

Optimizer step k within the rollout 512 4,096

ηS = 10−6 ηS = 5×10−6

ηS = 10−5

Figure 12: Clipped token fraction at each optimizer step k within a rollout for the 512 and 4,096 batches at three learning-rate levels with µ=8, averaged over the first 49k samples, before any of these runs fails. Color marks the rollout batch and line style the learning-rate level, and bands span two seeds where available. Nothing is clipped at k=0, both fractions then rise with k, and the large batch’s is 1.1–1.6 times the small batch’s at k=1 and 2–4 times from k=3 on.

Failure-Law Fits and Failure Points

Table 5 lists both failure criteria for every run in the active regimes. Using three or five times the early median instead of four shifts the detected gradient-norm onset by at most four rollouts, except for S at 5×10−6 (rollouts 151, 156 and 165 for seed 1, while seed 0’s excursion at rollout 187 does not reach 5×). Onsets are counted in rollouts, each of µ optimizer steps. Some runs fail without a gradient-norm excursion: accuracy falls while 16

β − γ/2

least squares (18 runs) Tobit (21 runs, 3 censored)

0.24 [0.00, 0.52] 0.47 [0.19, 0.72]

0.64 [0.29, 0.96] 0.72 [0.32, 1.04]

−0.08 [−0.23, 0.09] 0.11 [−0.09, 0.32]

fixed form √ b/η (Eq. 7, C=0.044) b/η 2 (random walk) 1/η (batch-blind) free (Tobit)

rms factor

log-lik.

LR test

1.53 2.32 1.68 1.57

−15.3 −24.7 −23.9 −14.0

p ≈ 0.29 p < 10−4 p < 10−4 –

free fit

Table 4: Fits of the failure law Nfail ∝ bβ η −γ . Brackets give 90% bootstrap intervals, and the square-root rule moves the failure point by the factor bβ−γ/2 . The rms factor is the geometric rms residual of the failing runs, or the fitted noise scale for Tobit, the log-likelihood is censored, and the LR test compares each fixed form with the free Tobit fit. clip onset rollout onset N acc. halves

run −5

S 10 seed 0 S 10−5 seed 1 S 5×10−6 seed 0 S 5×10−6 seed 1 S 1.7×10−5 S 2×10−5 S 2.8×10−5 L 5.7×10−5 S 10−5 seed 0 S 10−5 seed 1 M 1.7×10−5 S 10−5 seed 1♮ S 10−5 + grad-clip 1.0 S 5×10−6 S 2×10−5 S 5×10−6 seed 1 S 1.7×10−5 S 2.8×10−5 S 10−5 (µ=4) S 10−5 (µ=4) seed 1

on on on on on on on on off off off off off off off off off off off off

†

none 153 187 156 none none none none 81 none in 72 none in 40§ 61 68 145 35♯ 82 none 17 197 45

– 78k 96k 80k – – – – 41k >37k – 31k 35k 74k 18k 42k – 9k 101k 23k

– 83k – 92k 71k 36k 38k 87k 46k – 51k – 36k – 19k 45k 29k 13k 86k 26k

run −5

S 10 (µ=16) M 1.7×10−5 seed 1 M 10−5 M 10−5 seed 1 L 2.8×10−5 s0/s1 L 1.4×10−5 s0/s1 L 2.8×10−5 seed 0 L 2.8×10−5 seed 1 L 1.4×10−5 L 10−5 seed 0 L 10−5 seed 1 L 5.7×10−5 XL 4×10−5 DeepScaleR L 2.8×10−5 7B S 2×10−6 7B L 5.7×10−6 7B S 10−6 L 2.8×10−5 s0 L 2.8×10−5 s1

clip

onset rollout onset N acc. halves

off off off off on on off off off off off off off off off off off √ ε×√8 ε× 8

none 20 48 24 none in 24 none in 24 7 10 none in 15 none in 36 8 none none in 10 10 81♯ none†† none in 192 none‡ 22

– 31k 74k 37k >98k >98k 29k 41k >61k >147k 33k – >82k 41k 41k – >98k – 90k

34k 33k 81k 48k – – 29k 48k – – 29k†† 21k –♭ 52k 45k 53k – – 93k

Table 5: Failure points of every run in the active regimes under both parts of the failure criterion (§4). The columns give the sustained gradient-norm onset in rollouts and in samples, and the sample at which accuracy first halves from √ its best value, with – for runs that never halve, and clip ε× 8 is the widened radius. Marks denote a decay from the peak without a blow-up († ), a decay from 57k (‡ ), a quiet collapse in which accuracy falls while the gradient norm stays near baseline except for one spike (§ ), a second seed rerun on different hardware (♮ ), a 40% fall in accuracy from 41k without halving (♭ ), a collapse followed by recovery (†† ), and a KL blow-up with every response at the length cap and the gradient exactly zero afterwards, rather than a gradient-norm excursion (♯ ).

the gradient norm remains near baseline, or the KL divergence between the updated policy and the rollout policy blows up, after which every response hits the length cap and the gradient is exactly zero. Two large-batch runs collapse and then recover. The accuracy criterion detects these cases, so the failure definition in §4 gives it precedence.

samples). We also hold the exponents at three fixed forms and compare them with the free Tobit fit by a likelihood-ratio test with two degrees of freedom. The drift form of Eq. (7) passes the test (p ≈ 0.3), whereas the random-walk and batch-blind forms fail it (p < 10−4 ). Because single failure points vary across seeds, as Table 5 shows, our conclusions rest on the aggregate scaling across 21 runs rather than on any single pair.

Fitting the failure law. We regress log Nfail on log b, with b the responses per optimizer step, and on log η, in two ways, and report both in Table 4. Least squares uses the eighteen runs that fail. A right-censored normal likelihood (Tobit) adds the three runs that never fail, whose budget is a lower bound on their failure point. Intervals are 90% percentile bootstrap intervals over runs (2000 re-

Clipping Analyses

E.1

Replay

Figure 13 summarizes the replays of the final checkpoint of each µ=8 run at its training learning rate. Each replay starts from a fresh AdamW state for 17

Widened clip radius

0.25 10−2

Validation accuracy

1 − cos(clipped, unclipped gradient)

E.2 10−1

10−3

10−4

0.20 0.15 0.10 512, DAPO clip 4,096, DAPO clip

0.05

4,096, clip widened ×√ 8

0.00

Optimizer step k within the rollout 512 4,096 ηS = 10−6

ηS = 5×10−6 ηS = 10−5

100

Samples (thousands of responses)

4,096, clip widened clip off (control)

Figure 14: Widened clip for the 4,096 batch with ηS =10−5 and µ=8, against the 512 batch. Color marks the rollout batch and line style the clip radius, solid for the DAPO radius √ and dashed for the log-space radius multiplied by 8, and bands span two seeds. The widened radius gives the large batch the small batch’s shape, with an earlier rise and then decay, and one seed collapses.

Figure 13: Replayed rotation of the update by the clip, measured as one minus the cosine between the clipped and unclipped gradients, at each optimizer step k within a rollout with µ=8 (median over checkpoints). Color marks the rollout batch and line style the learning-rate level, the dash-dot line is the 4,096 batch with a widened clip, and the gray line is the clip-off control, which marks the numerical floor. At k=0 every ratio is one and the clip cannot act, and with ηS =10−5 the large batch’s rotation is about 0.2 from k=2 on, 6 to 50 times the small batch’s.

We multiplied the√ log-space clip radius of L at 2.8×10−5 by 8, giving (εlow , εhigh ) = (0.468, 1.01). This brings its clipped fraction down to the small batch’s level (4–7×10−4 instead of 2– 4×10−3 ). Both seeds learn faster early (15% accuracy after 10k instead of 14k samples) and then decay: seed 0 from about 57k samples to a final 0.202, below the plain L’s 0.23, and seed 1 collapses at 90k samples, ending at 0.078, as Fig. 14 shows. Replaying seed 0’s checkpoints explains this. The wider radius halves the clipped fraction but lets the ratios drift further (99th percentile of | log ρ| at k=7: 0.66–0.75 against 0.45–0.51), and the tokens that are still clipped carry the same share of the gradient (0.47–0.53 against 0.50–0.55). Widening the clip moves the large batch toward the small batch’s behavior: it learns faster early, but loses much of its stability advantage, and one seed eventually collapses.

both batches, so comparisons between batches are matched, although the fresh state makes the first updates somewhat larger, and absolute clipped fractions run about twice those in the training logs. As a numerical calibration, removing the clip gives dclip ≈ 2–3×10−5 at every k, which we treat as the measurement floor. At the intermediate rate, the small batch’s dclip is mostly 10−4 to 10−2 with isolated spikes. In one mini-batch where only 7×10−5 of tokens are clipped, dclip reaches 0.72, and Proposition 5 then implies that these few tokens carry a gradient component comparable in norm to the full mini-batch gradient, a small-batch instance of what the large batch shows at every step.

Gradient-noise scale. At the same checkpoints, with the unclipped objective, the squared norm of the gradient averaged over a whole rollout is within 4% of the pure-noise prediction from four-response micro-batches. The signal is therefore not resolvable against the noise, and the simple noise scale is above 6×104 responses at every checkpoint after the first few steps, far above both optimizer batches (64 and 512). This supports assumption (A1) of §3. The implied critical batches of Remark 7 lie well below this noise scale, as Eq. (6) allows.

Checkpoint Robustness

The progress result carries over to Qwen2.5-1.5BInstruct, the general-purpose checkpoint of the same family: the small batch leads early at both µ=1 and µ=8, although this checkpoint trains in a markedly different regime. At a matched learning rate, its per-step KL is about ten times larger, its within-rollout log-ratio spread is 2.7 times larger, and its entropy is 0.5–0.8 rather than about 0.2. At one step per rollout both batches learn at 5×10−6 , the small batch leading over the first 30k samples. 18

At µ=8 a 24-rollout ladder puts its stable window at 1–3×10−6 , an order of magnitude below the math model’s. At 2×10−6 S is stable for its whole budget, peaks at 0.102 after 25k samples and ends at 0.082. Its L partner is also stable but much slower: 0.058, 0.075 and 0.076 at 16k, 49k and 98k samples against 0.088, 0.102 and 0.082, as the right panel of Fig. 15 shows. The small batch leads at every evaluation, reaches 8% accuracy after 14k samples against 86k, and α=0.67.

Validation accuracy

μ = 1, ηS = 5×10−6

training. Each second stage starts from the corresponding first-stage checkpoint with a fresh optimizer state and a one-rollout warm-up, and its first evaluation is within 0.016 accuracy of the trajectory before the switch. The ramp is robust to the switch point: switches from 25k to 74k samples perform similarly, provided the switch precedes failure, and only the earliest switch, at 12k, gives up part of the small batch’s progress advantage.

μ = 8, ηS = 2×10−6

0.10 0.08 0.06 Rollout batch 512 4,096

0.04 0

100 0

100

Samples (thousands of responses)

Figure 15: Qwen2.5-1.5B-Instruct with one seed per run, comparing the 512 and 4,096 batches with µ=1 and ηS =5×10−6 (left) and with µ=8 and ηS =2×10−6 (right). Color marks the rollout batch. The small batch leads early in both settings.

Batch-Ramp Switch-Point Sweep

Validation accuracy

0.25 0.20 0.15 fixed 512 fixed 1,536 fixed 4,096 switch at 12k

0.10 0.05 0

switch at 25k switch at 49k switch at 74k

100

Samples (thousands of responses)

Figure 16: Switch-point sweep with ηS =5×10−6 and µ=8, in which one trajectory of the 512 batch (seed 0) switches to the 4,096 batch after 12k, 25k, 49k or 74k samples. Dashed lines are the switched runs, darker for later switches, solid lines the fixed 512, 1,536 and 4,096 batches (seed means), and dotted verticals the switches. Switches at 25k, 49k and 74k perform similarly, and only the earliest, at 12k, gives up part of the small batch’s lead.

Figure 16 varies only the switch point along a common small-batch trajectory, which separates the timing of the ramp from variation in first-stage 19

Two Clocks: Why Larger Rollout Batches Learn Slower but Fail on Schedule in Reinforcement Learning with Verifiable Rewards

Analyze & explore related research in Conceptio

Related documents