Tandem Reinforcement Learning with Verifiable Rewards Difan Jiao †∗
Raghav Singhal ‡
Robert West ‡
Ashton Anderson †∗
†
arXiv:2606.28166v1 [cs.AI] 26 Jun 2026
∗
‡ University of Toronto EPFL Contact: {difanjiao, ashton}@cs.toronto.edu
Abstract Reinforcement learning with verifiable rewards (RLVR) has significantly improved the reasoning capability of large language models, reaching expert or even superhuman performance in domains such as competition math. However, whether weaker agents and humans can actually harness this capability is far less certain, with RLVR documented to drift reasoning toward idiosyncratic patterns such as poor readability and language mixing. Tandem training is a recently introduced paradigm that targets this compatibility problem: a trained, stronger senior cogenerates each rollout with a frozen, weaker junior, and the two are rewarded as a team, so the senior is pushed to reason in ways the junior can follow. Yet this paradigm has so far been demonstrated only in proof-of-concept settings, leaving open whether it scales to the long chains of thought of the modern RLVR pipeline. In this work, we propose Tandem Reinforcement Learning (TRL), which carries the tandem training paradigm into RLVR. In TRL, the senior and a frozen junior alternate stochastically to co-generate the reasoning, the resulting generation is rewarded, and the standard GRPO loss is applied to the senior. Training Qwen34B-Instruct on competition math, we find that TRL matches vanilla GRPO on solo reasoning capability while three properties emerge together from the same rollout structure: stronger handoff robustness with the junior, reduced distributional drift from the junior, and a chain-of-thought more legible to the junior. Our results demonstrate a promising route for RLVR with practical payoffs in multi-model communication and human compatibility.1
1
Introduction
Reinforcement learning with verifiable rewards (RLVR) has emerged as a dominant post-training paradigm in language-model post-training. In RLVR, a model samples a solution, an external verifier scores the final answer, and policy optimization reinforces successful trajectories. This simple loop has proved remarkably effective, producing large gains on mathematical and competition-style benchmarks and eliciting long, self-correcting chains of thought without process supervision [Shao et al., 2024, Guo et al., 2025, Yu et al., 2025]. However, this objective has an important drawback: it can improve benchmark performance without requiring the reasoning trajectory to remain compatible with weaker agents. A model can become better at solving problems while moving toward reasoning patterns that its pre-RL base model, weaker partner models, or human overseers are less able to predict, continue, or understand. This concern is not hypothetical: RLVR is known to induce substantial distributional drift from the base policy [Guo et al., 2025, Li et al., 2025, Meng et al., 2026], and recent work suggests that some reasoning behavior may become concentrated in idiosyncratic token patterns rather than transparent surface explanations [Kirchner et al., 2024, Karpov et al., 2025, Skaf et al., 2025]. For oversight and multi-agent systems, this is a serious weakness, since a model’s reasoning is often useful only 1 The codebase is available at https://github.com/CSSLab/Tandem-RLVR.
Preprint.
1
Initialization
2
Tandem Rollout Question: 2 + 3 × 4 = ?
Qwen3-Instruct
co-author the response
½ at every boundary, a coin flip picks the next author
Senior
Junior
Active · trained model
3
Frozen · weaker proxy
Reward & Policy Update
✓ SUCCESSFUL TEAM
First, multiply 3 by 4: then add 2 to 12 Response:
Math Verifier
Group-relative Advantage
= 14
14
✓ reward = 1 ✗ FAILED TEAM
Let's think step by step: and then
● Update Senior
3 × 4 = 12,
Response:
5
2 + 3 = 5,
× 4 = 20
20
✗ reward = 0
Figure 1: Tandem Reinforcement Learning (TRL) at a glance. insofar as weaker humans, monitors, or partner models can still follow and act on it [Burns et al., 2023, Lightman et al., 2023, Davidson et al., 2025]. Standard defenses against this drift, such as KL penalties against a reference policy [Ziegler et al., 2019, Ouyang et al., 2022], supervised distillation [Hinton et al., 2015, Magister et al., 2023], or process supervision [Uesato et al., 2022, Lightman et al., 2023], require that the designer to commit, in advance, to an explicit specification of what “intelligible” reasoning looks like, encoded either as a reference distribution or as labeled traces. Outside narrow deployment contexts, such a specification is difficult to write down [Bai et al., 2022, Lightman et al., 2023]. Tandem training, a recentlyintroduced training paradigm [Hamade et al., 2024, West et al., 2026], offers a potential solution. Rather than penalize distance from a fixed reference, a trained senior model co-generates every rollout with a frozen junior partner model, and the team is rewarded as a whole. A successful trajectory is then, by construction, one the junior could have continued. Intelligibility is thus operationalized through outcomes rather than declared up front. Prior work has demonstrated proofs of concept in chess [Hamade et al., 2024] and in a simplified reasoning setting [West et al., 2026], but it remains unclear whether the paradigm can provide the intelligibility within the modern RLVR pipeline. In this work, we introduce Tandem Reinforcement Learning, which carries the tandem training paradigm into GRPO-style RLVR. As illustrated in Figure 1, a trainable senior and a frozen junior, both initialized from the same base model, co-generate each response by stochastically alternating at word boundaries. The completed response receives the usual binary verifier reward, and the senior is updated with the standard GRPO objective on the tokens it emitted. Thus, TRL only changes how rollouts are produced, not the reward, verifier, or policy-gradient loss, and differences between TRLand GRPO-trained seniors are attributable to rollout structure alone. Training Qwen3-4B-Instruct on competition math, we find that three properties emerge together from this single change to rollout structure. First, compared with a matched GRPO baseline, TRL preserves solo reasoning capability across competition benchmarks. Second, we find that TRL improves handoff robustness. When paired at inference with a frozen junior under a reasoning-step schedule, the TRL senior outperforms the GRPO senior by up to +6.6 pass@8 points on AIME. Third, distributional drift is substantially curbed and legibility improves. TRL’s marginal token distribution stays closer to the base model’s distribution across the vocabulary (14% lower KL divergence); among the 500 tokens GRPO most displaces, 87% shift back toward the base under TRL; and the senior’s
2
chain-of-thought becomes more legible to the base model, with per-token cross-entropy under the junior dropping up to 17%. An ablation that adds a per-token KL penalty toward the junior on top of vanilla GRPO does not reproduce these gains, showing they come from the tandem rollout structure rather than from regularization toward the junior. Our results show that compatibility with a weaker partner, distributional anchoring to a base model, and full RLVR capability are obtainable from the same intervention on rollout structure, without modifying the reward, the verifier, or the loss. This points to rollout structure as an underexplored design axis for the RLVR pipeline, with practical payoffs for multi-model communication and human compatibility.
2
Related work
Tandem training. Tandem training was first introduced by Hamade et al. [2024] in chess, where they showed that optimizing for partner compatibility, i.e., winning a team game alongside a weaker collaborator, is a distinct objective from optimizing for raw ability. In West et al. [2026], this tandem training paradigm is carried into language modeling, demonstrating on GSM8K that randomized handoffs to a frozen junior during RL teach a stronger senior to abandon jargon and adapt its language to weaker partners while keeping task accuracy high. Reinforcement learning with verifiable rewards (RLVR). Recent advances in reasoning posttraining have leveraged reinforcement learning with verifiable rewards (RLVR). Shao et al. [2024] introduced Group Relative Policy Optimization (GRPO) as a practical recipe for this setting, and Guo et al. [2025] demonstrated that outcome-only RL can elicit strong reasoning capabilities without process supervision. A growing follow-up literature expands the GRPO design space along two axes: optimization-side variants revisit stability, optimization biases, and ratio aggregation [Yu et al., 2025, Liu et al., 2025, Zheng et al., 2025a, Zhao et al., 2025, Chu et al., 2025a, Zeng et al., 2025], while rollout-side work targets sampling efficiency, prompt filtering, and replay of high-signal trajectories [Zhang et al., 2025, Zhan et al., 2025, Zheng et al., 2025b]. TRL operates at a finer granularity than either axis: it modifies the rollout structure at the level of who emits each token within a single rollout, leaving both the loss aggregation and the rollout-management strategy unchanged. RLVR-induced distributional drift. A documented empirical regularity of RLVR is that the trained policy drifts measurably away from the pretrained base distribution. DeepSeek-R1-Zero, a pure outcome-only RLVR model, exhibits poor readability and unprompted language mixing as a side effect of training [Guo et al., 2025]. Li et al. [2025] identify RLVR specifically as the post-training stage that triggers Chinese–English code-switching in bilingual reasoning models, and recent work has further uncovered steganographic chain-of-thought patterns in which reasoning becomes loadbearing yet undetectable to a downstream monitor [Kirchner et al., 2024, Karpov et al., 2025, Skaf et al., 2025]. Such drift also carries downstream costs, as recent work finds that compatible tokenlevel distributions between teacher and student are a governing condition for successful on-policy distillation [Li et al., 2026]. On the other hand, a recent complementary line of work argues that not all drift is unhelpful. Meng et al. [2026] show through token-level cross-sampling interventions that a small subset of high-divergence positions is functionally responsible for RLVR’s reasoning gains. Tandem training has so far been demonstrated only in proof-of-concept settings outside the RLVR pipeline. We carry the paradigm into RLVR, the mainstream post-training framework behind current reasoning LLMs, and observe how the resulting model behaves both on its own and in cooperation with weaker collaborators.
3
Tandem reinforcement learning
3.1
Preliminaries: tandem training
Tandem training [Hamade et al., 2024, West et al., 2026] is a recently-introduced training paradigm in which two language models jointly produce each rollout. A trainable senior policy πsen and a frozen junior policy πjun stochastically alternate generations, with coin flips determining the next active model at predetermined handoff boundaries such as tokens, words, or sentences (e.g. if we choose to 3
alternate at the token level, for every token we flip a coin to determine which model generates the following token). The co-constructed rollout that emerges is scored by a single reward function as one trajectory, and the senior is updated against that reward using a policy-gradient algorithm while the junior remains unchanged. Tandem training is designed to produce more compatible models. An operational definition of compatibility, or intelligibility, is handoff robustness [West et al., 2026]: a model’s output is intelligible to another agent if that agent can continue it without derailing the trajectory. If a tandem training rollout ends successfully, the senior must have been generating in such a way that the junior could continue without crashing the trajectory. Reinforcing such rollouts therefore selects for senior behavior that the junior can successfully collaborate with, thus promoting compatibility. In this setup, no explicit definition of “intelligibility” is required—since we are working with verifiable rewards, we can directly measure how successful the senior-junior tandem team is at any given point. This is appealing because intelligibility is otherwise hard to codify: explicit methods such as system prompts or supervised finetuning on canonical solutions each demand an a priori specification that depends on deployment context [West et al., 2026, Bai et al., 2022, Lightman et al., 2023]. In this work, we generalize the tandem training paradigm to Reinforcement Learning with Verifiable Rewards (RLVR), the cornerstone of contemporary large reasoning models. We refer to our instantiation as Tandem Reinforcement Learning (TRL). To adapt to RLVR, we make three design choices: the senior and junior are initialized from the same base model (§3.2); handoffs happen at the word level (§3.3); and the senior is updated in the same way as Group Relative Policy Optimization (GRPO) [Shao et al., 2024] on the senior-emitted tokens (§3.4). We now discuss each of these choices in turn. 3.2
Tandem pairs
In general tandem training, the senior agent is typically stronger, or at least stylistically distinct, from the junior, so that the senior must adapt to be compatible with a partner with weaker capabilities. In this work adapting tandem training to RLVR, such asymmetric pairings remain available, but we begin with the natural choice of setting the junior as the senior’s own pre-RL initialization—before training, the junior and senior models begin as identical copies of each other. This self-pairing offers methodological benefits. In a stronger-weaker pair, the trained senior is encouraged to learn how to adapt to potentially a very different style. With identical capabilities at the start, this pressure is eliminated, and instead the senior can focus on improving its capabilities while minimizing drift from its initial state, thus more directly addressing our motivation of developing a capable, compatible, and intelligible model. We therefore initialize πsen and πjun from the same base model and freeze πjun for the duration of training. 3.3
Tandem rollout
A tandem rollout produces a shared response y1:T from prompt x, where each step t is generated by an active model at ∈ {sen, jun}. At each step, both models receive the context x · y<t , the active model samples yt ∼ πat (· | x · y<t ), and the chosen yt is appended and then fed to both models so that both states condition on the same history. Following West et al. [2026], the active model is redrawn by an independent fair coin flip at every word boundary: if yt begins a new orthographic word, at+1 is set to the senior with probability p and to the junior otherwise; if yt does not begin a new word, at+1 = at . We use p = 0.5 in consistency with original tandem training. Details of this setup are included in Appendix A.1. We choose to stochastically alternate at word-level granularity over alternative choices (e.g., tokenlevel or sentence-level). First, at sentence- or paragraph-level the senior can dominate a rollout simply by suppressing boundary tokens (e.g., end-of-sentence or end-of-line tokens) that would force a potential handoff2 , collapsing the schedule back to vanilla GRPO. Second, at token level, alternation at subword granularity disrupts coherent utterances before they form. In contrast, wordlevel boundaries fire on every natural-language word and are difficult to suppress without sacrificing fluency, making them the coarsest granularity the senior cannot trivially erase by output choice. 2 This reward-hacking failure mode is induced by the schedule itself: as the senior strengthens, suppressing boundary tokens
becomes a low-cost route to extending its own control and avoiding handoffs.
4
Algorithm 1 Tandem Reinforcement Learning (TRL). Require: Base model π0 ; training set D; batch size B; group size G; max response length L; word-boundary set B; subword-span cap K; Bernoulli probability p; verifier r(·) 1: Initialize πsen ← π0 and πjun ← π0 ; freeze πjun 2: for τ = 1, 2, . . . do 3: Sample a batch of prompts {xj }B j=1 ∼ D 4: for each (j, i) with j ∈ [B] and i ∈ [G] do 5: // Tandem rollout (§3.3) 6: Draw a1 ∼ Bernoulli(p) over {sen, jun}; c ← 0 7: for t = 1, . . . , L do 8: Forward πsen and πjun on xj · y<t ; sample ytsen , ytjun independently 9: yt ← ytat ; mt ← 1[at = sen] 10: break if yt is the end-of-sequence token 11: if yt ∈ B or c ≥ K then 12: Draw at+1 ∼ Bernoulli(p) over {sen, jun}; c ← 0 13: else 14: at+1 ← at ; c ← c + 1 15: end if 16: end for 17: Record y (j,i) ← y1:t and authorship mask m(j,i) ← m1:t 18: end for 19: // Senior-only GRPO update (§3.4) 20: Compute rewards r(j,i) ← r(y (j,i) ) (j,i) 21: Compute group-relative advantages At from {r(j,i) }G i=1 22: Update πsen with the GRPO objective of §3.4, with response mask narrowed by m(j,i) 23: end for 3.4
Tandem policy update
Each tandem rollout receives a single binary reward r(y) ∈ {0, 1} from a math verifier, computed on the full response y1:T regardless of which model emitted each token. The senior is updated with the standard GRPO objective [Shao et al., 2024], with the response mask set to select only senior-emitted positions; the per-token policy-gradient contribution is " LTRL (sen) = − E
# X
At log πsen (yt | x · y<t )
,
(1)
t : at =sen
where At is the standard GRPO group-relative advantage; the clipped surrogate ratio and KL term are also identical to GRPO and omitted here for brevity. Junior-emitted tokens enter the response, and so shape the reward the senior receives, but contribute no gradient. Specifically, while the original tandem-training formulation also includes a soft junior-imitation term that pulls πsen toward πjun on junior-emitted positions with coefficient λjun , we set λjun = 0. Beyond removing a hyperparameter we have no principled value for, this keeps the per-token loss formally the same as vanilla GRPO. Thus, any difference between TRL- and GRPO-trained seniors is then attributable to the rollout structure that produces y, and not to an auxiliary regularization term. The end-to-end TRL pipeline is illustrated in Algorithm 1.
4
Results
4.1
Experimental setup
We fine-tune Qwen3-4B-Instruct-2507 [Yang et al., 2025] on DeepScaleR [Tan et al., 2025] with a binary correctness reward on the boxed final answer. Our baseline is vanilla GRPO [Shao et al., 2024] on the same base model, with optimization settings shared with TRL (Appendix A.3). We evaluate on AMC 23–25, AIME 24–26, and Minerva Math [Lewkowycz et al., 2022], and report pass@k 5
Base
AIME
60
GRPO
AMC
90
TRL
Minerva
50
80
45
60
40
70
40
50
30
60
35
40
20
50
30
Pass@k (%)
50
1
2
4
k
8 16 32
1
2
4
k
8 16 32
1
2
4
k
Macro Average
70
8 16 32
30
1
2
4
k
8 16 32
Figure 2: Reasoning capabilities (measured by pass@k, ↑) on mathematical reasoning benchmarks for Qwen3-4B-Instruct and its GRPO- and TRL-trained seniors.
AMC
90
GRPO
AIME
60
TRL
Minerva
50
50
45
50
70
40
40
45
60
30
35
40
Pass@k (%)
80
50
1
2
k
4
8
20
1
2
k
4
8
30
1
2
k
4
Macro Average
55
8
35
1
2
k
4
8
Figure 3: Reasoning-step handoff robustness (measured by pass@k, ↑) on mathematical reasoning benchmarks of GRPO and TRL seniors paired with the junior, frozen Qwen3-4B-Instruct. via an unbiased estimator following Chen et al. [2021]. We estimate bootstrap standard errors by resampling evaluation problems, reported as ± values in tables and as shaded bands in figures. This section is organized in four parts. In §4.2, we measure the resulting models’ solo reasoning capabilities, examining how TRL influences the reasoning gains RLVR delivers, with vanilla GRPO as the reference point. We then turn to handoff robustness in §4.3, the central behavioural property tandem training was originally proposed to elicit; we pair the trained models with the junior model at inference time and evaluate the team’s performance. In §4.4 we evaluate RL-induced distribution shift, a documented cost of RLVR, examining whether TRL’s co-generation structure measurably resists it. Finally, in §4.5 we measure whether this distributional anchoring makes the senior’s outputs more legible to the junior. 4.2
TRL retains the reasoning gains of RLVR
We first measure each model’s solo reasoning capability, with no junior in the loop at test time, to examine how the tandem rollout structure influences the gains RLVR delivers. Each senior generates n independent samples per problem; we report pass@k via the unbiased estimator for k ∈ {1, 2, 4, 8, 16, 32}. Figure 2 shows pass@k curves for the pre-RL base, vanilla GRPO, and TRL across benchmarks and their macro average. Two observations stand out. First, both trained seniors lift the base across the full k range on all three benchmarks, confirming that RLVR is delivering a non-trivial capability gain on this base. Second, the GRPO and TRL curves track each other closely throughout, with differences at any given k within typical run-to-run variance and no consistent direction across benchmarks. We read this as empirical evidence that the tandem rollout structure does not cost the senior any solo capability: the gains RLVR delivers under vanilla GRPO are retained under TRL. 4.3
TRL offers better handoff robustness
In §3.3 we argued against training with sentence- or paragraph-level handoffs, on the grounds that the senior could game such schedules by suppressing the boundary tokens that would force a switch. For evaluation, however, there is no risk of this type of gaming behavior, and we want to measure how 6
1.0
4
TRL drift from GRPO (nats)
Vanilla GRPO TRL
Fraction of tokens
0.8 0.6 0.4 0.2 0.00.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
Absolute drift from base (nats)
3 2 1 0 1 2 3 4
1.6
Recovers (87%) No recovery (13%)
4
3
2
1
0
1
2
GRPO drift from base (nats)
3
4
Figure 4: Distributional deviation from the base model for GRPO and TRL. Left: survival curves of absolute per-token log-ratio to the base over tokens. Right: for the top-500 most displaced tokens by GRPO, we show the drift versus TRL recovery. cooperation between language models actually unfolds in practice. In multi-agent reasoning systems, such as drafter-target setups for inference and human-AI handoffs, the natural unit of communication is a reasoning step rather than an individual word, with one party producing a step and a partner picking up where it left off [Chu et al., 2025b, Wang et al., 2025, Davidson et al., 2025]3 . This motivates our evaluation of handoff robustness under a reasoning-step schedule: at inference, the senior and the junior alternate at every \n\n-boundary token, each producing one reasoning step before yielding control to the other, with the team-produced response scored as a whole. Figure 3 shows pass@k for k ∈ {1, 2, 4, 8}. TRL leads on all three benchmarks, with the advantage most pronounced on AIME: +4.6 points at k=1 (29.3 vs. 24.7) widening to +6.6 at k=8 (44.4 vs. 37.8). The macro-average gap grows from +2.2 at k=1 to +3.1 at k=8. Read together with §4.2, the picture is clean: the TRL senior is no less capable than the GRPO senior reasoning alone, and is the stronger half of a team. We confirm the statistical significance of this gap, and also of the further TRL gains reported in following subsections, with paired t-tests in § 5.3. 4.4
TRL resists RLVR-induced distribution shift
TRL’s rollout structure carries an implicit consequence beyond handoff robustness. Because the junior is a frozen copy of the senior’s pre-RL base, every successful tandem rollout is one the junior could have continued at any word boundary. This creates a structural pressure against token choices that lie far outside the junior’s predictive support. Whether this pressure measurably anchors the senior’s output distribution is an empirical question and one worth asking, as distributional drift is a documented cost of RLVR at scale. Models develop idiosyncratic token patterns that diverge from the pre-RL base, including language mixing and syntactic irregularities [Guo et al., 2025, Yu et al., 2025], and recent work has uncovered steganographic chain-of-thought patterns whose reasoning is no longer recoverable from surface text [Karpov et al., 2025, Skaf et al., 2025]. Such drift undermines oversight, as supervision requires the stronger model’s reasoning to remain within the interpretive reach of weaker overseers [Burns et al., 2023, Lightman et al., 2023]. We therefore ask whether TRL’s co-generation structure measurably limits this drift. To operationalize, we estimate each model’s marginal token distribution from all senior generations on our evaluation benchmarks. Per-token deviation from the junior is then measured by the signed log-ratio log(psen (t) / pjun (t)) for each token t ∈ V (vocabulary space), positive when the senior over-uses t relative to the junior and negative when it under-uses it. Figure 4 shows that TRL curbs this drift. The left panel plots the survival curve of absolute deviation over all tokens appearing frequently in the junior’s own outputs. As shown, TRL’s curve lies uniformly below Vanilla GRPO’s at every threshold, indicating that fewer TRL tokens stray far from the junior at any scale. Full-vocabulary KL-divergence [Kullback, 1997] confirms the aggregate, where TRL is 14% closer to the junior by KL (0.022 vs. 0.026). The right panel zooms into the 500 tokens with the 3 also resembling a ply of chess games in Hamade et al. [2024].
7
Table 1: Junior legibility of senior generated chain-of-thoughts (Qwen3-4B-Instruct junior). Bold marks the better model per cell. Subscripts report ±1 standard error of the mean over 10,000 bootstrap resamples of problems (stratified by benchmark for the macro average). Metric
Model
AMC
AIME
Minerva
Avg.
Cross-entropy (nats, ↓)
GRPO TRL ∆
0.125±0.005 0.113±0.004 −0.012
0.154±0.006 0.156±0.007 +0.002
0.117±0.004 0.097±0.003 −0.020
0.132±0.003 0.122±0.003 −0.010
Distribution overlap (↑)
GRPO TRL ∆
0.961±0.001 0.973±0.001 +0.012
0.957±0.002 0.970±0.001 +0.013
0.963±0.001 0.976±0.001 +0.013
0.960±0.001 0.973±0.000 +0.013
largest GRPO drift4 . Among these, 87% show TRL shifting in the opposing direction: under-emitting what GRPO over-emits and vice versa (Spearman ρ = −0.58, p < 10−275 ), pushing back toward the junior’s distribution. Taken together, TRL stays closer to the junior than Vanilla GRPO: globally in aggregate, and on the tokens GRPO displaces most. A more detailed qualitative analysis of drifted tokens is included in Appendix B.1. 4.5
TRL yields a more junior-legible chain-of-thought
TRL’s distributional anchoring raises a question: does the junior (base) model find the senior’s reasoning more predictable? §4.4 establishes that the senior’s overall token usage stays nearer the junior’s, but it does not directly speak to whether the junior, reading the senior’s chain-of-thought left-to-right, would find the next token predictable at each step. The two are correlated but separable: a marginal-similar policy can still be conditionally surprising at most positions. In this part, we measure this junior legibility with two analyses, summarized in Table 1. The first is the junior’s per-token cross-entropy on the senior’s chain-of-thought: the average nats required to encode each senior token under the junior’s predictive distribution, where lower means the junior is less surprised. TRL’s cross-entropy is lower by 0.010 nats on average, with clear gains on AMC and Minerva and a near match on AIME. Specifically, a reduction of 0.010 nats represents a 7.6% decrease in per-token surprisal on average, reaching 17% on Minerva, meaning the junior finds TRL’s chain-of-thought that much easier to follow at each step. P The second is the token-level distribution overlap α = v min(psen (v | ·), pjun (v | ·)) averaged over all positions [Leviathan et al., 2023]: the probability mass both models assign to the same tokens at each step, ranging from 0 (fully disjoint) to 1 (identical). As shown, TRL’s distribution overlap is uniformly higher across all three benchmarks. Note the absolute gap between 0.973 and 0.960 may appear small, but expressed as shortfall from perfect agreement, TRL’s gap is around 0.03 against Vanilla GRPO’s 0.04: a roughly 30% reduction in the fraction of positions where the two models disagree.
5
Discussion
5.1
Training dynamics of TRL
TRL replaces every rollout with a joint two-model generation, so it is natural to ask whether this structural change disrupts training or imposes prohibitive overhead. Figure 5 reports the core training signals. TRL is operationalized end-to-end: checkpoint selection is driven by tandem-rollout accuracy rather than solo accuracy, so the two curves in panel (d) are plotted on separate axes to reflect that they measure different evaluation protocols. This is intentional, as selecting by solo accuracy would be inconsistent with the cooperative objective TRL optimizes. Panel (a) shows that training reward follows a stable trajectory under TRL, closely tracking GRPO throughout. Panel (b) shows that response lengths evolve similarly in both runs, without runaway growth or collapse to short responses, suggesting the senior neither offloads the full generation burden to the junior nor exploits the joint rollout to terminate early. Together, these signals indicate that the tandem rollout does not destabilize the basic RL learning dynamic. 4 tokens with highest log(p GRPO (t) / pjun (t)).
8
GRPO
TRL
0.75
Tokens
0.60 0.45
1600
200
(c) Training Wallclock
360
9.4h
300
0
7.8h 7.5
240
5.0
180
2.5 0
50
100
150
Checkpoint step
200
0.0
50
100
150
200
(d) Benchmark Accuracy
TRL (%)
150
50
47
48
46
46
45
44
50
100
150
Checkpoint step
200
Macro avg
100
GRPO (%)
50
Accumulated time (h)
0
Time / step (s)
2000 1200
0.30
120
(b) Response Length
2400
Macro avg
Mean Reward
(a) Training Reward
44
Figure 5: Training dynamics of TRL and GRPO. (a) Mean reward. (b) Average response length for rollouts. (c) Per-step wallclock (left axis) and accumulated wallclock to best checkpoints (right axis, shaded). (d) Macro-average benchmark accuracy under each run’s evaluation protocol. On training cost, running two full model forwards per step is an unavoidable overhead. A naive implementation cannot sustain the long contexts that RLVR requires and is impractical beyond toy settings (Appendix A.1); we therefore surgically extend the vLLM backend [Kwon, 2025] to both enable and accelerate TRL, reducing per-step latency to approximately 2× that of a single-model run, which we regard as a standalone technical contribution of this work. Empirically, TRL reaches its best checkpoint at step120 compared to step200 for GRPO, partially offsetting the per-step overhead; accumulated wall-clock to convergence is 9.4 hours for TRL versus 7.8 hours for GRPO (panel c). We also observe a sharp performance decline in TRL after sufficient training steps, a dynamic that West et al. [2026] similarly report for tandem training in language generation. We note that this pattern is consistent with what one might expect from prolonged co-generation training: as the senior diverges further from the frozen junior, the pressure that anchored its earlier checkpoints becomes harder to satisfy, and tandem rollouts may begin to fail more frequently. However, this does not affect the quality of the TRL senior at its best checkpoint, which is the model evaluated throughout §4, and we leave theoretical and mechanistic understanding of this behavior to future work.
5.2
Ablation study: KL-regularization towards the junior
TRL can be thought of as implicitly regularizing the senior toward the junior: every successful tandem trajectory is one the junior could have continued, so the training signal pushes the senior to stay near the junior’s distribution. KL regularization toward the junior is the standard explicit way to apply this kind of anchoring in RLHF, which makes it the natural ablation baseline. We call this baseline KL-Reg, with the reference fixed at the pre-RL base (the same checkpoint that serves as TRL’s frozen junior). Training data, optimizer settings, and evaluation protocol match TRL and vanilla GRPO. Table 2 reports macro averages across the four evaluation axes of §4. On solo capability, all three models are at parity. On handoff robustness, however, KL-Reg sits essentially at the vanilla GRPO level, while TRL leads by roughly three percentage points: adding a KL penalty toward the junior does not, on its own, make the senior produce text the junior can pick up and continue. On distribution overlap and junior legibility, KL-Reg moves partway from GRPO toward TRL but does not close the gap on either; TRL produces a chain-of-thought that the junior finds both more overlapping with its own distribution and more predictable token-by-token, by a margin KL-Reg does not match. We note that the KL regularization toward the junior and handoff success are correlated but structurally distinct: a senior can sit at low average KL to the junior but still produce tokens the junior cannot continue at handoff boundaries. Raising the regularization coefficient alone cannot resolve this 9
Table 2: Our ablation study: vanilla GRPO, junior-policy-regularized GRPO (KL-Reg), and TRL on the four quantitative evaluation axes of §4. Method GRPO KL-Reg TRL
Solo Capability
Handoff Robustness
Junior Legibility
Distribution Overlap
(pass@8, %, ↑)
(pass@8, %, ↑)
(nats, ↓)
(↑)
56.0±2.2 56.0±2.2 55.2±2.2
52.7±2.3 52.9±2.3 55.8±2.4
0.132±0.003 0.127±0.003 0.122±0.003
0.960±0.001 0.960±0.001 0.973±0.000
tension, since past a moderate value it also erodes the senior’s reasoning capability gained from RLVR. We provide detailed discussion in Appendix C. 5.3
Statistical significance of TRL gains
We complement the bootstrap standard errors of §4 with paired one-sided t-tests on per-problem statistics, mean-pooled across the benchmarks. For pass@k we test per-problem unbiased estimates [Chen et al., 2021]; for per-token cross-entropy and distribution overlap α we average per-problem measurements before testing. We note that each test is one-sided in the claimed direction, and asterisks denote significance levels: ∗ p < 0.05, ∗∗ p < 0.01, ∗∗∗ p < 0.001. Table 3: Paired t-tests for the evaluation gains of TRL over baselines. Comparison
∆
t
p
Handoff Robustness (pass@k, ∆ in percentage points, ↑) TRL > GRPO, k=1 +1.42 +2.61 0.005∗∗ TRL > GRPO, k=2 +1.72 +2.66 0.004∗∗ TRL > GRPO, k=4 +1.88 +2.32 0.010∗ TRL > GRPO, k=8 +2.07 +1.83 0.034∗ TRL > KL-Reg, k=1 +1.01 +1.73 0.043∗ TRL > KL-Reg, k=2 +1.24 +1.90 0.029∗ TRL > KL-Reg, k=4 +1.62 +2.13 0.017∗ TRL > KL-Reg, k=8 +2.07 +2.14 0.016∗ Junior Legibility (per-token cross-entropy, ∆ in nats, ↓) TRL < GRPO −0.0146 − 9.73 <10−20 ∗∗∗ TRL < KL-Reg −0.0114 −10.94 <10−24 ∗∗∗ Distribution Overlap (α, ↑) TRL > GRPO +0.0147 TRL > KL-Reg +0.0110
+36.16 +30.22
<10−139 ∗∗∗ <10−113 ∗∗∗
Table 3 reports paired one-sided t-tests for the three directional claims involving TRL in the main text. On handoff robustness (§4.3), TRL’s lead over both GRPO and KL-Reg holds at every k, with all eight tests significant at p < 0.05 and two tests at p < 0.001; the consistency across k rules out a single-k artifact. On junior legibility (§4.5), TRL’s per-token cross-entropy is significantly lower than both baselines, confirming that the senior’s chain-of-thought is genuinely more predictable to the junior. The distribution-overlap gap is the sharpest: TRL beats both baselines at p < 10−100 , providing essentially noise-free support for the rollout-driven anchoring described in §4.4.
6
Conclusion
In this work, we introduced Tandem Reinforcement Learning (TRL), which carries the tandem training paradigm into the RLVR pipeline that drives current reasoning LLMs. In TRL, each rollout is co-generated by a trainable senior and a frozen junior initialized from the senior’s pre-RL base, the two alternate at word boundaries, and the standard GRPO loss is applied to senior-emitted tokens. On competition math benchmarks with Qwen3-4B-Instruct, the TRL-trained senior is no less capable than the GRPO-trained one when reasoning alone. Beyond this parity, the same rollout structure that enabled training also leaves measurable traces in the senior’s behavior at inference: its co-generations 10
with the junior succeed more often than GRPO’s, its token-level distribution stays closer to the junior’s, and its reasoning steps are more readily predicted by the junior.
Future work Several directions for future work are worth highlighting. First, replacing the fixed self-paired junior with a pool of diverse juniors varying in capability, style, language, or tool use could regularize the senior toward more broadly intelligible behavior. Second, pairing the senior with juniors calibrated to specific human skill levels, analogous to Maia [McIlroy-Young et al., 2020] and Maia-2 [Tang et al., 2024] for chess, would provide a more direct bridge to the genuine human-AI handoff scenarios that originally motivated the tandem framework. We view this direction as currently bottlenecked by the absence of comparably well-calibrated, human-skill-level proxies for general-purpose language models, and a promising avenue once such models mature. Third, the mechanistic underpinnings of TRL warrant further study, including understanding the dynamics of TRL and why self-pairing produces the three observed properties.
Acknowledgments This research is funded by grants from the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada Foundation for Innovation, and the Ontario Research Fund. We are grateful to Zhenwei Tang and Julian Minder for helpful discussions and feedback on this work.
11
References Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, et al. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision. arXiv preprint arXiv:2312.09390, 2023. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021. Xiangxiang Chu, Hailang Huang, Xiao Zhang, Fei Wei, and Yong Wang. Gpg: A simple and strong reinforcement learning baseline for model reasoning. arXiv preprint arXiv:2504.02546, 2025a. Yuanlin Chu, Bo Wang, Xiang Liu, Hong Chen, Aiwei Liu, and Xuming Hu. Ssr: Speculative parallel scaling reasoning in test-time. arXiv preprint arXiv:2505.15340, 2025b. Tim R Davidson, Adam Fourney, Saleema Amershi, Robert West, Eric Horvitz, and Ece Kamar. The collaboration gap. arXiv preprint arXiv:2511.02687, 2025. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. Karim Hamade, Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Designing skill-compatible ai: Methodologies and frameworks in chess. arXiv preprint arXiv:2405.05066, 2024. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset, 2021. URL https://arxiv.org/abs/2103.03874. NeurIPS 2021 Datasets and Benchmarks Track. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. Artem Karpov, Tinuade Adeleke, Seong Hah Cho, and Natalia Perez-Campanero. The steganographic potentials of language models. arXiv preprint arXiv:2505.03439, 2025. Jan Hendrik Kirchner, Yining Chen, Harri Edwards, Jan Leike, Nat McAleese, and Yuri Burda. Prover-verifier games improve legibility of llm outputs. arXiv preprint arXiv:2407.13692, 2024. Solomon Kullback. Information theory and statistics. Courier Corporation, 1997. Woosuk Kwon. vLLM: An Efficient Inference Engine for Large Language Models. PhD thesis, UC Berkeley, 2025. Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR, 2023. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857, 2022. Yaxuan Li, Yuxin Zuo, Bingxiang He, Jinqian Zhang, Chaojun Xiao, Cheng Qian, Tianyu Yu, Huanang Gao, Wenkai Yang, Zhiyuan Liu, et al. Rethinking on-policy distillation of large language models: Phenomenology, mechanism, and recipe. arXiv preprint arXiv:2604.13016, 2026. Yihao Li, Jiayi Xin, Miranda Muqing Miao, Qi Long, and Lyle Ungar. The impact of language mixing on bilingual llm reasoning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32519–32536, 2025. 12
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. In The twelfth international conference on learning representations, 2023. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025. Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1773–1781, 2023. Reid McIlroy-Young, Siddhartha Sen, Jon Kleinberg, and Ashton Anderson. Aligning superhuman ai with human behavior: Chess as a model system. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1677–1687, 2020. Haoming Meng, Kexin Huang, Shaohang Wei, Chiyu Ma, Shuo Yang, Xue Wang, Guoyin Wang, Bolin Ding, and Jingren Zhou. Sparse but critical: A token-level analysis of distributional shifts in rlvr fine-tuning of llms. arXiv preprint arXiv:2603.22446, 2026. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback, 2022. URL https://arxiv.org/abs/2203.02155. NeurIPS 2022. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. Joey Skaf, Luis Ibanez-Lissen, Robert McCarthy, Connor Watts, Vasil Georgiv, Hannes Whittingham, Lorena Gonzalez-Manzano, David Lindner, Cameron Tice, Edward James Young, et al. Large language models can learn and generalize steganographic chain-of-thought under process supervision. arXiv preprint arXiv:2506.01926, 2025. Sijun Tan, Michael Luo, Justin Wong, Colin Cai, Xiaoxiang Shi, William Yuan Tang, Manan Roongta, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, et al. Deepscaler: Effective rl scaling of reasoning models via iterative context lengthening. 2025. Zhenwei Tang, Difan Jiao, Reid McIlroy-Young, Jon Kleinberg, Siddhartha Sen, and Ashton Anderson. Maia-2: A unified model for human-ai alignment in chess. Advances in Neural Information Processing Systems, 37:20919–20944, 2024. Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275, 2022. Xiao Wang, Jia Wang, Yijie Wang, Pengtao Dang, Sha Cao, and Chi Zhang. Mars: toward more efficient multi-agent collaboration for llm reasoning. arXiv preprint arXiv:2509.20502, 2025. Robert West, Ashton Anderson, Ece Kamar, and Eric Horvitz. Tandem training for language models. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8265–8278, 2026. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, et al. DAPO: An open-source LLM reinforcement learning system at scale, 2025. URL https://arxiv.org/abs/2503.14476. 13
Guanning Zeng, Zhaoyi Zhou, Daman Arora, and Andrea Zanette. Shrinking the variance: Shrinkage baselines for reinforcement learning with verifiable rewards. arXiv preprint arXiv:2511.03710, 2025. Runzhe Zhan, Yafu Li, Zhi Wang, Xiaoye Qu, Dongrui Liu, Jing Shao, Derek F Wong, and Yu Cheng. Exgrpo: Learning to reason from experience. arXiv preprint arXiv:2510.02245, 2025. Yuheng Zhang, Wenlin Yao, Changlong Yu, Yao Liu, Qingyu Yin, Bing Yin, Hyokun Yun, and Lihong Li. Improving sampling efficiency in rlvr through adaptive rollout and response reuse. arXiv preprint arXiv:2509.25808, 2025. Yuzhong Zhao, Yue Liu, Junpeng Liu, Jingye Chen, Xun Wu, Yaru Hao, Tengchao Lv, Shaohan Huang, Lei Cui, Qixiang Ye, et al. Geometric-mean policy optimization. arXiv preprint arXiv:2507.20673, 2025. Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.18071, 2025a. Haizhong Zheng, Yang Zhou, Brian R Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for llm reasoning via selective rollouts. arXiv preprint arXiv:2506.02177, 2025b. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019.
14
A
Reproducibility
A.1
Tandem rollout implementation
A naive realization of tandem rollout couples two HuggingFace models in an outer Python loop with manual KV-cache management. We built such a prototype and found it impractical for RL training: at 512 generated tokens it exhausts the memory of a single 80 GB GPU, making RLVR under long chain-of-thoughts for mathematical reasoning tasks infeasible. Moreover, per-step throughput at shorter lengths is already roughly 4× slower than a single-model vLLM [Kwon, 2025] rollout. Since RL training is dominated by rollout time, this slowdown alone makes large-scale tandem RL infeasible. We therefore implement the tandem rollout inside vLLM, with both πsen and πjun treated as pagedattention-aware models in the same engine. The result is a decoding path that pays only the dualforward cost (theoretically twice the single-model latency) and inherits every other vLLM optimization: paged KV cache, continuous batching, FlashAttention, CUDA graphs, and tensor parallelism. Empirically our throughput is approximately 0.5× that of a single-model vLLM rollout, matching the architectural lower bound for sequential dual decoding. A.1.1
Dual-decoder backend
In our implementation, πsen and πjun are loaded onto separate devices and registered in the same vLLM engine, with the junior’s layers carrying a fixed name prefix so that the engine’s forward context can route them without interfering with the senior’s. The two models share per-step attention metadata (slot mappings, block tables, position indices), since they process identical token sequences, but each writes to its own KV-cache tensor on its own device. At each generation step the engine performs one forward pass through πsen , then one through πjun on the same appended token, samples independently from each model’s logits, and a tandem sampler emits the chosen token according to the active-model schedule of §3.3; the chosen token is then fed to both models on the next step, keeping the two KV caches in lockstep with the shared response y<t . This isolates the tandem-specific logic to model loading, the second forward, and the sampler, while leaving every other vLLM runtime path untouched. For each request, the engine also accumulates a per-token authorship stream alongside the generated tokens, flagging which positions were emitted by πsen . The stream is propagated through the same output channels as the tokens themselves (model-runner output, engine-core output, per-request completion output) and is exposed to the verl trainer as a tensor field of the rollout batch. On the trainer side, the actor’s loss path consumes it by elementwise-multiplying it into the per-token response mask before the policy-gradient sum, which yields exactly the senior-only loss of §3.4. This is the only addition to the verl GRPO surface; the actor, critic, and trainer code paths are otherwise unchanged. A.1.2
Word-level handoff details
With the vLLM-native backend in place, we now detail the word-level handoff schedule of §3.3 as instantiated in our implementation. The set of word-boundary tokens is auto-resolved from the active tokenizer at engine initialization. For any BPE tokenizer using a leading-space marker (the Qwen3 / GPT-style word-initial prefix), The boundary set is exactly the set of vocabulary IDs whose surface form begins with that marker. For the Qwen3 tokenizer used in this work, this yields roughly 53k of 151k IDs. The boundary set alone, however, is not sufficient. Mathematical reasoning produces long runs of non-boundary tokens inside internal LaTeX expressions, variable identifiers, and code blocks, where leading-space tokens never appear. Without intervention, such runs would receive no handoff, allowing the active model to retain control across an entire derivation step and reintroducing exactly the reward-hacking surface that word-level granularity was meant to close. We therefore add a backstop: if more than K consecutive non-boundary tokens have elapsed since the last handoff, the active model is redrawn under the same Bernoulli(p) rule used at boundaries. We use K = 32, chosen as a default that prevented monopoly behaviour in our early rollouts. 15
Table 4: Hyperparameters for TRL training on DeepScaleR with Qwen3-4B-Instruct-2507. The GRPO block is identical to the vanilla-GRPO baseline; the Tandem block contains the TRL-specific configurations. Category
Hyperparameter
GRPO training Learning rate PPO clip ratio Train batch size PPO mini-batch size Group size G (rollouts per prompt) Max response length Entropy coefficient KL penalty in reward KL loss term Advantage estimator
1 × 10−6 0.2 16 8 8 3000 0 disabled disabled GRPO (group-relative, std-normalised)
Tandem (TRL-specific) Selection strategy Handoff probability p Subword-span cap K Junior-token loss weight λjun Senior πsen initialisation Junior πjun initialisation
word-level 0.5 32 0 Qwen3-4B-Instruct Qwen3-4B-Instruct (self-paired, frozen)
Generation Train temperature Validation temperature Validation top-p Validation samples per problem
0.6 0.6 0.95 4
The Bernoulli draws at boundary and cap-fallback positions use the per-request seeded random number generator that vLLM already maintains for sampling, so a given rollout’s authorship sequence is reproducible from its request seed. To avoid replaying the full token history at every generation step, the tandem sampler maintains a small per-request state (active model, tokens since the last handoff, last-seen response length) and resumes from this state on each subsequent step. A.2
Datasets
We train on DeepScaleR [Tan et al., 2025], a corpus of approximately 40,000 competition-math problems compiled from past olympiad and contest sources. Each example provides a problem statement and a verifiable final answer; reward is computed by extracting the final \boxed{} expression from the model’s response and comparing it against the gold answer with a sympy-based numeric or symbolic match adopted from Hendrycks et al. [2021]. For evaluation we use a panel of three competition-math benchmarks, each disjoint from the training corpus: • AMC 23–25 5 : 121 problems. • AIME 24–26 6 : 90 problems. • Minerva Math 7 : 272 problems. Each problem is formatted with an instruction requiring the model to output its final answer in \boxed{} form. We score every rollout with the same boxed-extraction grader used during training; aggregate metrics are reported as pass@k over k independent rollouts per problem following the unbiased estimator proposed in Chen et al. [2021]. 5 Compiled from AMC 10/12 problems (American Mathematics Competitions) of 2023, 2024, and 2025. 6 Compiled from AIME (American Invitational Mathematics Examination) of 2024, 2025, and 2026. 7 https://huggingface.co/datasets/math-ai/minervamath
16
A.3
Hyperparameter configuration
Table 4 summarises the hyperparameters used for TRL training and vanilla GRPO baseline. All TRL experiments reported in this paper were trained on 2× NVIDIA A100 80GB GPUs, with πsen on cuda:0 and πjun on cuda:1; both models run with tensor-parallel size 1.
B
Additional Results
B.1
Qualitative Analysis of Over-emitted Tokens
Figure 6 visualises the tokens each model over-emits most relative to the junior, sized by drift magnitude. GRPO’s dominant over-emitted tokens form two clusters. The first is a pure enumeration artifact: the -I, -H, -F, -G, -E, -C family, single-letter labels that GRPO uses 71–146 times per corpus while the junior uses them 0–11 times. These tokens carry no mathematical content: they are structural markers GRPO learned to impose on multi-case reasoning, labelling each case with a capitalised suffix. TRL recovers almost all of them to near-junior levels (1–30 occurrences). The second cluster is reward-induced answer signalling: Answer goes from 51 occurrences in the junior to 419 in GRPO (8×), reflecting a learned habit of explicitly opening the answer statement; TRL reduces this to 116. TRL’s remaining over-emitted tokens are qualitatively different. The two largest are dom (junior: 0, GRPO: 8, TRL: 103) and ct (junior: 4, GRPO: 40, TRL: 104), compact mathematical notation for domain and likely cotangent or a covariant transform, absent from the junior and barely present in GRPO. Beyond these, TRL develops a coordinate-subscript vocabulary that GRPO does not: _l (junior: 0, GRPO: 4, TRL: 55), -j (junior: 5, GRPO: 3, TRL: 49), +x (junior: 11, GRPO: 2, TRL: 53), +y (junior: 9, GRPO: 39, TRL: 70), the i, j, k, l component-labelling system of vector notation. The pattern is consistent with the structural account: GRPO’s most extreme over-emitted tokens are formatting and signalling habits with no grounding in the mathematical content of the problems. The co-generation constraint eliminates these—they are precisely the choices the frozen junior cannot continue—while TRL’s residual drift is concentrated in compact mathematical notation that the junior’s vocabulary cannot recover by construction.
Vanilla GRPO
TRL
Figure 6: Tokens most over-emitted by Vanilla GRPO (left) and TRL (right) relative to the junior, sized by displacement magnitude. B.2 B.2.1
Out-of-Distribution Junior Evaluation Handoff Robustness
The results in §4.3 concern only the training-partner junior (Qwen3-4B-Instruct). As a secondary observation with no strong claims attached, we ask what happens when the same trained senior is paired with Qwen3-family juniors of different sizes that were not seen during training. These results are supplementary to our main framing and should not be read as evidence that the cooperation property generalises broadly; we include them as an empirical data point. 17
Table 5 reports pass@4 for Qwen3-0.6B and Qwen3-1.7B juniors. The most consistent pattern appears on AIME, where TRL leads by +4.5 points with the 0.6B junior and +3.5 with the 1.7B junior. On AMC and Minerva the picture is more mixed: TRL leads with the weaker junior on AMC (+5.8) but falls marginally behind with the 1.7B junior (−0.8); Minerva differences are within 1.3 points in either direction. We do not interpret this as a reliable generalisation signal. The AIME pattern may reflect that on harder problems the alignment with the training partner provides a stronger lift, or it may be an artefact of sample size at these problem counts. The one cell where GRPO leads (Qwen3-1.7B, AMC) is a useful reminder that the advantage is not monotone across partner sizes. Table 5: Reasoning-step handoff robustness (pass@4, ↑) with out-of-distribution juniors not seen during TRL training. Bold marks the higher value per cell; ∆ = TRL − GRPO. AMC
B.2.2
AIME
Minerva
Junior
GRPO
TRL
∆
GRPO
TRL
∆
GRPO
TRL
∆
Qwen3-0.6B Qwen3-1.7B Qwen3-4B
65.7 76.9 74.1
71.5 76.1 76.6
+5.8 −0.8 +2.5
28.1 28.4 34.2
32.6 31.9 40.0
+4.5 +3.5 +5.8
40.9 41.1 39.6
41.0 42.4 39.8
+0.1 +1.3 +0.2
Avg.
72.2
74.7
+2.5
30.2
34.8
+4.6
40.5
41.1
+0.5
Legibility with out-of-distribution juniors
We also pair each trained senior with Qwen3-0.6B and Qwen3-1.7B juniors it never encountered during training, and report legibility results alongside the in-distribution partner in Table 6. Interestingly, the in-distribution legibility gain does not carry over to unseen juniors: with both smaller juniors, distribution overlap is matching or slightly lower for TRL than for GRPO, and cross-entropy differences are mixed and small in magnitude. However, this is a moderately expected property of a co-generation objective that is, by design, specific to the training partner8 . We report these results as an initial characterization of how TRL legibility behaves under partner shift, and leave a systematic investigation to future work. Table 6: Legibility of senior CoTs across junior models of varying capability. Qwen3-4B-Instruct is the in-distribution training partner; the remaining rows are out-of-distribution. Junior
Model
AMC
AIME
Minerva
Avg.
0.440 0.447 0.386 0.389 0.125 0.113
0.517 0.526 0.455 0.456 0.154 0.156
0.522 0.518 0.464 0.453 0.117 0.097
0.493 0.497 0.435 0.433 0.132 0.122
0.865 0.866 0.896 0.888 0.961 0.973
0.857 0.847 0.866 0.870 0.957 0.970
0.855 0.853 0.880 0.880 0.963 0.976
0.859 0.855 0.881 0.879 0.960 0.973
Cross-entropy (nats, ↓) GRPO TRL GRPO Qwen3-1.7B TRL GRPO Qwen3-4B TRL Qwen3-0.6B
Distribution overlap (↑) GRPO TRL GRPO Qwen3-1.7B TRL GRPO Qwen3-4B TRL Qwen3-0.6B
B.3
Stochastic reasoning-step handoff robustness
The main HR result in §4.3 alternates the senior and the junior strictly at every \n\n boundary. This deterministic round-robin is a clean evaluation protocol, but in scenarios such as agent-to-agent 8 Note that legibility is not translatable directly to handoff robustness.
18
collaboration, drafter-target inference, and human-AI handoffs, the timing of handoffs is normally more flexible and the next contributor is typically chosen by the situation rather than by a fixed cadence. As a robustness check, we replace the deterministic round-robin with an independent Bernoulli(p = 0.5) draw at every \n\n boundary, keeping every other setting identical. The expected senior share remains 0.5, but the per-rollout schedule now varies, and a team’s success has to be robust to that variation. Table 7: Reasoning-step handoff robustness under stochastic Bernoulli (p = 0.5) schedules at \n\n boundaries (pass@8, %, ↑). Junior is the Qwen3-4B-Instruct training partner. Benchmark
GRPO
TRL
∆
AMC AIME Minerva
78.5 38.9 44.5
80.2 42.2 44.9
+1.7 +3.3 +0.4
Macro Avg
54.0
55.8
+1.8
Per-rollout senior shares are well-controlled across methods (mean 50.0% for both, standard deviation in the 5%–9% range across benchmarks). TRL retains a directional advantage over vanilla GRPO on every benchmark and on macro, with the largest gap again on AIME (+3.3 at pass@8). The macro gap is smaller than under the strict schedule of §4.3 (+1.8 vs. +3.1), as expected: under a randomised schedule, individual rollouts vary in how much of the trajectory each model carries, and pass@8 aggregates over schedule variation in addition to the usual generation variation. The strict-alternation protocol of the main text is the more stringent measurement; the stochastic protocol here verifies that TRL’s compatibility lift is not an artefact of a fixed schedule.
C
Why KL regularization toward the junior is structurally weaker than tandem rollouts
A natural ablation to TRL is to keep solo GRPO rollouts and add a per-token KL penalty toward the frozen junior πjun . This likewise “anchors” the senior toward the junior, but as a soft additive regularizer on the senior’s own trajectories rather than by sampling rollouts from a tandem mixture. Empirically this baseline is strictly weaker than TRL on every anchoring axis we measure (handoff robustness, marginal drift to base, and conditional legibility). We give a structural account of why: the gradient of the KL-Reg objective has access, at any regularization coefficient β, to exactly two signals (solo senior reward and per-position divergence between πθ and πjun ), neither of which is a function of whether the junior could continue from ht to a correct answer. Sweeping β scales one of these two signals relative to the other, but does not introduce a third; the missing signal is supplied only by changing the rollout distribution, which is what TRL does. C.1
Setup and what each objective optimizes
Let V be the vocabulary, x a prompt, y = y1:T ∈ V T a response, ht = x · y<t the context at position t, and r : V T → [0, 1] a bounded verifier reward. Both πθ (senior, trainable) and πjun (junior, frozen) QT are autoregressive: π(y | x) = t=1 π(yt | ht ). We compare two objectives. (a) GRPO with KL regularization toward πjun . hP i hP i LKL-Reg (θ) = − Ey∼πθ A log π (y | h ) + β · E KL π (· | h ) π (· | h ) . t θ t t y∼π θ t jun t θ t t (2) Rollouts are sampled solo from πθ ; the regularizer penalizes per-position next-token disagreement with πjun along senior-natural trajectories. (b) TRL. Let Mp denote the tandem sampling: at each handoff boundary an independent Bernoulli(p) draw determines the active model at ∈ {sen, jun}, and yt ∼ πat (· | ht ). hP i LTRL (θ) = − Ey∼Mp A log π (y | h ) (3) θ t t . t : at =sen t 19
Since only senior-emitted positions depend on θ, the standard policy-gradient identity gives h i P ∇θ Ey∼Mp [r(y)] = Ey∼Mp r(y) t : at =sen ∇θ log πθ (yt | ht ) .
(4)
Eq. (3) with GRPO’s group-relative advantage is thus a variance-reduced unbiased gradient estimator of Ey∼Mp [r(y)]. TRL directly optimizes the expected reward of joint rollouts sampled under the tandem mixture schedule. Crucially, r here is evaluated on a trajectory that includes junior-emitted tokens, so the team-reward feedback at any senior-emitted position t depends on whether the junior’s contributions are compatible with the rest of the trajectory. The gradient on senior position t is therefore reward-weighted by a quantity that measures, for that position, whether the team carrying the junior’s tokens succeeds. This is the signal that the KL-Reg objective never receives. C.2
KL-Reg sees different signals than TRL at any β
The natural defense of KL-Reg is to argue that the empirical gap to TRL is an unfortunate choice of β and can be closed by tuning. We show it cannot: the KL-Reg gradient is, at any β, a function of a strict subset of the signals the TRL gradient sees, and that subset omits exactly the quantity handoff success depends on. No setting of β recovers the missing signal. From Eq. (2), the gradient of the KL-Reg objective decomposes as hP hP i i +β·∇ E π (· | h ) . ∇θ LKL-Reg (θ) = − ∇θ Ey∼πθ A log π (y | h ) KL π (· | h ) θ y∼π jun t t θ t t θ t θ t t {z } {z } | | (a) GRPO on solo senior rollouts
(b) per-position divergence from πjun
(5) Term (a) depends on r(y) for trajectories y ∼ πθ ; this is the senior’s solo reward, never evaluated on any trajectory in which πjun takes over. Term (b) depends on the next-token distributions πθ (·|ht ) and πjun (·|ht ) alone; it does not involve r at all. Neither term is a function of Ht , the probability that the junior’s continuation from ht reaches a correct answer. The coefficient β controls only the relative weight of (a) and (b); it cannot introduce the junior’s compatibility signal that is not present in either. Table 8 summarises the resulting behaviour at each position ht , organised by the per-position KL (what KL-Reg can see) and by whether the junior could continue successfully (what governs handoff success). The two methods agree on the off-diagonal and disagree on the diagonal; tuning β scales the row dimension without affecting the column. Table 8: What each method does at position ht , by per-position KL between πθ (· | ht ) and πjun (· | ht ) and by whether the junior could continue successfully (Ht ). Junior can continue (high Ht )
Junior cannot continue (low Ht )
High KL
KL-Reg penalises; TRL does not.
Both methods penalise.
Low KL
Neither method penalises.
TRL penalises (through reduced team reward); KL-Reg does not.
C.3
Implications
Tuning β cannot rescue KL-Reg on the axes the paper claims. A defender of the KL-Reg baseline could in principle argue that the empirical gap to TRL on handoff robustness and conditional legibility reflects an unfortunate choice of β. §C.2 closes this: raising β can only spend more pressure along the row axis of Table 8, while every position in the bottom-left cell remains invisible to the gradient. Beyond some moderate β, additional row-axis pressure also pulls πθ toward πjun on positions where the divergence carries the senior’s RLVR-acquired capability, potentially degrading solo accuracy without recovering the column-axis signal. Compute budgeting favors TRL. Polishing KL-Reg training with carefully-tuned β requires sweeping β, which multiplies the GRPO baseline cost linearly in the number of points swept. Given the orthogonality argument, this sweep is structurally guaranteed either to fail to close the gap on the column-axis-dependent metrics, or to close them only by collapsing solo accuracy. TRL has no 20
GRPO KL-Reg-GRPO TRL
Fraction of tokens
0.8 0.6 0.4
6
to GRPO (nats)
1.0
0.2 0.00.0
KL-Reg-GRPO (recovers 89%) TRL (recovers 93%) full recovery ( = drift)
4 2 0 2 4
0.5
1.0
1.5
6
2.0
Absolute drift from base (nats)
4
2
0
2
GRPO drift from base (nats)
4
Figure 7: Vocabulary drift from base for GRPO, KL-Reg, and TRL. Left: survival curve of the absolute per-token log-ratio to the base over tokens. Right: for the top-500 most displaced tokens by GRPO, the per-method drift relative to GRPO. analogous coefficient to sweep: the anchoring pressure is supplied by the rollout distribution rather than by a regularizer, and the only TRL-specific hyperparameters (p and K) are determined by the granularity of the handoff schedule rather than by a tradeoff with solo capability. The wall-clock comparison in §5.1 (TRL 9.4 hours, vanilla GRPO 7.8 hours, under the best-checkpoint protocol) therefore understates TRL’s compute advantage relative to a fully defended KL-Reg baseline, which would multiply the GRPO budget by the number of β values swept.
21