A Learning-Rate-Gated Failure of GRPO in a Small Language and Vision-Language Model Web Agent: A Controlled Null and Its Mechanism Chengguang Gan1 , Zhixi Cai2 , Yunhao Liang3 , Hanjun Wei3 , Shiwen Ni4 , Qinghao Zhang5 1 Independent Researcher 2 Monash University 3 University of Chinese Academy of Sciences 4 Shenzhen University of Advanced Technology 5 Pusan National University Correspondence: [email protected]
arXiv:2607.12640v1 [cs.AI] 14 Jul 2026
Abstract
Does GRPO add skill to a small web agent? Web agent task
Reinforcement learning with verifiable rewards, and Group Relative Policy Optimization (GRPO) in particular, is now run routinely on a supervised checkpoint in the hope of producing a stronger agent. We ask whether it adds skill to a small language and vision-language model web agent at the 4B to 8B scale, or whether it mostly reshapes behavior the supervised model already has. Across a control grid of 18 runs that varies learning rate, KL weight, seed, initialization, and clipping, no configuration credibly improves the success rate of a strong supervised baseline on tasks the agent has largely mastered. On the text track, moderate to high learning rates make it credibly worse. The null holds under paired testing, 25 evaluation seeds, 6 training seeds, changes to the recipe, both text and Set-of-Marks screenshot observations, and scaling the backbone to 8B; the credible harm is a text-track finding and is only nominal under Set-of-Marks. To show that the null reflects the setting and not a broken pipeline, we run the identical harness, reward, and recipe on tasks whose reward is reachable by sampling, and there the success rate rises by 22 points with a paired interval that excludes zero. GRPO therefore helps only when there is headroom to climb, meaning the sampled policy already succeeds more often than the greedy one. We then explain the failure. A middle learning rate degrades the agent and a high one collapses it, and the two regimes form a double dissociation: grafting localizes the degrade regime to the attention and MLP blocks, while the collapse regime cannot be traced to any single group, and the embedding change that dominates the weight movement is causally inert. At 4B, effective rank in the late layers tracks capability in both directions; at 8B the two come apart. This coupling is specific to the smaller model, so we report it as scale-dependent.
Instruction: "Click the Submit button" Submit text field click
a small model reads the page and emits one action per step
Qwen3-VL (4B/8B)
GRPO climbs only when the sampled policy can already beat the greedy one
mastered task greedy > sampled 49
33
headroom task sampled > greedy
49
greedy sampled GRPO
no credible gain
20
33
42
greedy sampled GRPO
gains +22 points
Same harness and reward.
The gap, not the algorithm, decides.
Figure 1: The question and the answer at a glance. GRPO adds no credible gain on tasks the agent has already mastered, but the same recipe gains 22 points where the reward is reachable by sampling.
1
Introduction
Post-training with reinforcement learning has become routine for language and multimodal agents. A supervised model is first taught to follow the task format, and GRPO or a close relative (Shao et al., 2024; Yu et al., 2026; Liu et al., 2025) is then run against a verifiable reward in the hope of pushing capability past what supervision alone reached. For web agents the appeal is direct, since rollouts are cheap to score against task success and the recipe that lifted mathematics and coding models is expected to lift an agent that clicks, types, 1
and navigates (Qi et al., 2025). What remains unclear is whether this second stage adds new skill to a small language and vision-language model agent, or whether it mostly sharpens choices the supervised model could already make (Chen et al., 2026). This matters in practice. If reinforcement learning adds capability, more of it is the right investment; if it only sharpens what the supervised model already does, the effort belongs in better supervision or distillation. We study one such agent under tight control: a 4B language and vision-language model (Bai et al., 2025) driving a web agent on the MiniWoB benchmark (Shi et al., 2017; Liu et al., 2018). Holding the harness, reward, and task set fixed, we sweep the reinforcement-learning choices a practitioner would tune and score every run with clean greedy decoding and a paired test against the supervised baseline. The result is a null. No configuration in the grid of 18 runs credibly beats supervision on tasks the agent has already largely mastered, and once the learning rate leaves a narrow low band the agent gets worse. The null is also robust. It survives adding evaluation and training seeds, adding the schedule and group-size choices the grid omitted, switching from a text observation to a Set-ofMarks screenshot, and scaling the backbone to 8B. Figure 1 previews the study and its outcome. One reading of a null is that the pipeline is broken or underpowered and would fail to climb anywhere. A positive control rules this out. Using the identical harness, reward, and recipe, we select tasks whose reward is reachable by sampling, where the model already succeeds under temperature more often than under greedy decoding, and retrain. There the agent climbs by up to 22 points, with a paired interval that excludes zero. GRPO improves the agent when the sampled policy can already beat the greedy one, and it stalls when greedy is the better of the two, which is the regime a competent supervised agent occupies. A practitioner can act on this by measuring the gap between sampled and greedy success first and expecting little from reinforcement learning when that gap is not positive. For a competent small web agent, extra skill comes from supervision or distillation, not from more reinforcement learning. We then open the model to ask why the harmful regimes fail. A low learning rate leaves an update that is real but too small to change behavior; a middle rate degrades the agent, and a high rate collapses it. The two failures are different lesions. The
degrade regime destroys the effective rank (Roy and Vetterli, 2007) of the late layers while leaving earlier layers intact, whereas the collapse regime holds that rank at or above supervision and instead destroys the readout, an inversion we confirm at nearly two thousand fixed states. To test whether this correlation is causal, we graft weights. Restoring the attention or MLP blocks from initialization repairs the degraded agent, while restoring the embedding, whose drift dominates the raw weight change, does nothing, which points to the compute blocks as the site of the failure and clears the embedding despite its large movement (Ilharco et al., 2022). Read in the other direction, the same probe shows late-layer rank rising with capability where the agent climbs and falling where it degrades, so at 4B rank tracks skill both ways. At 8B this coupling breaks down, and we say so. Our choice of a single benchmark is deliberate and follows from the interpretability goal. A mechanistic account requires holding everything except the intervention fixed, so that a change in effective rank or success rate can be attributed to the learning rate and not to uncontrolled variation in the environment. MiniWoB provides that control. Its tasks are deterministic under a seed, its success check is exact, and the breadth of interface skills it covers, from clicking and typing to selection and navigation, makes it a broad probe of web-agent behavior, not a narrow one. A benchmark such as WebArena (Zhou et al., 2024) is closer to deployment but introduces run-to-run variation that is hard to hold constant, and that variation would confound the attributions this study rests on. We treat MiniWoB as the controlled setting in which the mechanism can be read cleanly, and we are clear about what that costs in generality. Our contributions are the following. 1. A controlled null: across a grid of 18 runs and several robustness checks, GRPO does not credibly add skill to a competent small language and vision-language model web agent, and higher learning rates credibly remove it. 2. A positive control that identifies the cause of the null as a headroom condition, together with a cheap test of sampled against greedy success that can be applied beforehand. 3. A mechanistic account of the failure as a learning-rate-gated double dissociation, causally localized by weight grafting, with 2
the dominant embedding drift shown to be a correlational red herring.
success, which separates the parameters that carry a failure from those that merely move the most. Two gaps motivate this paper. The sharpening question has not been settled for an agent by a controlled comparison against its own supervised start with a matched positive control, and where a reinforcement learning null is reported it is seldom explained at the level of the weights. We address both. We establish a controlled null for a small language and vision-language model web agent, use a positive control to show the null is a property of the task rather than the training pipeline, and give a causal account of the failure at the level of individual weight groups. To our knowledge this is the first study to pair a controlled reinforcement learning null for a web agent with a mechanism for why it occurs.
4. Evidence that late-layer effective rank tracks capability in both directions at 4B and dissociates at 8B, reported as a property that depends on scale. 5. A full battery of paired statistics, equivalence tests, and interpretability measurements, released so others can check the null for themselves.
2
Related Work
Reinforcement learning with verifiable rewards has driven recent gains in reasoning models, and GRPO (Shao et al., 2024), with refinements such as decoupled clipping (Yu et al., 2026) and less biased advantage estimation (Liu et al., 2025) on top of the proximal policy optimization objective (Schulman et al., 2017), is now a standard recipe. Whether these gains reflect genuinely new capability or a sharpening of behavior the base model can already produce is contested, with evidence that reinforcement learning often fails to extend a model beyond what sampling already reaches (Chen et al., 2026). That question has been argued mostly on reasoning benchmarks through pass@k. We bring it to an interactive agent and ask it as a practitioner would, against the agent’s own supervised baseline. For web agents, both supervised and reinforcement learning pipelines have raised success rates across interactive benchmarks, from the MiniWoB environment (Shi et al., 2017; Liu et al., 2018) to the larger WebArena (Zhou et al., 2024), and online curriculum methods report further gains (Qi et al., 2025). This line of work shows that reinforcement learning can help a web agent. It does not settle whether it helps a small language and vision-language model agent on tasks the agent has already learned to solve, which is the regime a competent deployed system usually occupies and the one we isolate here. Our mechanistic analysis builds on tools for reading a model’s internal state. Effective rank (Roy and Vetterli, 2007) measures how many directions a representation actually uses, and editing in weight space shows that behavior can be moved by recombining parameter groups (Ilharco et al., 2022). We turn these into a causal test rather than a correlational one: we restore a single component group from its initialization, keep the rest, and re-measure
3
Experimental Setup
Our aim is to decide whether GRPO adds skill to a competent small language and vision-language web agent and, when it does not, to explain why. The design follows directly. We fix a single agent, environment, reward, and task set, so that any change in success rate traces to the reinforcement-learning choices and not to the setup, and on that fixed harness we run the three experiments summarized in Table 1. The rest of this section specifies the agent and reward (§3.1), the GRPO recipe and grid (§3.2), the evaluation and statistics that make the null credible (§3.3), and the interpretability probes (§3.4). 3.1
Agent, Environment, and Reward
The agent is Qwen3-VL at 4B and 8B (Bai et al., 2025), run on MiniWoB (Shi et al., 2017; Liu et al., 2018). At each step the policy πθ reads an observation and emits one action from a compact vocabulary of clicks, typing, selection, navigation, and an explicit finish, up to a fixed step budget. Two observation tracks share all other logic: a text track that serializes the page, and a Set-of-Marks track (Yang et al., 2023) that presents the screenshot with numbered marks. The control grid and the mechanism use the text track; the Set-of-Marks track is a modality check. The reward is a sparse binary terminal signal, broadcast to every step of the episode, r(τ ) = 1[success(τ )] ∈ {0, 1}.
3
(1)
Experiment
Principle
Purpose
A
Controlled null (§3.2, §3.3)
An 18-run grid over learning rate, KL weight, seed, initialization, and clip bound on one fixed harness, each arm paired against the supervised baseline (Eq. 2–9).
Decide whether any GRPO configuration credibly improves the agent’s success rate.
B
Positive control (§3.4)
The identical harness, reward, and recipe on tasks where sampled success already exceeds greedy (Eq. 15).
Separate a genuine null from a broken pipeline and identify the headroom condition.
C
Mechanism (§3.4)
Fixed-state rank and output-behavior probes, plus causal weight grafting on the trained checkpoints (Eq. 10–14).
Explain the failure at the level of individual weight groups.
Table 1: The three experiments of this study, the principle behind each, and what each is for. Section 3 specifies the methods and statistics; the results follow in later sections.
3.2
GRPO Recipe and Control Grid
3.3
For each training prompt the policy samples a group of G rollouts. Following Liu et al. (2025), the advantage is the group mean-centered reward, with no division by the group standard deviation, 1 Ai = ri − G
G X
Every run is scored with clean greedy decoding on the set E of 11 tasks at 5 seeds, giving 55 matched episodes. The success rate over E is SR =
rj .
(2)
j=1
πθ (yc | x, y<c ) , πθold (yc | x, y<c )
1 X 1[success(e)], |E|
(7)
e∈E
reported at level z with the Wilson score interval (Wilson, 1927) q p) z2 z2 + 4n pb + 2n ± z pb(1−b 2 n . (8) 2 1 + z /n
The per-token importance ratio between the current policy and the policy that produced the rollout is ρc (θ) =
Evaluation Protocol and Statistics
(3)
Because a MiniWoB reward is deterministic given the task seed, each (task, seed) pair is matched across C arms and every arm is compared to the super1 X vised LGRPO = − min ρc A, clip(ρc , 1−ϵlo , 1+ϵhi ) A , baseline with a paired test. On the discordant C pairs (b, c) the exact two-sided McNemar p-value c=1 (4) is (McNemar, 1947) with an asymmetric clip-higher bound (Yu et al., min(b,c) X b + c 2026), ϵlo = 0.20 < ϵhi = 0.28. An optional p=2 2−(b+c) , (9) b KL to the loss, using the nonKL anchor adds β D i i=0 negative k3 estimator against the frozen initializaand a task-clustered bootstrap resamples the 11 task tion πref with uc = log(πref /πθ ), clusters to give a 95% interval on the success-rate X b KL = 1 euc − uc − 1 . (5) difference ∆ = SRarm − SRSFT . We call an arm D C c credibly better or credibly worse only when this The learning rate follows a linear warmup over a interval excludes zero; a positive point estimate fraction f of the R rounds, then a cosine decay, whose interval touches zero is no credible differ( ence. Equivalence within a margin δ is a two one(t + 1)/w, t < w, sided test (Schuirmann, 1987): the 90% interval of w = ⌈f R⌉. ηt = η0 · 1 t−w 1 + cos π , t ≥ w, ∆ lies inside [−δ, +δ]. 2 R−w (6) 3.4 Interpretability Measurements The controlled null (experiment A) is a grid of 18 runs varying learning rate, KL weight β, seed, On a fixed cache of N hidden states we read a small initialization, and the clip bound, with a recipe- set of quantities. The effective rank of a layer is the ablation battery that adds the warmup and cosine exponential of the entropy of its normalized covarischedule and the group size G ∈ {8, 16, 32}. ance eigenvalues {λk } (Roy and Vetterli, 2007), and the clipped surrogate minimized over a segment of C completion tokens is
4
and we also report the participation ratio and the stable rank of the same state matrix X, X λk erank = exp − pk log pk , pk = P , λj j k (10) 2 P 2 λ ∥X∥F k PR = Pk 2 , sr = . (11) λ ∥X∥22 k k
from zero to about ten points and the exact McNemar test returns p between 0.50 and 0.69, so the nudge is indistinguishable from noise (Table 2). Nor does the result hinge on the learning rate alone. Varying the KL weight, the initialization, and the clip bound each moves the point estimate by at most two points and none of them credibly, so no axis of the 18-run grid turns into a gain. Even taking the best arm as the headline, an upward-biased choice, leaves an estimate that only matches supervision plus a statistically empty 3.6 points. A flat result invites the reading that the tasks leave no room to improve, but they do. Restricting the score to the frontier subset, defined as the tasks whose supervised success sits between 0.2 and 0.8, the baseline is 37.1% and the best configuration reaches 42.9%, a gain whose paired interval still includes zero, so the headroom that exists is not credibly converted into gain. Stated the other way, two one-sided tests place four of the five strongest configurations within five points of supervision, one of them episode for episode identical, which reads the outcome as an equivalence: on the mastered tasks GRPO matches the supervised policy but does not improve on it. The one variable that moves success is the learning rate, and it moves it only downward. Holding initialization and the KL weight fixed on the text track, clean success falls as the step size grows, from 52.7% at 3 × 10−6 , through the supervised baseline near 5 × 10−6 , down to 33.3% at 1 × 10−5 and zero at 2 × 10−5 (Figure 2). Three regimes fall out. At the low end the update is a functional no-op that leaves behavior at supervision. A middle rate degrades the agent by a credible 15 points, and the highest rate collapses it to zero, both with intervals that exclude the baseline and McNemar p below 0.04 and 0.001 (Table 2). The only rates that leave the agent unharmed are the ones too small to change behavior, and no rate above that band clears the baseline. This monotone shape is specific to the 4B text track; on the Set-of-Marks track and at 8B the nominal peak sits elsewhere, so we keep the shape scoped to 4B text. The null does not rest on any single choice in the grid, and the full battery is collected in Appendices C and D. Adding 25 evaluation seeds lowers the baseline to 44.7% and still surfaces no credible winner; the one arm that reaches p = 0.001 under an unclustered test loses that status once episodes are clustered by task, and a replication over 6 training seeds averages 49.7%, back on the baseline, so
Three probes read the output distribution pn = softmax(ℓn ). Argmax agreement with the initialization counts how often the top next-token prediction is unchanged, i 1 X h agr = 1 arg max ℓθn = arg max ℓref , n N n (12) while the mean entropy and the degeneracy, the fraction of states whose mass collapses onto one token above a threshold τ , are 1 X X H= − pn,v log pn,v , N n v (13) 1 X 1 maxv pn,v > τ . deg = N n A causal graft restores one component group G, attention, MLP, or embedding, from initialization and re-evaluates, θ′ = θ,
θG′ ← θGinit ,
recovery = SR(θ′ ). (14) Finally, the headroom of a task is the gap between the sampled and greedy success of the supervised policy, ∆head = SRsample − SRgreedy .
(15)
GRPO can climb only when ∆head > 0; the positive control (experiment B) selects tasks on this criterion, and the grafts drive the mechanism analysis (experiment C).
4
When GRPO Fails
We begin with the regime a deployed agent occupies, the tasks it has already learned to solve. The supervised policy clears the 11-task grid at a success rate of 49.1% (27 of 55 matched episodes), an aggregate that mixes tasks it solves almost every time with a harder frontier. The question is whether any reinforcement learning configuration moves that number upward, and none does. The two strongest arms are nominally above supervision by 3.6 points, yet their paired intervals stretch 5
Regime
LR
no-op 3 × 10−6 baseline – degrade 1 × 10−5 collapse 2 × 10−5
SR
∆SR
52.7 +3.6 [+0.0, +10.9] 49.1 – 33.3 −15.0 [−20.9, −9.5] 0.0 −49.1 [−62.7, −36.4]
5
McNemar p 0.50 – ≤0.04 <0.001
When GRPO Succeeds
The previous section ended on the obvious worry that the pipeline simply cannot climb anywhere. We test it by keeping the harness, reward, and recipe exactly as before, still on the 4B text track, and changing only the tasks. We profiled the supervised policy on all 119 MiniWoB tasks, recording both its greedy success and the success it reaches under temperature sampling, and selected ten tasks with reachable headroom, where sampling already succeeds more often than greedy and there is therefore a policy for GRPO to move toward that greedy does not already reach. Two examples fix the idea: drawing a line, where greedy never succeeds but sampling does more than half the time, and filling an autocomplete field, where sampling roughly doubles the greedy rate. On this set the same recipe climbs. Scored on the ten tasks at 5 seeds, 50 matched episodes, the supervised baseline solves 20.0%, and the strongest configuration reaches 42.0%, a gain of 22 points whose task-clustered interval runs from 8 to 40 and excludes zero, with McNemar p = 0.007 (Figure 3). The climb is not one lucky run. Five of the six configurations rise between 12 and 22 points, four of them credibly, and only a single run at a middle rate degrades. The low learning rate that was a pure no-op on the mastered grid gains 12 to 18 points here, so the variable that decides the outcome is the task and not the step size. Table 3 puts the two settings side by side. The recipe that cannot separate from supervision on the mastered grid produces a credible 22-point gain on the headroom set, so the null is not a broken or underpowered pipeline; it is GRPO failing where a working GRPO would also fail, on tasks whose reward the greedy policy already reaches. This explains the null in terms of the objective itself. GRPO optimizes sampled rollouts, so it can only climb toward a policy that sampling reaches. On the mastered grid the gap runs the wrong way: greedy is at 0.49 against a sampled 0.32 to 0.38, so there is nothing above greedy to move toward. On the headroom set the gap is positive by construction, and the agent climbs. The gap of Eq. 15 is therefore a cheap screen a practitioner can run before committing compute, since it predicts both the null and the climb from the supervised policy alone.
Table 2: Learning rate is the only dial that moves success, and only downward (4B, text track, supervised initialization, no KL). The degrade and collapse rows pool the three and two seeds run at those rates, and ∆SR is the task-clustered paired difference against the supervised baseline, so for a pooled cell it need not equal the difference of the marginal rates. The low-rate no-op is indistinguishable from supervision; the moderate and high rates are credibly worse.
its apparent edge was a lucky training seed. The warmup and cosine schedule the grid omitted remove the collapse pathology, since moderate rates now dip and recover instead of dying, but a recovery is not a gain and the schedule still produces no credible improvement. A larger sampling group of 16 or 32 does not help, and at a high rate it breaks the agent sooner, since a larger group is a larger effective step. The Set-of-Marks screenshot raises the baseline to 63.6% and again yields no credible gain, and the 8B backbone repeats the pattern, a nominal low-rate nudge that never clears the interval and credibly worse success at the higher rates. The null holds across every axis a practitioner would turn. A null could also be an artifact of rewardhacking or of scoring a checkpoint after it broke, and neither is at work here. If the agent were gaming the sparse reward by finishing early, the flat runs would fill with premature terminations, but the perepisode taxonomy puts reward-hacking near zero in every regime, and the degrade and collapse runs fail because they emit invalid output, not because they stop short. If the null came from scoring the final checkpoint after it had already broken, then selecting the best checkpoint by held-out success would surface a hidden win. It does not. Scoring every saved round and running a paired test on the best one, the strongest flat run still gains only 3.6 points with an interval that touches zero, and the brief early rise of the degrade run falls apart before it is credible. The null comes from the setting itself: it is not an artifact of when we stopped, and it is not the agent gaming the reward. That still leaves the worry that the pipeline cannot climb anywhere, which the next section settles.
6
Learning rate is the master knob: no learnable band (4B, text) clean greedy SR (%)
60 no-op
no-op
50 40
degrade
30 20 10 0
SFT baseline (49.1)
collapse
5e-6
2e-5
3e-6
1e-5 learning rate
0
10
Per-task success: 2e-5 arms collapse to 0
SFT 49.1
20 30 40 50 Clean SR (%) with Wilson 95% CI
60
70
click-button [ceil]
5
5
5
0
5
0
click-checkboxes-transfer [front]
3
4
3
0
3
0
click-color [front]
1
1
0
0
1
0
click-link [front]
2
2
0
0
2
0
click-menu [front]
1
1
1
0
1
0
click-tab [ceil]
5
5
3
0
5
0
click-tab-2-medium [front]
2
2
0
0
2
0
count-sides [hard]
0
1
0
0
0
0
navigate-tree [front]
2
2
0
0
2
0
sign-agreement [ceil]
4
4
4
0
4
0
stock-market [front]
2
2
0
0
2
0
se -ba SFT
s2
e6_
_lr3
sw
ov_
s2
e5_
_lr1
sw
ov_
s2
e5_
_lr2
sw
ov_
o
e5
_lr2
05
_kl0
ab v2_
e5
4 3 2 1 0
_lr2
ase
c_b
_fa ov2
5
successes (of 5 seeds)
Control grid: no CI clears the SFT baseline low-lr (<=5e-6) flat 1e-5 degrade 2e-5 collapse
ov2_fac_base_lr3e6 ov_sw_lr3e6_s2 ov_fac_base_lr5e6 ov2_ab_clipsym_lr5e6 ov_sw_lr5e6_s2 ov_klR_kl05_lr5e6 ov_fac_base_lr1e5 ov2_ab_clipsym_lr1e5 ov_sw_lr1e5_s2 ov_sw_lr1e5_s3 ov2_sw_lr1e5_s4 ov_klR_kl01_lr1e5 ov_klR_kl05_lr1e5 ov2_ab_kl010_lr1e5 ov2_fac_base_lr2e5 ov_sw_lr2e5_s2 ov2_sw_lr2e5_s3 ov2_ab_kl005_lr2e5
Figure 2: The null across the grid (4B, text). Left: clean greedy success against learning rate, flat at the low rates where the update is a functional no-op, then degrading and collapsing as the step grows, with no setting clearing the supervised baseline (dashed). Middle: a forest plot of all 18 arms with Wilson intervals, none of which clears the baseline. Right: the per-task success matrix, where the moderate and high rates drive whole tasks to zero while the low-rate arm tracks supervision.
clean greedy SR (%)
Positive control: same recipe climbs on headroom-reachable tasks 70 60 50 40 30 20 10 0
nine at supervision to near one while the earlier layers keep their rank. The high rate that collapses the agent leaves that late-layer rank at or above supervision yet drives the argmax agreement with initialization to zero (Figure 4, left). Both regimes push the readout down, collapse completely and degrade in part, but only degrade also destroys the representation. This rank inversion is stable across the full cache of 1999 states, so it is not an artifact of thin sampling. Rank and agreement are correlations, and to make them causal we graft. Restoring one component group from initialization into the trained model, keeping the rest, and re-scoring isolates the weights that carry the failure. For the degraded agent, restoring the attention or the MLP alone lifts frontier success from 11% to 37% and 40%, at or just above the supervised level, while restoring the embedding leaves it near 14% (Figure 5; the full recovery table for both regimes is in Appendix E). Either compute pathway is sufficient, so the failure is localizable, and the direction holds across three degrade checkpoints. This is where a correlational reading would have gone wrong. The embedding drifts more than any other group in the raw weight difference, so an argument from movement alone would blame it, yet the graft shows the embedding is causally inert while the attention and MLP carry the damage. To rule out a generic undoing of drift, we restore a random group of matched parameter count as a null; across two dozen draws it recovers to a mean of 19% and a 95th percentile of 23%, and the real attention and MLP grafts sit well above that band, so the localization is specific and not a side effect of moving weights back toward their start. The collapsed agent behaves differently under
credible +22pp (CI [+8,+40], p=0.007)
32 20
34
38
38 42
14
SFT lr1e-5 lr3e-6 lr5e-6 lr1e-5 lr3e-6 lr5e-6 baseline s1 s1 s1 s2 s2 s2
Figure 3: Positive control on the headroom set. The supervised baseline (dashed) solves the ten tasks at 20%; five of six configurations climb, and the best reaches 42% with a paired interval that excludes zero. The low rate that does nothing on the mastered grid climbs here. Task set
SFT
Best
∆SR
McNemar p
Mastered grid Headroom set
49.1 20.0
52.7 42.0
+3.6 +22.0
0.50 0.007
Table 3: The same harness, reward, and recipe on two task sets, the mastered 11-task grid and the 10-task headroom set, each at 5 seeds. On the mastered grid the best arm gains only +3.6 points, with a task-clustered 95% interval [+0.0, +10.9] that includes zero. On the headroom set, where sampling already beats greedy, it gains a credible +22.0 points, interval [+8.0, +40.0]. The null is a property of the task, not of the pipeline.
6
Why GRPO Fails
The headroom condition says when GRPO fails to help, but not how the harmful learning rates damage the model, so we open it up. Reading a small set of quantities on a fixed cache of hidden states shows that the two harmful regimes damage the model in different ways. The middle rate that degrades the agent destroys the effective rank of the late layers, read at layer 35, which falls from about 7
9.2
10.1
7.0
1.2
base
SFT
Rank tracks capability in both directions (4B, pc task family) 8
no-op (3e-6)
climb (GRPO gain)
SFT (start)
7
5 degrade (GRPO loss)
3 5
degrade collapse (1e-5) (2e-5)
10
15
20 25 30 35 clean greedy SR (%)
40
45
1.0
20
6
4
8B: late-layer rank collapses without behavior loss (decoupled)
50
0.8
argmax agreement
13.9
L35 effective rank
16 14 12 10 8 6 4 2 0
L35 effective rank
L35 effective rank (N=1999)
Degrade destroys late-layer rank; collapse preserves it
15
0.6
10
0.4
5 0
0.2 SFT
3e-6
5e-6 8e-6 learning rate
1e-5
2e-5
0.0
Figure 4: The rank account (layer 35). Left: effective rank by regime at 4B, where degrade collapses the rank and collapse preserves or raises it. Middle: on the positive-control family, rank stays healthy where the agent climbs and collapses where it degrades. Right: at 8B, rank and argmax agreement never fall together across the learning-rate sweep, so the coupling does not transfer. Full probes, including argmax agreement and degeneracy by regime, are in Appendix E.
frontier SR after restoring init component (%)
Grafting localizes degrade: attn/mlp exceed the random-null p95 40
random-null spread (mean to p95) 37.1
family rank health and capability move together, with the caveat that these are single checkpoints and the upward move is small. This coupling is a property of the 4B model and does not survive a change of scale. Reading the same probe across the learning-rate sweep on the 8B backbone, on the text track, we find no setting where rank and argmax agreement fall together (Figure 4, right). At the rates that degrade the 4B model the 8B late-layer rank collapses to about two while the argmax agreement stays between 0.78 and 0.95, so the readout is untouched even though the representation is not, and only at the highest rate does the agreement collapse, with the rank left high. The larger model absorbs a late-layer rank collapse without changing what it predicts. We therefore scope the account: the link between rank and capability holds at 4B, shown in both directions by grafting and by the positive control, and it breaks at 8B. We treat that as a limit of the 4B account. The mechanism, then, is a pair of learning-rategated lesions. A middle rate erases the late-layer rank that the attention and MLP carry, and a high rate erases the readout in a way spread across the whole model. Neither adds anything the supervised agent lacked.
40.0
30 null p95 = 22.9
19.3
20 10 0
11.4
none (as-is)
random null
attn graft
mlp graft
Figure 5: Causal grafting on the degraded agent. Restoring the attention or MLP from initialization recovers frontier success past the random-null band (its mean to 95th percentile), while restoring the embedding does not, so the damage is localized to the compute pathways and not to the group that moved most.
the same test. No single group restores it, attention and MLP alone recover almost nothing, and only restoring every group together returns it to the supervised level, so collapse is a distributed corruption of the readout rather than a lesion in one place. This completes a double dissociation: degrade destroys the late-layer rank and is localizable to a single compute pathway, while collapse spares the rank and cannot be localized at all. For rank to be a meaningful measure, it should rise when the agent improves, not only fall when it breaks. On the family used for the positive control it does. Reading the same late-layer rank on three checkpoints from that family, the supervised start sits at rank 6.9 with 20% success, the configuration that climbs to 42% keeps its rank healthy at 7.1, and the configuration that degrades to 12% has its rank cut to 3.2 (Figure 4, middle). The loss shows up as a sharp rank collapse and the gain as a rank that stays healthy rather than falling, so within this
7
Discussion
The result carries a direct lesson for practice. On a small web agent that already performs its tasks, the next increment of skill does not come from a second stage of reinforcement learning, because the agent’s greedy policy already sits above what its own sampling reaches and GRPO can pursue only the sampled policy. The headroom gap of Eq. 15 is the screen to run first: a value at or below zero on a held-out set is the signal to spend the budget on 8
Limitations
supervision instead. In a distillation pilot, expert supervision lifted the base model by 30 points on a task set where reinforcement learning on top of any initialization added nothing (Appendix C). Two boundaries on the claim should be read alongside it. The study is one benchmark and one model family, chosen so the mechanism can be read without the confounds a noisier environment adds, and the robustness battery covers the recipe choices a reviewer would raise (schedule, group size, seeds, observation track, and model scale), none of which changes the verdict. The credible harm from a large step is established on the 4B text track and is only nominal on the Set-of-Marks track, so we state it as a text-track result and not a universal one. The mechanism has its own scope. The link between late-layer rank and capability that grafting and the positive control establish at 4B does not hold at 8B, where a rank collapse and a change in behavior never coincide, so the rank account is a property of the smaller model and not a law across scale. The finding brings the sharpening debate, so far settled largely on reasoning tasks by pass@k (Chen et al., 2026), to an interactive agent with a controlled answer. Where that debate asks whether reinforcement learning extends a model beyond its base, we ask the same of an agent against its own supervised start and find that it does not, then show with the positive control that the boundary is not the method but the headroom of the task. The contested general claim becomes a specific one we can test: reinforcement learning extends the agent only when its sampled policy already beats its greedy policy.
8
This study is deliberately narrow, and several boundaries should be kept in mind. It uses a single benchmark, MiniWoB, and a single model family, Qwen3-VL at 4B and 8B; the null and the mechanism may look different on a more varied environment such as WebArena or on a different backbone. The credible harm from a large learning rate is established on the 4B text track, and on the Set-ofMarks track it is only nominal, so the harm side of the result is text-track evidence and not a universal claim. The mechanism has a scale boundary of its own: the link between late-layer rank and capability holds at 4B but breaks at 8B, so the rank account is specific to the smaller model. The causal grafting and rank-health readings rest on a small number of checkpoints per regime and their point estimates are noisy, so we report them qualitatively. Finally, the null concerns the regime where the agent has already mastered its tasks, and the positive control shows that the same recipe does climb once the task has headroom; we therefore make no claim that reinforcement learning is useless for web agents in general, only that it adds no skill in the competent-agent regime we isolate.
References Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. 2025. Qwen3vl technical report. arXiv preprint arXiv:2511.21631. Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. 2026. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? Advances in Neural Information Processing Systems, 38:57654– 57689.
Conclusion
We asked whether GRPO adds skill to a small language and vision-language web agent on tasks it has already learned. Across the grid and the robustness checks it does not, and past a narrow band of learning rates it subtracts skill instead. The same pipeline climbs 22 points once the task has headroom, so the null is about the task, not the method. The failure has a clear mechanism: a middle rate erases the late-layer rank that grafting traces to attention and MLP, a high rate erases the readout that no single group repairs, and the dominant embedding drift is inert. For a competent small agent, more skill comes from better supervision, and GRPO earns its compute only where sampling beats greedy.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089. Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. 2018. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint arXiv:1802.08802. Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. 2025. Understanding r1-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783.
9
The appendices below collect the full result tables, a gallery of the supporting figures, and the algorithms referenced in the main text. All numbers are transcribed from the same frozen result tables used in the body.
Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Jiadai Sun, Xinyue Yang, Yu Yang, Shuntian Yao, Wei Xu, et al. 2025. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning. In International Conference on Learning Representations, volume 2025, pages 79791–79821.
A
The Control Grid in Full
The controlled null is an 18-run grid on the 4B text track that varies the learning rate, the KL weight, the seed, the initialization (supervised or base), and the clip bound. Every arm is scored by clean greedy decoding on the 11 tasks at 5 seeds, giving 55 matched episodes, and compared to the supervised baseline with the paired test of the main text. Table 4 lists the representative arms. Classified by their training dynamics the 18 runs split into 9 functional no-ops, 6 degrade runs, and 3 collapse runs, and no arm’s paired interval clears the baseline. Reporting the best of eighteen is itself an upwardbiased statistic, and even that best matches supervision plus a statistically empty 3.6 points. The “no headroom” reading is refuted on the frontier subset, the tasks whose supervised success lies in [0.2, 0.8]: the supervised frontier baseline is 37.1% [23.2, 53.7] (13 of 35) and the best configuration reaches only 42.9% [28.0, 59.1] (15 of 35), whose interval lower bound of 28.0 sits below the baseline point estimate.
Olivier Roy and Martin Vetterli. 2007. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pages 606–610. IEEE. Donald J Schuirmann. 1987. A comparison of the two one-sided tests procedure and the power approach for assessing the equivalence of average bioavailability. Journal of pharmacokinetics and biopharmaceutics, 15(6):657–680. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. 2017. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages 3135–3144. PMLR.
B
Edwin B Wilson. 1927. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158):209–212.
Paired Statistics, Equivalence, and Checkpoint Selection
The verdict uses the exact McNemar test on the discordant pairs and a task-clustered bootstrap on the 11 task clusters, both over episode-matched arms. Beyond the failure to reject, a two one-sided test states a positive equivalence: four of the five strongest configurations lie within ±5 points of supervision (Table 5), one of them episode-identical. Because the reported success is the final checkpoint, we also retrained the three regime exemplars saving every round and paired-tested the single best-by-eval checkpoint of each; none is credibly superior (Table 6), so the null is not an artifact of scoring an already broken checkpoint.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. 2023. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.11441. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. 2026. Dapo: An open-source llm reinforcement learning system at scale. Advances in Neural Information Processing Systems, 38:113222–113244. Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. 2024. Webarena: A realistic web environment for building autonomous agents. In International Conference on Learning Representations, volume 2024, pages 15585–15606.
C
Robustness Battery
The null is unchanged by every recipe and evaluation choice a reviewer would raise. Table 7 collects the axes. Widening evaluation to 25 seeds lowers the baseline to 44.7% and leaves no credible winner. The one borderline arm (McNemar p = 0.001 10
Arm
Init
LR
KL β
Clean SR [95%]
∆SR [95% CI]
McNemar p
Verdict
Supervised baseline Base initialization Best, SFT init Best, base init Low rate Middle rate, pooled High rate, pooled KL rescue Best KL
SFT base SFT base SFT SFT SFT SFT SFT
— — 3 × 10−6 3 × 10−6 5 × 10−6 1 × 10−5 2 × 10−5 2 × 10−5 1 × 10−5
0 0 0 0 0 0 0 0.05 0.10
49.1 [36.4, 61.9] 50.9 [38.1, 63.6] 52.7 [39.8, 65.3] 52.7 [39.8, 65.3] 49.1 [36.4, 61.9] 33.3 [26.6, 40.8] 0.0 [0.0, 3.4] 49.1 [36.4, 61.9] 50.9 [38.1, 63.6]
— +1.8 +3.6 [+0.0, +10.9] +3.6 [+0.0, +9.1] +0.0 −15.0 [−20.9, −9.5] −49.1 [−62.7, −36.4] +0.0 +1.8
— — 0.50 0.69 — ≤0.04 <0.001 — —
baseline no-diff no-diff no-diff no-diff worse worse no-diff no-diff
Table 4: Representative arms of the 18-run control grid (4B, text). Clean SR is greedy success over 55 matched episodes with a Wilson interval; ∆SR is the task-clustered paired difference against the supervised baseline. The middle- and high-rate rows pool 3 and 2 seeds. The KL rescue arm is episode-identical to supervision (discordant pairs b/c = 0/0). No arm is credibly better than supervision.
90% CI of ∆
Configuration Base-init flat KL 0.10 Recipe, G=16 Base-init sanity lr 3 × 10−6 , s2
[+0.0, +4.1] [+0.0, +0.0] [−2.7, +4.1] [−2.3, +0.9] [+0.0, +15.0]
Equiv. (±5pp)
Robustness axis
Base Best ∆ Verdict
yes yes, identical yes yes only ±15pp
Grid, 11 tasks × 5 seeds Frontier subset 25 evaluation seeds 6 training seeds, mean Warmup + cosine schedule Group size G ∈ {16, 32} Set-of-Marks observation 8B backbone (Set-of-Marks)
49.1 37.1 44.7 49.1 49.1 49.1 63.6 63.6
Table 5: Two one-sided equivalence at the 25-seed grain. Four of five top configurations are statistically equivalent to supervision within ±5 points. The lone exception is the lucky-seed arm, whose six-seed replication averages 49.7% (Table 10). Regime
Best-ckpt SR −6
Flat (3 × 10 ) Degrade (1 × 10−5 ) Collapse (2 × 10−5 )
52.7 ∼51 (r1) 49.1 (r0)
∆SR [95% CI]
+3.6 +5.7 +5.1 +0.6 +3.6 – +7.3 +9.1
no credible gain no credible gain no credible gain no credible gain no credible gain no credible gain no credible gain no credible gain
Table 7: The null survives every robustness axis. Base is supervised success and Best ∆ is the largest nominal point-estimate gain of any arm, in percent, except the training-seed row, which reports the six-seed mean. Every listed gain has a paired interval that includes zero.
p
+3.6 [+0.0, +9.1] 0.500 +1.8 [−9.1, +12.7] 1.000 −49.1 final <0.001
∆SR
Verdict
Warmup+cosine, G=8 3 × 10 52.7 +3.6 Warmup+cosine, G=8 8 × 10−6 — recover Warmup+cosine, G=8 1.5 × 10−5 — recover Warmup+cosine, G=8 2 × 10−5 0.0 / 32.7 worse Larger group, G=16 1.5 × 10−5 0.0 collapse
no-diff no-diff no-diff worse worse
Schedule / group
LR
SR −6
Table 6: Best-by-eval checkpoint selection. Scoring every saved round and paired-testing the maximum-SR checkpoint of each regime surfaces nominal nudges (flat drifts to 52.7%, degrade peaks near 51% before breaking) but none survives the paired test.
Table 8: Recipe ablation. A standard warmup and cosine schedule turns the constant-rate collapse into a dip-andrecover at moderate rates but yields no credible gain, and a larger sampling group is a larger effective step that breaks high rates faster.
under an unclustered test) loses credibility once episodes are clustered by task, and a six-seed replication of it averages 49.7% (Table 10), so the edge was a lucky training seed. Adding the warmup and cosine schedule removes the collapse pathology without producing a gain, and a larger group breaks the agent sooner at high rates (Table 8). A per-episode taxonomy shows the failures are output degeneration, not reward-hacking: premature finishing is near zero in every regime (Table 9). On a separate distillation pilot (5 tasks, 6 seeds, n = 30), expert supervision lifts the base model from 50% to 80% while GRPO on top of any initialization adds nothing, which is the evidence that the constructive lever is supervision rather than reinforcement learning.
Regime
Reward-hack
Invalid output
Correct/valid
Flat / KL Degrade Collapse
∼0% ∼0% ∼0%
low 63% 98%
∼50% / ∼40% rest rest
Table 9: Per-episode failure-mode taxonomy. Rewardhacking (premature finish) is near zero in every regime, so the null is not reward gaming; degrade and collapse fail by emitting invalid output while flat and KL preserve the supervised structure.
11
Seed
1
2
3
4
5
mean rel. L2
Regime
6
sparse frac.
embed drift
−6
Clean SR
49.1
52.7
45.5
49.1
50.9
No-op (3 × 10 ) 0.00017 0.0023–0.0031 ∼0.0002 Degrade (1 × 10−5 ) 0.0010–0.0011 — ∼0.011 −5 Collapse (2 × 10 ) 0.0008–0.0010 up to 0.068 0.0002–0.0010 KL rescue (2 × 10−5 ) 0.00198 0.115 ∼0.006
50.9
Table 10: Six-seed replication of the nominal-best configuration (SFT init, constant lr 3 × 10−6 , G = 8). The mean is 49.7% (±2.5 sd, 95% CI [47.7, 51.7]), on the supervised baseline of 49.1%, so the +5.1-point singleseed result was training-seed noise. Config (4B, SoM)
SR
∆SR
McNemar p
Supervised baseline lr 3 × 10−6 lr 1 × 10−5 lr 2 × 10−5
63.6 67.3 70.9 52.7
— +3.6 +7.3 −10.9
— no-diff no-diff 0.070
Table 13: Weight movement by regime. The outcome is not monotone in movement: collapse moves less than degrade yet destroys more, and the KL rescue arm moves the most of all while staying at the supervised baseline. Regime base SFT no-op degrade collapse
Table 11: Set-of-Marks track at 4B. No configuration credibly beats the supervised baseline, and the highrate degradation is nominal (p = 0.070, task-clustered interval touches zero), not credible as it is on the text track. Config (8B, SoM)
SR
∆SR
McNemar p
Supervised baseline lr 3 × 10−6 lr 1 × 10−5 lr 2 × 10−5
63.6 72.7 45.5 38.2
— +9.1 −18.2 −25.5
— 0.062 0.002 0.001
— 1.00 1.00 0.07–0.50 0.00
0.00 0.00 0.00 0.50–0.93 0.80–1.00
0.002–0.011 ∼0.01 0.078 7.2–10.3 0.17–9.04
which is behaviorally at the baseline (Table 13). The rank and output-behavior probes give the double dissociation in full, including next-token entropy (Table 14). Causal grafting localizes degrade to the attention or MLP and refutes the embedding, whose drift is largest; a magnitude-matched random-component null over 24 draws recovers only to a mean of 19.3% and a 95th percentile of 22.9%, below the real grafts, while for collapse no single group recovers (Table 15). The rankto-capability coupling that holds at 4B does not transfer to 8B, where rank and argmax agreement never fall together across the sweep (Table 16). The KL anchor is a mode-preserving stabilizer, not a teacher: every KL arm holds argmax agreement at 1.0 while next-token entropy inflates to 4.9–7.7, and KL suppresses embedding drift (∼0.006 with KL against ∼0.011 without, at 1 × 10−5 ) rather than freezing the weights.
Observation Track and Model Scale
The no-gain null generalizes across observation track and model scale, while the credible harm from a large rate is established on the 4B text track and is only nominal elsewhere. On the Set-ofMarks track the supervised baseline is higher, at 63.6%, and no configuration credibly beats it; the high-rate degradation is directional but not credible (Table 11). At 8B on Set-of-Marks the baseline is also 63.6%, the low-rate nudge is again nominal, and the higher rates are credibly worse (Table 12). The 8B mechanism cache (372 states, text) reproduces the late-layer rank inversion: supervised 13.6, flat 15.1, degrade 1.9, collapse 19.6.
E
7.0 9.2 10.1 1.2 13.9
entropy
Table 14: Full fixed-state probes at N = 1999 (4B, text). Degrade destroys the late-layer rank while collapse preserves or raises it; both drive the readout down, collapse completely and degrade in part. Entropy is not monotone across regimes, so entropy inflation alone is harmless under greedy decoding.
Table 12: Set-of-Marks track at 8B. The low-rate nudge is nominal and single-seed (p = 0.062), the same luckyseed pattern seen at 4B, and the higher rates are credibly worse.
D
L35 rank argmax agr degeneracy
F
The Positive Control in Detail
The positive control keeps the harness, reward, and recipe of the 4B text grid and changes only the tasks. We profiled the supervised policy on all 119 MiniWoB tasks and selected ten with a positive sampled-minus-greedy gap (Table 17). The same GRPO recipe, run over a small learning-rate sweep at two seeds, climbs on this set: five of six configurations rise and the strongest gains a credible
Mechanism in Detail
The interpretability probes read a fixed cache of hidden states. Movement magnitude does not predict the failure mode: collapse moves less in L2 than degrade yet is more destructive, and the single largest total movement in the grid is the KL rescue arm, 12
Restored group
degrade
collapse
none (as trained) random null (mean) random null (95th pct.) embedding attention MLP all groups
11.4 19.3 22.9 14.0 37.1 40.0 37.1
0.0 0.0 — 0.0 0.0 2.9 37.1
Table 15: Frontier success (%) after restoring a component group from initialization. For degrade, attention or MLP alone recovers above the random-null 95th percentile, and the direction holds across three degrade checkpoints (recovery 33–111% of the none-to-all gap). For collapse no single group recovers. Values are over 35 episodes and noisy in the point estimate. LR (8B, text)
L35 rank
argmax agr
SFT 3 × 10−6 5 × 10−6 8 × 10−6 1 × 10−5 2 × 10−5
13.63 15.14 14.39 1.80 1.90 19.63
— 1.00 0.73 0.78 0.95 0.00
draw-line count-sides click-menu use-autocomplete
∆SR
McNemar p
Supervised baseline lr 1 × 10−5 , seed 1 lr 3 × 10−6 , seed 1 lr 5 × 10−6 , seed 1 lr 1 × 10−5 , seed 2 lr 3 × 10−6 , seed 2 lr 5 × 10−6 , seed 2
20.0 14.0 32.0 34.0 38.0 38.0 42.0
— −6.0 +12.0 +14.0 +18.0 +18.0 +22.0 [+8, +40]
— — 0.070 0.016 0.012 0.004 0.007
Checkpoint
SR
L35 rank
argmax agr
Supervised start Climb (lr 5 × 10−6 , s2) Degrade (lr 1 × 10−5 , s1)
20.0 42.0 12.0
6.88 7.14 3.21
1.00 0.96 0.56
Table 19: Rank health on the positive-control family. The climbing checkpoint keeps its late-layer rank at the supervised level while success rises, and the degrading checkpoint’s rank collapses as success falls, so rank tracks capability in both directions at 4B. These are single checkpoints and the upward move is small.
tion states otherwise, and all intervals are Wilson 95% or task-clustered paired bootstrap intervals as noted.
greedy SR
sampled SR
∆head
G.1
Controlled null
0.00 0.00 0.20 0.20
0.58 0.33 0.50 0.46
+0.58 +0.33 +0.30 +0.26
G.2
Robustness
G.3
Mechanism
G.4
Generalization
H
Algorithms and Additional Definitions
Table 17: Example headroom tasks. The reward is reachable by sampling that greedy decoding misses, so ∆head = SRsample − SRgreedy > 0, the criterion the ten selected tasks satisfy and the mastered grid does not.
This section states the three procedures the main text refers to by name, and recaps two estimators used in the evaluation. Algorithm 1 is the update we sweep; the only knobs the grid varies are the learning rate ηt , the KL weight β, the clip pair (ϵlo , ϵhi ), the seed, and the initialization of θ. Algorithm 2 is the interval behind every “credible” verdict, and Algorithm 3 is the causal test behind the localization claim. Two estimators complete the evaluation. The headroom criterion that selects the positive-control tasks compares the greedy success to the mean success under temperature sampling of the supervised policy, P ∆head = K1 k 1[success(τk )]−1[success(τgreedy )], (16)
22 points (Table 18). Reading the late-layer rank on three checkpoints from this family shows rank health accompanying the gain and rank collapse accompanying the loss (Table 19), the constructive complement to the degrade lesion of the mastered grid.
G
SR
Table 18: Per-configuration positive control on the ten headroom tasks (50 matched episodes). Five of six configurations climb, four credibly, and the low rate that is a no-op on the mastered grid gains 12 to 18 points here.
Table 16: The 8B rank-to-behavior map across the learning-rate sweep. No rate drops rank and argmax agreement together: at 8 × 10−6 and 1 × 10−5 the rank collapses while agreement stays high, and at 2 × 10−5 agreement collapses while the rank is preserved, so the coupling seen at 4B does not transfer. Task
Configuration
Figure Gallery
This section collects the supporting figures that do not appear in the main text, grouped by the role they play in the argument. All success rates are clean greedy decoding on the 11-task text track (11 tasks by 5 seeds, 55 episodes per run) unless a cap13
Seed sweep: +5.1pp was training-seed noise
Paired deltas: worse* arms, none credibly better worse* (CI excludes 0, <0) no-diff
54 final clean SR (%)
ov2_fac_base_lr3e6 ov_fac_base_lr5e6 ov_fac_base_lr1e5 * ov2_fac_base_lr2e5 * ov_sw_lr3e6_s2 ov2_ab_clipsym_lr5e6 ov_sw_lr5e6_s2 ov_klR_kl05_lr5e6 ov2_ab_clipsym_lr1e5 * ov_sw_lr1e5_s2 * ov_sw_lr1e5_s3 * ov2_sw_lr1e5_s4 ov_klR_kl01_lr1e5 ov_klR_kl05_lr1e5 ov2_ab_kl010_lr1e5 ov_sw_lr2e5_s2 * ov2_sw_lr2e5_s3 * ov2_ab_kl005_lr2e5
52 50 48 46 mean 95% CI [47.7, 51.7] training-seed mean 49.7%
44 1 70
60
50 40 30 20 10 0 Paired vs SFT (pp), task-clustered 95% CI
2
3 4 training seed
SFT baseline 49.1% per-seed final SR
5
6
10
Figure 8: Training-seed replication of the one borderline arm (+5.1pp at 25 eval seeds). The exact configuration (SFT-init, constant lr 3 × 10−6 , G = 8, 15 rounds) was retrained across six training seeds and clean-evaluated at the final checkpoint: 49.1, 52.7, 45.5, 49.1, 50.9, 50.9%. The mean is 49.7% ± 2.5 (sd), 95% CI [47.7, 51.7], sitting on the SFT baseline of 49.1%, so the +5.1pp was training-seed noise.
Figure 6: Paired re-analysis of every grid arm against SFT on the 55 matched (task, seed) episodes, using a task-clustered bootstrap 95% interval and an exact McNemar test on the discordant pairs. No arm is credibly better than SFT. The nominal winners at lr 3 × 10−6 (+3.6pp) have intervals [+0.0, +10.9] and [+0.0, +9.1] with McNemar p = 0.688 and 0.500. The moving regimes are credibly worse: pooled lr 1 × 10−5 = −15.0pp [−20.9, −9.5] and pooled lr 2 × 10−5 = −49.1pp [−62.7, −36.4]. The KL-rescued arms have discordant counts b/c = 0/0, i.e. per-episode outcomes identical to SFT on all 55 episodes.
Recipe ablation: none beats SFT, large G breaks first
clean SR (%)
50
25 seeds: none credibly better than SFT (44.7%) ov_sw_lr3e6_s2 49.8%
40 30 20
ov2_fac_base_lr3e6 46.9%
10
ov2_ab_kl010_lr1e5 45.1%
0
G=8 G=16 G=32 SFT 49.1%
3e-6 4 × 10 6 6 × 10 6 8e-6 1.5e-5 2e-5 peak learning rate (warmup+cosine)
rec_g16_lr3e6_s1 44.1% BASELINE_base 44.7%
2.5
Figure 9: Recipe ablation adding a warmup (15%) plus cosine schedule and sweeping group size G ∈ {8, 16, 32}, paired against SFT (49.1%). No run credibly beats SFT. The schedule removes the collapse pathology at moderate learning rate (constant-lr 1 × 10−5 degraded and 2 × 10−5 collapsed in the grid; with the schedule, 8×10−6 and 1.5×10−5 at G = 8 become no-diff), but recovery is not learning. Larger G does not help and makes high-lr collapse worse.
0.0 2.5 5.0 7.5 10.0 12.5 paired clean SR vs SFT (pp), 95% CI
Figure 7: The null under a five-fold increase in evaluation seeds (25 seeds, 275 episodes per config). The 25-seed SFT baseline is 44.7% [39.0, 50.6] (123/275), slightly below the 5-seed 49.1%. Under the taskclustered paired bootstrap no config is credibly better: the largest nominal gap is +5.1pp with McNemar p = 0.001 but a bootstrap interval whose lower bound touches 0, so it is not credible and is a single training seed.
reports; a real graft is a specific cause only when its recovery exceeds that percentile.
averaged over evaluation seeds, and a task is climbable when ∆head > 0. The randomcomponent null for grafting draws a set R of nonattention parameters with |R| equal to the attention parameter count, restores R from initialization, and repeats the draw 24 times to form the null distribution whose mean and 95th percentile Table 15
14
TOST: equivalent to SFT except the lucky-seed arm
Late-layer rank collapse is degrade-specific
flat lr3e-6 (sft-init, lucky)
15.0 Effective rank (N=1999)
flat lr3e-6 (base-init) recipe w+cos G16 lr3e-6 KL0.10 lr1e-5 ±5pp equivalence band
5.0
2.5 0.0 2.5 5.0 7.5 10.0 12.5 15.0 90% CI of (config SFT) clean SR (pp)
Fraction (agreement / degeneracy)
clean SR (%)
40 30 SFT baseline 49.1 flat (lr3e-6) degrade (lr1e-5) collapse (lr2e-5)
0 1
3
5 7 training round
9
2.5
degrade L35 -> ~1
18 Transformer layer
30
35
Output corruption differs by regime; entropy not monotone
50
10
BASELINE_base (base) BASELINE_sft (SFT) ov_sw_lr3e6_s2 (flat) ov_sw_lr1e5_s2 (degrade) ov_sw_lr1e5_s3 (degrade) ov2_sw_lr2e5_s3 (collapse) ov_sw_lr2e5_s2 (collapse)
5.0
Figure 12: Fixed-state effective rank by network depth (layers 6, 12, 18, 24, 30, 35) on identical inputs, localizing the degrade lesion to the late layers. Baselines and flat runs hold a late-layer (L35) effective rank of about 6.4 to 6.7, whereas the no-KL degrade runs collapse L35 to about 1.0 to 1.2 while earlier layers (L6 to L18) stay healthy. Collapse runs preserve L35 rank, the opposite lesion. The dissociation survives at N = 1999 states, so it is not a small-N artifact.
Best-by-eval checkpoints do not credibly beat SFT
20
7.5
6
Figure 10: Two one-sided equivalence at the 25-seed grain: a config is declared equivalent to SFT within ±δ when its task-clustered paired 90% interval of (config minus SFT) lies inside [−δ, +δ]. Four of five top configs are equivalent within ±5pp: base-init flat [+0.0, +4.1]; KL0.10 [+0.0, +0.0] (episode-identical); recipe G16 [−2.7, +4.1]; base-init sanity [−2.3, +0.9]. The lone exception is the lucky-seed arm, [+0.0, +15.0], the same seed noise resolved by the six-seed replication (Fig. 8). This is a positive equivalence claim, not merely a failure to reject.
60
10.0
11
1.2
argmax agreement
degeneracy
next-token entropy
12
Next-token entropy (nats)
base-init sanity (no GRPO)
12.5
1.0
10
0.8
8
0.6
6
0.4
4
0.2
2
0.0
base BASELINE_base
SFT no-op/flat degrade collapse BASELINE_sft ov2_ab_clipsym_lr5e6ov_sw_lr1e5_s2 ov2_sw_lr2e5_s3
0
Figure 13: Fixed-state output behavior on identical decision states: baseline argmax agreement, degeneracy (unparseable or no-op fraction), and next-token entropy. Baselines and flat runs preserve the policy (agreement about 0.95 to 1.0, degeneracy near 0) even when entropy is inflated, so entropy inflation alone is harmless under greedy decoding. Degrade corrupts the argmax (agreement 0.07 to 0.5, degeneracy 0.5 to 0.93); collapse destroys it entirely (agreement 0, degeneracy 0.8 to 1.0) in two flavors, high-entropy garble and low-entropy mode collapse.
Figure 11: Best-by-eval checkpoint trajectory for the three regime exemplars, clean-evaluated at every saved round. Best-by-eval selection surfaces nominal nudges (the flat run drifts to a max of 52.7% by rounds 7 to 11; the degrade run reads 50.9% at round 1 before falling to 35–40%), but none survives the episode-matched paired test: flat max +3.6pp [+0.0, +9.1] (p = 0.500), degrade r1 +1.8pp [−9.1, +12.7] (p = 1.000), collapse −49.1pp. The null is not an artifact of scoring an already-broken final checkpoint.
15
Movement magnitude does not predict failure mode
80
degrade flat collapse
0.2 0.1 0.0
65 60 55 45
3e-6
1e-5 peak learning rate
2e-5
Failure taxonomy: invalid output, not reward hacking
80
60
60
50
40
40
20 0
SoM SFT baseline 63.6 SoM GRPO run
Figure 16: Set-of-Marks (vision) learning-rate sweep for the 4B model, paired against the SoM SFT baseline of 63.6% [50.4, 75.1] (35/55), higher than the text baseline and itself a text-versus-SoM asymmetry. No SoM config credibly beats the baseline: lr 3 × 10−6 is +3.6pp (p = 0.500), lr 1 × 10−5 is a nominal +7.3pp (p = 0.219), and lr 2 × 10−5 is −10.9pp (p = 0.070, interval touching 0), a nominal-but-not-credible degrade. The no-gain null generalizes to SoM; the credible lrcollapse established on text is only nominal here.
clean SR (%)
Episode share (%)
70
50
0.25 0.50 0.75 1.00 1.25 1.50 1.75 2.00 Weight movement mean_rel_l2 (x1e-3)
Figure 14: Weight-movement magnitude relative to initialization (mean relative L2 and sparse fraction) versus learning rate and regime, showing that magnitude does not predict the failure mode. Low-lr flat runs move least. Collapse moves less in L2 than degrade yet is far more destructive. The single largest total movement in the grid belongs to the KL-rescued run, which is behaviorally at the SFT baseline (49.1%). The outcome is not monotone in how far the weights move; direction and structure decide it.
100
SoM GRPO: no credible gain, nominal lr collapse
75
0.3
clean SR (%)
Clean SR (final)
0.4
base
SFT-base
flat
KL
degrade
30 20
8B text GRPO: null at low lr, collapse at 2e-5
8B SFT baseline 47.3 8B GRPO run
10
collapse
0 correct valid but wrong
truncated premature finish
invalid output
3e-6
Figure 15: Per-episode failure-mode taxonomy by regime (correct, valid-but-wrong target, prematurefinish, invalid or unparseable, truncated). Rewardhacking by premature finishing stays at roughly 0 to 2% across every regime, so the null is not explained by reward-gaming. The moving regimes fail through output degeneration: degrade is 63% invalid and collapse is 98% invalid. Flat and KL runs preserve the SFT structure (about 50% correct and 40% valid-but-wrong), the base capability ceiling rather than a new failure.
5e-6
8e-6 1e-5 peak learning rate
2e-5
Figure 17: Text-track learning-rate sweep for the 8B model, paired against the 8B-SFT baseline of 47.3% [34.7, 60.2] (26/55). No 8B config credibly beats the baseline. The low-lr arms give the familiar nominal nudge (lr 3 × 10−6 up to 52.7%, not credible), while the moving regimes break: lr 1 × 10−5 seed 2 is credibly worse (36.4%, p = 0.031) and both lr 2 × 10−5 runs collapse to 0.0% (p < 0.001). The null and the high-lr breakage carry over from 4B to 8B.
16
Algorithm 1 GRPO update for the web agent Require: policy πθ , frozen initialization πref , prompts, group size G, clip (ϵlo , ϵhi ), KL weight β, schedule ηt 1: for each round t and prompt x do 2: sample G rollouts {τi } from πθold (· | x) 3: ri ← 1[success(τi )] ▷ sparse terminal reward P 4: Ai ← ri − G1 j rj ▷ group mean-centered, no std 5: for each completion token c of each rollout do 6: ρc ← πθ (yc | x, y<c )/πθold (yc | x, y<c ) 7: gc ← min ρc A, clip(ρc , 1 − ϵlo , 1 + ϵhi ) A 8: end for P b KL (πθ ∥ πref ) 9: L ← − C1 c gc + β D 10: θ ← θ − ηt ∇θ L 11: end for
8B L35: rank preserved but agreement lost L35 effective rank (left) argmax agreement (right)
1.2 1.0
20
argmax agreement
L35 effective rank
25
0.8
15
0.6
10
0.4
5 0
0.2 SFT
flat
degrade regime
collapse
0.0
Figure 18: 8B mechanism interpretability on N = 372 fixed states. The late-layer (L35) effective rank matches the 4B pattern: SFT 13.6, flat 15.1, degrade collapsed to 1.9, collapse preserved at 19.6. The honest scale nuance is that the 8B degrade keeps argmax agreement high (0.954) with intact success despite the rank collapse, whereas the 4B degrade drops agreement to 0.07 to 0.5. So the rank leg of the dissociation reproduces at 8B while the rank-to-behavior coupling does not.
80
Algorithm 2 Task-clustered paired bootstrap for ∆SR Require: matched per-episode outcomes for an arm and for SFT over tasks T (|T | = 11) and seeds, resamples B 1: for b = 1 to B do 2: draw |T | task clusters from T with replacement 3: ∆b ← SRarm − SRSFT over the drawn clusters 4: end for 5: return the 2.5 and 97.5 percentiles of {∆b } Verdict: credibly better/worse iff the interval excludes 0; equivalent within δ iff the 90% interval lies inside [−δ, +δ]
8B SoM GRPO: no credible gain, lr collapse
clean SR (%)
70 60 50 40 30
8B-SoM SFT baseline 63.6 8B-SoM GRPO run
3e-6
1e-5 peak learning rate
2e-5
Figure 19: 8B Set-of-Marks learning-rate sweep, paired against the 8B-SoM-SFT baseline of 63.6% [50.4, 75.1] (35/55). No 8B-SoM config credibly beats the baseline: lr 3 × 10−6 is the same low-lr nominal nudge (+9.1pp, p = 0.062, single seed), while lr 1 × 10−5 (−18.2pp, p = 0.002) and lr 2 × 10−5 (−25.5pp, p = 0.001) are both credibly worse. The lowest learning rate gives a nominal positive nudge that never clears the paired interval, here as at 4B.
Algorithm 3 Causal component grafting Require: trained weights θ, initialization θinit , component group G (attention, MLP, embedding, or a random parameter set of matched size) 1: θ ′ ← θ ′ ← θ init ▷ restore one group, keep the rest 2: θG G 3: return SR(πθ′ ) ▷ recovery attributable to G
17