General Preference Reinforcement Learning

Unknown

Conceptio › Archive › arXiv CS

arXiv CSopen access

General Preference Reinforcement Learning

Unknown · 2026 · arxiv_cs

arXiv CS · Papers · License: Open Access · 2026

Open Source ↗Direct PDF ↓

neuralnetworks

machine learning, deep learning, neural networks

General Preference Reinforcement Learning

Muhammad Umer Stanford University [email protected]

arXiv:2605.18721v1 [cs.LG] 18 May 2026

Arslan Chaudhry Independent Researcher

Muhammad Ahmed Mohsin Stanford University [email protected] Andreas Haupt Stanford University [email protected]

Emily Fox Stanford University [email protected]

Ahsan Bilal The University of Oklahoma [email protected] Sanmi Koyejo Stanford University [email protected]

John M. Cioffi Stanford University [email protected]

Abstract Post-training has split large language model (LLM) alignment into two largely disconnected tracks. Online reinforcement learning (RL) with verifiable rewards drives emergent reasoning on math and code but depends on a programmatic verifier that cannot reach open-ended tasks, while preference optimization handles open-ended generation yet forgoes the continuous exploration that powers online RL. Closing this gap requires a verifier for open-ended quality, but a scalar reward model is the wrong shape for the job. Quality is multi-dimensional, and any scalar score is an incomplete proxy that lets online RL collapse onto whichever axis the score is most sensitive to. We turn instead to the General Preference Model (GPM), which embeds responses into k skew-symmetric subspaces and represents preference as a structured, intransitivity-aware comparison. Building on this, we propose General Preference Reinforcement Learning (GPRL), which carries the k-way structure through to the policy update. GPRL computes perdimension group-relative advantages, normalizes each on its own scale so no axis can dominate, and aggregates them with context-dependent eigenvalues. The same structure powers a closed-loop drift monitor that detects single-axis exploitation and corrects it on the fly by reweighting dimensions and tightening the trust region. Starting from Llama-3-8B-Instruct, GPRL reaches a length-controlled win rate of 56.51% on AlpacaEval 2.0 while also outperforming SimPO and SPPO on Arena-Hard, MT-Bench, and WildBench by resisting reward hacking across extended training runs.

Introduction

Post-training now decides what a large language model (LLM) can actually do, since it turns a pretrained base model into a system that follows instructions, reasons through hard problems, and aligns with human values [1]. The dominant recipe, Reinforcement Learning from Human Feedback (RLHF) [2], fits a scalar reward model (RM) to human preferences and optimizes a policy against it with Proximal Policy Optimization (PPO) [3]. Although this pipeline has unlocked capabilities such as instruction following, multi-turn dialogue, and tool use, it remains complex, trains unstably, and gives way to reward hacking once pushed at scale [4, 5].

Preprint.

enilnO

reward hacking In response, the field has split into two largely scalar RM disconnected tracks. The first avoids explicit reverifier GRPO GPRL ward modeling and optimizes the policy directly online RL needs a DAPO +RM reward signal on preference data. Offline methods such as DiGRPO PPO+RM rect Preference Optimization (DPO) [6] dominate this track alongside game-theoretic variants such online DPO GPO Nash-MD as Self-Play Preference Optimization (SPPO) [7], Simple Preference Optimization (SimPO) [8], and self-rewarding SPPO ReST RAFT LM Nash-MD [9]. These methods align well on openended tasks, yet they train on static or iteratively RewardIPO KTO DRO refreshed preference pairs and forgo the continuweighted ous online exploration that defines reinforcement DPO SimPO SFT RFT learning (RL) [10, 11]. The second track keeps an Reward-model-free Reward-model-based explicit reward signal and operates online, sampling fresh responses from the current policy at Figure 1: Landscape of LLM post-training every step. Group Relative Policy Optimization methods, organized by supervision source and (GRPO) [4] has emerged as the standard algorithm training regime. Online RL with a scalar RM here. It provides a critic-free variant of PPO that reaches open-ended tasks but suffers reward estimates advantages from a group of sampled re- hacking; GPRL fills the gap with a structured, sponses and admits any scoring function as the multi-dimensional reward. reward. When the reward is a programmatic verifier such as a math checker or a unit test, the resulting reinforcement learning with verifiable rewards (RLVR) framework powers modern LLMs such as DeepSeek-R1 [12] and elicits emergent behaviors, including self-reflection and dynamic strategy adaptation, that supervised fine-tuning (SFT) alone does not produce.

evitaretI

enifflO

A natural way to combine the two tracks is to run GRPO with a learned scalar RM in place of the verifier, and Figure 1 shows where this combination sits. While the idea looks attractive on paper, since open-ended tasks admit no programmatic verifier and a learned RM seems to fill the gap, in practice it breaks down under extended online RL because of reward hacking. The deeper issue is that any scalar score is an incomplete proxy for what humans want from a response, in the sense formalized by Zhuang et al. [13]. Quality has many latent dimensions including helpfulness, factual accuracy, safety, and style, and folding them into one number always discards information about how they trade off [14]. Optimizing the proxy then amplifies whichever dimension the RM is most sensitive to and silently degrades the rest [5, 12]. We trace the root cause to the lack of representational structure in scalar rewards, and find a principled alternative in the General Preference Model (GPM) [15], which embeds each response into R2k and computes preferences with a block-diagonal skew-symmetric operator on k independent twodimensional subspaces, each capturing a different facet of quality.1 This delivers a multi-dimensional preference signal, represents intransitive preferences (A ≻ B ≻ C ≻ A) that scalar models cannot express, and preserves the linear query complexity of the Bradley-Terry (BT) model [17]. Building on this, we propose General Preference Reinforcement Learning (GPRL), which treats GPM as a structured, multi-dimensional reward source for GRPO-style online RL on open-ended tasks. Rather than collapsing each response to a scalar advantage, GPRL computes per-dimension grouprelative advantages across GPM’s k subspaces, normalizes each on its own scale, and aggregates them with context-dependent eigenvalues. After rescaling, every subspace contributes on the same unit-variance scale, so no axis can inflate its share by simply growing in magnitude, and any policy that improves on one axis at the expense of others sees its aggregate advantage held in check by the rest. This vector-derived advantage replaces GRPO’s scalar advantage in an otherwise identical objective. The same structure also lets us correct reward hacking online through a drift monitor that tracks the variance profile across preference dimensions during training, closing the loop with a controller that reweights the dimensions and tightens the KL trust region whenever drift crosses a threshold. Should the policy concentrate probability mass on, say, the verbosity axis, the controller downweights verbosity in the next aggregate advantage, and the policy’s incentive shifts back toward neglected axes such as factuality and safety. 1 The best k is a property of the supervision corpus, not of GPM. With k = 3 on Skywork-Reward [16], the subspaces specialize on roughly distinguishable axes such as helpfulness against verbosity, factual accuracy against fluency, and safety against directness; we use k = 3 throughout because it is where this corpus saturates.

Contributions. (i) We propose GPRL, which extends GRPO-style online RL to open-ended tasks through a multi-dimensional, intransitivity-aware preference embedding and recovers GRPO at k=1. Unlike multi-objective RLHF, which returns a Pareto family from independent scalar rewards [18, 19], GPRL trains a single online policy whose axis structure comes directly from the GPM. (ii) We introduce per-dimension group-relative advantage estimation and identify a sufficient condition under which the aggregate advantage rejects single-axis exploitation that any scalar reward must prefer, formally separating GPRL from scalar GRPO. (iii) We design a closed-loop drift monitor that flags reward hacking from the shape of the advantage variance profile and corrects it online by reweighting dimensions and tightening the KL trust region, all without any external evaluation signal. (iv) Starting from Llama-3-8B-Instruct, GPRL reaches a 56.51% length-controlled win rate on AlpacaEval 2.0, beating GRPO with a BT reward model by 14.59 points and also outperforming DPO, SimPO, SPPO, GPO, and GRPO+BT on Arena-Hard, MT-Bench, and WildBench.

Preliminaries

2.1

Group Relative Policy Optimization

GRPO [4] is a critic-free variant of PPO [3] that estimates advantages from a group of sampled responses rather than a learned value function. Given a prompt x, GRPO samples G responses {y1 , . . . , yG } from πθold (· | x), scores each with a reward R, and normalizes within the group to form advantages Âi = R(x, yi ) − meanj R(x, yj ) / stdj R(x, yj ) + ϵ . With importance ratio ri = πθ (yi | x)/πθold (yi | x) and its clipped counterpart r̄i = clip(ri , 1−ϵ, 1+ϵ), GRPO minimizes " # G 1 X LGRPO (θ) = −Ex,{yi } min ri Âi , r̄i Âi − βKL πθ ∥πref , (1) G i=1 where β controls KL regularization toward a reference policy πref . In RLVR [12, 20], R is a binary verifier R(x, y) ∈ {0, 1} that checks correctness, which works well for math and code but does not extend to open-ended tasks; swapping in a learned scalar RM reintroduces reward hacking [5, 21]. 2.2

General Preference Model and General Preference Optimization

The BT model [17] sets P(yi ≻ yj | x) = σ(R(x, yi ) − R(x, yj )) with σ the logistic function and R(·) ∈ R a scalar reward. This formulation imposes a total ordering and cannot represent intransitive preferences such as A ≻ B ≻ C ≻ A that appear in human judgments and rule out any scalar utility [22, 23]. The classical fix models preferences with a skew-symmetric bilinear form on a convex set of options, generalizing von Neumann-Morgenstern utility and admitting intransitive cycles by construction [24, 25]. GPM [15] brings this idea into modern alignment by instantiating the bilinear form via learned response embeddings. Preference embeddings. GPM maps each response y to an embedding vy|x ∈ R2k with ∥vy|x ∥2 = 1, scoring pairs through a fixed block-diagonal skew-symmetric operator R≻ ∈ R2k×2k consisting of (1) (2k) k blocks Rl = 01 −10 . Writing the embedding components as (vi , . . . , vi ), the per-subspace score is (2l) (2l−1) (2l−1) (2l) sl (yi , yj | x) = vi vj − vi vj , (2) and the overall score aggregates across subspaces with context-dependent eigenvalues2 λl (x) ≥ 0, s(yi ≻ yj | x) =

k X

λl (x)sl (yi , yj | x),

P(yi ≻ yj | x) = σ s(yi ≻ yj | x) .

(3)

l=1

Since Eq. (2) is antisymmetric in (i, j), the aggregate score is too, and the resulting preference probabilities define a constant-sum two-player game [7, 9]. General Preference Optimization. Zhang et al. [15] pair GPM with an iterative SPPO-style scheme called General Preference Optimization (GPO). Given an opponent policy µ, the algorithm samples K 2 Real skew-symmetric matrices have purely imaginary eigenvalues in conjugate pairs ±iλ ; the λ (x) ≥ 0 in Eq. (3) l l are their moduli. They are produced for each prompt by a learned “eigenvalue scale gate” on the language model’s prompt encoding [15].

drift monitor adapt responses

each GPM

agg. advantage

GPM

policy

GPM KL

loss

reference

score matrix

Figure 2: Overview of GPRL. The policy πθ samples G responses per prompt, GPM embeds them, and R≻ produces k pairwise score matrices that yield per-dimension advantages. The aggregate drives the GRPO-style clipped surrogate, while a drift monitor D(t) adapts the dimensional weights and β to suppress reward hacking. PK 1 responses per prompt, estimates ŝ(yi ≻ µ | x) = K k=1 s(yi ≻ yk | x), and regresses log πθ /πθt onto these scores at each iteration t. The procedure remains offline and iterative, since one fixes πθt , collects a batch, trains to convergence, and only then refreshes the rollout policy, mirroring the regime that limits SPPO and DPO-style methods [26, 27]. GPRL keeps GPM as the reward source but replaces GPO’s iterative regression with an online policy-gradient update that samples freshly from πθ at every step, computes per-dimension group-relative advantages instead of collapsing to the scalar ŝ, and adds a mechanism to control drift.

General Preference Reinforcement Learning

GPRL replaces GRPO’s scalar reward with GPM’s multi-dimensional preference signal and carries the k-way structure through to the policy update. Averaging the k subspace scores into a single number would discard the very structure that makes GPM worth using, so GPRL keeps each subspace separate during advantage estimation, normalizes each on its own scale, and only then aggregates. 3.1

Per-dimension advantage estimation and GPRL objective

Given a prompt x, GPRL samples G responses {y1 , . . . , yG } from πθold and passes them through a frozen GPM, which returns embeddings vyi |x ∈ R2k and eigenvalues λl (x). We then build the advantage in three steps. First, we form the k pairwise score matrices using sl (yi , yj | x) from Eq. (2), as shown in Figure 2. Averaging each row against the rest of the group yields a per-dimension P (i) 1 population score ŝl (x) = G−1 j̸=i sl (yi , yj | x) that measures how much yi beats the rest of the group along dimension l. Second, since different subspaces operate at different scales, we normalize within each dimension rather than across all of them, (i)

(i)

Âl (x) =

ŝl (x) − µl (x) , σl (x) + ϵ

(i)

Âl (x) = 0 if σl (x) = 0,

(4)

(i)

where µl (x) and σl (x) are the mean and standard deviation of {ŝl (x)}G i=1 . A single global normalization would let whichever subspace carries the largest magnitudes drown out the rest, whereas per-dimension rescaling places every signal on a common unit-variance footing. Finally, we combine the rescaled advantages with GPM’s eigenvalues to form the aggregate advantage, Â(i) (x) =

k X

(i)

λl (x)Âl (x),

(5)

l=1

and plug this into a GRPO-style clipped surrogate to obtain the GPRL objective, " # G 1 X LGPRL (θ) = −Ex,{yi } min ri Â(i) , r̄i Â(i) − βKL πθ ∥πref , G i=1 4

(6)

1 (t)

Variance share ,l

(t)

Variance share ,l

0.6 0.4 0.2 0

0.8

2000

healthy hacked

0.8

0.6 over-optimized l?

0.4 0.2

4000 6000 Training step

8000

(a) Healthy run

corrected BT RM score

0.4

0.4 t0

0.2

0 0

2000

4000 6000 Training step

(b) Hacked run

8000

0.8 0.6

0.6

0.2 threshold =

0 0

(t)

(t) ,1

Drift D(t)

(t)

BT RM score

1 0.8

0 0

2000

4000 6000 Training step

8000

Figure 3: Dimensional drift distinguishes healthy training from reward hacking. (a) The variance profile α(t) holds its initial shape on a healthy GPRL run. (b) Under hacking, it collapses onto a single dimension l⋆ . (c) D(t) stays near zero on the healthy run and crosses τ at t′ on the hacked one, allowing the corrected trajectory to engage the controller at t′ and pull back as the profile rebalances, while a BT reward model climbs monotonically and offers no comparable signal. where ri and r̄i match those of Eq. (1). Since Eq. (6) is structured identically to the GRPO objective and only the computation of Â(i) differs, GPRL drops into existing large-scale RL infrastructure such as vLLM rollouts [28] and distributed training seamlessly, with the only added cost being a frozen GPM forward pass per group that is comparable to any learned RM call. PG (i) Proposition 1 (Zero-mean advantages). For every prompt x and every dimension l, i=1 Âl (x) = PG 0, hence i=1 Â(i) (x) = 0 for any weights λl (x) ∈ R. Proposition 1 implies that aggregating k subspace advantages does not break the variance-reduction property that makes group-relative methods well-behaved [4]. The aggregate retains a zero baseline within every group, the policy gradient stays unbiased, and we are free to mix dimensions through any weights λl (x) without rederiving anything.3 Proofs for this and what follows appear in Appendix C. 3.2

Drift control

The aggregate advantage of Eq. (5) acts as a lever the policy gradient pulls on, so understanding why a multi-dimensional reward resists reward hacking requires asking what it takes for the lever to flip the wrong way. Suppose a candidate policy π † improves a single axis l∗ relative to a reference policy π ∗ while regressing on the rest. The gradient direction depends on which policy carries the larger aggregate advantage, and Proposition 2 pins down when GPRL prefers the balanced policy. Proposition 2 (Multi-dimensional structure rejects single-axis hacking). Fix a prompt x and a group of responses, and let Âπl (x) denote the dimension-l advantage of a response drawn from π, as in † ∗ † ∗ Eq. (4). Let π ∗ and π † satisfy, for some axis l∗ , Âπl∗ (x) > Âπl∗ (x) and Âπl (x) < Âπl (x) for all ∗ l ̸= l . If X ∗ † † ∗ λl (x) Âπl (x) − Âπl (x) > λl∗ (x) Âπl∗ (x) − Âπl∗ (x) , (7) l̸=l∗ †

∗

then Âπ (x) < Âπ (x). †

∗

The verdict Âπ (x) < Âπ (x) is the consequential half. GPRL gives the smaller aggregate advantage to the single-axis exploiter, so the clipped surrogate in Eq. (6) pushes the policy gradient away from π † and toward π ∗ . Under any scalar reward R with R(x, π † ) > R(x, π ∗ ), by contrast, the GRPO advantage is monotone in R within a fixed group, so π † always wins and the gradient always rewards the exploit. The per-dimension normalization in Eq. (4) is what makes Eq. (7) likely to hold, since (i) rescaling every Âl to unit variance bounds how much any single dimension can grow its contribution to the aggregate. We use Eq. (7) as the design foundation of the controller below, ensuring that whenever the inequality fails, the controller intervenes on the very weights λl (x) that govern it. (i)

3 Fixing k = 1 and λ (x) ≡ 1 and defining R 1 GPM (x, yi ) := ŝ1 (x) collapses Eq. (5) to the GRPO advantage under RGPM and Eq. (6) to Eq. (1), so GPRL strictly generalizes GRPO under a preference-model-based scalar reward, and any improvement we observe with k > 1 is attributable to the multi-dimensional structure rather than a different optimizer.

Table 1: AlpacaEval 2.0. LC. WR, WR, and average response length against the gpt-4-turbo reference, using Llama-3-8B-Instruct as the base policy and gpt-4-turbo as judge. SPPO and GPO follow the published three-iteration protocol, while GRPO and GPRL are reported at the matched third epoch so that all reward-model-based methods are compared at similar compute. Group

Method

Size

RM Type

Iter./Ep.

LC. WR

Avg. Len

No reward model

DPO [6] SimPO [8]

– –

40.30 44.70

37.90 40.50

1837 1825

SPPO [7]

2B 2B 8B 8B

BT GPM BT GPM

3 3 3 3

40.01 36.06 42.55 39.45

42.12 45.61 40.92 41.64

2136 2498 1948 2385

GPO [15]

2B 2B 8B 8B

BT GPM BT GPM

3 3 3 3

42.21 37.74 40.37 38.98

44.20 48.25 38.56 41.54

2151 2582 1969 3249

GRPO [4]

2B 8B

BT BT

3 3

39.87 41.92

38.21 40.51

1925 1893

GPRL (ours)

2B 8B

GPM GPM

3 3

51.08 56.51

45.21 48.33

1699 1600

Reward model

Online RL with reward model

While Eq. (7) can be verified response by response, acting online requires a scalar that summarizes the inequality across a training step. We use the variance profile of the per-dimension advantages, since when a policy starts reward hacking l∗ , the between-response variance concentrates on l∗ and collapses on the rest, which is exactly the imbalance the controller has to push back on. Define (i) Varyi ∼πt ŝl (x) (t) (t) (0) αl = Pk , (8) (i) , l = 1, . . . , k, D(t) = KL α ∥ α l′ =1 Varyi ∼πt ŝl′ (x) with α(t) = 1/k in the degenerate all-zero case. In healthy training α(t) holds its initial shape, but under hacking it spikes on the exploited axis, causing D(t) to rise accordingly. We feed D(t) back into a closed-loop controller that retunes the aggregation weights and the KL coefficient, and Figure 3 sketches the qualitative drift trajectories it acts on. Controller. The controller acts through a per-dimension multiplier ml (t) ≥ 0, initialized to 1 and applied on top of the eigenvalues, so the effective weight becomes λ̃l (x, t) = ml (t) λl (x). While D(t) > τ for threshold τ , the controller tightens according to !γ # " (0) αl , β ← min κ · β, βmax , (9) ml ← N ml · (t) αl + ε where N [·] renormalizes the multipliers to mean 1, γ ∈ (0, 1] sets the redistribution strength, and κ > 1 tightens the trust region. The update targets dimensions that need adjusting. Over-grown axes receive ml < 1 and stagnated axes receive ml > 1, so the aggregate steers back toward balance while the increase in β slows further excursions in the meantime. Once D(t) ≤ τ , the controller relaxes via ml ← δml + (1 − δ) and β ← max(β0 , β · δ) with recovery rate δ ∈ (0, 1), allowing both quantities to decay toward their baselines as the profile rebalances.

Experiments

Setup. Every method starts from the same base policy, Llama-3-8B-Instruct, so any difference in scores reflects the optimization signal rather than pretraining. To separate the effect of supervision structure from supervision content, we train two reward models on Skywork-Reward [16], namely a scalar BT model and a GPM with embedding dimension 2k = 6, each at two scales (Gemma-2B-it for 2B and Llama-3.1-8B-Instruct for 8B). The policy then trains under GPRL with online rollouts on prompts drawn from UltraFeedback [29], using G = 8 generations per prompt, completion length 512, and temperature 1.0. We train for three epochs with AdamW at peak learning rate 6

Table 2: Arena-Hard v2 and MT-Bench. Win rate (WR) against the Arena-Hard gpt-4-0314 reference (AH2) and MT-Bench score, same setup as Table 1. Group

Method

Size

RM Type

Iter./Ep.

AH2

MT-Bench

No reward model

DPO [6] SimPO [8]

– –

0.9 1.0

7.90 8.03

SPPO [7]

8B 8B

BT GPM

3 3

0.9 0.9

8.02 8.12

GPO [15]

8B 8B

BT GPM

3 3

0.8 0.9

7.87 8.03

GRPO [4]

0.8

7.69

GPRL (ours)

GPM

1.3

8.33

Reward model

Online RL with reward model

Table 3: WildBench. Calibrated WB-Score and task-balanced WB-Reward against the three official baseline references (gpt-4-turbo, claude-3-haiku, llama-2-70b-chat), same setup as Table 1. Group

Method

Size

RM Type

Iter./Ep.

WB-Score

WB-Reward

No reward model

DPO [6] SimPO [8]

– –

35.90 37.05

8.16 9.57

SPPO [7]

8B 8B

BT GPM

3 3

36.14 35.21

8.33 7.18

GPO [15]

8B 8B

BT GPM

3 3

35.87 35.98

8.25 7.42

GRPO [4]

35.41

8.62

GPRL (ours)

GPM

37.98

11.15

Reward model

Online RL with reward model

1×10−6 , cosine schedule with 0.03 warmup, weight decay 0.1, gradient clipping at 1.0, KL coefficient β = 0.01, and one inner iteration per rollout group. This setting matches the three-iteration style of SPPO and GPO so that all reward-model-based methods sit on a similar compute footing, and it also reaches the regime where the drift controller actively engages, as Appendix D.4 characterizes further. Benchmarks. We evaluate on three open-ended instruction-following benchmarks. AlpacaEval 2.0 [30] reports length-controlled win rate (LC. WR) and WR against gpt-4-turbo4 on 805 general prompts, with the LC variant explicitly debiasing against verbosity. Arena-Hard v2 [32] draws 500 adversarial queries from real Chatbot Arena traffic and currently shows the highest correlation with human Arena rankings among public benchmarks. WildBench (WB) [33] scores 1024 in-the-wild user instructions on a calibrated 0 to 100 WB-Score and a task-balanced WB-Reward in [−100, 100]; we additionally report MT-Bench [34] on 80 multi-turn prompts. Baselines. Five baselines span both post-training tracks. DPO [6] reparameterizes RLHF as a closed-form classification loss on offline pairs, while SimPO [8] drops the reference model and uses a length-normalized log-likelihood margin. SPPO [7] iteratively regresses against pairwise win rates from a preference oracle, and GPO [15] runs the same iterative recipe on GPM scores. GRPO [4] paired with a scalar BT reward model is the same-paradigm online RL baseline that shares everything with GPRL except the multi-dimensional reward. Following published practice, SPPO and GPO are run for three iterations and we report iteration 3 results, since prior work establishes that this setting yields their strongest scores. GRPO and GPRL likewise report the third epoch against the matched 8B BT and 8B GPM reward models. Main results. Tables 1, 2, and 3 report the scores, and Appendix D.4 traces the full per-epoch trajectory. Three patterns repeat across all four benchmarks: (a) Static and iteratively-refreshed methods plateau early. DPO and SimPO sit at 40 to 45 LC. WR on AlpacaEval 2.0, while SPPO and 4 Our reported numbers come from an automated gpt-4-turbo judge rather than human raters, and the well-known length

and stylistic biases of such judges apply to every method in our tables [30, 31].

Writing

AlpacaEval 2.0 LC. WR

60 8B GPM 55

Humanities

+3.67 2B GPM

+3.95

STEM

Roleplay DPO SimPO SPPO GPO GRPO GPRL

GPO GRPO GPRL

Coding

Creative

DPO SimPO SPPO

Planning

Reasoning

w/ controller No controller

40 1

3 4 Training epoch

(a) Extended-training scaling

Extraction

Math Coding

(b) MT-Bench, per-category

Info/Advice

Math/Data

Figure 4: Scaling and per-category breakdown. (a) AlpacaEval 2.0 LC. WR across five training epochs at both reward-model scales, with the controller enabled holding near its peak through epoch 5 and the controller disabled degrading once drift develops. (b, c) Per-category scores on MT-Bench and WildBench, where GPRL leads on the categories that match the supervision and on structural categories while remaining within noise of the strongest baseline elsewhere. GPO at iteration 3 stay in the same band. They cluster around 0.8 to 0.9 on Arena-Hard v2,5 7.9 to 8.1 on MT-Bench, and 35 to 37 on WildBench. (b) Online RL with a scalar BT reward improves and then unwinds. GRPO+BT peaks earlier in training than the third epoch we report and then regresses to 41.92 LC. WR on AlpacaEval 2.0, 7.69 on MT-Bench, and 35.41 on WildBench, indicative of a scalar proxy collapsing onto whichever axis it tracks most sensitively. (c) Swapping the scalar reward for GPM under the same online paradigm produces the largest jump and continues to improve through epoch 3. GPRL using the 8B GPM reaches 56.51 LC. WR on AlpacaEval 2.0, beating GRPO+BT by 14.59 points and the strongest iterative baseline (SPPO with 8B BT at 42.55 LC. WR) by 13.96 points on the same metric, while the same ordering holds on the other three benchmarks. The length numbers in Table 1 also tell a consistent story. Since the gpt-4-turbo judge favors longer responses [30, 34, 35], iterative GPM-driven methods exploit this drift and inflate to 2400 to 3300 tokens by iteration 3 even as their LC win rates stall. GPRL, by contrast, holds length nearly flat at 1600 tokens, the shortest of any reward-model-based method in the table. This is what per-dimension normalization (Eq. (4)) is designed to produce, since after rescaling, no single subspace can dominate the aggregate by simply growing in magnitude, so any axis that correlates with length cannot pull the policy gradient on its own as long as the others retain comparable variance. The drift controller reinforces this by reweighting whichever subspace’s variance share grows past the initial profile, length-correlated or otherwise, and collectively delivers the strongest LC. WR in the table at the shortest average length, rather than the usual trade-off where one of the two has to give for the other to improve. 4.1

Analysis and ablations

Per-category gains and behavior under extended training. Figure 4 pairs the per-category breakdowns of MT-Bench and WildBench with the controller’s behavior across epochs. Less expected is that from supervision on Skywork-Reward and rollouts on UltraFeedback GPRL’s largest single-category gains land on the structural categories that lie outside both datasets, namely MTBench coding (+1.00 over the strongest iterative baseline) and WildBench math/data (+2.84 over GRPO+BT), with a comparable lead on extraction and a tie with SPPO on STEM. We attribute the pattern to the same mechanism that drives our main results, since online RL with group-relative advantages elicits emergent reasoning whenever the reward signal is rich enough to discriminate good from bad chains of thought [36], and GPM’s k subspaces appear to provide enough discriminatory signal on the structural axes to keep the policy from collapsing onto stylistic mimicry on these prompts. This is consistent with reports from RLVR-style training [12, 4] that online RL with a structured reward lifts the categories where the supervision discriminates well rather than uniformly improving every axis. Subfigure 4a traces the same process across five epochs at both reward-model scales, where the controller-on runs hold within roughly half a point of their peak through epoch 5 while the controller-off runs degrade once drift develops. 5 AH2 absolute scores are uniformly low because the base model is not strong on adversarial reasoning regardless of

post-training. The ordering is what we test, and Appendix D.7 reports stronger base policies.

Ablations. Appendix D provides the complete evaluation tables and analyses for each configuration; we summarize the primary findings here. (i) Number of subspaces k. LC. WR climbs from 44.21 at k = 1, where GPM collapses to a single skew-symmetric block and GPRL recovers GRPO under the resulting BT-style scalar reward, to 56.51 at k = 3. It then plateaus and slightly regresses by k = 6, with the jump from k = 1 to k = 3 being the largest single contributor to GPRL’s gains, though the location of the plateau likely reflects Skywork-Reward’s facet structure rather than a universal optimum. (ii) Per-dimension vs. global normalization. Replacing Eq. (4) with a single global normalization that pools (µ, σ) across subspaces drops LC. WR by roughly 4 points and re-introduces length drift to 2104 tokens, recovering the verbose pattern of the iterative GPM methods. (iii) Drift controller. The controller has little effect at the first epoch, where the variance profile sits close to its initial shape and on/off scores stay within 0.59 LC. WR points, but it becomes load-bearing under extended training as the profile starts to lean on a single axis. By the third epoch it is worth 3.67 LC. WR points on the 8B GPM and 3.95 on the 2B. Appendix D.4 shows the gap widening to 8.65 and 10.91 points by epoch 5 as controller-off runs follow the same downward trajectory we observe for GRPO+BT. (iv) Drift threshold τ . Setting τ very low underperforms our default and even the disabled controller, because the controller starts redistributing weight before the policy has had a chance to legitimately concentrate variance on whichever subspace the prompt distribution rewards, and aggressive early correction drags the policy back toward the initialization profile and loses real signal alongside the spurious one. (v) Group size G. Larger G improves the variance estimates the drift monitor depends on, with gains saturating at G = 8, our default. Limitations. GPRL inherits the assumptions of the GPM it consumes, so if the underlying preference model is poorly calibrated on a quality axis or simply does not represent it as a subspace, GPRL cannot recover the missing signal, and the per-dimension normalization will faithfully amplify whatever biases the GPM already encodes. Our experiments use an 8B GPM trained on Skywork-Reward with rollouts on UltraFeedback prompts, so the per-category gains on math and coding are best read as a transfer effect rather than a substitute for verifier-based RLVR. The drift controller is a feedback loop rather than a contraction, and we do not give convergence guarantees for the closed-loop dynamics in Eq. (9) and the relaxation rule that follows it, so pathological choices of (γ, κ, τ ) can produce oscillatory behavior in D(t). We also evaluate on a single base policy, a single reward-model corpus, and a single rollout-prompt corpus, so the dimensional plateau at k = 3 should be read as a property of Skywork-Reward’s facet structure rather than of GPM in general.

Related Work

Direct and game-theoretic preference optimization. The standard RLHF pipeline [2] fits a scalar BT reward model and optimizes with PPO, but the proxy score reliably climbs while true quality degrades once training is pushed at scale [5, 13]. DPO [6] sidesteps the explicit reward stage by reparameterizing the RLHF objective as a closed-form classification loss, sparking a family of offline and iterative variants such as IPO [37], KTO [38], SimPO [8], SPPO [7], and Nash-MD [9]. The gametheoretic methods in this group step beyond BT by working directly on pairwise win probabilities and casting alignment as a constant-sum two-player game, but they all rely on static or iterativelyrefreshed batches and cannot keep exploring once the iterate is fixed, so the improvement ceiling tracks the quality of the collected preference data rather than the compute spent on optimization [26]. Online RL with verifiable rewards. GRPO [4] introduced critic-free group-relative advantage estimation, replacing PPO’s learned value function with within-group normalization. Pairing it with binary verifiers yields the RLVR paradigm that powers DeepSeek-R1 [12] and subsequent refinements such as DAPO [39] that further stabilize entropy and trust-region dynamics over long training horizons. These methods elicit emergent reasoning behaviors including self-reflection and dynamic strategy adaptation [36], but they require a programmatic verifier and so do not extend to open-ended tasks, while substituting a scalar learned reward model reintroduces hacking, which is precisely the gap GPRL addresses with a structured multi-dimensional reward. Multi-dimensional preferences. A separate line of work, including Fine-Grained RLHF [40], Rewarded Soups [41], and MODPO [42], handles multi-faceted preferences through several independently learned scalar rewards combined by scalarization or weight interpolation, returning a Pareto family along pre-specified axes with no per-step signal for monitoring single-axis exploitation. GPM [15] instead models preferences through a skew-symmetric bilinear form on k subspaces, admitting intransitive cycles by construction [24, 25, 22], and its companion optimizer GPO is iterative 9

and SPPO-style. GPRL keeps GPM as the reward source and brings its full k-dimensional output into a GRPO-style online update, with per-dimension normalization and a drift controller that build on the structural properties separating GPM from scalar BT.

Conclusion

We argued that the gap between online RL and open-ended alignment is a question of reward shape rather than reward strength, since a scalar reward model is an incomplete proxy for multi-dimensional human quality and online RL against it will reliably collapse onto whichever axis the proxy is most sensitive to. GPRL leverages GPM’s structured, k-subspace preference signal and carries that structure through to the policy update, which both discourages single-axis exploitation by construction and exposes a closed-loop signal for our proposed controller to correct it when it threatens to occur. The same idea, that supervision structure is a first-class design variable rather than a fixed property of the loss function, may prove useful well beyond preference optimization, in any setting where a learned proxy stands in for an unmeasurable target.

LLM Usage ChatGPT 5.4 by OpenAI was used for editing, proofreading, and resolving LATEX formatting issues in this document. Codex by OpenAI was used to assist with code writing in the implementation of gprl.

References [1] Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21321, 2025. [2] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022. [3] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. [4] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. [5] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, pages 10835–10866. PMLR, 2023. [6] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. [7] Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, and Quanquan Gu. Self-play preference optimization for language model alignment. arXiv preprint arXiv:2405.00675, 2024. [8] Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. Advances in Neural Information Processing Systems, 37:124198–124235, 2024. [9] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, et al. Nash learning from human feedback. In Forty-first International Conference on Machine Learning, 2024. 10

[10] Junkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, and Xiangnan He. Alphadpo: Adaptive reward margin for direct preference optimization. arXiv preprint arXiv:2410.10148, 2024. [11] Qi Gou and Cam-Tu Nguyen. Mixed preference optimization: Reinforcement learning with data selection and better reference model. arXiv preprint arXiv:2403.19443, 2024. [12] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. [13] Simon Zhuang and Dylan Hadfield-Menell. Consequences of misaligned ai. Advances in Neural Information Processing Systems, 33:15763–15773, 2020. [14] Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for llms. Advances in Neural Information Processing Systems, 37:75522–75558, 2024. [15] Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, and Quanquan Gu. Beyond bradley-terry models: a general preference model for language model alignment. In Proceedings of the 42nd International Conference on Machine Learning, ICML’25. JMLR.org, 2025. [16] Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410.18451, 2024. [17] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952. [18] Nuoya Xiong and Aarti Singh. Projection optimization: A general framework for multi-objective and multi-group rlhf. arXiv preprint arXiv:2502.15145, 2025. [19] Qiang He and Setareh Maghsudi. Pareto multi-objective alignment for language models. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 257–272. Springer, 2025. [20] Mengyu Zhang, Siyu Ding, Weichong Yin, Yu Sun, and Hua Wu. Extending rlvr to open-ended tasks via verifiable multiple-choice reformulation. arXiv preprint arXiv:2511.02463, 2025. [21] Jiayi Fu, Xuandong Zhao, Chengyuan Yao, Heng Wang, Qi Han, and Yanghua Xiao. Reward shaping to mitigate reward hacking in rlhf. arXiv preprint arXiv:2502.18770, 2025. [22] Peter C Fishburn. Nontransitive measurable utility. Journal of Mathematical Psychology, 26(1): 31–67, 1982. [23] Peter C Fishburn. Nontransitive preferences in decision theory. Journal of risk and uncertainty, 4(2):113–134, 1991. [24] Peter C Fishburn. An axiomatic characterization of skew-symmetric bilinear functionals, with applications to utility theory. Economics Letters, 8(4):311–313, 1981. [25] Yutaka Nakamura. Skew-symmetric additive representations of preferences. Journal of Mathematical Economics, 30(3):367–387, 1998. [26] Yuda Song, Gokul Swamy, Aarti Singh, J Bagnell, and Wen Sun. The importance of online data: Understanding preference fine-tuning via coverage. Advances in Neural Information Processing Systems, 37:12243–12270, 2024. [27] En Wang, Xingyu Lin, Chenfu Du Su, Zhonghou Lv Bao, Funing Yang, Yuanbo Xu, and Wenbin Liu. Indirect online preference optimization via reinforcement learning. In Proceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence, pages 538–546, 2025. 11

[28] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626, 2023. [29] Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377, 2023. [30] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. [31] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. The Innovation, 2024. [32] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939, 2024. [33] Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking llms with challenging tasks from real users in the wild. arXiv preprint arXiv:2406.04770, 2024. [34] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623, 2023. [35] Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. Judging the judges: A systematic study of position bias in llm-as-a-judge. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics, pages 292–314, 2025. [36] Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. Emergent hierarchical reasoning in llms through reinforcement learning. arXiv preprint arXiv:2509.03646, 2025. [37] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455. PMLR, 2024. [38] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306, 2024. [39] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source llm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476, 2025. [40] Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36: 59008–59033, 2023. [41] Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36:71095–71134, 2023. [42] Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10586–10613, 2024. 12

[43] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, 2023. [44] Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460– 9471, 2022. [45] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023. [46] Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. Odin: Disentangled reward mitigates hacking in rlhf. arXiv preprint arXiv:2402.07319, 2024. [47] Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4998–5017, 2024. [48] Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, et al. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548, 2023. [49] Thomas Kwa, Drake Thomas, and Adrià Garriga-Alonso. Catastrophic goodhart: regularizing rlhf with kl divergence does not mitigate heavy-tailed reward misspecification. Advances in Neural Information Processing Systems, 37:14608–14633, 2024. [50] Chaoqi Wang, Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, et al. Beyond reward hacking: Causal rewards for large language model alignment. arXiv preprint arXiv:2501.09620, 2025. [51] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. [52] Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118, 2024. [53] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388, 2025.

Appendix A

More on reward hacking

This appendix locates GPRL within the existing reward-hacking literature, focusing on the specific failure modes that motivated the per-dimension structure and drift controller of Section 3. Reward hacking refers to the phenomenon in which a policy increases a learned proxy reward R̂ while the underlying quantity R⋆ that R̂ was meant to estimate stays flat or degrades [44, 21]. Gao et al. [5] characterized this empirically in RLHF as reward over-optimization, showing that as the policy spends KL budget against a learned RM, the gold reward traces a hill-shaped curve that initially climbs and then falls, with the peak depending on RM size, KL coefficient, and amount of preference data. The same qualitative shape, namely a peak followed by sustained degradation, is what we observe in Appendix D.4 for GRPO+BT and for GPRL with the controller disabled, which is the empirical signature GPRL is designed to interrupt. Three contributing factors explain the hill-shaped curve in scalar RLHF. Goodhart’s law [5] states that any optimization pressure against an imperfect proxy will eventually find the gap between proxy and target, and the larger the optimization budget the wider the gap that gets exploited. Spurious correlations in the supervision corpus give the RM surface features that correlate with quality on the training distribution but generalize poorly. Response length is the canonical example, since longer responses are systematically preferred in many preference datasets and judges, so the RM learns that long loosely implies good and the policy then exploits this by inflating length without improving content [45–47]. Other documented spurious axes include sycophancy [48], stylistic mimicry, and position or formatting biases in LLM-as-judge evaluation that propagate into RM training [34, 30]. Heavy-tailed misspecification compounds these effects, since a scalar RM has no degree of freedom to express that several axes should improve simultaneously, so once the policy finds the dominant axis the gradient stays pointed at it for the rest of training [49]. None of these failure modes is detectable from the proxy alone, since the proxy by construction agrees with the policy that things are improving, which is precisely the asymmetry GPRL’s variance profile α(t) is built to expose. Existing mitigations split into three families, each addressing one part of the problem. Reward model improvements such as ODIN [46], length-debiased preference data construction [47], and causal regularization [50] reduce the proxy gap on a spurious axis the designer anticipates, but they do not change the shape of the proxy. Optimization-side defenses such as KL regularization [3], reward clipping, and early stopping cap the available gain rather than separating clean from dirty progress, since they cannot distinguish a policy that earned its reward from one that hacked it. Multi-objective RLHF approaches such as Fine-Grained RLHF [40], Rewarded Soups [41], and MODPO [42] expose multi-dimensional structure but optimize a scalar combination per policy, so the same hacking dynamics apply within each combination, and the returned Pareto family does not give a per-step signal for monitoring single-axis exploitation during a single run. GPRL combines elements of all three families while addressing the underlying shape problem directly. The reward is structurally vector-valued through the k subspaces of GPM, the per-dimension normalization in Eq. (4) prevents any one axis from inflating its share of the aggregate by simply growing in magnitude, and Proposition 2 formalizes when the multi-dimensional structure tips the gradient against a single-axis exploiter. The drift controller of Section 3.2 then plays the optimizationside role by adjusting both the dimension weights and the KL coefficient β, but it does so based on the variance profile α(t) , which is the per-step signal that scalar pipelines cannot construct. This does not eliminate reward hacking in the broadest sense, since the GPM itself can be miscalibrated on axes outside its supervision (we acknowledge this in Section 4 under Limitations), but it changes what the policy can hack from a scalar to a configuration of axes whose imbalance is observable. Figure 3 and Tables 6 and 7 measure this effect quantitatively, with the controller acting on the dimensional fingerprint before the policy settles into any single-axis exploit.

Proofs

We collect here the full proofs of the propositions stated in Section 3, together with the calculation that establishes GRPO as the k = 1 special case of GPRL. C.1

Zero-mean property of advantages

Proof of Proposition 1. Fix a prompt x and a dimension l ∈ {1, . . . , k}, and consider two cases. (i)

When σl (x) = 0. By the convention fixed below Eq. (4), Âl (x) = 0 for all i ∈ {1, . . . , G}, so PG (i) i=1 Âl (x) = 0 trivially. When σl (x) > 0. By definition, G

µl (x) =

1 X (i) ŝ (x), G i=1 l

G X (i) ŝl (x) = Gµl (x).

i=1

Applying Eq. (4) term by term and factoring the common denominator, G X

(i) Âl (x) =

G (i) X ŝ (x) − µl (x) l

σl (x) + ϵ

i=1

1 = σl (x) + ϵ

G G X X (i) ŝl (x) − µl (x) i=1

! (10)

i=1

1 (Gµl (x) − Gµl (x)) = 0. σl (x) + ϵ PG (i) Combining both cases, i=1 Âl (x) = 0 for every l. For the aggregate advantage, linearity of summation in Eq. (5) gives =

G X

(i)

Â (x) =

i=1

G X k X

(i) λl (x)Âl (x) =

i=1 l=1

k X

λl (x)

l=1

G X i=1

(i) Âl (x) =

k X

λl (x) · 0 = 0,

l=1

where we used that λl (x) does not depend on i. The conclusion holds for any choice of weights λl (x) ∈ R, since the inner sum vanishes regardless of sign, which establishes the claim and the variance-reduction property that GPRL inherits from GRPO. C.2

GRPO as a special case of GPRL

The main text states without a formal proposition that fixing k = 1 and λ1 (x) ≡ 1 recovers GRPO under a preference-model-based scalar reward, which we verify here. Setting k = 1 and λ1 (x) ≡ 1, we define the induced scalar reward 1 X (i) RGPM (x, yi ) := ŝ1 (x) = s1 (yi , yj | x), G−1 j̸=i

and show in turn that (i) the per-dimension normalization reduces to the group-relative normalization of GRPO, (ii) the aggregate advantage equals this normalized quantity, and (iii) the GPRL objective reduces to the GRPO objective. (i) Normalization. With k = 1, Eq. (4) specializes to (i)

(i)

Â1 (x) =

ŝ1 (x) − µ1 (x) RGPM (x, yi ) − µ1 (x) = , σ1 (x) + ϵ σ1 (x) + ϵ

where, by the definitions of µl (x) and σl (x) in Section 3.1, µ1 (x) =

G G 1 X (j) 1 X ŝ1 (x) = RGPM (x, yj ) = meanj RGPM (x, yj ), G j=1 G j=1

(j) σ1 (x) = stdj ŝ1 (x) = stdj RGPM (x, yj ) . 16

(i)

Substituting these into the expression for Â1 (x) gives (i)

Â1 (x) =

RGPM (x, yi ) − meanj RGPM (x, yj ) , stdj RGPM (x, yj ) + ϵ

which is precisely the GRPO advantage from Eq. (1) computed under the reward RGPM . (ii) Aggregation. With λ1 (x) ≡ 1, Eq. (5) reduces to Â(i) (x) =

1 X

(i)

λl (x)Âl (x) = 1 · Â1 (x) = Â1 (x),

l=1

so the aggregate advantage equals the GRPO group-relative advantage of step (i). (iii) Objective. Comparing Eq. (6) with Eq. P (1), the two loss expressions share the same clipped-ratio 1 structure, the same uniform average G i , and the same KL-regularization term, with their only (i) formal difference being the symbol Â versus Âi in the clipped surrogate. By step (ii), these two quantities coincide in the present limit, so substitution yields LGPRL (θ) = LGRPO (θ) under the reward RGPM , which proves the claim. C.3

Resistance to single-axis reward hacking †

∗

Proof of Proposition 2. We express the gap Âπ (x) − Âπ (x) in a form that isolates the contribution of the hacked axis l∗ , then apply the hypothesized inequality. Applying Eq. (5) to each policy and using linearity in the per-dimension advantages, π†

π∗

Â (x) − Â (x) =

k X

† λl (x)Âπl (x) −

l=1

k X

∗ λl (x)Âπl (x) =

l=1

k X

† ∗ λl (x) Âπl (x) − Âπl (x) ,

l=1

and splitting the sum into the hacked axis and its complement while factoring signs gives X † ∗ † ∗ † ∗ Âπ (x) − Âπ (x) = λl∗ (x) Âπl∗ (x) − Âπl∗ (x) + λl (x) Âπl (x) − Âπl (x) l̸=l∗ †

∗

= λl∗ (x) Âπl∗ (x) − Âπl∗ (x) −

∗ † λl (x) Âπl (x) − Âπl (x) .

(∗)

l̸=l∗

P † ∗ ∗ † Letting H := λl∗ (x) Âπl∗ (x) − Âπl∗ (x) and D := l̸=l∗ λl (x) Âπl (x) − Âπl (x) denote the weighted gain on the hacked axis and the weighted degradation on the other axes, (∗) reads †

∗

Âπ (x) − Âπ (x) = H − D. †

∗

The hypothesis Âπl∗ (x) > Âπl∗ (x) together with λl∗ (x) ≥ 0 gives H ≥ 0, while the hypothesis † ∗ Âπl (x) < Âπl (x) for every l ̸= l∗ makes each summand in D non-negative, so D ≥ 0. Since the † ∗ proposition’s condition (7) is precisely D > H, combining the expression Âπ (x)− Âπ (x) = H −D with D > H yields † ∗ Âπ (x) − Âπ (x) = H − D < 0, †

∗

so Âπ (x) < Âπ (x), as claimed. Contrast with scalar rewards. The main text contrasts this conclusion with the scalar-reward case, and we sketch the argument here for completeness. Under any scalar reward R(x, y) and the GRPO normalization of Eq. (1), the advantage Âi = (R(x, yi ) − meanj R(x, yj ))/(stdj R(x, yj ) + ϵ) is an affine, strictly increasing function of R(x, yi ) for the fixed group {yj } whenever the group has † ∗ non-zero spread, so if R(x, π † ) > R(x, π ∗ ) within the same group, then Âπ > Âπ regardless of how the two policies behave on any latent quality axes. Proposition 2 exhibits the opposite verdict for GPRL whenever k ≥ 2 and Eq. (7) holds, which is the formal basis for the drift monitor of Section 3.2.

Additional ablations

This appendix expands on the ablations summarized in Section 4.1. Unless stated otherwise, all runs use the 8B GPM, G = 8, k = 3, the drift controller with our default τ = 0.2, and the three-epoch training budget that the main results use. We also report 2B GPM numbers throughout, since the drift behavior we want to characterize is more visible at the smaller reward-model scale where the variance profile is noisier and single-axis exploitation is correspondingly easier to start. D.1

Number of GPM subspaces k

Table 4 sweeps k from 1 to 6 at both reward-model scales, and the k = 1 row deserves a clarification before we discuss the trend. With a single skew-symmetric block the GPM score (2) (1) (1) (2) s1 (yi , yj | x) = vi vj − vi vj defines a pairwise comparison along one axis, and the perdimension normalization of Eq. (4) reduces to GRPO’s group-relative normalization under the (i) induced scalar reward RGPM (x, yi ) = ŝ1 (x), which is the special case formalized in Appendix C.2 with λ1 (x) ≡ 1. This row is GRPO under a BT-style scalar reward derived from the GPM block rather than from a separately trained scalar BT model. Although the two are not numerically identical they land in the same neighborhood (44.21 for k = 1 versus 43.18 for GRPO+BT at the 8B scale), which is a consistency check we want, and the rest of the ablation isolates the effect of k rather than of the reward-model architecture. Adding a single subspace (k = 1 → 2) buys 7 to 8 LC. WR points at either scale, while adding a second (k = 2 → 3) buys another 4 to 5 and is the largest individual jump in the table. After this the gain disappears and the run mildly regresses by k = 6, which we read as a property of the supervision rather than of GPRL itself. Since Skywork-Reward aggregates preferences along a small set of facets, this puts a soft ceiling on how many independently informative subspaces a GPM trained on it can carve out, and subspaces beyond that ceiling pick up noise that the per-dimension normalization then amplifies, which makes scaling k jointly with the reward-model supervision corpus the natural next experiment. Table 4: Ablation on the number of GPM subspaces k. AlpacaEval 2.0 LC. WR, raw WR, and average response length, with all other hyperparameters fixed at their main-results values and the drift controller enabled. At k = 1 the GPM collapses to a single skew-symmetric block, and GPRL with λ1 (x) ≡ 1 recovers GRPO under a BT-style scalar reward RGPM . Subspaces k k = 1 (GRPO under RGPM ) k=2 k=3 k=4 k=6

D.2

LC. WR

2B GPM WR Avg. Len

LC. WR

8B GPM WR Avg. Len

39.92 47.30 51.08 50.62 49.81

38.74 42.96 45.21 44.83 44.27

44.21 51.86 56.51 56.18 55.74

41.07 45.92 48.33 47.94 47.41

1812 1742 1699 1714 1758

1738 1671 1600 1612 1648

Per-dimension vs. global normalization

Table 5 replaces the per-dimension normalization of Eq. (4) with a single global normalization that pools (µ, σ) across all k subspaces. LC. WR drops by roughly 4 points at both reward-model scales while average response length jumps by 400 to 500 tokens, recovering the verbose pattern of the iterative GPM-based methods. The mechanism is exactly the one Section 3.1 predicts, since whichever subspace happens to carry the largest raw magnitudes drives both µ and σ under global normalization, so the rescaled advantages of the other subspaces collapse toward zero and their contribution to the aggregate vanishes. If that dominant subspace correlates with length, as it tends to under judges that favor longer responses, the policy gradient inherits the correlation and response length climbs. D.3

Drift controller

The controller’s effect is least visible in any single epoch and most visible across the training trajectory, and Table 6 reports two complementary checkpoints that probe the regime where the controller actually does work. At the first epoch on the 8B GPM the controller is essentially decorative, 18

Table 5: Ablation on the normalization scheme. Replacing the per-dimension normalization of Eq. (4) with a single global normalization that shares (µ, σ) across all k subspaces. AlpacaEval 2.0 with k = 3 and the drift controller enabled, at both reward-model scales. Normalization Global (shared µ, σ) Per-dimension (ours)

LC. WR

2B GPM WR Avg. Len

LC. WR

8B GPM WR Avg. Len

47.13 51.08

44.86 45.21

52.34 56.51

47.86 48.33

2168 1699

2104 1600

since LC. WR moves by less than a point and terminal drift D(tend ) stays near 0.02 in either case, well under the default threshold τ = 0.2, which is the right outcome for healthy training and a useful sanity check that the controller does not interfere when it has nothing to push back on. By the third epoch the picture has reversed on both reward models, since the controller is worth 3.67 LC. WR points and 0.92 WB-Score on the 8B GPM and 3.95 and 1.61 on the 2B GPM, with the controller-off run developing measurable drift (D(tend ) = 0.18 on 2B, 0.11 on 8B) while the on-run holds D(tend ) near 0.04 throughout. The 2B GPM enters this regime earlier than the 8B because its variance estimates are likely noisier and single-axis exploitation is correspondingly easier to start, while the 8B GPM gets there through accumulated drift across three epochs, and Appendix D.4 extends the comparison to epoch 5 and shows that the gap continues to widen. Table 6: Drift controller ablation. Disabling the controller corresponds to τ = ∞ and ml (t) ≡ 1. The first-epoch row on the 8B GPM serves as the no-drift sanity check, while the third-epoch rows capture the regime our main results use, with the 2B GPM included to show that drift develops earlier under noisier supervision.

D.4

Setting

Controller

LC. WR

WB-Score

Avg. Len

D(tend )

8B GPM, epoch 1 8B GPM, epoch 1

off on

49.74 49.83

36.71 36.84

1718 1721

0.02 0.02

2B GPM, epoch 3 2B GPM, epoch 3

off on

47.13 51.08

35.45 37.06

1864 1699

0.18 0.05

8B GPM, epoch 3 8B GPM, epoch 3

off on

52.84 56.51

37.06 37.98

1742 1600

0.11 0.03

Extended training

The drift controller is a feedback loop rather than a one-shot adjustment, so the cleanest way to see what it buys us is to run the same configuration with and without it across an extended horizon. Table 7 reports five epochs of training on both reward-model scales while Figure 4a (in Section 4.1) plots the LC. WR trajectories. Three regimes emerge across the table. Through the first epoch the controller is essentially invisible, since the variance profile sits close to α(0) on either reward model, D(t) never crosses τ , and the on and off curves are within a point of each other, which matches the design intent of Section 3.2 that the controller should remain dormant when the training signal is healthy. From epochs 2 to 3 the curves split, with the noisier 2B GPM losing ground from epoch 2 onward as drift develops earlier, while the 8B GPM holds slightly past epoch 2 before turning over at epoch 3. In both cases the controller-on run continues to improve because whenever the variance profile starts to lean on a single subspace, the controller redistributes weight back toward the rest before the policy gradient can lock in. Past epoch 3 the controller-off curves enter sustained degradation, with the 2B run dropping 9.35 LC. WR points between epochs 2 and 5 and the 8B run dropping 8.10 over the same span as D(tend ) climbs from 0.04 to 0.27 on the 8B model. The controller-on curves plateau and oscillate within roughly half a point of their peak, with the 8B GPM peaking at 56.84 at epoch 4 and settling at 56.47 by epoch 5, and the 2B GPM peaking at 51.08 at epoch 3 and settling at 50.18 by epoch 5. This oscillation is expected rather than being a sign of instability, since the closed-loop dynamics of ml (t) produce damped fluctuations (an empirical observation) rather than monotone convergence. 19

Table 7: LC. WR on AlpacaEval 2.0 across five epochs. GPRL with the drift controller enabled and disabled, both reward-model scales, all other hyperparameters fixed at their main-results values.

D.5

Controller

Epoch 1

Epoch 2

Epoch 3

Epoch 4

Epoch 5

2B GPM 2B GPM

off on

45.39 45.62

48.62 48.93

47.13 51.08

43.85 50.81

39.27 50.18

8B GPM 8B GPM

off on

49.74 49.83

55.92 53.96

52.84 56.51

51.46 56.84

47.82 56.47

Drift threshold τ

Table 8 sweeps the drift threshold τ at the 8B GPM with three epochs of training, the regime where the controller actually does work (Appendix D.3), and the shape of the table answers the natural question of why we do not just set τ as small as possible. At τ = 0.05 the controller engages almost from the first step and continuously redistributes weight away from any subspace whose variance share exceeds the initial profile by even a small margin. Since legitimate optimization concentrates variance on whichever subspace the prompt distribution actually rewards (helpfulness on a summarization prompt, faithfulness on a factual one, and so on), this aggressive correction drags the variance profile back toward the initialization α(0) before any genuine signal can accumulate, which amounts to running with the eigenvalues frozen and produces a result 4.07 LC. WR points below our default at an average length close to a barely post-trained model. As τ grows, the controller gives the policy room to concentrate variance for the right reasons before stepping in, and performance peaks at our default τ = 0.2, beyond which the controller engages too late and the run drifts back toward the no-controller numbers. The default τ = 0.2 therefore balances two distinct failure modes, namely premature collapse of the variance profile at small τ and unchecked single-axis exploitation at large τ , rather than trading off two equally good behaviors. Table 8: Ablation on the drift threshold τ . AlpacaEval 2.0 LC. WR, average response length, and terminal drift D(tend ) with the 8B GPM, three-epoch training, and all other hyperparameters fixed.

D.6

Threshold τ

LC. WR

Avg. Len

D(tend )

τ = 0.05 (very early) τ = 0.10 τ = 0.20 (default) τ = 0.40 τ = ∞ (disabled)

52.44 55.21 56.51 55.62 52.84

1538 1571 1600 1684 1742

0.02 0.03 0.03 0.08 0.11

Group size G

Table 9 sweeps the rollout group size G. Since the drift monitor depends on the per-dimension (i) variance estimates Varyi ∼πt [ŝl (x)] computed from the G responses in the group, the noise of the variance estimate scales as 1/(G − 1), so at G = 2 those estimates are too noisy for the controller to act on reliably and D(t) runs several times its G = 8 value at both reward-model scales. The LC. WR cost is steep at 7 to 9 points below our default, while doubling to G = 4 recovers most of the drop but still leaves 2 to 3 points on the table, and gains saturate at G = 8 with G = 16 producing only a marginal further improvement at twice the rollout cost. We therefore use G = 8 throughout the main results. D.7

Stronger base policies

In the main text, all results start from Llama-3-8B-Instruct for direct comparison with the SPPO and GPO settings of Wu et al. [7] and Zhang et al. [15], but a natural follow-up question is whether GPRL’s gains hold when the base policy itself is stronger. We report preliminary results on three additional base policies that span the open 8 to 9B class, namely Llama-3.1-8B-Instruct [51], Gemma-2-9B-it [52], and Qwen3-8B [53], with everything else held at the same setting (8B GPM, G = 8, k = 3, and so on). To put the numbers in context, the published AlpacaEval 2.0 LC. WR for the bases themselves sits at roughly 22.6 for Llama-3-8B-Instruct, 24.7 20

Table 9: Ablation on group size G. AlpacaEval 2.0 LC. WR and terminal drift D(tend ) at both reward-model scales, with the controller enabled. Group size G

2B GPM LC. WR D(tend )

8B GPM LC. WR D(tend )

G=2 G=4 G=8 G = 16

41.94 48.07 51.08 51.36

47.83 53.62 56.51 56.74

0.16 0.08 0.05 0.04

0.09 0.05 0.03 0.03

for Llama-3.1-8B-Instruct, 51.1 for Gemma-2-9B-it, and 52.4 for Qwen3-8B, so each successive base shrinks the headroom that any post-training method can claim. Table 10 reports GPRL applied to each base model. As a useful reference point, SimPO on Gemma-2-9B-it with on-policy ArmoRM-annotated UltraFeedback data reaches 72.4 LC. WR at 1833 tokens [8], which we take as the strongest publicly reported preference-optimization number on this base, and GPRL improves on it by roughly 1.9 points at a comparable length. These runs use the same Skywork-Reward-trained 8B GPM as the main results, so the dimensional plateau at k = 3 inherits all the corpus-specific caveats from Section 4 and Appendix D.1, and a properly matched GPM trained on a corpus richer in math and reasoning supervision would likely shift the per-category breakdowns in Figures 4b and 4c further toward the structural axes where Qwen3-8B already excels. Table 10: GPRL across base policies. AlpacaEval 2.0 LC. WR, WR, and average response length, with the 8B GPM, G = 8, k = 3, drift controller enabled, and three epochs of training. The ∆ column reports the LC. WR gain over the base policy. Base policy

LC. WR

Avg. Len

∆

Llama-3-8B-Instruct [51] Llama-3.1-8B-Instruct [51] Gemma-2-9B-it [52] Qwen3-8B [53]

56.51 58.27 74.31 63.79

48.33 50.04 67.42 60.18

1600 1612 1791 2186

+33.91 +33.55 +23.21 +11.39

Qwen3-8B starts from a verbose base of 3100 tokens on AlpacaEval 2.0 and GPRL pulls this down to 2186 while still improving the LC. WR, which is the same pattern we predicted in Section 4 where per-dimension normalization prevents the length-correlated subspace from dominating the aggregate. Gemma-2-9B-it lands at 1791 tokens, which is slightly under the 1833 that the SimPO [8] checkpoint produces, and the controller engages on roughly the same schedule as on Llama-3-8B-Instruct despite the much stronger base. We read this as evidence that the variance profile α(t) is a property of the GPM and the rollouts rather than of the policy’s initial capability.

Implementation details

This appendix collects the practical settings used to produce the results presented in Section 4 and the corresponding supplementary tables. We aim to ensure that the reader can reliably reproduce our findings or extend our methodology without needing to infer experimental specifics from the broader narrative. Where a hyperparameter is shared with GRPO [4] or the SPPO and GPO baselines [7, 15], we deliberately retain the previously published values to facilitate a rigorous and equitable comparison, and we flag the cases where we deviate. Hardware. All training experiments were executed on a dedicated cluster designated as REDACTED_FOR_SUBMISSION, which utilizes NVIDIA DGX H100 nodes equipped with eight H100 80GB GPUs each. A single computation node provides sufficient processing capacity to derive the primary results at the 8B parameter scale. Unlike the training experiments, all evaluation procedures are computationally lightweight and operate efficiently on a single H100 80GB GPU, since only forward passes through the policy are required for response generation. Reproducibility. To provide a clear overview of the experimental framework, Table 11 contains key hyperparameters. Code will be released at the URL given in the abstract. Table 11: Implementation details for GPRL. Hyperparameter values used to produce the main results in Section 4 unless an ablation overrides them. Group

Setting

Value

Policy and dataset

Base policy Rollout prompts GPM reward source GPM embedding dimension 2k Eigenvalue source Prompt head

Llama-3-8B-Instruct UltraFeedback 8B GPM trained on Skywork-Reward 6 (k = 3 subspaces) uniform (λl (x) ≡ 1) disabled

Optimization

Optimizer Peak learning rate LR scheduler Warmup ratio Weight decay Gradient clipping Initial KL coefficient β Per-device batch size Gradient accumulation steps Effective batch size Training epochs Inner iterations per group Mixed precision

AdamW 1 × 10−6 cosine 0.03 0.1 1.0 0.01 1 8 64 3 1 bfloat16

Rollouts

Group size G Max completion length Sampling temperature Backend

8 512 1.0 vLLM

Drift controller

Threshold τ Redistribution exponent γ KL multiplier κ KL ceiling βmax Relaxation rate δ

0.20 0.5 1.5 0.20 0.99

Hardware

Training Evaluation

1 DGX H100 node (8× H100 80GB) 1× H100 80GB

NeurIPS Paper Checklist 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and Section 3 introduce GPRL as a multi-dimensional, intransitivity-aware extension of GRPO, and Section 4 reports the AlpacaEval 2.0, ArenaHard v2, MT-Bench, and WildBench results that back the quantitative claims. The contributions listed at the end of Section 1 map to Sections 3.1, 3.2, and 4. Guidelines: • The answer [N/A] means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A [No] or [N/A] answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 2. Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: A dedicated Limitations paragraph at the end of Section 4.1 discusses inheritance of GPM biases. Appendix B additionally clarifies what kinds of reward hacking GPRL does and does not address. Guidelines: • The answer [N/A] means that the paper has no limitation while the answer [No] means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate “Limitations” section in their paper. • The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren’t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 3. Theory assumptions and proofs 23

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: Propositions 1 and 2 state their assumptions explicitly in Section 3, and full proofs appear in Appendix C. The k = 1 special-case calculation that establishes GPRL as a strict generalization of GRPO is given in Appendix C.2. Guidelines: • The answer [N/A] means that the paper does not include theoretical results. • All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems. • The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Section 4 specifies the base policy, reward-model corpus, rollout corpus, optimizer, learning-rate schedule, group size, completion length, KL coefficient, and training horizon. Appendix E consolidates every hyperparameter into Table 11. The codebase is included as a zipped attachment in the Supplementary Material. Guidelines: • The answer [N/A] means that the paper does not include experiments. • If the paper includes experiments, a [No] answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). 24

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 5. Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: GPRL codebase is included as a zipped attachment in the Supplementary Material accompanying this submission. Guidelines: • The answer [N/A] means that paper does not include experiments requiring code. • Please see the NeurIPS code and data submission guidelines (https://neurips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so [No] is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). • The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //neurips.cc/public/guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 6. Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer) necessary to understand the results? Answer: [Yes] Justification: Section 4 describes the data corpora, evaluation benchmarks, and shared training protocol, while Appendix E (Table 11) lists the optimizer, schedule, KL coefficient, group size, drift-controller parameters, and other settings used to produce the reported numbers. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 7. Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] 25

Justification: Following common practice for AlpacaEval 2.0, Arena-Hard v2, MT-Bench, and WildBench reporting [30, 32–34], we report single-run point estimates rather than error bars across seeds, since each judge-based evaluation is itself costly and the gaps over baselines are several percentage points wide on the core metrics. The benchmark scoring procedures (LC. WR debiasing, calibrated WB-Score) provide their own internal robustness mechanisms that the cited benchmark papers analyze in detail. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The authors should answer [Yes] if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g., negative error rates). • If error bars are reported in tables or plots, the authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 8. Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Appendix E reports that all training experiments run on a single NVIDIA DGX H100 node with eight H100 80GB GPUs and that evaluation runs on a single H100 80GB GPU. The training horizon (three epochs, G = 8 rollouts per prompt, completion length 512, effective batch size 64) given in Table 11 together with the standard UltraFeedback prompt set is sufficient to estimate wall-clock cost. Guidelines: • The answer [N/A] means that the paper does not include experiments. • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn’t make it into the paper). 9. Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: The work uses only publicly released models and datasets under their respective licenses, does not involve human subjects or new data collection, and the research direction is alignment of language models, which is consistent with the responsible-research expectations laid out in the NeurIPS Code of Ethics. 26

Guidelines: • The answer [N/A] means that the authors have not reviewed the NeurIPS Code of Ethics. • If the authors answer [No], they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [N/A] Justification: GPRL is a post-training method that improves an existing class of methods (online RL with reward models) along the well-studied axis of reward-hacking robustness, and it does not introduce new capabilities, modalities, or release artifacts beyond what RLHF-aligned models already provide. The improvements GPRL targets, namely better preservation of factuality, safety, and balanced quality across axes, are themselves a contribution to safer alignment rather than an enabler of new misuse pathways. Guidelines: • The answer [N/A] means that there is no societal impact of the work performed. • If the authors answer [N/A] or [No], they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate Deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 11. Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We plan to release the GPRL-trained Llama-3-8B-Instruct checkpoint after publication, under the same Llama Community License of the base model. This model inherits the safety fine-tuning of Llama-3-8B-Instruct and is post-trained on the public UltraFeedback prompt set with a reward model derived from Skywork-Reward, neither of which adds capabilities outside the scope of standard RLHF-aligned 8B-class chat models, and the per-dimension structure of GPRL is itself designed to preserve safety-correlated subspaces during training. Guidelines: 27

• The answer [N/A] means that the paper poses no such risks. • Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All datasets, models, and benchmarks used such as Llama-3-8B-Instruct, Llama-3.1-8B-Instruct, Gemma-2-9B-it, and Qwen3-8B models, as well as Skywork-Reward, UltraFeedback, AlpacaEval 2.0, Arena-Hard v2, MT-Bench, WildBench dataset and benchmarks, are cited to their original publications in Sections 4 and D.7, and we use them within the terms of their respective public licenses. Guidelines: • The answer [N/A] means that the paper does not use existing assets. • The authors should cite the original paper that produced the code package or dataset. • The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset. • For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset’s creators. 13. New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: At submission time, the new asset attached to this work is the GPRL codebase, included as an anonymized zipped archive in the Supplementary Material with a top-level README that documents installation, dataset preparation, training commands. A GPRL-posttrained Llama-3-8B-Instruct checkpoint is not attached to the submission to preserve anonymity, and will be released after publication. Guidelines: • The answer [N/A] means that the paper does not release new assets. • Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. 28

• At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 14. Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [N/A] Justification: This work does not involve crowdsourcing or research with human subjects. All evaluation is performed automatically by LLM-based judges (e.g., gpt-4-turbo for AlpacaEval 2.0 and the standard judges shipped with Arena-Hard v2, MT-Bench, and WildBench), and the preference data used to train the GPM comes from the publicly released Skywork-Reward corpus. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [N/A] Justification: No human subjects were involved in the study, so IRB review or an equivalent process did not apply. All preference data and prompts used come from previously released public corpora. Guidelines: • The answer [N/A] means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 16. Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigor, or originality of the research, declaration is not required. Answer: [Yes] Justification: The LLM Usage statement preceding the bibliography discloses that ChatGPT 5.4 was used for editing, proofreading, and LATEX formatting assistance, and that Codex was used to assist with code writing for the gprl implementation. Guidelines: 29

• The answer [N/A] means that the core method development in this research does not involve LLMs as any important, original, or non-standard components. • Please refer to our LLM policy in the NeurIPS handbook for what should or should not be described.

Record · ID 200476 · SHA-256 961d9318d05d8b5e
Retrieved via Conceptio — every document is proof-bundled with source, license, and retrieval metadata.