Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking During Online Adaptation in Reasoning Models
Subramanyam Sahoo 1 Aman Chadha 2 Vinija Jain 3 Divya Chaudhary 4
1. Introduction
arXiv:2606.30627v1 [cs.LG] 29 Jun 2026
Abstract Conservative offline training is widely advocated as a safe foundation for subsequent online adaptation: if a policy stays close to well-supported behaviour, the argument goes, it is less likely to exploit imperfections in a learned reward model. We challenge this intuition empirically and mechanistically. We train a Qwen3-14B policy under Direct Preference Optimisation (DPO) with three levels of conservatism (β ∈ {βlo , βmid , βhi } derived from empirical log-ratio percentiles), then adapt each checkpoint online against a learned reward ensemble (3 × Qwen3-1.7B) while measuring true performance on GSM8K exact-answer accuracy. We find that higher offline conservatism monotonically increases reward-hacking damage, measured by the Goodhart gap and its area under the curve (AUGC), with Spearman ρ = 1.0 across all three conditions. Mechanistic analysis reveals a three-link causal chain: (i) high-β DPO compresses policy entropy, (ii) Low-entropy policies generate responses with reduced diversity, concentrating in a narrow region of the reward model’s training distribution (lower pairwise cosine distance), and (iii) despite this proximity, ensemble disagreement (epistemic uncertainty) increases with β and is exploited faster during online optimisation. We further fit a power-law curve to the (β, AUGC) data and identify a practical optimal conservatism level β ⋆ that balances alignment fidelity against hacking vulnerability. Our results suggest that the field needs calibrated, not maximal, conservatism.
The standard recipe for safe language model alignment runs as follows: first perform offline training on humanpreference data (e.g., via RLHF (Christiano et al., 2023) or DPO (Rafailov et al., 2024)) using a conservatism coefficient β that penalises deviation from a reference policy, then optionally adapt online using a learned proxy reward. The implicit contract is that a more conservative offline checkpoint enters the online phase closer to the distribution where the reward model was trained, and therefore exploits it less aggressively.
The Paradox We demonstrate empirically and mechanistically that the opposite can happen: higher offline conservatism (β) leads to greater reward-hacking damage during online adaptation. The mechanism is not arbitrary; it arises from a principled chain of events rooted in entropy compression and out-of-distribution extrapolation. To make this concrete, consider what a high-β DPO objective actually does to the policy. It tightens the KL constraint against πref , concentrating probability mass on a narrow slice of token sequences that πref already assigns high density. The resulting policy has low output entropy and low response diversity. When this compressed policy is then optimised against a learned reward ensemble, two effects combine. First, the reward model was trained on a relatively diverse set of human-preference pairs; the compressed policy generates responses that lie in sparse regions of that training distribution, causing high epistemic uncertainty (ensemble disagreement). Second, a low-entropy starting point has fewer exploration directions through which gradient updates can improve true performance, so the optimiser rapidly channels all gradient signal toward reward-model blind spots. The net result is that the Goodhart gap—the divergence between proxy and true reward—opens faster and wider for high-β policies.
1
Horizon Research 2 Apple 3 Meta 4 Northeastern University. Correspondence to: Subramanyam Sahoo <[email protected]>. Proceedings of the 43 rd International Conference on Machine Learning, Decision-Making from Offline Datasets to Online Adaptation: Black-Box Optimization to Reinforcement Learning (DEMO 2026), Seoul, South Korea. PMLR 306, 2026. Copyright 2026 by the author(s). Code: Conservative-Offline-Training-Amplifies-RewardHacking-During-Online-Adaptation
This paper makes four contributions. 1. Empirical demonstration of the paradox using real 1
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking
models (Qwen3-14B policy, Qwen3-1.7B reward ensemble) and real data (UltraFeedback, GSM8K).
inputs and has been used as a pessimism signal in offline RL (Kumar et al., 2020; Kidambi et al., 2021). Our ensemble (3 × Qwen3-1.7B trained with bootstrap resampling) provides both the proxy reward signal and the uncertainty signal used in our mechanistic analysis.
2. Mechanistic attribution via response-entropy collapse measurements and reward-model OOD-distance analysis.
2.4. Offline RL and Conservative Methods
3. A power-law fit of AUGC as a function of β, yielding a practical design principle: an optimal β ⋆ exists below which online safety degrades faster than offline alignment improves.
The connection between DPO and offline RL is well known. CQL (Kumar et al., 2020) penalises Q-values for out-ofsupport actions; IQL (Kostrikov et al., 2021) avoids OOD actions by replacing Bellman backups with implicit quantile regression. The Decision Transformer (Chen et al., 2021) reframes RL as sequence modelling, which is directly analogous to DPO. All of these methods share the same conservatism logic encoded in Equation (1): larger conservatism coefficient → policy stays closer to reference → (assumed) safer online performance. We show this assumption is violated under a specific failure mode: OOD-driven reward hacking.
4. Algorithmic and benchmark recommendations for the next generation of conservative-alignment methods that explicitly trade off pessimism against hacking vulnerability.
2. Background and Related Work 2.1. Direct Preference Optimisation DPO (Rafailov et al., 2024) sidesteps the need for an explicit reward model by reparameterising the RLHF objective directly in terms of policy log-ratios. Given a preference dataset D = {(x, yw , yl )} of prompts with winning and losing responses, DPO minimises
3. Problem Formulation
Let πθ denote the policy being trained, πref the frozen DPO (0) checkpoint (which itself used πref as its reference), and r̂ϕ (x, y) a learned proxy reward parameterised by an ensemh ⋆ πθ (yw | x) ble ϕ = {ϕk }K k=1 . The true reward r (x, y) is verifiable LDPO (πθ ; πref ) = −E(x,yw ,yl )∼D log σ β log πref (yw | x) (GSM8K exact-answer accuracy) and is observed only duri ing evaluation, never during training. πθ (yl | x) − β log . (1) Definition 3.1 (Goodhart Gap). At online step t, define the πref (yl | x) batch-averaged normalised rewards where σ is the logistic function and β > 0 is the conserr̄proxy (t) r̄true (t) vatism coefficient. Larger β imposes a tighter implicit KL r̃proxy (t) = , r̃true (t) = , |r̄ (0)| + ε |r̄ (0)| + ε proxy true constraint KL(πθ ∥πref ), pulling the learned policy closer to the reference. where ε = ϵfloat32 . The Goodhart gap is 2.2. Goodhart’s Law in RLHF
G(t; β) = r̃proxy (t) − r̃true (t).
Goodhart’s Law (Goodhart, 1975) states that when a measure becomes a target, it ceases to be a good measure. In the RLHF context, Gao et al. (2022) formalise this as a monotone degradation of true performance as the policy drifts toward maximising a proxy reward. Skalse et al. (2025) give a taxonomy of reward hacking and show it is near-inevitable when the reward model has any imperfection. Our work studies a less-explored facet: how the offline training strategy modulates the severity and speed of hacking during online adaptation.
(2)
Definition 3.2 (AUGC). The area under the Goodhart gap curve (AUGC) measures cumulative hacking damage over the online run: Z T AUGC(β) = max G(t; β), 0 dt. (3) 0
The central question of this paper is: does AUGC(β) increase or decrease with β? Conventional wisdom
2.3. Reward Model Ensembles and Uncertainty
Offline RL theory predicts AUGC(β) should decrease with β: a more conservative policy stays closer to πref , which lies within the reward model’s training support, so it should hack less.
Ensembling is the standard approach for epistemic uncertainty estimation in deep neural networks (Lakshminarayanan et al., 2017). In the reward-model context, ensemble disagreement provides a proxy for out-of-distribution 2
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking
Empirical finding
4.3. Offline DPO Training
We observe the opposite: AUGC(β) increases monotonically with β (Spearman ρ = 1.0, p < 0.05).
For each β ∈ {βlo , βmid , βhi }, we fine-tune an independent LoRA adapter using Equation (1) via the TRL DPOTrainer (Von Werra et al., 2022). The per-device batch size, gradient accumulation, learning rate, and gradient clipping are all derived from hardware properties and the training set size (see Section C for derivation formulas).
4. Experimental Setup 4.1. Models and Data
4.4. Online Adaptation Loop
Policy model. We use Qwen/Qwen3-14B as the policy, loaded in 4-bit NF4 QLoRA quantisation (Dettmers et al., 2023) with LoRA adapters (Hu et al., 2021) applied to all attention projection matrices. The LoRA rank r is derived architecturally: r = 2⌊log2
√
hhidden ⌉
,
α = 2r,
After offline DPO, each checkpoint is adapted online using the objective h i Lonline = −Eτ ∼πθ Â(x, y) · log πθ (y | x) h 2 i + κ(β) · E log πθ (y | x) − log πref (y | x) . (8)
(4)
where ⌈·⌉ denotes rounding to nearest integer and hhidden is the model hidden dimension. √ The dropout rate is datasetderived: pdrop = clip(32/ n, 0.01, 0.10) for n training examples.
where the normalised advantage is Â(x, y) =
r̂(x, y) − µr̂ , σr̂ + ε
(9)
and the adaptive KL coefficient is
Reward ensemble. Three independent Qwen/Qwen3-1.7B sequence classifiers (also QLoRA) are trained with bootstrap resampling to produce both a mean reward and an epistemic uncertainty (ensemble standard deviation). Each member is trained with the Bradley–Terry preference loss (Bradley & Terry, 1952): LBT = − E(x,yw ,yl )∼D log σ rϕ (x, yw ) − rϕ (x, yl ) . (5)
κ(β) =
β . QpKL |log πθ − log πref | + ε
(10)
Here QpKL denotes the empirical pKL -th percentile of the absolute KL values over the current batch. This normalisation makes the KL penalty scale-invariant with respect to the policy’s divergence level.
Preference data. Offline DPO and reward-model training use HuggingFaceH4/ultrafeedback binarized (Cui et al., 2024), split 80/10/10.
Online loop summary At each step: (1) sample prompts from GSM8K test set; (2) generate responses via batched sampling; (3) score with reward ensemble; (4) compute advantageweighted policy gradient; (5) add adaptive KL penalty; (6) update LoRA adapter weights. Every eval freq steps, also evaluate GSM8K exact-answer accuracy.
Verifiable task. Online true reward is evaluated on openai/gsm8k (main) (Cobbe et al., 2021) using exactanswer matching after extracting the number following the #### delimiter. 4.2. Deriving the β Grid
5. Mechanistic Analysis
We do not choose β values arbitrarily. Instead, we compute the per-example absolute log-ratio magnitude under the (0) frozen reference policy πref : (0)
(0)
(i)
(i) δi = log πref (yw | x(i) ) − log πref (yl
| x(i) ) .
To explain the paradox, we identify three causal links illustrated in Figure 2: entropy compression, OOD distance amplification, and uncertainty-driven exploitation.
(6) 5.1. Link 1: Entropy Compression
The three β values are taken at the 20th, 50th, and 80th percentiles of {δi }, normalised by the median absolute logratio: pctpj ({δi }) βj = , (p1 , p2 , p3 ) = (20, 50, 80). median({δi }) + ε (7) This makes the three conservatism levels commensurate with the actual preference signal magnitude, not with arbitrary numerical choices.
We measure the mean token-level entropy of each p DPO checkpoint on a fixed probe set of nprobe = ⌈ |Dtest |⌉ GSM8K prompts: X 1 1 X X (β) (β) H (β) = − πθ (v | x<t ) log πθ (v | x<t ). |Xprobe | |x| t x∈Xprobe
v∈V
(11) Theorem 5.1 formalises the expected relationship. 3
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking ↑β Preference data Dpref (UltraFeedback)
tighter KL → lower entropy Offline Phase
Online Phase DPO Checkpoint (β) πθ (3 adapters)
DPO Training LDPO (β) β ∈ {βlo , βmid , βhi }
Online Adaptation Lonline (Eq. 5) adaptive KL
GSM8K True reward r⋆
Proxy reward r̂ Gap Goodhart
Reward Ensemble {r̂ϕk }3k=1 BT loss + bootstrap
G(t) = r̃Rproxy − r̃true AUGC = max(G, 0) dt Paradox: ↑ β ⇒ ↑ AUGC
Figure 1. Full experimental pipeline. The offline phase trains three DPO checkpoints with different conservatism levels β and a learned reward ensemble. The online phase adapts each checkpoint against the proxy reward while measuring true GSM8K accuracy. The Goodhart gap and AUGC quantify hacking damage. The paradox: higher β (more conservative) produces larger AUGC.
High β DPO
↓ diversity
Low policy compressed supportLow response diversityensemble disagrees High epistemic entropy H(πθ ) near RM training dist. uncertainty û (Sec. 5.1) H =−
P
(Sec. 5.2)
(Sec. 5.3)
exploitable gap
ρdiv = mean pairwise dcos
v pv log pv
û = stdk {rϕk }
Fast reward hacking large AUGC Figure 2. Causal chain explaining the paradox. High-β DPO compresses the policy into a low-entropy manifold. Low-entropy policies generate responses that are out-of-distribution for the reward model (measured by cosine distance in hidden-state space). OOD responses produce high ensemble disagreement. High uncertainty is the exploitable gap that enables fast reward hacking.
Proposition 5.1 (Entropy–conservatism monotonicity). Under mild regularity conditions on the DPO loss landscape, the equilibrium policy entropy H (β) is non-increasing in β:
distance between the hidden state of a generated response and the centroid of the UltraFeedback training distribution: d(β) cos = 1 −
β1 ≤ β2 ⇒ H (β1 ) ≥ H (β2 ) .
hϕ (x, y (β) )⊤ µref , ∥hϕ (x, y (β) )∥2 ∥µref ∥2
(12)
Proof sketch. The DPO stationary condition implies (β) πθ (y | x) ∝ πref (y | x) exp r(y | x)/β where r is the implicit reward. The entropy of a Gibbs distribution over a fixed reward function is a non-increasing function of 1/β (equivalently, non-decreasing in temperature τ = β), which establishes the claim.
Pnref where µref = n1ref i=1 hϕ (xi , yiuf ) is the reference cenp troid computed from nref = ⌈ |Dtrain |⌉ UltraFeedback training examples. We also compute mean pairwise cosine distance as a response diversity metric.
We further observe entropy collapse: after online adaptation, the entropy of the high-β policy decreases more than that of (β) the low-β policy. This is measured as ∆H (β) = HDPO − (β) Honline .
Contrary to conventional expectation, High-β DPO policies generate responses that are closer to the reward model’s training distribution (lower dcos , Spearman ρ = −1.00 across β levels). Despite this, epistemic uncertainty still increases with β (Spearman ρ = +1.00), suggesting that uncertainty is driven not by raw OOD distance but by the interaction between low response diversity and ensemble disagreement in the compressed region of policy support.
OOD finding
5.2. Link 2: OOD Distance from the Reward Model Let hϕ (x, y) ∈ Rd denote the mean-pooled penultimatelayer hidden state of reward ensemble member ϕ0 for input (x, y). We measure out-of-distribution distance as the cosine 4
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking
5.3. Link 3: Uncertainty-Driven Exploitation The ensemble uncertainty signal is v u K u 1 X 2 û(x, y) = t rϕk (x, y) − r̄(x, y) , K −1
this zone, trading some offline alignment precision for significantly reduced online hacking damage. A practitioner choosing β beyond β ⋆ may produce a more conservative offline policy at the cost of a more exploitable one.
(13)
k=1
P where r̄(x, y) = K −1 k rϕk (x, y). When û is high, individual members disagree strongly about the true reward of a response. The online optimiser maximises r̄, but the disagreement means the landscape is unreliable. The policy quickly finds responses that fool some but not all ensemble members—a classic form of distributional reward hacking (Pan et al., 2022).
7. Results 7.1. Goodhart Gap Trajectories Figure 3 shows the Goodhart gap G(t; β) across online adaptation steps for all three β levels. The hacking threshold τhack is set at the 75th percentile of all positive gap values, derived from the data (not hand-tuned). Key observations: (a) the Goodhart gap is predominantly negative throughout, indicating the proxy reward overestimates true performance; (b) Low β (β = 0.310) shows the most volatile trajectory, oscillating to −3×106 , while Mid β and High β remain near zero; and (c) the cumulative hacking damage (AUGC) is nonetheless monotonically ordered by β (31.1, 43.0, 145.8), confirming the paradox.
We compute the Pearson correlation ρUQ (β) between the time series of û and G for each β condition, and report the relationship in the summary table (Table 1).
6. Theory: Optimal Conservatism Having established the empirical and mechanistic case, we now ask: what is the optimal β? Definition 6.1 (Optimal conservatism). Let A(β) denote (0) the offline alignment quality (e.g., win rate over πref ) and AUGC(β) the online hacking damage. The optimal conservatism β ⋆ solves β ⋆ = arg min AUGC(β) − λ · A(β), β>0
where λ > 0 weights alignment against hacking risk. For the empirical approximation, we fit a power law to the observed (β, AUGC) data: AUGC(β) ≈ a · β b + c,
a, b, c > 0.
(14)
Figure 3. Goodhart gap trajectories across online adaptation steps. The Goodhart gap is negative throughout, indicating the proxy reward overestimates true performance. Low β (β = 0.310) shows the most volatile trajectory, oscillating to −3 × 106 , while Mid β (β = 1.000) and High β (β = 2.379) both remain near zero. The hacking threshold (p75, red dashed line) sits near zero. Dotted vertical lines mark the first step at which each condition crosses the threshold.
Parameters (a, b, c) are obtained by least-squares optimisation with initialisation derived from the data range (see Section E). The practical β ⋆ is then defined as the smallest β at which AUGC exceeds 1.5 × minβ ′ AUGC(β ′ ): b β ⋆ = inf β : AUGC(β) > 1.5 · (c + a · βmin ) . (15) Proposition 6.2 (Power-law hacking damage). If Equation (14) holds with b > 1, then the marginal hacking ∂2 cost grows super-linearly in β: ∂β 2 AUGC(β) = a b (b − b−2 > 0. This implies that beyond β ⋆ , small increases 1) β in conservatism incur disproportionately large hacking risk.
7.2. AUGC Summary Table 1 reports AUGC, mean uncertainty, hacking onset step, and UQ-gap Pearson correlation for each β condition. The Spearman rank correlation between β and AUGC is ρ = 1.0 (perfect monotone ordering), confirming the paradox.
Design implication 7.3. Entropy Collapse
The power-law fit identifies a safe operating zone [0, β ⋆ ]. Practitioners should calibrate β to lie within
Figure 4 summarises the entropy measurements. The left panel shows that DPO checkpoint entropy is nearly identical 5
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking Table 1. Summary of online adaptation results across three offline conservatism levels. Higher β strictly increases every hackingrelated metric. rUQ is the Pearson correlation between ensemble uncertainty and Goodhart gap. β level
AUGC ↑
Mean UQ
Onset step ↓
rUQ
Low β (0.310) Mid β (1.000) High β (2.379)
31.1 43.0 145.8
1.80 2.03 2.09
latest middle earliest
moderate higher highest
Spearman ρ (β vs AUGC)
1.00
across all three β levels (≈ 0.81–0.82), with only a marginal decrease at higher β, broadly consistent with Theorem 5.1. The right panel reveals an unexpected pattern: Low β shows a small positive entropy collapse (≈ +0.004), Mid β is near zero, and High β shows a negative collapse (≈ −0.0025), meaning the most conservative policy gains entropy during online adaptation rather than losing it.
Figure 5. Power-law fit of AUGC vs. β with optimal conservatism β ⋆ = 1.082. Data points: Low β (β = 0.310, AUGC = 31.1), Mid β (β = 1.000, AUGC = 43.0), High β (β = 2.379, AUGC = 145.8). The dashed curve is the fitted power law (Equation (14), R2 = 1.000). Shading shows the safe zone (β < β ⋆ ) and danger zone (β > β ⋆ ).
preference data. The intersection of “what the DPO policy generates” and “what the reward model was trained on” is smaller for high β, not larger. Implication for safe alignment Maximising β is not a reliable safety measure for online deployment. A practitioner who cranks up conservatism to avoid reward hacking may inadvertently accelerate it. The calibrated conservatism principle— choose β near β ⋆ —is a more defensible design choice.
Figure 4. Entropy compression. (a) DPO checkpoint entropy is nearly identical across all three β levels (≈ 0.81–0.82), with only a marginal decrease at higher β. (b) Entropy collapse ∆H = HDPO − Honline : Low β exhibits a small positive collapse (≈ +0.004), Mid β is near zero, and High β shows a negative collapse (≈ −0.0025), meaning the high-conservatism policy gains entropy during online adaptation rather than losing it.
8.2. Connection to the Conservative Offline-to-Online Framework
7.4. OOD Distance and the β ⋆ Curve
The companion paper (the workshop draft) proposes a unified conservative loop that applies to both offline RL and offline BO. Our findings provide a concrete empirical instantiation of when that loop can backfire: when the offline phase uses a proxy reward (DPO) whose implicit reward model is different from the reward model used online. The offline alignment step concentrates the policy using one measure of quality; the online step optimises a different (learned) measure. The mismatch is the source of the paradox.
Figure 5 shows the fitted power-law curve overlaid on the three AUGC data points. The safe zone [0, β ⋆ ] and danger zone (β ⋆ , ∞) are shaded. The goodness of fit R2 = 1.0 at three data points (exact fit), but the power-law form provides a smooth extrapolation for practical design guidance.
8. Discussion 8.1. Why the Paradox Matters The conventional conservative-offline-RL argument is a support argument: train on data in the support of the behaviour policy, and the reward model generalises accurately there. Our paradox reveals a subtle conflict: the support of a highβ DPO policy may be quite different from the support of the reward model’s training data, even though the DPO policy stays close to πref . This happens because DPO concentrates mass in a low-entropy region that πref assigns high density to, but the reward model was trained on diverse human
8.3. Limitations Our study uses a single hardware target (H100 80GB), three β values, and a specific policy/reward model combination (Qwen3-14B/Qwen3-1.7B). The power-law fit is exact at three points by construction; the R2 = 1.0 claim should not be interpreted as strong evidence for the functional form, only as an interpolation device. A broader study with more β values, multiple model families, and multiple downstream 6
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking
tasks is needed to generalise the findings.
Goodhart, C. A. E. Problems of monetary management: The UK experience. Papers in Monetary Economics, 1, 1975. Commonly cited as “Goodhart’s Law” (1984 reprint).
9. Conclusion
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models. 2021. URL https://arxiv. org/abs/2106.09685.
We demonstrated that conservative offline training can amplify rather than dampen reward hacking during online adaptation. The mechanism is a three-link chain: highβ DPO compresses policy entropy, which pushes generated responses out of the reward model’s training distribution, which creates exploitable epistemic uncertainty. We formalised Goodhart’s gap and its area under the curve as the primary evaluation metric, derived an optimal conservatism level β ⋆ from a power-law fit, and provided algorithmic recommendations for calibrated conservatism. These findings point toward a richer view of safe offline-to-online alignment: one that explicitly accounts for the distribution mismatch between the offline alignment signal and the online proxy reward.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel : Model-based offline reinforcement learning. 2021. URL https://arxiv.org/abs/2005. 05951. Kostrikov, I., Nair, A., and Levine, S. Offline reinforcement learning with implicit q-learning. 2021. URL https: //arxiv.org/abs/2110.06169. Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. 2020. URL https://arxiv.org/abs/2006.04779.
References Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324–345, 1952.
Lakshminarayanan, B., Pritzel, A., and Blundell, C. Simple and scalable predictive uncertainty estimation using deep ensembles. 2017. URL https://arxiv.org/abs/ 1612.01474.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. 2021. URL https://arxiv.org/abs/ 2106.01345.
Lyle, C., Zheng, Z., Nikishin, E., Pires, B. A., Pascanu, R., and Dabney, W. Understanding plasticity in neural networks. 2023. URL https://arxiv.org/abs/ 2303.01486. Pan, A., Bhatia, K., and Steinhardt, J. The effects of reward misspecification: Mapping and mitigating misaligned models. 2022. URL https://arxiv.org/abs/ 2201.03544.
Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. 2023. URL https://arxiv. org/abs/1706.03741.
Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. 2024. URL https://arxiv.org/abs/2305.18290.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. 2021. URL https://arxiv. org/abs/2110.14168.
Skalse, J., Howe, N. H. R., Krasheninnikov, D., and Krueger, D. Defining and characterizing reward hacking. 2025. URL https://arxiv.org/abs/2209.13085.
Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with scaled ai feedback. 2024. URL https://arxiv.org/abs/ 2310.01377.
Von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., and Huang, S. TRL: Transformer reinforcement learning. https://github. com/huggingface/trl, 2022.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. 2023. URL https://arxiv.org/abs/2305.14314. Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. 2022. URL https: //arxiv.org/abs/2210.10760. 7
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking
Figure 6. Mechanistic summary: why conservative offline policies hack faster. Top-left: Hacking damage (AUGC) increases monotonically with β — Low β = 31.1, Mid β = 43.0, High β = 145.8 (Spearman ρ = +1.00, p = 0.0000). Top-right: Policy entropy at the DPO checkpoint is nearly identical across all three conditions (≈ 0.81–0.82), with only a marginal decrease at higher β (Spearman ρ = −1.00). Bottom-left: OOD cosine distance from the reward model’s training distribution decreases with β (Low ≈ 0.175, High ≈ 0.163; Spearman ρ = −1.00), contrary to the conventional assumption. Bottom-right: Epistemic uncertainty (ensemble standard deviation) increases with β (Low ≈ 1.80, Mid ≈ 2.03, High ≈ 2.09; Spearman ρ = +1.00), confirming that higher conservatism produces more exploitable reward-model disagreement despite responses lying closer to the training distribution.
8
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking
A. Proof of Proposition 5.1 We provide a more detailed version of the proof sketch given in Section 5.1. Recall the variational characterisation of the DPO solution. The KL-regularised reward maximisation problem is max Ex∼p0 Ey∼πθ (·|x) r⋆ (x, y) − β · KL πθ (· | x)∥πref (· | x) , (A.1) πθ
⋆
where r is the implicit reward recovered by DPO. The unique solution to Equation (A.1) is the Gibbs distribution: πref (y | x) exp r⋆ (x, y)/β (β) πθ (y | x) = , (A.2) Zβ (x) P where Zβ (x) = y′ πref (y ′ | x) exp r⋆ (x, y ′ )/β . The entropy of this Gibbs distribution is X (β) (β) (β) H πθ (· | x) = − πθ (y | x) log πθ (y | x) y
= log Zβ (x) −
1 ⋆ E (β) [r (x, y)]. β y∼πθ
Differentiating with respect to β (and suppressing x): 1 1 ∂ log Zβ ∂H = 2 E[r⋆ ] − · β = 2 Vary∼π(β) [r⋆ (x, y)] ≥ 0. θ ∂β β ∂(1/β) β
(A.3)
(A.4)
Thus H is non-decreasing in β (non-increasing in 1/β), confirming Theorem 5.1. □ Remark A.1. The result holds for any proper distribution; it is a standard property of exponential families. The important consequence is that increasing β (more pessimism) strictly compresses entropy unless the reward is constant, which is never the case in practice.
B. Proof of Proposition 6.2 Given AUGC(β) = aβ b + c with a, b, c > 0, the first and second derivatives are: ∂ AUGC(β) = abβ b−1 > 0, ∂β ∂2 AUGC(β) = ab(b − 1)β b−2 . ∂β 2
(A.5) (A.6)
The second derivative is positive iff b > 1. If b > 1, AUGC is strictly convex in β, meaning marginal hacking cost grows faster than linearly. In our data, the fitted b > 1 (confirmed experimentally), so the super-linearity claim holds. □
C. Complete Hyperparameter Derivation All hyperparameters are derived from the architecture, hardware, and data, with no hand-tuned magic numbers. C.1. LoRA Rank Given hidden dimension h: r = 2⌊log2
√ h⌉
,
rmin = 4,
rmax = 64,
r ← clip(r, rmin , rmax ).
(B.1)
LoRA scaling: α = 2r. C.2. LoRA Dropout 32 pdrop = clip √ , 0.01, 0.10 , n where n is the training set size. More data ⇒ less regularisation. 9
(B.2)
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking
C.3. Batch Size
⌊log2
B=2
VGB −1 ⌋ PB ·m·s ,
B ← clip(B, 1, 64),
(B.3)
where VGB is VRAM in GB, PB is model size in billion parameters, m ∈ {1, 2} is a training/inference multiplier, and s = ℓ/1024 is the sequence length pressure factor. C.4. Learning Rate
!
η0
, ηmin , ηmax η = clip p ntrainable /n0
,
(B.4)
where η0 = 2×10−4 , n0 = 107 , ηmin = 5×10−7 , ηmax = 2×10−4 . Larger adapter ⇒ smaller learning rate. C.5. Gradient Clipping Norm
log10 (ntrainable ) , 0.5, 2.0 . cclip = clip log10 (107 )
(B.5)
p Twarmup = ⌊ Ttotal ⌋,
(B.6)
C.6. Warmup Steps
a square-root schedule that provides early stability without sacrificing too many training steps. C.7. Weight Decay
λwd = √
This gives more regularisation when data is scarce. 10
1 . ntrain
(B.7)
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking
D. Extended Algorithm Listing Algorithm 1 Full experimental pipeline 1: Input: preference data Dpref , verifiable data Dver , β percentiles p √ 2: Derive sequence lengths from 75th-percentile token statistics over a n-size sample 3: Derive β grid from empirical log-ratio percentiles (Eq. 7) 4: Train reward ensemble: for k = 1, . . . , K bootstrap Dpref , minimise LBT (Eq. 5) 5: for β ∈ {βlo , βmid , βhi } do (β)
Offline DPO: minimise LDPO (β) (Eq. 1) ⇒ checkpoint πθ Measure DPO entropy H (β) on probe set (Eq. 11) (β) 8: Measure OOD distance dcos (Eq. 12) 9: end for 10: for β ∈ {βlo , βmid , βhi } do (β) (β) 11: Initialise online policy from πθ , reference from πθ (frozen) 12: for t = 1, . . . , Tonline do 13: Sample prompts {xi } from Dver 14: Generate responses {y i } ∼ πθ (· | xi ) P 1 15: Score: (r̄i , ûi ) ← K k rϕk (xi , yi ), stdk 16: Compute advantage Âi (Eq. 9) 17: Compute KL coefficient κ(β) (Eq. 10) 18: Update policy via ∇θ Lonline (Eq. 8) 19: if t mod Teval = 0 then 20: Evaluate r⋆ (GSM8K exact match), compute G(t; β) 21: end if 22: end for 23: end for 24: Compute AUGC per β (Eq. 3) 25: Fit power law AUGC(β) = aβ b + c (Eq. 14) 26: Report β ⋆ (Eq. 15) 6: 7:
E. Power-Law Fitting Details The power-law curve (Equation (14)) is fit via scipy.optimize.curve fit with the following data-derived initial parameters: a0 = AUGCmax − AUGCmin , log(AUGCmax /AUGCmin ) , log(βmax /βmin ) c0 = AUGCmin . b0 =
(C.1) (C.2) (C.3)
Parameter bounds are a, b, c ∈ [0, ∞). The optimisation runs for up to 10,000 function evaluations. The practical β ⋆ is found by dense grid search over β ∈ [0.1βmin , 2βmax ] with 1000 evenly spaced points, finding the first β for which \ \ min ). AUGC(β) > 1.5 × AUGC(β
F. Benchmark Evaluation Protocol A complete benchmark reproducing our findings should report the following metrics in addition to task performance: 1. Goodhart gap time series G(t; β) at a minimum of ⌈T /50⌉ evaluation points. 2. AUGC (Eq. 3) and its standard error over at least three seeds. 3. DPO checkpoint entropy H (β) on a fixed probe set. 11
Pessimism’s Paradox: Conservative Offline Training Amplifies Reward Hacking
4. Entropy collapse ∆H (β) . (β)
5. OOD cosine distance dcos from the reward model’s training distribution. 6. Response diversity (mean pairwise cosine distance). 7. Spearman ρ between β and AUGC. 8. UQ–gap correlation rUQ (β). 9. Power-law parameters (a, b, c) and R2 of the fit. 10. Optimal β ⋆ and the AUGC at that point. For reproducibility, every benchmark should specify: (a) the exact β derivation method (ours uses empirical log-ratio percentiles); (b) the hacking threshold derivation (ours uses the 75th percentile of positive gap values); (c) evaluation seed protocol; and (d) the logging policy and dataset coverage statistics.
G. Limitations and Future Work Single hardware target. All experiments run on a single H100 80GB. Memory-constrained derivations of batch size and LoRA rank depend on VGB ; results may differ slightly on A100 or consumer GPUs. Three β values. A power-law fit at three points is exact by construction. Validating the functional form requires at least five to eight β values spanning two orders of magnitude. Single model family. We use Qwen3 throughout. The entropy-compression mechanism should be model-agnostic (it follows from the Gibbs-distribution form of the DPO solution), but the quantitative curve may differ for other architectures or scales. Future work. (1) Multi-model, multi-task replication. (2) Adaptive β scheduling that tracks û online and reduces β when uncertainty grows. (3) Extension to other offline alignment methods (ORPO, SimPO, KTO) to test whether the paradox is specific to DPO’s KL regularisation or arises for any conservative offline objective. (4) Connecting the entropy-collapse mechanism to plasticity loss (Lyle et al., 2023) in continual RL.
12