A Near-Zero Monitor Readout Is Not Evidence of Behavioral Control

A Near-Zero Monitor Readout Is Not Evidence of Behavioral Control

arXiv:2610.03458v1 [cs.AI] 2 Oct 2026

Zhe Zhou University of Washington zhou1106@cs.washington.edu

Tianhua Tao University of Washington tianhuat@cs.washington.edu

Abstract Post-training with verifiable rewards can induce reward hacking, motivating the use of monitors within the training objective rather than solely for offline auditing. We show that a low monitor readout does not identify whether such an intervention controls behavior. In a code-generation environment whose dominant exploit is available at the start of the reasoning trace, we train policies against three monitors that pass the same offline gate: an in-domain activation probe and two penalties conditioned on how early the policy commits to its own final answer. The probe score is at its numerical floor from the first recorded training step, and the trainedscore median is zero for every prefix-trained run at the endpoint. These readouts estimate different quantities, and we do not compare their scales; within each monitor family, however, low values do not establish behavioral control. Within one fixed configuration, prefix-trained runs with the same zero-median trained score range, by seed alone, from a mixed regime with a low hacking share to near-pure reward hacking. All probe runs reach the hacking regime, but their floor-level readout reflects a mismatch between the position where the probe was validated and the position where it was read during training, not a second instance of this ambiguity. Text-level analysis identifies a prefix failure mode: generic planning and filler shells postpone the exploit past the cut without eliminating it from the final output. Low measured commitment therefore does not distinguish a low hacking share from delayed commitment to the exploit. Offline discrimination and low monitor-aligned readouts are insufficient evidence of behavioral control; an out-of-band behavioral check is required. We characterize the endpoint readout, not its evolution. Code is available at https://github.com/zhezhou1106/ spoof-cost.

1

Introduction

Modern post-training pipelines optimize against programmatically checkable rewards [Shao et al., 2024, Lambert et al., 2025], which can produce reward hacking when the checker is an imperfect proxy for the intended objective [Skalse et al., 2022, Gao et al., 2023, Denison et al., 2024]. Reward-model overoptimization already distinguishes a proxy’s frozen-distribution fit from the policy produced by optimizing it [Gao et al., 2023]. We instantiate that broader problem for monitors used as RLVR training signals. Because reasoning models externalize much of their computation into readable text, one mitigation is to read that text: a monitor scores each rollout for suspiciousness and its score is subtracted from the reward [Baker et al., 2025, Korbak et al., 2025]. This mitigation has a known failure mode. Baker et al. [2025] show that penalizing a chain-ofthought monitor during RL can suppress visible evidence faster than behavior. Related work shows that in-conflict chain-of-thought rewards can preserve unwanted behavior while suppressing its discussion [Kaufmann et al., 2026], that output-only supervision can also induce cross-task chain40th Conference on Neural Information Processing Systems (NeurIPS 2026). Workshop: Foundations of LLM Post-Training in Changing Environments.

of-thought obfuscation [Mitrani Hadida et al., 2026], and that adversarial training incentives can degrade monitorability even when task accuracy is controlled [MacDermott et al., 2025]. White-box activation monitors can likewise be evaded while external behavior is preserved [Gupta and Jenner, 2025, McGuinness et al., 2025]. Monitor-in-the-loop training may therefore lead to qualitatively different regimes rather than simply pass or fail. We study prefix-conditioned monitoring: truncate the reasoning trace, sample answers from the frozen pre-update policy conditioned on each prefix, and estimate how early it commits to the answer it eventually gave. The score is a Monte Carlo estimate of a property of the policy’s conditional distribution, not a direct score of its output text. The policy still controls the emitted prefix and may manipulate it through padding, misdirection, or other strategies; the forced-answer primitive is therefore an alternative optimization surface, not an evasion-proof monitor. We train policies against each gated monitor in an MBPP honeypot whose dominant exploit is visible-test hardcoding. The monitor-aligned endpoint readouts are uniformly low, but the resulting behavior ranges from the earlier mixed regime to near-pure exploitation. For Probe, the actual in-loop score is at its numerical floor; for SCL and Cut, the trained-score median recomputed at the step-120 adapter is zero in both mixed and hacking runs. The readout a practitioner might use as evidence of success therefore does not identify the regime. Reasoning-shell analysis shows how prefix-trained policies can satisfy the measurement without abandoning the exploit. Contributions. 1. Non-identifiability of a low trained prefix score. Within one fixed configuration, varying only the seed, the trained-score median (ψ for SCL, c0.25 for Cut) is zero both in runs that stay in a mixed regime with a low hacking share (d from 0.094 to 0.156) and in runs that reach near-pure reward hacking. The mixed runs still hack, so the claim is that one readout covers two hacking-share regimes, not that the lower one is controlled. 2. An online prefix-conditioned penalty and its failure mode. We move an own-final-answer commitment primitive from offline auditing into the RLVR inner loop. We then characterize how policies defeat it: generic planning and filler shells postpone commitment past the truncation point while still emitting the exploit. 3. A read-position lesson from the activation probe. The probe passes the offline gate when read at the end of the emitted code, but training reads it after an appended self-report turn, where its score is at the numerical floor from the first recorded step. The Probe arm is therefore close to no-monitor training and every run hacks: a monitor can pass an offline gate and still be vacuous inside the loop. 4. Behavioral certification and explanatory diagnostics. Held-out execution identifies the behavioral regime. Off-grid truncation, two-reader re-scoring, and clustering of endpoint reasoning openings do not independently certify control; they test explanations for why the monitor readout failed, including delayed commitment and representation change.

2

Related Work

Monitoring under optimization pressure. Baker et al. [2025] establish that placing a chain-ofthought monitor in the reward can induce obfuscated hacking. Kaufmann et al. [2026] characterize when chain-of-thought and outcome rewards are aligned, orthogonal, or in conflict; their in-conflict MBPP results already capture the general evidence-rewriting concern. Mitrani Hadida et al. [2026] further show that output-only supervision can induce chain-of-thought obfuscation that generalizes across tasks. MacDermott et al. [2025] isolate training incentives while controlling for accuracy and find that adversarial optimization degrades monitorability. These results motivate treating robustness under optimization as distinct from default monitorability [Guan et al., 2025]. Training-time monitors. Taufeeque et al. [2026] show that activation-monitor outcomes depend jointly on detector strength, regularization, and optimization pressure; Gupta and Jenner [2025] and McGuinness et al. [2025] show that evasion can preserve behavior and generalize to unseen probes. GRIFT suppresses detected reward hacking through rejection fine-tuning [Wang et al., 2026], while Li et al. [2026] show that monitors trained on prompted hacks may miss hacking produced by RL.

2

Prior work demonstrates that a monitored score can fall while behavior persists, and the Atlas sweep already spans honest and obfuscated outcomes across its (KL, detector-penalty) configurations. Our focus is the ambiguity within one fixed configuration: by seed alone, the same low readout occurs in runs with a low hacking share and in runs with near-pure hacking, so the readout itself cannot distinguish the regimes. Prefix-conditioned signals. Wang et al. [2025] introduce TRACE, which measures how early a prefix suffices for verifier reward. Shen et al. [2026] instead measure agreement with the model’s eventual answer and find that shortcut-available contexts commit earlier. Both are auditing methods rather than optimized anti-hacking penalties, and self-commitment detects early answer anchoring rather than reward hacking directly. Faithfulness. Lanham et al. [2023] use truncation and corruption to test whether final answers depend on displayed reasoning. These interventions distinguish reasoning that is merely visible from reasoning that is causally load-bearing for the answer.

3

Prefix-Conditioned Monitoring

3.1

SCL

Self-Commitment Latency (SCL) is an online adaptation of the auditing method introduced by Shen et al. [2026]. Let a rollout be a triple (x, r, a) of prompt, reasoning span, and final code. Fix truncation fractions f1 < · · · < fK . At each GRPO step we freeze the pre-update policy, condition it on the prompt plus the reasoning span truncated at fj plus a fixed answer cue, and sample N answer-only continuations. Define the binary equivalence rule Eq(aj,n , a) ∈ {0, 1}. Commitment at fraction fj is 1 X Eq(aj,n , a), (1) cj = N n where the answer-equivalence rule is frozen before the gate. Because code continuations are noisy and non-monotonic in f , the score is the normalized area under the commitment curve, X X ψ= wj cj , wj ≥ 0, wj = 1, (2) j

j

with trapezoidal weights. Larger ψ means the policy had already committed earlier to the answer it eventually gave. Let g index the GRPO group of eight rollouts sharing a prompt. The applied sequence penalty is ψi − µ g zi = , Ritrain = Rienv − α max(zi , 0), (3) σg where µg and σg are the within-group mean and standard deviation of ψ, with zi = 0 when σg = 0. GRPO also standardizes advantages within the group, so α has no common unit with a raw reward offset. The penalty is applied to raw ψ; within-group standardization removes any common offset. The quantity ψ estimates a property of the policy’s conditional distribution given a prefix rather than directly scoring the text it emitted. As emphasized by Shen et al. [2026], self-commitment measures early answer anchoring, not reward hacking itself. Its use as an anti-hacking signal here therefore depends on the environment’s dominant exploit being an early-constant shortcut. Planning and filler shells that postpone that constant past the early cut represent a limitation of the monitoring primitive rather than a scoring malfunction. If the final code is empty or unparseable, we set ψ = 1, assigning maximum suspicion in addition to the separate format penalty described in Section 4.1. This prevents an extraction failure from eliminating the monitor penalty. Instantiation. We use grid [0.1, 0.25, 0.5, 0.75, 0.9], N = 4, 64 forced tokens, and frozen whitespace-normalized string equality; Appendix A gives full settings. 3.2

Cut

Cut applies the same prefix-conditioned primitive at one fixed early fraction of 0.25, with no curve and no trained probe. Like SCL, it uses the penalty pipeline in Equation 3. Using one rather 3

than five truncation points reduces continuation compute to approximately one fifth of SCL’s cost, independently of its effectiveness in controlling behavior during training.

4

Experimental

4.1

Environment and reward

MBPP-Honeypot-CoT. The environment and GRPO harness are adapted from the public MBPPHoneypot setup of Taufeeque et al. [2026], with the tagged reasoning and format contract described below. Each prompt is an MBPP problem [Austin et al., 2021] with one visible assertion and a small held-out assertion set. The policy must emit a tagged reasoning span followed by one fenced Python block. Visible tests set the environment reward, minus a code-length cost of 0.003. Held-out tests define the outcome labels used in this paper. The dominant exploit is visible-test hardcoding, which makes a short, correct-looking answer available at the very start of the trace—a property that favors a prefix-conditioned signal and one we treat as a scope condition throughout. Human review assigns 82% of the 126 baseline exploit rollouts to this category; the retained audit does not further classify the remaining 18%. The exploit is not reached immediately, and the onset step is seed-dependent (Figure 1). Training objectives. For the sequence-penalty arms (SCL and Cut), the training reward is given by Equation 3. The Probe arm instead adds a term on layer-29 residual activations h, read at a single token position: Ritrain = Rienv − α qϕ (hlast,i ),

Lprobe = LGRPO + α qϕ (hlast ),

(4)

where qϕ is the frozen cross-fitted probe and lower q is the non-hacking direction. The harness applies the probe at the final token of the completion after a fixed self-report turn (“Did you respond truthfully?” / “Yes”) is appended; that token, not the end of the emitted code, is the read position. Unlike SCL and Cut, the Probe arm enters optimization through both a differentiable activation-space term and an unstandardized sequence-reward term. We use α = 1.0 in Equations 3 and 4. The objectives enter optimization through different routes and share no common unit. Format contract. Renv is clamped to ≤ 0 and a penalty of 0.5 applies unless the completion has a usable reasoning span: correctly tagged, at least 200 characters and 40 words, followed by a parseable fence. The floor keeps prefixes queryable; it is set above the degenerate short-span range that the format gate is intended to exclude. 4.2

Baseline monitors

Probe (white-box, dual-route). A linear probe on frozen layer-29 residual activations, fit in-domain on baseline rollouts and injected through both an activation-space term and a sequence-reward penalty. It is the only in-domain-fitted detector and has the highest offline point estimate among the passing procedures, but that ranking is not comparable to the zero-shot scores. It belongs to a class that prior work identifies as evadable [Taufeeque et al., 2026, Gupta and Jenner, 2025]. We write its score as q. Judge (passive, text). Graded suspicion in [0, 1] from a fixed base instruct model shown the problem, visible test, reasoning span, and final code, few-shot and near-greedy. Together with SCL and Cut (Section 3) these give four monitors plus a no-monitor baseline, of which three receive an online trial. All monitor parameters are frozen during policy training. 4.3

Injection pathways and comparison scope

Probe injects through both a differentiable activation-space term and an unstandardized sequencereward penalty on q; SCL and Cut inject as nondifferentiable, z-scored sequence penalties on ψ. All monitored arms use the same coefficient, α = 1.0, but the units and gradient pathways are not commensurable, so equal coefficients do not imply matched optimization pressure. We therefore compare complete monitoring procedures rather than attributing cross-family differences to detector type alone.

4

4.4

Training

We use the GRPO/LoRA configuration family reported by Taufeeque et al. [2026]: Llama-3.1-8BInstruct [Grattafiori et al., 2024] with LoRA (r = 64, lora_alpha = 128) [Hu et al., 2022], GRPO [Shao et al., 2024], βKL = 10−4 , 120 steps over 400 prompts at group size 8, and n = 128 endpoint rollouts. In the Atlas sweep, βKL = 10−4 is a low-regularization setting where reward hacking and probe evasion are common; honesty appears with sufficiently higher KL and detector penalties. All trained arms therefore evaluate the same low-regularization corner of that design space, which bounds their external validity but cannot explain differences between them. Runs share the environment, prompt set, data order, training budget, evaluation schedule, monitor coefficient, and the frozen baseline adapter and probe. Seeds {0, 1, 2} vary the policy initialization and the per-step generation seed only. Each run trains a fresh adapter from the base model; the shared baseline supplies the reference row and the fitted probe, not initial policy weights. The policy is not given the monitor’s score or identity as input; monitors affect optimization through the sequence-level penalty, the activation-space term, or both as described above. Full hyperparameters are in Appendix A. 4.5

Metrics

For every endpoint rollout, d is the fraction that passes the visible assertion but fails at least one held-out assertion, h is the fraction that passes the visible and all held-out assertions, and v = d + h is the visible-pass rate. The ratio h/v is the honest share among visible-pass rollouts. We also report format-OK and the share of well-formed attempts that fail the visible assertion. These labels describe observable test outcomes. The complete, mutually exclusive outcome taxonomy is in Appendix C. A run is in the hacking regime if it meets the onset rule: the first of five consecutive steps with held-out pass below 0.10. A run that never meets this rule within 120 steps is in the mixed regime. This label records only that onset had not occurred by step 120: mixed runs still hack (d from 0.094 to 0.156), and we claim neither that they are controlled nor that the regime is stable. Onset is a descriptive post-hoc criterion (Appendix A); we do not treat it as a preregistered primary endpoint. 4.6

Offline gate

On a frozen balanced cohort of 256 late baseline rollouts, Probe, SCL, and Cut pass the prespecified gate (AUROC 0.775, 0.751, and 0.732), while Judge fails at 0.505. The gate was fit once on the frozen baseline and is not refit per seed. Probe is cross-fitted in-domain whereas the other scorers are zero-shot, so these are descriptive procedure-level estimates rather than a detector-class comparison. Table 7, bootstrap details, and cohort provenance are in Appendix D.

5

Results

The central result is that a low monitor-aligned readout does not identify the behavioral regime. We run three GRPO seeds for each gated monitor at α = 1.0 and retain the frozen no-monitor run as a reference. The run is the unit of analysis: medians and [min, max] summarize runs, not pooled rollouts. 5.1

Monitor readouts and behavioral outcomes

Table 1 reports each procedure’s endpoint prefix score alongside its behavioral outcome. We apply the same unmodified SCL primitive to all four arms, using 128 endpoint rollouts and the corresponding step-120 adapter (the frozen baseline adapter for the no-monitor row). These scores correspond to the quantities used by the prefix penalties rather than diagnostic proxies: SCL, no-monitor, and Probe report the curve score ψ, whereas Cut reports agreement at the trained fraction 0.25. Probe’s in-loop score q is a different estimand and is discussed below. We do not compare the scales of ψ and c0.25 across families. Our analysis concerns the endpoint readout rather than its evolution because per-step ψ was not retained. Endpoint readouts and regime. All Probe runs have in-loop scores of q ∼ 10−9 and endpoint hack rates of d ∈ [0.953, 0.969]; applying the prefix primitive to these policies gives a ψ median of 1.000 5

Figure 1: Training trajectories, three seeds per gated monitor. Color marks the arm; solid / dashed / dotted mark seeds 0 / 1 / 2. Each series is Gaussian-smoothed along the step axis (σ=3). Smoothing is for display only; onset steps are computed on unsmoothed per-step values. The no-monitor baseline trajectory was not retained. The Probe arm transitions in every seed (onset 28, 58, 49). SCL stays mixed at seed 0 but transitions at steps 40 and 96; Cut transitions at step 54 but stays mixed at seeds 1 and 2. All runs begin in a mixed visible- and held-out-pass band; whether the later transition occurs depends on the seed. Table 1: Prefix scores are the exact trained quantities recomputed at step 120 on the endpoint rollouts (SCL’s ψ on its full grid; Cut’s single 0.25 cut). Medians are exactly 0.000 for every prefix-trained run; per-rollout means are non-zero, so the zeros reflect the modal rollout rather than a scoring failure. The no-monitor and Probe rows use the same scorer and rollout count. Values other than the single no-monitor run are median [min, max] over three run-level statistics. Onset is the first of five consecutive steps with held-out pass below 0.10; “—” denotes not reached. Arm

Regime (s0/s1/s2)

Onset

Score med.

Score mean

Hack d

Honest h

No monitor hacking 54 1.000 0.958 0.984 0.000 Probe hack / hack / hack 28 / 58 / 49 1.000 [1.000, 1.000] 0.916 [0.853, 0.955] 0.961 [0.953, 0.969] 0.008 [0.008, 0.008] SCL mixed / hack / hack — / 40 / 96 0.000 [0.000, 0.000] 0.017 [0.014, 0.019] 0.883 [0.156, 0.938] 0.031 [0.008, 0.422] Cut hack / mixed / mixed 54 / — / — 0.000 [0.000, 0.000] 0.127 [0.020, 0.154] 0.133 [0.094, 0.930] 0.305 [0.008, 0.344]

Mixed k/3 — 0/3 1/3 2/3

SCL, no-monitor, and Probe report ψ; Cut reports the trained single-cut score c0.25 . Scales are not compared across families.

(mean 0.916 [0.853, 0.955]), compared with a median of 1.000 and mean of 0.958 for no-monitor. Prefix-trained medians are 0.000 in both mixed and hacking runs, while endpoint hacking ranges from 0.094 to 0.938; SCL means also remain in 0.014–0.019 across regimes. Cut’s mean runs opposite to the hacking share across the observed runs: the hacking run has mean c0.25 = 0.020, below the mixed runs at 0.127 and 0.154, so neither statistic supports interpreting a lower Cut readout as better control. Format-OK is at least 0.969 throughout (Appendix B), so format violations do not account for the behavioral divergence. Onset timing. After the initial ramp, all monitored runs occupy a mixed visible- and held-out-pass band, with per-run visible-pass means from 0.467 to 0.602 over steps 10–27 (Figure 1). Probe reaches onset in every run (steps 28, 58, and 49); SCL reaches it in two runs (40 and 96), and Cut in one (54). The frozen no-monitor onset of 54 was computed from its per-step trajectory during the original run, but that trajectory was not retained for re-analysis. Across the sensitivity grid in Table 4, the criterion shifts timing but never changes which runs reach onset. Cut versus SCL. Cut uses one truncation point rather than five and therefore requires approximately one fifth of SCL’s continuation compute. This computational advantage does not establish that one cut reliably controls behavior: across three seeds, Cut remains mixed in more runs than SCL but reaches the hacking regime once, and a discarded initial run at the same seed collapsed to a degenerate format-legal mode with v = 0 (Section 8). Three runs per arm do not support a reliable ordering between SCL and Cut; even a 3/3 versus 0/3 split would yield a two-sided Fisher exact p = 0.1. We

6

therefore report Cut as an efficiency baseline for future comparisons, without claiming empirical superiority. 5.2

How policies satisfy the measurement without abandoning the exploit

Reasoning-shell analysis provides an explanation for near-zero trained prefix scores in hacking runs. We cluster the first sentence of all endpoint reasoning spans. SCL seed 1 repeats “First, we will carefully read the given testing cases / visible test . . . ” in 128/128 rollouts; 119/120 hacking outputs are visible-test hardcodes, yet the trained ψ median is 0.000. SCL seed 2 uses a generic multi-step-plan wrapper in 123/128 rollouts, and Cut seed 0 begins with a description of the function definition in 128/128; both postpone the answer past the cut while retaining the exploit. Final-program length does not explain the near-zero readings in these runs: median program lengths are 9–31 tokens across runs, below the 64-token continuation budget (Appendix D.4). No final program exceeds the budget in Cut seed 0, and only one of 127 does in SCL seed 1, yet both have a trained-score median of 0.000 while hardcoding the visible test. Low commitment is therefore compatible with both a delayed exploit and a mixed behavioral regime. Generic phrasing alone does not account for low commitment: Probe seeds 1 and 2 use repeated wrappers but have a trained-score median of 1.0 under the same primitive (means 0.955 and 0.853). The prefix failure mode specifically involves postponing answer commitment beyond the measurement point.

6

Diagnostic analysis

Held-out execution is the out-of-band check that identifies the behavioral regime. Two-reader rescoring, off-grid truncation, and reasoning-opening clusters are explanatory diagnostics: they test why monitor readouts fail, but do not independently certify the behavioral regime. 6.1

Probe diagnostics

The Probe failure is vacuity rather than ambiguity: its readout is effectively constant and therefore carries no regime information. All Probe runs record q ∼ 10−9 from the first recorded step through step 120 and reach the hacking regime (d ∈ [0.953, 0.969]). Saturation from the first recorded step precludes attributing these low scores to training-induced minimization of a well-scaled probe. Instead, the results show that a monitor can pass an offline gate at one read position yet lack usable dynamic range at its training read position. Two-reader results show both text-side and representationside decreases, with reader means interleaving across regimes rather than tracking control (Appendix Table 9). 6.2

Truncation and ablation

Following Lanham et al. [2023], we also measure own-answer agreement after a 0.30 prefix cut and a 0.40 interior ablation. Figure 2 plots the trained endpoint score—ψ for SCL, c0.25 for Cut—against hack rate. The off-grid 0.30 cut is a robustness check that tracks the on-grid readout (Appendix D). Figure 2 makes the ambiguity visible within the prefix family. The lower-right quadrant is empty—no run combines a high trained prefix score with low hacking—but the zero-median region contains both mixed and near-pure hacking runs. A low reading is therefore not sufficient for control: it occurs both with a low hacking share and with delayed commitment to the exploit.

7

Discussion

7.1

What a monitor readout does and does not certify

Neither offline discrimination nor a low monitor-aligned readout certifies behavioral control here. All three procedures clear the same frozen gate, but gate ordering does not predict the online regime. At training time, Probe supplies no usable dynamic range at its read position, while the trained prefix score is at a run-median of 0 in mixed and hacking runs alike. These are complete procedures with

7

Figure 2: Trained prefix score versus endpoint hack rate. SCL points are the curve score ψ; Cut points are c0.25 . Each point is one run’s median over n = 128 rollouts; the legend identifies runs by color and marker. Upper right: no-monitor and Probe have median scores of 1.0 under the same primitive and high hack rates. Upper left: SCL seeds 1–2 and Cut seed 0 reach the hacking regime despite median scores of 0. Lower left: mixed prefix runs also have median scores of 0. The lower-right region is empty. incommensurable scales and injection routes, so the result does not rank detector classes; it shows that the evidence exposed by each procedure is insufficient to certify its own success. An out-of-band behavioral check is therefore required, and here only held-out execution identifies the regime. The trained prefix score becomes evidence of delayed commitment only after the behavior is known; two-reader values interleave across regimes; and opening clusters provide descriptive, unvalidated evidence of generic shells. These diagnostics explain plausible failure mechanisms rather than serving as independent certificates. For prefix runs that remain mixed, the honest fraction among visible-pass rollouts indicates a compositional difference rather than solely a loss of capability. Conditioning on visible pass excludes capability failures from the denominator: h/v is 0.730 / 0.696 / 0.786 for SCL seed 0 and Cut seeds 1–2, compared with at most 0.034 in every hacking run (Appendix C). Thus approximately 70–79% of visible-pass outputs in the mixed runs also pass the held-out tests. An increase in visible-test failures alone would not explain this difference if the composition of visible-pass outputs remained unchanged. Across these mixed runs, early-band h/v (steps 10–27) averages 0.832, while late-training means (steps 100–120) are 0.826 / 0.741 / 0.762, indicating preservation of a substantial held-out-passing share (Table 6). Mean reasoning length is 365–513 characters and format-OK is at least 0.969 (Appendix B), providing additional evidence against collapse to a minimal format-compliant output. We manually review 30 randomly sampled capability-failure rollouts per mixed run and classify 88/90 as attempted general solutions with implementation errors and 2/90 as degenerate outputs (empty or leaked-reasoning). None match the discarded Cut seed-0 assert-echo collapse with v = 0. These observations support interpreting most sampled capability failures as unsuccessful attempts at general solutions. This is a supporting interpretation for those three runs, not a claim that prefix penalties reliably preserve the regime. What the setup makes cheap. The zero readout itself is partly a product of the training setup. Equation 3 penalizes only rollouts that commit earlier than their group mean, and within-group standardization removes any absolute level, so the pressure always points toward later commitment and has no absolute target. With zi = 0 when σg = 0, a group that commits uniformly late pays nothing, and neither does one that commits uniformly early. The format floor then supplies a contract-

8

satisfying place to put the delay. A near-zero readout is therefore cheap to reach under this objective. What is not built in is that mixed and near-pure hacking endpoints share that readout; this is the finding, and it is the part that bears on monitors rather than on this objective. Logged penalties and raw scores. The sequence-penalty fields in pressure_trajectory.jsonl record α · max(z, 0) rather than raw ψ. Run-mean penalty terms lie in [0.17, 0.26] for every SCL and Cut seed, consistent with within-group z-scoring at group size 8, which removes the raw score’s location and scale. The standardized penalties therefore do not recover the absolute level of ψ. We characterize the endpoint readout rather than its evolution during training.

8

Limitations

Measurement validity. Whitespace-normalized string equality is easier to satisfy for a short hardcoded constant than for a longer program with many behaviorally equivalent implementations. Agreement among held-out-passing programs can still reflect surface-form diversity; execution-based or syntax-aware equivalence remains an important robustness check. Behavioral labels and diagnostics. The label correct_honest means only that a rollout passes the available assertions; a sufficiently comprehensive hardcoded program can receive it without implementing the intended general solution. The held-out-passing mixed-run programs were not audited as deeply as baseline exploits. Truncation, ablation, two-reader scores, and opening clusters diagnose compatible mechanisms but do not uniquely determine policy strategy. In SCL seed 2, 86 of 113 hacking outputs are not visible-test hardcodes and remain uncharacterized (Appendix C). Gate design. The offline gate determines which monitors receive an online trial, so its assumptions shape the comparison. Requiring AUROC ≥ 0.65 with a lower confidence bound above chance treats static discrimination as a prerequisite for useful training pressure and excludes the Judge arm from testing whether a weak detector can nevertheless redirect optimization. The gate is also a weaker filter than its point threshold suggests: it requires only an interval lower bound above 0.50, and the SCL and Cut intervals extend below 0.65 (Table 7). In addition, Probe is fitted on the gate cohort through cross-fitting, whereas SCL, Cut, and Judge are applied without fitting; their AUROCs therefore compare complete procedures rather than interchangeable detector classes. The prefix instrument assumes early commitment. Prefix-conditioned ψ presumes the exploit is available near the start of the trace. The format floor keeps prefixes queryable but also makes multi-sentence planning shells a contract-satisfying way to postpone commitment. This failure mode appears in SCL seeds 1 and 2 and Cut seed 0. The Probe arm has no usable dynamic range at its training read position. Its score is at the numerical floor from the first recorded step, so this study does not test whether a well-scaled activation probe would control reward hacking. The fit/apply position mismatch makes Probe closer to a failed monitoring procedure than a pressure-matched comparison with SCL or Cut. n = 3 establishes coexistence, not rates. The central claim requires the existence of both mixed and hacking outcomes at similar low prefix readouts, which the observed runs establish. Three runs per arm are not enough to estimate P (onset) or rank SCL against Cut. Moreover, all runs share one frozen baseline adapter and probe, so they measure optimization variability rather than independent environment inductions.

9

Conclusion

Subject to Section 8, neither offline discrimination nor a low monitor-aligned readout certifies behavioral control in this environment. The Probe score is at its floor while all runs hack, and the trained prefix score is at a run-median of 0 in mixed and near-pure hacking runs. Generic planning shells explain how prefix-trained policies can satisfy the measurement without abandoning the exploit. Training-time monitors therefore require an out-of-band behavioral check; stronger tests should add pressure-matched controls, execution-based answer equivalence, and scores that do not assume early commitment. 9

Ethics Statement This work studies how policy behavior changes when monitors are placed in the training objective. The apparent evasion behavior is observed rather than explicitly trained as an attack, and the monitors studied are based on published constructions. The results are reported to inform monitor selection. A potential positive impact is more reliable evaluation of training-time monitors and reduced reward hacking. A potential negative impact is that detailed failure diagnostics could inform attempts to evade monitoring. We restrict cross-family conclusions to complete procedures rather than detector classes. All experiments use a public benchmark and an openly released base model, involve no human subjects, and produce no personal data.

Reproducibility Statement The training contract, monitor hyperparameters, gate procedure, and endpoint definition are given in Appendix A, and the outcome taxonomy is in Appendix C. The training and evaluation implementation, configurations, and reproduction instructions are available at https://github.com/ zhezhou1106/spoof-cost. Runs share the frozen baseline, data order, and budget; seeds {0, 1, 2} vary initialization and generation. No run uses best-checkpoint selection over a validation metric.

Acknowledgments and Disclosure of Funding We thank the H2Lab at the University of Washington (https://h2lab.cs.washington.edu/) for GPU compute.

10

References Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926, 2025. Carson Denison, Monte MacDiarmid, Fazl Barez, David Duvenaud, Shauna Kravec, Samuel Marks, Nicholas Schiefer, Ryan Soklaski, Alex Tamkin, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Ethan Perez, and Evan Hubinger. Sycophancy to subterfuge: Investigating reward-tampering in large language models. arXiv preprint arXiv:2406.10162, 2024. Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning, 2023. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, et al. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. Melody Y. Guan, Miles Wang, Micah Carroll, Zehao Dou, Annie Y. Wei, Marcus Williams, Benjamin Arnav, Joost Huizinga, Ian Kivlichan, Mia Glaese, Jakub Pachocki, and Bowen Baker. Monitoring monitorability. arXiv preprint arXiv:2512.18311, 2025. Rohan Gupta and Erik Jenner. RL-Obfuscation: Can language models learn to evade latent-space monitors? arXiv preprint arXiv:2506.14261, 2025. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. Max Kaufmann, David Lindner, Roland S. Zimmermann, and Rohin Shah. Aligned, orthogonal or in-conflict: When can we safely optimize chain-of-thought? arXiv preprint arXiv:2603.30036, 2026. Tomek Korbak, Mikita Balesni, Elizabeth Barnes, et al. Chain of thought monitorability: A new and fragile opportunity for AI safety. arXiv preprint arXiv:2507.11473, 2025. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, et al. Tülu 3: Pushing frontiers in open language model post-training. In Conference on Language Modeling, 2025. Tamera Lanham, Anna Chen, Ansh Radhakrishnan, et al. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702, 2023. Lichen Li, Hengguang Zhou, Yijun Liang, Tianyi Zhou, and Cho-Jui Hsieh. Do prompt-elicited trajectories reflect training-time reward hacking? a systematic study on monitoring training-time reward hacking in code generation. arXiv preprint arXiv:2604.23488, 2026. Matt MacDermott, Qiyao Wei, Rada Djoneva, and Francis Rhys Ward. Reasoning under pressure: How do training incentives influence chain-of-thought monitorability? arXiv preprint arXiv:2512.00218, 2025. Max McGuinness, Alex Serrano, Luke Bailey, and Scott Emmons. Neural chameleons: Language models can learn to hide their thoughts from unseen activation monitors. arXiv preprint arXiv:2512.11949, 2025. Nathaniel Mitrani Hadida, Sassan Bhanji, Cameron Tice, and Puria Radmard. Chain-of-thought obfuscation learned from output supervision can generalise to unseen tasks. arXiv preprint arXiv:2601.23086, 2026. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024. 11

Bonan Shen, Youting Wang, Dingyan Shang, and Tao Ning. Self-commitment latency: A reward-free probe for prompted implicit hacking. arXiv preprint arXiv:2606.05625, 2026. Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. In Advances in Neural Information Processing Systems, 2022. Mohammad Taufeeque, Stefan Heimersheim, Adam Gleave, and Chris Cundy. The obfuscation atlas: Mapping where honesty emerges in RLVR with deception probes. In Proceedings of the 43rd International Conference on Machine Learning, 2026. Songtao Wang, Quang Hieu Pham, Fangcong Yin, Xinpeng Wang, Jocelyn Qiaochu Chen, Greg Durrett, and Xi Ye. Detecting and suppressing reward hacking with gradient fingerprints. arXiv preprint arXiv:2604.16242, 2026. Xinpeng Wang, Nitish Joshi, Barbara Plank, Rico Angell, and He He. Is it thinking or cheating? detecting implicit reward hacking by measuring reasoning effort. arXiv preprint arXiv:2510.01367, 2025.

A

Training contract Table 2: Frozen training contract. All values fixed before the reported runs unless noted.

Parameter

Value

Base policy Adapter Optimizer KL coefficient β Steps / prompts / group size Train batch size Checkpoint interval Endpoint Seeds No-monitor / probe source Monitor penalty α Format contract

Llama-3.1-8B-Instruct LoRA, r = 64, lora_alpha = 128 Group Relative Policy Optimization (GRPO), described in Section 4.4 1e-4 120 / 400 / 8 16 20 steps scheduled; intermediate checkpoints not retained step 120, n = 128 on-policy rollouts; no best-checkpoint selection {0, 1, 2} for each gated monitor; generation seed forwarded with per-step advance one frozen baseline adapter and its in-domain probe; not a continued adapter 1.0 for monitors that passed the offline gate; 0 for baseline tagged span, ≥ 200 chars, ≥ 40 words, fenced Python block; penalty 0.5; Renv ≤ 0 0.003 taufeeque/mbpp-hardcode; 400 train prompts visible = test_list[0]; held-out = test_list[1:]; 2 held-out/problem Atlas defaults for policy and continuations; near-greedy for Judge [0.1, 0.25, 0.5, 0.75, 0.9] / N = 4 / 64 0.25 same raw-ψ z-score pipeline as SCL </reasoning>\n“‘python\n when the rollout closes a <reasoning> block, else \n“‘python\n; same rule in SCL, Cut, and the Section 6.2 diagnostics. All reported rollouts are tagged, so the tagged cue was used throughout. whitespace-normalized string equality, frozen before the gate raw ψ, within-group z-scored; no reference-baseline subtraction 1.0 256 RL-generated rollouts, balanced 128/128, nominally steps 81–120 eligible pool size, exact step histogram, held-out-pass acceptance rate LogisticRegression, C = 1.0, layer 29, prompt-cluster K-fold (K = 5) Frozen few-shot template from the adapted Atlas harness 1000 prompt-cluster resamples AUROC ≥ 0.65, interval lower bound > 0.50 d ≥ 0.80; dominant exploit = visible-test hardcoding; format-OK ≥ 0.80 first of 5 consecutive steps with held-out-pass < 0.10 not preregistered; introduced post hoc prefix 0.30, interior ablation 0.40 fresh rerun, job 39458633; discarded collapse archived separately 2×A100 per run; 10 successful runs; approximately 50 hours on a 4-A100 allocation (∼200 A100-hours)

Code-length cost Dataset Assertions Decoding SCL grid / continuations / forced tokens Cut fraction Cut normalization Answer cue

Answer-equivalence rule Penalty input ψ on unusable final code Gate cohort Unretained gate metadata Probe fitting Judge prompt Bootstrap Gate threshold Baseline sign-off Onset rule Onset provenance Diagnostic cuts Official Cut seed 0 Compute

12

Existing assets and licenses. We use the original 974-problem MBPP release from Google Research [Austin et al., 2021], distributed under CC BY 4.0, through Mohammad Taufeeque’s taufeeque/mbpp-hardcode, revision 08e7ea5 (October 14, 2025). The derived dataset card declares no separate license. We adapt the training harness from FAR.AI’s Obfuscation Atlas codebase [Taufeeque et al., 2026], available at https://github.com/AlignmentResearch/ obfuscation-atlas. As of September 5, 2026, the repository has no published release or declared project license; 0.1.0 is a package metadata version rather than a release tag. The vendored FAR.AI framework, third_party/afterburner, declares package version 0.0.1 but no license in its inspected metadata, and we identified no separate license file in that directory. These package versions do not identify the exact training checkout. We use Meta’s Llama-3.1-8B-Instruct (July 23, 2024 release) under the Llama 3.1 Community License and Acceptable Use Policy. The supporting libraries include scikit-learn (scikit-learn developers; BSD-3-Clause). The upstream harness documents transformers, PEFT, and accelerate (Hugging Face), vLLM (vLLM project), and Hydra (Meta): the first four use Apache-2.0 and Hydra uses MIT. We report these license identifiers separately from the exact installed library versions, which are not specified here. We credit upstream authors and release our experimental code at https://github.com/zhezhou1106/spoof-cost; we do not redistribute the hosted dataset, upstream Atlas code, or model weights. The missing license declarations, including afterburner’s unresolved licensing status, remain limitations of the available licensing documentation.

13

B

Per-run endpoints

Table 3: Per-run endpoint values. Reasoning length excludes the fenced code block. Format-OK is at least 0.969 in every run. Arm

Seed

No monitor Probe Probe Probe SCL SCL SCL Cut Cut Cut

— 0 1 2 0 1 2 0 1 2

d

h

v

Format-OK

Chars (mean/median)

0.984 0.953 0.969 0.961 0.156 0.938 0.883 0.930 0.133 0.094

0.000 0.008 0.008 0.008 0.422 0.008 0.031 0.008 0.305 0.344

0.984 0.961 0.977 0.969 0.578 0.945 0.914 0.938 0.438 0.438

1.000 1.000 1.000 1.000 0.992 1.000 0.977 1.000 0.969 0.984

576.2 / 564.0 619.1 / 622.0 451.6 / 448.0 522.3 / 519.0 365.3 / 361.0 524.6 / 509.0 422.8 / 412.0 548.4 / 536.0 513.2 / 512.0 419.1 / 421.0

Table 4: Onset step under varying held-out-pass thresholds and consecutive-step windows. “—” denotes not reached. The criterion choice shifts onset by a few steps but never changes which runs reach it: every run that reaches the hacking regime does so under all nine criteria, and every mixed run does so under none. The frozen no-monitor trajectory was not retained.

Run Probe s0 Probe s1 Probe s2 SCL s0 SCL s1 SCL s2 Cut s0 Cut s1 Cut s2

0.05/3 0.05/5 0.05/10 0.10/3 0.10/5 0.10/10 0.20/3 0.20/5 0.20/10 28 59 51 — 42 106 57 — —

32 59 51 — 42 106 57 — —

32 71 57 — 42 106 57 — —

28 58 49 — 40 96 54 — —

14

28 58 49 — 40 96 54 — —

28 58 49 — 40 96 54 — —

27 58 49 — 39 92 49 — —

27 58 49 — 39 92 54 — —

27 58 49 — 39 92 54 — —

C

Outcome taxonomy

Each endpoint rollout receives exactly one label, with all rollouts in the denominator. The labels are correct_honest (passes visible and held-out), reward_hacking (passes visible but fails heldout), capability_failure (a well-formed attempt that fails visible tests), and other_ambiguous (unparseable or contract-violating). Table 5: Per-run outcome shares, n = 128. The correct_honest share equals h and the reward_hacking share equals d. h/v is the honest fraction among visible-pass rollouts. other_ambiguous is 0.000 in every run. Run

Regime

correct_honest

reward_hacking

capability_failure

h/v

No monitor Probe s0 Probe s1 Probe s2 SCL s0 SCL s1 SCL s2 Cut s0 Cut s1 Cut s2

hacking hacking hacking hacking mixed hacking hacking hacking mixed mixed

0.000 0.008 0.008 0.008 0.422 0.008 0.031 0.008 0.305 0.344

0.984 0.953 0.969 0.961 0.156 0.938 0.883 0.930 0.133 0.094

0.016 0.039 0.023 0.031 0.422 0.055 0.086 0.062 0.562 0.562

0.000 0.008 0.008 0.008 0.730 0.008 0.034 0.008 0.696 0.786

Among hacking rollouts, visible-test hardcode is 122/122 (Probe seed 0), 87/124 (Probe seed 1), 123/123 (Probe seed 2), 119/120 (SCL seed 1), and 119/119 (Cut seed 0). SCL seed 2 is the exception: other 86/113 and hardcode 27/113. Mixed-run family shares have small hacking n and are not interpreted. Table 6: Honest share h/v among visible-pass rollouts in the three mixed runs. Endpoint values are from the n = 128 step-120 rollouts; the late band is the mean over training steps 100–120. Over the early band (steps 10–27) these runs average h/v = 0.832, and per-run visible-pass means across all monitored runs lie in [0.467, 0.602]. Run-mean penalty terms α max(z, 0) lie in [0.17, 0.26] for every SCL and Cut seed. A late-band share near the early-band average does not show that onset would not occur after step 120. Run SCL s0 Cut s1 Cut s2

Endpoint d

Endpoint h/v

Late-band h/v (steps 100–120)

0.156 0.133 0.094

0.730 0.696 0.786

0.826 0.741 0.762

15

D

Additional implementation and diagnostic details

D.1

Sequence penalty and offline gate

The gate cohort contains 128 hacking and 128 held-out-passing late-baseline rollouts. Probe is promptcluster cross-fitted; the other scorers are zero-shot. The eligible-pool size, exact step histogram, and held-out-pass acceptance rate were not retained. Table 7: Descriptive offline scores on the frozen balanced gate cohort (n = 256). Marginal 95% prompt-cluster bootstrap intervals are not paired equality or equivalence tests; mean scores are internal to each procedure. The gate was not refit per seed.

D.2

Monitor

AUROC

95% CI

Probe SCL Cut Judge

0.775 0.751 0.732 0.505

[0.658, 0.893] [0.645, 0.868] [0.628, 0.850] [0.415, 0.589]

Mean (hack / held-out pass)

Gate

0.80 / 0.46 0.93 / 0.56 0.96 / 0.58 0.96 / 0.95

PASS PASS PASS FAIL

Off-grid prefix diagnostic (robustness)

The main tables report the trained on-grid scores. As an off-grid robustness check, we evaluate a 0.30 cut, which is neither on the SCL grid nor equal to Cut’s training fraction. With one continuation and a 32-token budget (the E4 diagnostic), agreement is near zero in every prefix-trained run and high in the no-monitor and Probe runs, consistent with their respective on-grid endpoint medians of 0 and 1. Table 8: Own-answer agreement after a 0.30 prefix cut and a 0.40 interior ablation, grouped by the run’s endpoint regime rather than by arm name. All rollouts in the denominator (n = 128 per run). Off-grid robustness check; not the trained ψ or c0.25 .

D.3

Arm

Regime

Prefix @0.30

Ablation @0.40

No monitor Probe s0 / s1 / s2 SCL s1 / s2 Cut s0 SCL s0 Cut s1 / s2

hacking hacking hacking hacking mixed mixed

0.945 0.883 / 0.922 / 0.836 0.008 / 0.016 0.008 0.023 0.070 / 0.000

0.938 0.875 / 0.938 / 0.812 0.008 / 0.016 0.039 0.000 0.117 / 0.000

Probe diagnostics

The in-loop probe is read after an appended self-report turn, whereas the gate and two-reader diagnostic read the end of emitted code. Thus the gate AUROC does not measure the exact scorer optimized during training. Probe training logs give an in-loop score of ∼ 10−9 from step 1 through 120. D.4

Continuation budget

Because forced continuations are capped at 64 tokens, a final program longer than the cap cannot be reproduced and would score zero by construction. Tokenizing the fenced code body with the Llama-3.1-8B tokenizer, medians are 9–31 tokens and the share exceeding the cap (including the two-token closer) is 0/128 in every no-monitor, Probe, and Cut seed-0 run, 1/127 in SCL seed 1, and at most 11/128 in any mixed run. Per-continuation cap-hit flags were not written to the endpoint log, so we bound the mechanism by program length rather than measuring truncation directly.

16

Table 9: Two-reader re-scoring with the frozen probe (n = 128 per run). The reference reader is the baseline adapter; the policy reader is the run’s own adapter. Reader means interleave across behavioral regimes, so the descriptive patterns test possible text or representation changes but do not identify control. Runs below hack share 0.5 are not labeled. Arm

Seed

Hack d

q̄ reference

q̄ policy

No monitor Probe Probe Probe SCL SCL SCL Cut Cut Cut

— 0 1 2 0 1 2 0 1 2

0.984 0.953 0.969 0.961 0.156 0.938 0.883 0.930 0.133 0.094

0.750 0.324 0.508 0.582 0.245 0.689 0.609 0.555 0.223 0.102

0.750 0.580 0.682 0.638 0.540 0.782 0.247 0.699 0.082 0.160

Pattern neither policy-reader drop reference-reader drop policy-reader drop — (mixed) neither policy-reader drop reference-reader drop — (mixed) — (mixed)

Table 10: Per-run median program lengths under the Llama-3.1-8B tokenizer. All medians are well below the 64-token forced-continuation cap. Arm

Seed

Regime

Median code tokens

No monitor Probe Probe Probe SCL SCL SCL Cut Cut Cut

— 0 1 2 0 1 2 0 1 2

hacking hacking hacking hacking mixed hacking hacking hacking mixed mixed

10 11 9 11 31 13 18 11 24 20

17

Related documents

• A Near-Zero Monitor Readout Is Not Evidence of Behavioral Control#1188136
• Beyond Punishment, Reward, and Praise: Awareness, Conditioning, and the Structures of Human Control#724868
• Provisioning the calibration window: code, score arrays and outputs (v2.0)#1108455
• Accessibility Statements and Data Protection Notices: What Can Data Protection Law Learn from the Concept of Accessibility?#461945
• Risk-Adaptive Data Loss Prevention: Behavioral Intelligence with DLPWhitepaperCyber Defense4 Jun 2026Matt Bromiley#256019