RH Y VE: C OMPETENCE -AWARE V ERIFICATION AND P HASE -AWARE D EPLOYMENT FOR LLM-G ENERATED R EWARD H YPOTHESES A P REPRINT Xu Zheng Zhuocheng Wang Yi ming Dai Hui Li∗ School of Cyber Engineering, Xidian University [email protected], [email protected], [email protected] [email protected], [email protected]
arXiv:2604.28056v1 [cs.AI] 30 Apr 2026
Feiyu Wu
A BSTRACT Large language models (LLMs) make reward design in reinforcement learning substantially more scalable, but generated rewards are not automatically reliable training objectives. Existing work has focused primarily on generating, evolving, or selecting reward candidates, while paying less attention to when such candidates can be verified and deployed during policy optimization. We study this deployment-time problem by treating generated rewards as reward hypotheses whose utility depends on the competence of the current policy and the phase of training. We propose RH Y VE, a competence-aware verification and phase-aware deployment protocol that compares small sets of reward hypotheses from shared policy checkpoints using short-horizon fork verification. Our experiments show that reward rankings are unreliable at low competence but become informative after task-dependent thresholds. On a sparse manipulation task, phase-aware deployment improves peak and retained performance under a locked protocol. Updated LLM-generated reward-candidate experiments show candidate-family-dependent behavior: generated pools can exhibit phase-dependent winner changes, but no fixed warm-up schedule is universally optimal. Held-out schedule selection, conservative selector baselines, compute-matched controls, and scale controls further show that RH Y VE is best understood as a verification-informed deployment protocol rather than a universal scheduler. Dense and all-failure boundary experiments delimit the scope of the method. Together, these results suggest that reward generation and reward deployment should be studied as coupled problems: generated rewards must be verified and deployed under changing policy competence.
1
Introduction
Reward design remains a central bottleneck in reinforcement learning: good shaping rewards can accelerate learning, while misspecified rewards can induce proxy optimization, reward hacking, and poor generalization Ng et al. [1999], Wiewiora [2003], Devlin and Kudenko [2012], Amodei et al. [2016], Everitt et al. [2017], Everitt and Hutter [2019], Skalse et al. [2022], Pan et al. [2024]. A broad literature studies how to obtain rewards, including inverse RL, preferencebased RL, RLHF, and structured reward specification Ng and Russell [2000], Abbeel and Ng [2004], Ziebart et al. [2008], Finn et al. [2016], Christiano et al. [2017], Ibarz et al. [2018], Stiennon et al. [2020], Ouyang et al. [2022], Bai et al. [2022], Toro Icarte et al. [2018], Camacho et al. [2019], Toro Icarte et al. [2022], Bukharin et al. [2025]. Yet once a candidate reward is available, an equally practical question remains: when is it reliable enough to drive optimization? This question is especially important for LLM-based reward design. Recent systems synthesize reward code, dense shaping functions, and online reward selectors from language Ma et al. [2024], Xie et al. [2024], Li et al. [2024], Hazra et al. [2025], Zhang et al. [2025], Li et al. [2025], Rocamonde et al. [2024]. These methods can generate plausible candidates, but usually treat reward quality as a property of the candidate alone: generate a set of rewards, evaluate or refine them, and deploy the selected one. Our central observation is that this view is incomplete. The apparent quality ∗
Corresponding author: [email protected]
RHyVE
A P REPRINT
of a generated reward can depend strongly on the competence of the current policy and on the phase of training in which the reward is used. We therefore argue that generated rewards should be treated as reward hypotheses rather than immediately reliable objectives. A reward that is useful for late-stage optimization may be too sparse, too sharp, or too poorly scaled to bootstrap learning from a weak policy. Conversely, a dense reward that exposes coarse progress signals early in training may later become misaligned with the task objective. This view connects reward deployment to curricula and adaptive objectives Bengio et al. [2009], Florensa et al. [2017], Graves et al. [2017], Matiisen et al. [2017], Jaderberg et al. [2017], Wang et al. [2020], Xu et al. [2018], Zheng et al. [2018], but sharpens the deployment question in the reward-design setting: the issue is not only which reward is best, but also when the learner is competent enough to verify that fact. A second deployment challenge arises when the training objective changes. Switching rewards mid-training can create non-stationary targets for the actor and critic, produce reward-scale shocks, and destabilize value estimation. Potential-based shaping and trust-region methods motivate caution, but they do not guarantee that an immature learned critic can safely support an online reward switch Schulman et al. [2015], Achiam et al. [2017], Ng et al. [1999], Wiewiora [2003], Devlin and Kudenko [2012]. Reward deployment therefore requires attention both to verification timing and to switching stability. We therefore propose RH Y VE, a competence-aware verification and phase-aware deployment protocol for small sets of reward hypotheses. RH Y VE compares candidates from shared policy checkpoints using short-horizon fork verification, records when winner identity becomes reliable, and uses the resulting phase profile to decide whether a single reward suffices or whether a two-stage schedule is warranted. The method is deliberately local in scope: it is not a new reward generator and not a universal adaptive scheduler, but a protocol for deciding when already available candidates are trustworthy enough to compare and deploy. In this sense, RH Y VE addresses the missing middle between reward generation and reward commitment. This paper makes the following contributions: • We formulate generated rewards as hypotheses whose reliability depends on policy competence and training phase, separating reward generation from reward verification and deployment. • We introduce shared-checkpoint fork verification and phase-profile construction for small candidate sets, together with deployment rules that choose a single reward, a two-stage schedule, or conservative fallback depending on verification reliability. • We show on the locked F RANKAC ABINET protocol that phase-aware deployment improves both peak and recomputed terminal performance in a sparse phase-sensitive manipulation regime. • We extend the evidence to LLM-generated reward-candidate families under a reduced 6×3090 protocol, showing candidate-family-dependent deployment behavior rather than universal warm-up optimality. • We provide controls and scope evidence, including held-out schedule selection, conservative online selector baselines, compute-matched direct training, reward-scale controls, dense-oracle boundary behavior, candidatepool stress tests, and an all-failure extra-task pilot.
2
Related Work
Recent work shows that large language models and foundation models can automate substantial parts of reward engineering. Eureka, Text2Reward, Auto MC-Reward, REvolve, and R* synthesize or refine executable reward programs and shaping functions; ORSO is the closest operational prior because it studies online reward selection during training; and pretrained vision-language models can act as zero-shot reward models. These works mainly study how to generate, refine, search over, or reactively select reward candidates. RH Y VE studies a complementary deployment-time question: when does a small candidate set become reliable to compare from the current learner state, and should the resulting evidence lead to a single reward, a phase-aware schedule, or conservative fallback? Our conservative selector baselines are included to test reactive alternatives, but we do not claim to reproduce or dominate the full ORSO system. Learned and structured reward methods reduce manual reward engineering by inferring or constraining reward signals from demonstrations, preferences, human feedback, or formal task structure Ng and Russell [2000], Abbeel and Ng [2004], Ziebart et al. [2008], Finn et al. [2016], Christiano et al. [2017], Ibarz et al. [2018], Stiennon et al. [2020], Ouyang et al. [2022], Bai et al. [2022], Hadfield-Menell et al. [2017], Toro Icarte et al. [2018], Camacho et al. [2019], Toro Icarte et al. [2022], Bukharin et al. [2025]. Curriculum, teacher-student, population-based, and meta-gradient methods likewise exploit the fact that useful supervision can change during learning Bengio et al. [2009], Florensa et al. [2017], Graves et al. [2017], Matiisen et al. [2017], Jaderberg et al. [2017], Wang et al. [2020], Xu et al. [2018], Zheng et al. [2018]. Our focus is narrower: given already available rewards, we ask when the current learner is competent 2
RHyVE
A P REPRINT
enough for local comparison, when reward utility changes across phases, and when a fixed phase-aware deployment rule is warranted. Reward shaping and safe optimization are directly relevant once the objective changes. Potential-based shaping gives policy-invariance guarantees under suitable assumptions Ng et al. [1999], Wiewiora [2003], Devlin and Kudenko [2012], Grzes and Kudenko [2010], while trust-region, constrained, and safety-oriented perspectives stress that changing objectives can induce instability, proxy optimization, or reward tampering Schulman et al. [2015], Achiam et al. [2017], Amodei et al. [2016], Everitt et al. [2017], Everitt and Hutter [2019], Skalse et al. [2022], Pan et al. [2024]. RH Y VE does not propose a universal switch operator; instead, it separates competence-aware verification from the trade-offs of how a reward change is executed, which is why our experiments treat hard switching, PBRS, and critic reset as conditional mechanisms rather than interchangeable defaults.
3
Problem Setup and Method
In RH Y VE, we study the deployment-time problem that arises after an upstream reward-design procedure, such as an LLM-based reward generator, has produced a small set of candidate rewards. Our goal is not to generate new reward code, but to decide when candidate rewards are reliable to compare, when they should be deployed, and how objective changes should be introduced without destabilizing training. The central distinction is that candidate rewards are treated as hypotheses whose usefulness may depend on learner competence rather than as immediately trustworthy objectives. The method is intentionally modest in scope. RH Y VE is designed for the practically common regime in which a small number of structured candidate rewards is already available and the learner’s competence changes over training. 3.1
Reward Hypotheses
We consider an episodic Markov decision process M = (S, A, P, ρ0 , γ), where S and A are the state and action spaces, P is the transition kernel, ρ0 is the initial-state distribution, and γ ∈ (0, 1) is the discount factor. Instead of assuming a single fixed reward, we are given a small candidate set H = {h1 , . . . , hK }, where each hypothesis hk induces a reward function rhk : S × A × S → R. We use the term reward hypothesis deliberately. A generated reward is not assumed to be immediately trustworthy, globally optimal, or useful at every training phase. Rather, it is a candidate training signal whose value may depend on the competence of the current policy. Let zt = (πt , Vt , Ωt ) denote a training checkpoint at optimization step t, consisting of the current policy πt , critic Vt , and optimizer or training state Ωt . Associated with each checkpoint is a task-dependent competence proxy ct = c(zt ), such as success rate, task progress, or another downstream metric. We do not assume a universal competence scale across tasks; competence is used only to determine whether the current learner is in a regime where reward comparisons are informative. 3.2
Shared-Checkpoint Fork Verification
The central operation in RH Y VE is shared-checkpoint fork verification. At a checkpoint zt , each reward hypothesis hk is evaluated by cloning the same learner state, continuing training for a short fork horizon L under reward rhk , and evaluating the resulting forked policy with a common downstream metric: (L)
Jt
(hk ) = Eval(Train(zt , rhk , L)) .
The local winner at checkpoint t is (L)
ĥt = arg max Jt hk ∈H
3
(hk ),
RHyVE
A P REPRINT
Figure 1: Overview of RH Y VE. Reward candidates are treated as hypotheses, compared from shared checkpoints using fork verification, and deployed as a single reward, two-stage schedule, or conservative fallback depending on the phase profile. and its margin over the runner-up is (L)
mt = Jt
(L)
(ĥt ) − max Jt
(hj ).
hj ̸=ĥt
Shared checkpoints avoid confounding reward quality with independent initialization, exploration history, and optimizer state. The fork horizon L is intentionally short: RH Y VE estimates which reward is locally useful for the current learner rather than fully optimizing every candidate. This keeps verification aligned with the practical question of whether a reward should be deployed from the current policy state. If M checkpoints are probed, K hypotheses are compared, and each fork runs for L update steps, the additional verification cost scales as O(M KL). Thus RH Y VE is designed for small candidate sets, sparse checkpoint probes, and short local verification windows. 3.3
Verification-Informed Phase Profiles
Fork verification produces a phase profile over training: n o (L) P = (t, ct , ĥt , mt , {Jt (hk )}K , k=1 ) : t ∈ T where T = {t1 , . . . , tM } is a sparse set of probed checkpoints. This profile records how apparent reward utility changes as the policy becomes more competent and provides a compact diagnostic of whether the candidate family exhibits a stable phase structure. Early checkpoints may be uninformative because a weak policy has not yet reached states where reward differences matter. We therefore treat a checkpoint as verification-informative only when the comparison shows sufficient separation and stability, assessed using mt , Agreet , Entt , where mt is the winner margin, Agreet is repeated-fork winner agreement, and Entt is winner entropy across repeated forks or nearby horizons. In practice, these diagnostics are used to distinguish true phase structure from noisy local rank fluctuations. If winner identity is unstable or margins are negligible, RH Y VE abstains from aggressive reward commitment; if a winner transition stabilizes, the profile suggests phase-dependent reward utility. 4
RHyVE
3.4
A P REPRINT
Phase-Aware Deployment
Given a phase profile P, RH Y VE deploys a single reward when one hypothesis is stable from the first informative checkpoint onward, a two-stage schedule when an early winner is later overtaken by a stable later winner, and a conservative no-switch rule when the profile remains ambiguous. The two-stage case uses r (1) t < ts , rt = h rh(2) t ≥ ts , where ts is the first stable later-phase switch point. In practice, we prefer deployment rules that are selected from phase profiles and then evaluated on held-out seeds, rather than rules chosen after inspecting test performance. When the phase profile is stable, RH Y VE may deploy a two-stage schedule such as a fixed warm-up followed by a task-faithful reward. When a single candidate is stable from the first informative checkpoint onward, RH Y VE deploys that single reward. When margins, agreement, or winner identity remain ambiguous, RH Y VE falls back to a conservative rule rather than forcing a switch. This framing is important: RH Y VE is not a universal scheduler, and fixed warm-up is not assumed to be optimal for every candidate family. 3.5
Switch Operators
Once a later-phase reward has been identified, the remaining question is how to execute the objective change. Hard switching directly replaces the early reward with the later reward, r (1) t < ts , rt = h rh(2) t ≥ ts . and serves as the practical default in our experiments because it introduces no additional learned shaping terms. A smoothing alternative uses potential-based shaping Ng et al. [1999], Wiewiora [2003], Devlin and Kudenko [2012]: given a source critic Vtold trained before the switch, we define s r̃h(2) (s, a, s′ ) = rh(2) (s, a, s′ ) + γΦ(s′ ) − Φ(s),
Φ(s) = Vtold (s). s
This operator is useful only when the source critic is sufficiently informative; otherwise the shaping term can inject stale or noisy value estimates. A third operator keeps the actor but reinitializes the critic at the switch point, (πtnew , Vtnew ) = (πtold , Vreset ). s s s These operators are therefore analyzed as conditional mechanisms. The experiments treat hard switching, PBRS, and critic reset as stability–retention trade-offs rather than as interchangeable safe-switching guarantees.
4
Experiments
We evaluate RH Y VE as a deployment-time protocol for small sets of reward hypotheses. The experiments are designed to answer four questions: when fork verification becomes informative, whether phase-aware deployment helps in a sparse phase-sensitive regime, whether the phenomenon appears for LLM-generated reward candidates, and which controls delimit the scope of the method. 4.1
Experimental Setup
The main task is F RANKAC ABINET, a sparse and phase-sensitive manipulation task in Isaac Gym. Unless otherwise stated, each task uses a small candidate set with K = 3 reward hypotheses and PPO-style actor–critic optimization. The structured F RANKAC ABINET family contains an early dense bootstrap reward, a later task-faithful oracle-like reward, and a later alternative reward. Fork verification compares candidates from shared policy checkpoints over short horizons and scores the resulting forked policies with a common downstream metric. We report peak success (consec_max) together with recomputed final success and tail metrics when available. This distinction is important because a method can achieve a transient success spike and later collapse. Main-text tables report seed counts; evidence status is made explicit in the appendix taxonomy and appendix tables. Locked structured results support the main sparse-task claim; reduced 6×3090 LLM results support candidate-family-dependent deployment behavior; selector, compute-matched, scale, and extra-task experiments are used as controls or scope evidence. 5
RHyVE
A P REPRINT
Figure 2: Competence-limited fork verification. Reward rankings are unstable at low competence and become informative only after task-dependent checkpoints and sufficient fork horizon. Table 1: Locked FrankaCabinet structured-reward results. Values are means over fixed seeds. Final and tail metrics are recomputed from learning curves when available.
Method
n
Max / consec_max ↑
Recomputed final ↑
Last-5 mean ↑
Tail AUC ↑
Direct oracle (wu=0) One-shot oracle Fixed wu=50 hard switch Fixed wu=100 hard switch wu=50 + Vold PBRS
8 8 8 8 8
0.215 0.228 0.389 0.233 0.314
0.022 0.022 0.163 0.089 0.036
0.028 0.025 0.161 0.087 0.037
0.034 0.034 0.157 0.056 0.040
4.2
Competence-Limited Reward Verification
Figure 2 shows that fork-based reward rankings are competence-limited. On F RANKAC ABINET, early checkpoints are unstable or favor the dense bootstrap reward, while later checkpoints reveal a stable later-phase winner. The first informative checkpoint also depends on fork horizon. This motivates phase-aware deployment rather than one-shot reward commitment. Two details are important. First, an early checkpoint can produce a modal winner without being verification-informative, because the winner margin or repeated-fork agreement may still be weak. Second, longer fork horizons can expose later-phase utility earlier than very short forks, but at higher verification cost. Thus reward verification is itself a competence-dependent diagnostic rather than a static evaluation of the candidate set. 4.3
Phase-Aware Deployment on F RANKAC ABINET
Table 1 reports the locked F RANKAC ABINET structured comparison. The fixed wu=50 hard-switch schedule achieves the strongest mean peak performance and the strongest recomputed final performance among the main locked methods. Figure 3 shows the corresponding learning curves. The result should be interpreted as a sparse-regime deployment finding rather than a universal scheduler claim. Peak paired tests remain affected by high seed variance, so we avoid claiming broad benchmark dominance. The practical conclusion is narrower: in this sparse phase-sensitive candidate family, a short competence-informed warm-up is a strong deployment default, and its gain is visible in both peak and retained behavior. 4.4
LLM-Generated Reward Candidate Families
Table 2 reports the updated F RANKAC ABINET LLM-generated candidate results under the reduced 6×3090 protocol. These results replace the older pilot-style source-generalization table. The updated rows show that generated candidate pools can exhibit phase-dependent deployment behavior, but the best deployable rule is candidate-family-dependent. 6
RHyVE
A P REPRINT
Figure 3: Locked FrankaCabinet learning curves. The phase-aware wu=50 hard-switch schedule improves both peak and retained behavior relative to direct and one-shot deployment under the structured reward family. Table 2: FrankaCabinet LLM-generated reward-candidate results under the reduced 6x3090 protocol. The best deployment rule varies across candidate families, supporting a verification-informed rather than universal-schedule interpretation. Oracle rows are non-deployable references and are not guaranteed to upper-bound deployable rows when seed sets or comparison protocols differ. Candidate source
Method
n
Max / consec_max ↑
Flip rate
Status
llm_lightly_curated llm_lightly_curated llm_lightly_curated
Direct wu=50 hard Held-out selected
8 8 5
0.145 0.207 0.298
0.875 0.875 –
deployable deployable deployable
llm_lightly_curated
Best adaptive (oracle reference)
3
0.580
0.875
oracle
llm_minimally_filtered llm_minimally_filtered llm_minimally_filtered
Direct wu=50 hard Held-out selected
8 8 5
0.211 0.639 0.800
0.500 0.500 –
deployable deployable deployable
llm_minimally_filtered
Best adaptive (oracle reference)
2
0.369
0.500
oracle
The lightly curated and minimally filtered LLM families should not be read as evidence that a fixed wu=50 schedule is universally optimal. Instead, they support the central diagnostic claim: generated rewards are hypotheses whose usefulness depends on policy competence, candidate source, and deployment rule. Oracle-style adaptive rows, when shown, are separated from deployable methods and are not used as main method claims. 4.5
Held-Out Schedule Selection
Table 3 audits schedule selection using dev seeds and held-out report seeds. This is included to reduce the risk that a fixed switch time is interpreted as post-hoc test-set tuning. The schedule selected from dev-seed phase profiles is frozen before evaluation on held-out seeds. If the phase profile is ambiguous, RH Y VE uses a conservative fallback rather than forcing an aggressive switch. This held-out audit is not meant to prove that a single switch time is universal. Its role is to show that deployment choices can be made from verification evidence without inspecting held-out performance, and that the resulting behavior can be reported separately from oracle upper bounds. 4.6
Robustness and Fairness Controls
Table 4 summarizes three important controls. First, compute-matched direct training tests whether the advantage of phase-aware deployment is simply due to the extra fork-verification budget. The compute-matched direct baseline narrows the gap, showing that verification overhead is a real fairness consideration, but it does not by itself establish that direct deployment is preferable. Second, conservative online selector baselines test whether reactive reward selection 7
RHyVE
A P REPRINT
Table 3: Held-out schedule-selection audit. Schedules are selected using dev seeds and evaluated on held-out report seeds; only deployable held-out rows are shown in the main text. Candidate source
Deployable rule
n
Max / consec_max ↑
structured_handcrafted_K3 llm_lightly_curated
fixed_wu50_hard → fixed_wu50_hard direct_oracle_or_direct_best → direct_oracle_or_direct_best fixed_wu50_hard → fixed_wu50_hard
5 5
0.394 0.125
5
0.800
llm_minimally_filtered
Table 4: Robustness and fairness controls. Compute-matched direct training narrows the gap but should be interpreted separately from deployable phase-aware scheduling; reactive selectors underperform the stable schedule in this sparse small-candidate regime; scale matching helps but does not fully explain the phase-aware gain. Method
n
Max ↑
Final ↑
Tail AUC ↑
A. Compute-matched fairness direct_oracle_or_direct_best direct_oracle_plus_verification_steps fixed_wu50_hard rhyve_heldout_selected
8 8 8 5
0.215 0.339 0.389 0.394
0.022 0.019 0.163 –
0.045 0.023 0.130 –
B. Selector baseline fixed_wu50_hard conservative_periodic_selector moving_average_selector naive_last_selector direct_oracle_or_direct_best
8 8 8 8 8
0.389 0.051 0.113 0.092 0.215
– – – – –
– – – – –
C. Scale control direct_oracle fixed_wu50_hard direct_oracle_scale_matched_to_early
8 8 8
0.215 0.389 0.235
0.022 0.163 –
0.034 0.157 –
can replace stable phase-aware deployment in this sparse small-candidate regime. These selectors underperform the stable schedule in the updated support comparison, and their failure modes are analyzed in the appendix. Third, scale controls show that reward scale is an important confound, but scale matching does not fully explain the phase-aware gain. Together, these controls support a cautious interpretation: RH Y VE is useful because it separates verification, deployment, and switching stability. It should not be interpreted as a claim that compute is irrelevant, that reactive selectors never work, or that scale mismatch is unimportant. 4.7
Boundary Conditions and Scope
BALL BALANCE provides a dense-oracle boundary condition: all locked methods saturate the peak metric, so consec_max is not discriminative. We therefore treat it as evidence that warm-up is not universally necessary rather than as another performance-dominance benchmark. We also attempted an additional F RANKAC UBE S TACK sparse-manipulation pilot under the reduced optional-scope budget. All compared methods failed to reach nontrivial success, so we report it only as an all-failure boundary in the appendix rather than as a method-ranking comparison. This boundary result does not support a generalization claim, but it clarifies the scope of the current method: RH Y VE requires a learnable task/reward setup under the available budget. 4.8
Additional Evidence and Scope
The appendix provides full fork-verification grids, per-seed results, tail audits, switch-operator analyses, LLM candidate-source details, selector failure taxonomy, compute-matched metrics, candidate-pool stress tests, and the F RANKAC UBE S TACK boundary pilot. The overall evidence supports a task-aware conclusion: RH Y VE is strongest as a small-candidate verification-and-deployment protocol for sparse, phase-sensitive regimes. It is not a universal adaptive scheduler, not a large-pool reward-search method, and not a guarantee that warm-up improves every task. 8
RHyVE
5
A P REPRINT
Discussion and Limitations
The main lesson is that automated reward design does not end at reward generation. Even plausible LLM-generated rewards are not immediately reliable objectives: they must be verified against the competence of the current learner and deployed with attention to when objective changes are actually justified. RHYVE is therefore best understood as a verification-informed deployment protocol rather than as a new reward generator. The empirical scope is task-aware rather than universal. F RANKAC ABINET benefits from phase-aware deployment because the useful reward changes with competence, while BALL BALANCE acts as a dense-oracle boundary condition in which warm-up is unnecessary or metric-dependent. The additional F RANKAC UBE S TACK pilot is an all-failure boundary under the optional budget, showing that RHyVE cannot produce meaningful rankings when the underlying task/reward setup is not learnable. These boundary cases are part of the method’s diagnostic role rather than failures of a universal scheduler. Reward switching also remains a trade-off rather than a solved subproblem. Hard switching is the strongest practical default in the locked setting, while PBRS and critic reset can help in some regimes but expose different stability– retention costs. The method therefore separates the question of whether a reward change is justified from the question of how that change should be executed. Compute and scale must also be interpreted carefully. Compute-matched direct training narrows the gap to phase-aware deployment, confirming that verification overhead is a real fairness consideration. Reward-scale controls likewise show that scale mismatch contributes to cold-start failures. However, neither extra compute nor scale correction fully subsumes the phase-composition interpretation in the reported sparse regime. Finally, RHYVE is a small-candidate local-verification protocol. Its conclusions depend on fork horizon, checkpoint selection, competence proxies, and candidate-family quality. Larger candidate pools will likely require uncertaintyaware ranking, coarse-to-fine verification, clustering, or more efficient candidate pruning. In practice, RHYVE should be used diagnostically: first test whether reward rankings are reliable under the current competence regime, then decide whether a single reward, a two-stage schedule, or conservative fallback is appropriate.
6
Conclusion
We introduced RHYVE, a competence-aware protocol for verifying and deploying generated reward hypotheses. The central finding is that reward verification can be competence-limited: a reward that appears weak from a low-competence policy can become useful later, while dense early shaping can cease to be the right objective once the learner reaches task-relevant states. On the locked F RANKAC ABINET protocol, phase-aware deployment improves peak and retained performance. Updated LLM-generated candidate experiments show that this issue also arises beyond the hand-designed reward family, but that the best deployment rule depends on the candidate pool. Controls with held-out schedule selection, conservative selectors, compute-matched direct training, scale matching, dense boundary tasks, and an all-failure extra-task pilot delimit the claim. RHYVE is not a universal scheduler or a safe-switching guarantee; it is a verification-informed deployment protocol for small candidate sets in regimes where reward utility may change with policy competence. Future reward-design systems should therefore reason not only about how to generate rewards, but also about when they become reliable and useful to deploy.
References Andrew Y. Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the 16th International Conference on Machine Learning, pages 278–287, 1999. Eric Wiewiora. Potential-based shaping and q-value initialization are equivalent. Journal of Artificial Intelligence Research, 19:205–208, 2003. Sam Devlin and Daniel Kudenko. Dynamic potential-based reward shaping. In Proceedings of the 11th International Conference on Autonomous Agents and Multiagent Systems, pages 433–440, 2012. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv:1606.06565, 2016. Tom Everitt, Victoria Krakovna, Laurent Orseau, and Shane Legg. Reinforcement learning with a corrupted reward channel. In Proceedings of the 26th International Joint Conference on Artificial Intelligence, pages 4705–4713, 2017. Tom Everitt and Marcus Hutter. Reward tampering problems and solutions in reinforcement learning: A causal influence diagram perspective. arXiv preprint arXiv:1908.04734, 2019. 9
RHyVE
A P REPRINT
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward hacking. arXiv preprint arXiv:2209.13085, 2022. Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. Feedback loops with language models drive in-context reward hacking. arXiv preprint arXiv:2402.06627, 2024. Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. In Proceedings of the 17th International Conference on Machine Learning, pages 663–670, 2000. Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the 21st International Conference on Machine Learning, page 1, 2004. Brian D. Ziebart, Andrew L. Maas, J. Andrew Bagnell, and Anind K. Dey. Maximum entropy inverse reinforcement learning. In Proceedings of the 23rd AAAI Conference on Artificial Intelligence, pages 1433–1438, 2008. Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the 33rd International Conference on Machine Learning, pages 49–58, 2016. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299–4307, 2017. Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. Reward learning from human preferences and demonstrations in atari. In Advances in Neural Information Processing Systems, pages 8022–8034, 2018. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325, 2020. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. Rodrigo Toro Icarte, Toryn Q. Klassen, Richard A. Valenzano, and Sheila A. McIlraith. Using reward machines for high-level task specification and decomposition in reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, pages 2112–2121, 2018. Alberto Camacho, Rodrigo Toro Icarte, Toryn Q. Klassen, Richard A. Valenzano, and Sheila A. McIlraith. Ltl and beyond: Formal languages for reward function specification in reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 6065–6073, 2019. Rodrigo Toro Icarte, Toryn Q. Klassen, Richard A. Valenzano, and Sheila A. McIlraith. Reward machines: Exploiting reward function structure in reinforcement learning. Journal of Artificial Intelligence Research, 73:173–208, 2022. Alexander Bukharin, Yuchen Li, Pengcheng He, and Tuo Zhao. Deep reinforcement learning from hierarchical preference design. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 5875–5893, 2025. Yecheng Jason Ma, William Liang, Guanzhi Wang, De-An Huang, Osbert Bastani, Dinesh Jayaraman, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Eureka: Human-level reward design via coding large language models. In International Conference on Learning Representations, 2024. Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu. Text2reward: Reward shaping with language models for reinforcement learning. In International Conference on Learning Representations, 2024. Hao Li, Xue Yang, Zhaokai Wang, Xizhou Zhu, Jifeng Zhou, Yu Qiao, Xiaogang Wang, Hongsheng Li, Liwei Lu, and Jifeng Dai. Auto mc-reward: Automated dense reward design with large language models for minecraft. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16426–16435, 2024. Rishi Hazra, Alexandros Sygkounas, Andreas Persson, Amy Loutfi, and Pedro Zuidberg Dos Martires. Revolve: Reward evolution with large language models using human feedback. In International Conference on Learning Representations, 2025. Chen Bo Calvin Zhang, Zhang-Wei Hong, Aldo Pacchiano, and Pulkit Agrawal. Orso: Accelerating reward design via online reward selection and policy optimization. In International Conference on Learning Representations, 2025. 10
RHyVE
A P REPRINT
Pengyi Li, Jianye Hao, Hongyao Tang, Yifu Yuan, Jinbin Qiao, Zibin Dong, and Yan Zheng. R*: Efficient reward design via reward structure evolution and parameter alignment optimization with large language models. In Proceedings of the 42nd International Conference on Machine Learning, volume 267 of Proceedings of Machine Learning Research, pages 34509–34527, 2025. Juan Rocamonde, Vicente Montesinos, Elvis Nava, Ethan Perez, and David Lindner. Vision-language models are zero-shot reward models for reinforcement learning. In International Conference on Learning Representations, 2024. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning, pages 41–48, 2009. Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Proceedings of the 1st Conference on Robot Learning, pages 482–495, 2017. Alex Graves, Marc G. Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In Proceedings of the 34th International Conference on Machine Learning, pages 1311–1320, 2017. Tambet Matiisen, Avital Oliver, Taco Cohen, and John Schulman. Teacher-student curriculum learning. arXiv preprint arXiv:1707.00183, 2017. Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, Iain Dunning, Karen Simonyan, et al. Population based training of neural networks. arXiv preprint arXiv:1711.09846, 2017. Xin Wang, Yudong Chen, and Wenwu Zhu. A comprehensive survey on curriculum learning. arXiv preprint arXiv:2010.13166, 2020. Zhongwen Xu, Hado P. van Hasselt, and David Silver. Meta-gradient reinforcement learning. In Advances in Neural Information Processing Systems, pages 2402–2413, 2018. Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. In Advances in Neural Information Processing Systems, pages 4649–4659, 2018. John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning, pages 1889–1897, 2015. Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel. Constrained policy optimization. In Proceedings of the 34th International Conference on Machine Learning, pages 22–31, 2017. Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel, and Stuart Russell. Inverse reward design. In Advances in Neural Information Processing Systems, pages 6765–6774, 2017. Marek Grzes and Daniel Kudenko. Online learning of shaping rewards in reinforcement learning. Neural Networks, 23 (4):541–550, 2010. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. Viktor Makoviychuk, Lukasz Wawrzyniak, Yunrong Guo, Michelle Lu, Kier Storey, Miles Macklin, David Hoeller, Nikita Rudin, Arthur Allshire, Ankur Handa, and Gavriel State. Isaac gym: High performance gpu-based physics simulation for robot learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
A
Protocol, Metrics, and Reproducibility
This appendix describes the reporting protocol, metric definitions, implementation details, and reproducibility conventions used throughout the paper. The goal is to make clear which results support the main claims, which results are mechanism or boundary evidence, and how the reported metrics are computed. A.1
Evidence Status and Reporting Protocol
The experiments in this paper are organized by evidence status. This distinction is important because the paper combines locked main comparisons, reduced 6×3090 updates, mechanism-oriented ablations, stress tests, and boundary pilots. Only the locked structured F RANKAC ABINET comparison is used for the primary sparse-task quantitative claim. Reduced-main LLM rows support candidate-family-dependent deployment behavior. Selector, compute-matched, scale, stress-test, and boundary rows are used to explain mechanisms, establish scope, or delimit the method. 11
RHyVE
A P REPRINT
Table 5: Evidence-status taxonomy used throughout the paper. Only the locked structured F RANKAC ABINET comparison supports the main sparse-task quantitative claim. Other categories explain mechanisms, establish scope, or delimit the method. Status
Purpose
Example role
locked_main_existing
Primary locked structured comparison used for the main sparse-task quantitative claim. Reduced-main LLM evidence supporting candidate-family-dependent deployment behavior.
Structured F RANKAC ABINET main result LLM F RANKAC ABINET reduced-main rows
Dev-selected schedule evaluated on held-out report seeds.
Held-out schedule audit
Conservative selector support evidence rather than full online-selector reproduction.
Periodic or moving-average selectors
Fairness-control rows with verification-equivalent extra training budget.
Compute-matched direct training
Mechanism-oriented top-up rows that clarify confounds without becoming main baselines.
Scale top-up controls
Optional extra-task pilot used only as scope or boundary evidence.
F RANKAC UBE S TACK all-failure pilot
Non-deployable reference separated from deployable methods. Scope-setting experiments outside the strongest claim regime.
Best adaptive oracle row
Older breadth or pilot evidence retained only for traceability.
Legacy breadth rows
Incomplete, flat, or blocked evidence not used for ranking claims.
Flat or blocked appendix entries
reduced_main_6x3090 heldout_test_ 6x3090 selector_support_ 6x3090 compute_matched_ 6x3090 appendix_support_ topup optional_extra_task_ pilot_6x3090 oracle_upper_ bound stress_tests historical_ legacy failed_or_ blocked
Candidate-pool scaling
For each experimental block, we retain a manifest that records the task, reward-hypothesis family, seed list, training budget, fork horizon, checkpoint set, switch rule, switch operator, metrics, and evidence status. The main paper reports only compact summaries. Full per-seed values, paired tests, and diagnostic tables are included in the appendix. Locked comparisons. A locked comparison is defined before aggregation by fixing the set of methods, seeds, reward family, evaluation protocol, and reporting metrics. For F RANKAC ABINET, the locked main comparison uses eight seeds and compares direct oracle deployment, one-shot oracle deployment, fixed warm-up schedules, and value-aligned PBRS. For BALL BALANCE, the locked boundary comparison uses the same dense-oracle reward family and reports peak saturation together with AUC and tail metrics. Support and stress experiments. Appendix-support experiments are not treated as additional main baselines. For example, reward-scale controls mix locked and mechanism-only evidence, and are therefore explicitly marked by evidence status. Candidate-pool scaling is reported as a stress test because it probes whether the small-candidate local-verification assumption continues to hold as K increases. A.2
Metric Definitions
The main paper reports both peak and tail metrics. This is necessary because a method can achieve a high transient success value and later collapse. We therefore recompute terminal and tail metrics from full learning curves whenever possible. Recomputed terminal metrics. Some result files contain summary fields produced by different logging paths. To avoid inconsistent terminal reporting, we recompute final and tail metrics directly from the full learning curves. The main text therefore uses recomputed final success, last-5 mean, and tail AUC when discussing terminal retention. Older summary fields are retained only for traceability and are not used as main evidence. 12
RHyVE
A P REPRINT
Table 6: Metric definitions. Peak metrics summarize whether a method ever reaches high performance, while tail metrics measure whether performance is retained near the end of training. Metric
Definition
Primary use
consec_max consec_auc
Peak performance Training efficiency
Last-5 mean Tail AUC
Maximum consecutive-success score observed during training. Area under the consecutive-success curve over the full training trajectory. Final success value recomputed from the full logged learning curve. Mean success over the final five evaluation points. Area under the success curve over the final training window.
Best-checkpoint success
Maximum evaluation success over all saved checkpoints.
Collapse indicator
Indicator for high best-checkpoint success with weak terminal retention. Peak absolute critic magnitude or critic-instability proxy. Absolute mean reward shift induced by a reward switch.
Recomputed final success
critic_peak_abs |∆r̄|
Terminal performance Tail stability Retained late-stage performance Best attainable checkpoint performance Detecting transient spikes Switching stability Reward shock / scale mismatch
Table 7: Task roles and reporting status. The main claims are based on F RANKAC ABINET locked results and BALL BALANCE boundary evidence. Other tasks are used for breadth, scope, or all-failure boundary evidence. Task
Regime
Role in paper
Evidence status
F RANKAC ABINET
Sparse, phase-sensitive manipulation Dense-oracle boundary
Main competence-aware verification and phase-aware deployment task Boundary evidence where warm-up is not meaningfully ranked by peak success Breadth and candidate-pool scaling boundary evidence Weak or fragile breadth evidence
locked_main_existing
BALL BALANCE C ART P OLE A NT F RANKAC UBE S TACK
Appendix breadth / stress task Appendix breadth / negativescope task Optional extra-task pilot
All-failure boundary; not used for method ranking
historical_legacy / boundary historical_legacy / stress_tests historical_legacy optional_extra_task_pilot_6x3090
Collapse rate. We define collapse as a diagnostic rather than a primary optimization objective. A run is marked as collapsed when it reaches a nontrivial best checkpoint but has low terminal retention. In our analysis scripts, the default rule is Icollapse = 1 [best_success ≥ τbest ∧ last5 ≤ τtail ] , where τbest and τtail are task-dependent thresholds. The thresholds are reported in the corresponding audit tables. Bootstrap intervals and paired tests. For aggregate tables, we report means, standard deviations, and bootstrap confidence intervals over seeds. For locked paired comparisons, we use exact sign-flip permutation tests when the number of paired seeds is small. We additionally report paired standardized effect sizes. Because the main sparsecontrol task exhibits high seed variance, we interpret statistical tests together with effect magnitude, tail metrics, and mechanism-consistent ablations. A.3
Tasks, Reward Families, and Seed Protocol
Table 7 summarizes the tasks and their role in the paper. F RANKAC ABINET is the main sparse, phase-sensitive manipulation task. BALL BALANCE is a dense-oracle boundary condition. C ART P OLE and A NT are appendix-only breadth or stress tasks. F RANKAC UBE S TACK is an optional extra-task pilot; because all compared methods failed to reach nontrivial success under the reduced optional budget, it is reported as an all-failure boundary rather than a method-ranking benchmark. Reward-hypothesis families. The main experiments use small candidate sets, typically K = 3. For F RANKAC ABI NET , the reward family contains an early dense bootstrap reward, a later task-faithful oracle-like reward, and a later alternative. For BALL BALANCE, the reward family contains an early speed-biased reward, an oracle-like position-speed reward, and a distance-based alternative. LLM-generated reward families are evaluated separately in the generalization and scope section of the appendix. 13
RHyVE
A P REPRINT
Table 8: High-level configuration summary. Exact low-level hyperparameters are stored in the released configuration files. Experiment block
Task
Candidate family
Main reporting role
Locked main comparison
F RANKAC ABINET
Small structured triplet.
Main phase-aware deployment result.
Boundary comparison
BALL BALANCE
Small structured triplet.
Dense-oracle boundary condition.
Fork verification sweep
F RANKAC ABINET, BALL - Same task-specific triplets. BALANCE
Reward-scale controls
F RANKAC ABINET
Early/late reward pair and scale Mechanism and confound analvariants. ysis.
Switch-operator grid
F RANKAC ABINET
Early/late reward pair.
Stability–retention trade-off.
Source generalization
F RANKAC ABINET, BALL - Hand-crafted and BALANCE, C ARTPOLE generated families.
LLM- Phenomenon generalization.
Pool scaling
F RANKAC ABINET, C ART- Expanded candidate pools.
Competence and fork-horizon reliability.
Stress test and scope.
POLE
Seed protocol.
The locked F RANKAC ABINET comparison uses the seed set {42, 123, 456, 789, 999, 1234, 2025, 3031}.
The locked BALL BALANCE boundary comparison uses a fixed subset of seeds from the same experimental suite. Some appendix-support experiments use smaller seed sets because they serve as diagnostic or stress evidence rather than main claims. Each table reports its own n. A.4
Training and Verification Configuration
The main experiments use actor-critic policy optimization with PPO-style updates Schulman et al. [2017]. The simulator and task implementation follow Isaac Gym-style GPU-accelerated control environments Makoviychuk et al. [2021]. Table 8 gives the high-level configuration summary. Exact hyperparameters and reward definitions are provided in the supplementary configuration files. Fork verification configuration. At each probed checkpoint zt , every candidate reward hypothesis is evaluated by cloning the same policy, critic, and optimizer state, training each clone for a short fork horizon, and scoring the resulting policies with a common downstream metric. The fork horizon is varied in the reliability analysis to test whether winner identity is robust to the verification budget. Checkpoint spacing. The checkpoint set is task-dependent. Sparse tasks use checkpoints that cover early, middle, and late training phases. Boundary tasks such as BALL BALANCE use earlier checkpoints because they reach high success much faster. The exact checkpoint list for each experiment is recorded in the manifest. Computational complexity. If M checkpoints are probed, K reward hypotheses are compared, and each fork uses horizon L, the verification overhead scales as O(M KL). The intended regime of RH Y VE is small K, sparse M , and short-to-medium L. Candidate-pool scaling experiments reported later in the appendix show that larger K increases both verification cost and ranking difficulty. A.5
Algorithms
For completeness, we give the procedural form of the two main operations used by RH Y VE. We write the algorithms in text form to avoid dependency on additional algorithm packages. Algorithm A1: Shared-checkpoint fork verification. 1. Input: candidate reward hypotheses H = {h1 , . . . , hK }, checkpoint set T , fork horizon L, evaluation function Eval. 14
RHyVE
A P REPRINT
2. For each checkpoint t ∈ T : (a) Load the shared learner state zt = (πt , Vt , Ωt ). (b) Compute or record the task competence proxy ct . (c) For each reward hypothesis hk ∈ H: i. Clone the learner state zt . ii. Continue training the clone for L updates using reward rhk . (L)
iii. Evaluate the resulting forked policy and record Jt (d) Compute the local winner
(hk ). (L)
ĥt = arg max Jt hk ∈H
(hk ).
(e) Compute the winner margin (L)
mt = Jt (L)
(f) Store (t, ct , ĥt , mt , {Jt
(L)
(ĥt ) − max Jt
(hj ).
hj ̸=ĥt
(hk )}K k=1 ).
3. Output: phase profile P. Algorithm A2: Phase-aware deployment. 1. Input: phase profile P, reward hypotheses H, optional switch-operator set. 2. Identify checkpoints that satisfy the verification-informative criterion, using margin, agreement, and entropy diagnostics. 3. If a single reward hypothesis wins consistently from the first informative checkpoint onward, deploy that reward throughout training. 4. If an early winner h(1) is later overtaken by a stable later winner h(2) , construct a two-stage schedule: r (1) t < ts , rt = h rh(2) t ≥ ts . 5. Choose the switch operator: (a) use hard switching as the default practical operator; (b) use value-aligned shaping only as a conditional mechanism; (c) use critic reset only as a diagnostic or trade-off operator. 6. If no stable phase structure is observed, abstain from aggressive switching and retain a conservative deployment rule. 7. Output: deployed training schedule and switch-operator choice. A.6
Artifact and Table-Generation Discipline
All paper-ready tables and figures are generated from CSV summaries that are themselves derived from per-seed raw logs. To avoid accidental mixing of reporting modes, each generated table records its source file and evidence status. Before submission, the following consistency checks are applied: 1. all main-text tables use only locked_main or explicitly marked boundary evidence; 2. all appendix-support and stress-test results are labeled as such; 3. no table mixes locked and appendix evidence without an explicit evidence-status column; 4. all terminal metrics in the main text use recomputed final or tail metrics rather than inconsistent summary fields; 5. all incomplete values are removed before final submission. Table 9 summarizes the main artifact groups used to generate the paper-ready results. 15
RHyVE
A P REPRINT
Table 9: Artifact groups used for paper-ready tables and figures. Paths are represented by logical artifact names rather than machine-specific absolute paths. Artifact group
Purpose
Experiment manifests
Define tasks, methods, seeds, reward families, switch rules, and evidence status.
Per-seed raw logs
Store learning curves, checkpoint metrics, critic diagnostics, and rewardshift statistics.
Locked result CSVs
Generate main comparison tables and locked paired tests.
Appendix-support CSVs
Generate reward-scale, switching, verification, trigger, and sourcegeneralization tables.
Stress-test CSVs
Generate candidate-pool scaling and compute–reliability plots.
Paper-ready table scripts
Convert source CSVs into LaTeX tables with evidence-status annotations.
Paper-ready figure scripts
Generate learning curves, heatmaps, trade-off plots, and reliability figures.
Table 10: First reliable checkpoint by task and fork horizon. Reliability depends on both policy competence and verification horizon. Task
B
Fork horizon First reliable checkpoint
BALL BALANCE BALL BALANCE BALL BALANCE BALL BALANCE
50 100 200 400
30 300 300 30
F RANKAC ABINET F RANKAC ABINET F RANKAC ABINET F RANKAC ABINET
50 100 200 400
1500 1500 1000 1000
Full Verification and Main-Task Results
This appendix provides the full evidence behind the main experimental claims. The following subsections expand the competence-limited verification analysis, report the locked F RANKAC ABINET results with per-method and statistical details, and document the dense-oracle BALL BALANCE boundary condition. B.1
Full Competence and Fork-Horizon Reliability
The main text summarizes the central verification phenomenon: reward rankings obtained from fork verification are not uniformly reliable throughout training. Here we provide the full checkpoint and fork-horizon results. For each task, we evaluate candidate reward hypotheses from shared checkpoints using multiple fork horizons. We record the modal winner, repeated-fork top-1 reliability, winner margin, margin lower confidence bound, and whether the checkpoint satisfies the verification-informative criterion. Verification-informative criterion. Unless otherwise stated, a checkpoint is treated as verification-informative when the modal winner is sufficiently stable across repeated forks and has a positive margin lower bound: Relt ≥ 0.75
LCB95 (mt ) > 0.01.
and
For F RANKAC ABINET, margins are small because success metrics are normalized, and the same criterion is used only as a diagnostic threshold rather than a universal rule. The important qualitative pattern is that early checkpoints are less reliable and later checkpoints become more informative under sufficiently long fork horizons. The F RANKAC ABINET grid supports two observations. First, the earliest checkpoint is not verification-informative despite occasionally producing a modal winner. Second, later checkpoints become informative only under sufficiently reliable fork horizons. This justifies the main text’s competence-aware interpretation: reward ranking is not a static property of the candidate set, but depends on the current learner and the verification budget. The BALL BALANCE grid shows that reward winner identity can still change across checkpoints, but the downstream peak metric is less discriminative because all main deployment methods reach the ceiling. We therefore use BALL BALANCE primarily to show that warm-up is not universally needed when the oracle-like reward is already dense. 16
RHyVE
A P REPRINT
Figure 4: Full fork-verification reliability. Each cell aggregates repeated forks from a shared checkpoint and reports modal winner identity, top-1 reliability, and winner margin. The reliable region appears only after sufficient policy competence and depends on fork horizon. Table 11: Full F RANKAC ABINET fork-verification grid. The table reports modal winner and top-1 reliability for each checkpoint and fork horizon. Margins and entropy are reported in the released CSV. Checkpoint Fork horizon
B.2
Modal winner
Top-1 reliability Margin mean Informative
500 500 500 500
50 100 200 400
cand_b7d2e641 cand_b7d2e641 cand_b7d2e641 cand_b7d2e641
0.75 0.50 0.50 0.50
0.00017 0.00007 0.00057 0.01552
no no no no
1000 1000 1000 1000
50 100 200 400
cand_bc49b15b cand_6ef57213 cand_6ef57213 cand_6ef57213
1.00 0.50 1.00 1.00
0.01419 0.00755 0.09161 0.08832
no no yes yes
1500 1500 1500 1500
50 100 200 400
cand_bc49b15b cand_bc49b15b cand_6ef57213 cand_6ef57213
0.75 1.00 0.50 1.00
0.05101 0.05602 0.02488 0.09381
yes yes no yes
2000 2000 2000 2000
50 100 200 400
cand_bc49b15b cand_bc49b15b cand_6ef57213 cand_6ef57213
0.50 0.50 0.50 1.00
0.04170 0.01654 0.00329 0.04448
no no no no
FrankaCabinet Locked Main Results
This section expands the locked F RANKAC ABINET comparison from the main text. The locked comparison uses eight seeds: {42, 123, 456, 789, 999, 1234, 2025, 3031}. All methods use the same task, reward-family protocol, training budget, and evaluation pipeline. Table 13 reports the aggregate peak results. Tail-metric audit. The main text reports recomputed final and tail metrics rather than older summary fields. Table 15 summarizes the tail audit. These values are computed from full learning curves and are used to distinguish transient peaks from retained performance. Additional last-5, tail-AUC, and collapse diagnostics are visualized in Figure 6 when available from the audit logs. The tail audit strengthens the main result. The wu=50 hard-switch schedule is not only the strongest peak method; it also improves recomputed terminal and tail metrics over direct and one-shot deployment. Therefore the main result is not merely a transient success spike. 17
RHyVE
A P REPRINT
Table 12: Full BALL BALANCE fork-verification grid. BALL BALANCE reaches high task competence quickly, and peak metrics saturate, making the task a boundary condition rather than a main performance benchmark. Checkpoint Fork horizon
Modal winner
Top-1 reliability Margin mean Informative
30 30 30 30
50 100 200 400
cand_bb_speed_bias cand_bb_speed_bias cand_bb_speed_bias cand_bb_speed_bias
0.75 0.75 0.50 0.75
54.10 52.59 39.96 26.16
yes no no yes
100 100 100 100
50 100 200 400
cand_bb_speed_bias cand_bb_speed_bias cand_bb_speed_bias cand_bb_speed_bias
0.50 0.50 0.50 0.50
30.01 19.16 26.27 25.82
no no no no
300 300 300 300
50 100 200 400
cand_bb_dist_only cand_bb_pos_speed cand_bb_pos_speed cand_bb_pos_speed
0.50 0.75 0.75 0.50
17.12 37.02 46.08 30.73
no yes yes no
500 500 500 500
50 100 200 400
cand_bb_dist_only cand_bb_dist_only cand_bb_pos_speed cand_bb_pos_speed
0.75 0.75 0.50 0.50
22.98 18.56 8.45 17.03
yes yes no no
Table 13: Locked F RANKAC ABINET peak results. wu=50 hard switching achieves the highest mean consec_max. Method
n consec_max mean
Direct oracle One-shot oracle Fixed wu=50 hard Fixed wu=100 hard wu=50 + PBRS
8 8 8 8 8
0.21498 0.22815 0.38894 0.23263 0.31341
Std.
95% CI
0.28109 0.27984 0.22560 0.19816 0.27950
[0.05100, 0.41471] [0.06691, 0.42390] [0.23248, 0.53150] [0.11569, 0.37862] [0.13502, 0.49579]
Table 14: Per-seed F RANKAC ABINET consec_max values for locked main methods. These values are included to make the seed variance visible. Method
42
123
456
789
999
1234
2025
3031
Direct oracle One-shot oracle Fixed wu=50 hard Fixed wu=100 hard wu=50 + PBRS
0.251 0.251 0.033 0.634 0.663
0.026 0.026 0.505 0.206 0.013
0.146 0.251 0.602 0.205 0.451
0.000 0.000 0.180 0.271 0.341
0.003 0.003 0.655 0.233 0.389
0.602 0.602 0.380 0.306 0.650
0.689 0.689 0.210 0.004 0.001
0.002 0.002 0.546 0.002 0.000
Table 15: Tail-metric audit for locked F RANKAC ABINET runs. Final values are recomputed from the full learning curves and are used to distinguish transient peaks from retained performance. Method
n Recomputed final ↑
Direct oracle One-shot oracle Fixed wu=50 hard Fixed wu=100 hard wu=50 + PBRS
8 8 8 8 8
0.0223 0.0216 0.1634 0.0893 0.0364
Paired tests. Table 16 reports paired tests for the main peak metric. Additional paired tests for final, last-5, tail AUC, and collapse diagnostics are generated from the paper-ready audit files. B.3
BallBalance Boundary Details
BALL BALANCE is used as a dense-oracle boundary condition. In the locked comparison, all methods reach the peak ceiling, so consec_max alone is not informative. The available locked boundary evidence shows that all methods reach the peak ceiling of 499. Therefore, this task is used to establish a dense-oracle boundary condition rather than to rank methods by peak performance. 18
RHyVE
A P REPRINT
Table 16: Paired sign-flip tests for F RANKAC ABINET consec_max. Effect sizes are moderate, but high seed variance prevents strong significance claims on the peak metric. Comparison
Mean diff.
wu50 vs direct wu50 vs one-shot wu50 vs wu100
0.174 0.161 0.156
p
dz
0.281 0.408 0.305 0.384 0.258 0.426
Figure 5: Per-seed learning curves for locked F RANKAC ABINET runs. The figure visualizes the high seed variance and shows why we report both peak and tail metrics. Table 17: Locked BALL BALANCE peak results. All methods saturate the peak metric, so this table should not be used for ranking methods. Method
n consec_max mean Std.
Direct oracle (wu=0) 5 Fixed wu=50 hard switch 5 Fixed wu=100 hard switch 5
C
499.0 499.0 499.0
0.0 0.0 0.0
Mechanism and Ablation Analyses
This appendix provides the mechanism and ablation evidence behind the main experimental interpretation. The following subsections study reward-scale controls, analyze switching operators as stability–retention trade-offs, and report selector and trigger ablations, including failure modes of reactive alternatives. C.1
Reward Scale and Phase-Composition Controls
A possible alternative explanation for the main result is that direct oracle deployment fails only because of reward-scale mismatch or reward shock. To test this, we compare direct oracle deployment, scale-normalized direct deployment, scale-matched direct deployment, reward interpolation, early-only training, late-only training, and the phase-aware wu=50 hard-switch schedule. 19
RHyVE
A P REPRINT
Figure 6: Tail retention diagnostics for locked F RANKAC ABINET runs. The wu=50 schedule improves recomputed terminal and tail behavior relative to direct and one-shot deployment. Table 18: Updated reward-scale top-up summary on F RANKAC ABINET. These rows are appendix_support_topup evidence and replace the older stale appendix scale aggregates. Method
n Max ↑
Std.
direct_oracle_running_norm direct_oracle_scale_matched_to_early early_dense_only late_reward_only
5 8 5 5
0.170 0.281 0.025 0.026
0.078 0.235 0.013 0.013
95% CI low 95% CI high Status 0.001 0.074 0.001 0.001
0.230 0.432 0.036 0.036
appendix_support_topup appendix_support_topup appendix_support_topup appendix_support_topup
Table 19: Locked F RANKAC ABINET switching summary. Final values use recomputed terminal metrics. No operator uniformly dominates across metrics; the wu=50 hard switch gives the strongest peak and recomputed final performance with low critic instability, whereas critic reset trades retention for high critic instability. Method
n Max ↑ Final ↑ Critic peak ↓
|∆r̄|
Direct oracle Fixed wu=50 hard Fixed wu=100 hard wu=50 + V-old PBRS Hard switch + critic reset
8 8 8 8 8
– 0.304 4.189 0.214 0.927
0.215 0.389 0.233 0.313 0.258
0.022 0.163 0.089 0.036 0.142
0.480 0.064 0.019 0.077 3.113
Table 18 retains only the updated scale-topup summary aligned with the revised main-text control semantics. We do not keep the older appendix tables that still reported stale scale-matched direct rows from the earlier appendix snapshot. The updated scale-control appendix still supports the same narrow interpretation: reward scale is a real confound, but scale correction alone does not subsume the phase-aware deployment result reported in the main text. Interpretation. These controls should be read as mechanism evidence rather than as additional main baselines. They rule out a simple explanation in which warm-up only acts as reward-scale normalization. Scale and shock matter, but the failure of early-only and late-only training shows that correct phase composition is also necessary. C.2
Switching Operator and Critic-Maturity Analysis
We next analyze how the reward switch is executed. The main paper emphasizes that switching safety is not solved by a universal operator. Here we provide the full switching summary and the operator-by-epoch grid. Table 19 reports the locked F RANKAC ABINET safe-switching summary. PBRS is not a universal improvement, and critic reset improves terminal retention in some settings but induces much larger critic peaks. 20
RHyVE
A P REPRINT
Figure 7: Reward-shift diagnostic. Reward-scale and reward-shock effects influence training stability, but they do not fully account for the phase-aware deployment gain.
Table 20: Switch operator × switch epoch grid on F RANKAC ABINET. Values are means over eight seeds. No operator uniformly dominates across timing and metrics. n Final ↑ Max ↑ Critic peak ↓
|∆r̄|
Switch epoch
Regime
Operator
50 50 50
immature immature immature
hard switch 8 0.1200 0.1845 PBRS 8 0.0892 0.2263 critic reset 8 0.0868 0.2959
0.5430 1.2868 2.4217
0.2073 0.2115 0.5279
500 500 500
sweet spot hard switch 8 0.0150 0.0182 sweet spot PBRS 8 0.0597 0.0607 sweet spot critic reset 8 0.1421 0.2582
0.4797 0.4877 3.1131
0.9122 0.9091 0.9268
hard switch 8 0.0776 0.4190 PBRS 8 0.1265 0.3103 critic reset 8 0.1466 0.3284
1.1653 1.2125 2.3612
1.2253 1.2513 1.2370
1000 1000 1000
later later later
Operator-by-epoch grid. Table 20 reports the multiseed switch-operator grid. We evaluate hard switching, PBRS with Φ = Vold , and critic reset at three switch epochs. The best operator depends on both the target metric and the switch timing. This reinforces the stability–retention interpretation: no switch operator dominates across all regimes.
Interpretation. The switch-operator results should not be interpreted as evidence for a universal safe-switching mechanism. Hard switching is the practical default in the locked setting because it combines strong peak and tail performance with low critic instability. PBRS can be useful in some regimes but is not robustly dominant. Critic reset can improve terminal retention, but its large critic peaks make it a trade-off rather than a default solution. 21
RHyVE
A P REPRINT
Figure 8: Switching operator heatmaps. Operator performance depends on switch timing and metric. PBRS is not a universal improvement, and critic reset can increase retention while producing high critic instability. Table 21: Updated selector support comparison on F RANKAC ABINET. These rows are selector_support_6x3090 evidence and replace the older small-n selector ablation table.
Method
n
Mean
CI low CI high No-switch rate Mean switches
conservative_periodic_selector direct_oracle_or_direct_best fixed_wu50_hard moving_average_selector naive_last_selector one_shot_early one_shot_oracle
8 8 8 8 8 8 8
0.0508 0.2150 0.3889 0.1127 0.0919 0.1252 0.0175
0.0143 0.0586 0.2362 0.0680 0.0521 0.0785 0.0068
0.0967 0.4019 0.5226 0.1595 0.1374 0.1752 0.0304
0.50 1.00 1.00 0.125 0.125 1.00 1.00
0.50 0.00 0.00 0.875 0.875 0.00 0.00
Table 22: Table 22: Selector failure taxonomy. The appendix records a fuller failure taxonomy than the representative failure patterns visualized in Figure 9. Failure mode
Description
Never switch
Selector fails to trigger a reward change despite later-phase evidence.
Switch without gain
Selector switches, but the chosen transition does not improve downstream performance.
No reliable checkpoint
Verification signal does not satisfy the reliability criterion.
Over-conservative trigger Trigger delays switching until the useful phase window has passed. Wrong reward identity
C.3
Selector commits to a candidate whose early local advantage does not translate to downstream utility.
Selector and Trigger Ablations
A natural question is whether fixed warm-up can be replaced by an automatic online selector or competence-triggered switch. To avoid keeping the older small-n selector pilot table alongside the revised main-text control framing, we retain only the updated 6x3090 selector-support comparison here. Table 21 summarizes the updated selector-support comparison. On F RANKAC ABINET, the fixed wu=50 hard switch remains stronger than the selector variants in the revised support comparison. Figure 9 visualizes the dominant selector failure patterns, while Table 22 records the fuller failure taxonomy used in the appendix diagnostics. 22
RHyVE
A P REPRINT
Figure 9: Selector and trigger diagnostics. The figure visualizes the dominant reactive-selector failure patterns: never switching, switching too late, or failing to find a reliable checkpoint. Table 23: LLM candidate-generation statistics. The table summarizes generated, parsed, valid, and executable candidates. These statistics are used to characterize candidate-source diversity, not to support main performance claims. Task F RANKAC ABINET BALL BALANCE C ARTPOLE
Generated Parsed Executable rate Eval family size Pool size Semantic diversity 105 108 108
105 108 108
0.905 0.787 0.972
3 3 3
10 10 10
10.76–34.41 11.02–12.10 6.88–13.47
Blocked BallBalance selector rows. Two BALL BALANCE selector rows remain documented as blocked appendix entries: the moving-average selector and a legacy globally tuned critic-loss threshold selector. They are not part of the locked main evidence and do not affect the boundary conclusion, since BALL BALANCE peak success saturates for all main methods. Interpretation. The selector ablations support the paper’s main positioning. RH Y VE is not strongest as an increasingly reactive scheduler. Its value lies in competence-aware verification and stable phase-aware deployment. In the sparse small-candidate setting, a simple fixed schedule can be more reliable than reactive selectors that depend on noisy early verification signals.
D
Generalization, Stress Tests, and Scope
This appendix reports evidence that delimits the scope of RH Y VE. The following subsections cover LLM-generated candidate families, cross-task breadth, candidate-pool scaling, and support-only or negative evidence retained for transparency. D.1
LLM-Generated Candidate Families
The main experiments use small structured reward-hypothesis families so that verification and deployment effects can be cleanly analyzed. To test whether the observed phase-dependence is limited to hand-crafted reward families, we also evaluate LLM-generated candidate pools. For each task, candidate rewards are generated using multiple prompt variants, temperatures, and replicates. For analysis, we distinguish a hand-crafted reference family and two updated LLM source families (lightly curated and minimally filtered). The hand-crafted reference family is reported in the locked main comparison and in the held-out detail table, while Table 24 focuses on the updated LLM source families. The purpose of this experiment is not to prove that a fixed warm-up schedule is universally optimal for LLM-generated reward pools. Rather, it tests whether phase-dependent winner changes also arise when candidate rewards are generated by an upstream language model. Table 24 reports the updated reduced 6x3090 LLM source-generalization summary, with deployable rows separated from oracle references by an explicit status field. Table 25 records the corresponding held-out schedule-selection details for the deployable source families. 23
RHyVE
A P REPRINT
Table 24: Updated reduced 6x3090 LLM source-generalization summary. Deployable rows and oracle references are separated explicitly so that the appendix does not preserve the older pilot-style completed-only summary as if it were current main evidence. Oracle rows are non-deployable references and are not guaranteed to upper-bound deployable rows when seed sets or comparison protocols differ. Candidate source
Method
n Max / consec_max ↑ Flip rate
llm_lightly_curated llm_lightly_curated llm_lightly_curated
Direct wu=50 hard Held-out selected
8 8 5
0.145 0.207 0.298
0.875 0.875 –
deployable deployable deployable
llm_lightly_curated
Best adaptive (oracle reference) 3
0.580
0.875
oracle
8 8 5
0.211 0.639 0.800
0.500 0.500 –
deployable deployable deployable
llm_minimally_filtered Best adaptive (oracle reference) 2
0.369
0.500
oracle
llm_minimally_filtered Direct llm_minimally_filtered wu=50 hard llm_minimally_filtered Held-out selected
Status
Table 25: Held-out schedule-selection details for the reduced 6x3090 source families. Non-deployable oracle references are recorded explicitly but kept separate from deployable held-out rows in the main text. Candidate source
Dev seeds
Test seeds
structured_handcrafted_K3 structured_handcrafted_K3 llm_lightly_curated llm_lightly_curated llm_minimally_filtered llm_minimally_filtered
42,123,456 42,123,456 42,123,456 42,123,456 42,123,456 42,123,456
789,999,1234,2025,3031 fixed_wu50_hard fixed_wu50_hard dev_prefers_fixed_wu50 789,999,1234,2025,3031 oracle_upper_bound best_adaptive_upper_bound non-deployable reference 789,999,1234,2025,3031 direct_oracle_or_direct_best direct_oracle_or_direct_best dev_prefers_direct 789,999,1234,2025,3031 oracle_upper_bound best_adaptive_upper_bound non-deployable reference 789,999,1234,2025,3031 fixed_wu50_hard fixed_wu50_hard dev_prefers_fixed_wu50 789,999,1234,2025,3031 oracle_upper_bound best_adaptive_upper_bound non-deployable reference
Selected schedule
Evaluated method
Notes
Figure 10: Left: Winner-flip rates across candidate sources. LLM-generated F RANKAC ABINET reward families exhibit phase-dependent winner changes, while dense or ceiling-prone tasks show little variation. Right: Method comparison across reward-candidate sources. The best deployment rule can vary across generated pools, so this experiment is used as phenomenon evidence rather than a fixed-schedule superiority claim. Takeaway. The source-generalization experiments support the hypothesis that phase-dependent reward utility is not an artifact of a hand-designed reward family. However, they also show why RH Y VE should be framed as a verification protocol rather than a fixed universal schedule: different generated pools can favor different deployment rules. D.2
Cross-Task Breadth
Table 26 summarizes the cross-task phenomenon scorecard. The goal of this table is not benchmark dominance. Instead, it records whether the main phenomena appear across tasks and where the method’s assumptions break down, including the optional F RANKAC UBE S TACK all-failure boundary. The cross-task results support a task-aware interpretation. RH Y VE is most useful when the task is sparse or phasesensitive, such that early and late rewards serve different roles. When a dense reward already provides sufficient guidance, or when reward rankings do not exhibit meaningful phase changes, warm-up may be unnecessary. 24
RHyVE
A P REPRINT
Table 26: Cross-task phenomenon scorecard. The strongest practical evidence is on F RANKAC ABINET; other tasks are used to establish boundary conditions and phenomenon breadth. Task
Competence threshold
F RANKAC ABINET BALL BALANCE C ART P OLE A NT F RANKAC UBE S TACK
task-dependent not limiting weak weak not meaningful
Winner flip
Warm-up helps
Evidence
Interpretation
observed yes locked_main_existing/reduced_main_6x3090 main sparse phase-sensitive evidence not central not meaningful (ceiling) historical_legacy dense boundary control fragile not reliable historical_legacy appendix breadth boundary fragile not reliable historical_legacy appendix breadth boundary not meaningful not meaningful optional_extra_task_pilot_6x3090 all-failure boundary; not used for method ranking
Table 27: Candidate-pool scaling stress test. Larger pools increase verification cost and can reduce top-1 reliability. This supports the small-candidate local-verification scope of RH Y VE.
Task
K Top-1 hit rate Hit@3 Rank corr. Overhead (s)
F RANKAC ABINET 3 F RANKAC ABINET 5 F RANKAC ABINET 10
1.00 1.00 0.50
1.00 1.00 1.00
0.833 0.950 0.911
1732 2909 5850
C ART P OLE C ART P OLE C ART P OLE
0.50 0.00 0.00
1.00 0.50 0.50
0.500 0.700 0.600
60 100 201
3 5 10
Figure 11: Left: Candidate-pool reliability versus pool size. Top-1 reliability weakens as the candidate set grows, especially on the lightweight stress task. Right: Verification overhead grows with candidate-pool size. This motivates using RH Y VE as a small-candidate local-verification protocol rather than exhaustive large-pool search. D.3
Candidate-Pool Scaling Stress Test
The main method is designed for small candidate sets. To test the boundary of this regime, we perform candidatepool scaling stress tests with K ∈ {3, 5, 10} on F RANKAC ABINET and C ARTPOLE. These results are marked as stress_tests and are not used for main performance claims. Takeaway. The pool-scaling experiment narrows the paper’s claim. RH Y VE is currently best supported for small candidate sets. Extending it to larger pools will likely require coarse-to-fine search, uncertainty-aware ranking, clustering, or more efficient candidate pruning. D.4
Additional Limitations, Blocked Runs, and Negative Evidence
We include negative and blocked evidence for transparency. These results are not used for main quantitative claims, but they clarify the boundary of the current method. Scope of the current evidence. The most defensible interpretation of the full experimental suite is as follows. RH Y VE identifies and operationalizes a real deployment issue for LLM-generated rewards: reward hypotheses may become reliable and useful only after sufficient learner competence. On sparse phase-sensitive tasks, this can make a simple phase-aware schedule stronger than direct or one-shot deployment. However, the method is not a universal adaptive scheduler, not a large-pool reward-search algorithm, and not a guarantee that warm-up improves every task. 25
RHyVE
A P REPRINT
Table 28: Support-only and negative evidence summary. These results are documented to clarify scope and prevent overclaiming. Evidence block
Status
Interpretation
BallBalance blocked selectors
documented, not rerun
Two selector rows remain blocked; they are appendixlevel support baselines and do not affect the locked BallBalance boundary conclusion.
Trigger-based switching
support only
Competence-triggered switching does not replace the fixed wu=50 schedule on F RANKAC ABINET.
Reactive selectors
support only
Selectors can fail by never switching, switching too late, or selecting unstable winners.
Candidate-pool scaling
stress test
Larger pools weaken top-1 reliability and increase compute cost.
Cartpole and Ant breadth
boundary / negative
These tasks show that phase dependence and warmup gains are task-dependent rather than universal.
Historical singleton runs
legacy only
Retained for traceability but excluded from main and support claims.
Table 29: F RANKAC UBE S TACK optional pilot boundary table. All rows are optional_extra_task_pilot_6x3090 evidence and are retained only to document the all-failure boundary. Method
n Max / consec_max ↑ Final ↑ Last-5 mean ↑ Full AUC ↑ Tail AUC ↑ Collapse rate
direct_late_task_faithful 3 fixed_wu100_hard 3 fixed_wu50_hard 3
Recommended use.
0.0000 0.0028 0.0001
0.0000 0.0000 0.0000
0.0000 0.0000 0.0000
0.0000 0.000084 0.0000003
0.0000 0.000005 0.0000
0.0 0.0 0.0
Status optional_extra_task_pilot_6x3090 optional_extra_task_pilot_6x3090 optional_extra_task_pilot_6x3090
In its current form, RH Y VE is most appropriate when:
1. a small set of plausible reward hypotheses is available; 2. the task is sparse or phase-sensitive; 3. early reward comparisons are suspected to be unreliable; 4. the practitioner can afford sparse shared-checkpoint fork verification; 5. the deployment goal is to choose a stable phase-aware schedule rather than continuously chase a reactive selector. When the oracle-like reward is already dense, or when the candidate set is too large for reliable local ranking, RH Y VE should be used as a diagnostic rather than as an automatic deployment rule. D.5
FrankaCubeStack Optional Pilot
The reduced optional-scope F RANKAC UBE S TACK pilot is retained only as all-failure boundary evidence. It is not used for method ranking, and it does not change the main task-aware interpretation of the paper.
26
RHyVE
A P REPRINT
Figure 12: F RANKAC UBE S TACK optional pilot: all-failure boundary. All compared methods remain effectively flat under the reduced optional-scope budget, so this pilot is retained only as boundary evidence and not used for method ranking.
27