arXiv:2605.20164v1 [cs.AI] 19 May 2026
Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR Utkarsh Tyagi1 , Xingang Guo1 , MohammadHossein Rezaei1 , Daniel George2 , Anas Mahmoud1 , Jackson Lee1 , Bing Liu1 , Yunzhong He1 1 Scale AI, 2 Persona
Abstract Reinforcement learning with verifiable rewards has made post-training highly effective when correctness can be checked automatically. However, many important model behaviors require satisfying several qualitative criteria at once. Rubric-based rewards address this setting by grading prompt-specific criteria and aggregating them into a scalar reward. Yet standard static aggregations conflate a criterion’s human-assigned importance with its current usefulness as an optimization signal. We show that this assumption breaks down in rubric RL: many important criteria are already saturated or currently unreachable, while criteria that distinguish rollouts are not necessarily those with the largest human weights. We introduce POW3R, a policy-aware rubric reward framework that preserves human weights and category balance as the rubric objective while adapting criterion-level reward weights during training. POW3R uses rollout-level contrast to emphasize criteria that currently separate the policy’s outputs, making the GRPO reward more informative without changing the underlying evaluation target. Across three base policies on two datasets spanning multimodal and text-only settings, POW3R wins 24 of 30 base-policy/metric comparisons, improving both mean rubric reward and strict completion (the fraction of prompts whose response satisfies every required rubric criterion) over vanilla GRPO with rubric rewards, and reaches the same plateau in 2.5–4× fewer training steps. Rubric rewards should therefore distinguish what should matter in the final answer from what can teach the current policy.
1 Introduction Reinforcement learning with verifiable rewards (RLVR) has become a central recipe for post-training language models on tasks where success can be cheaply and reliably checked. Group-relative methods such as GRPO have made this practical at scale by replacing a learned value model with within-prompt rollout comparison [1–3]. The strength of the recipe is also its limitation: it works best when target behavior can be reduced to a single outcome score, and recent RLVR diagnostics and imperfect-verifier analyses already document that scalar rewards can hide heterogeneous failure modes and noise [4, 5].
1
Scale AI Research
(b) Pressure moves to contrastive rubrics
(a) Signal state by static weight MM
HB 100%
17
18
18
51
52
50
50%
31
0%
30
Low 1--3
24
24
Sat.
Mixed
59
55
55
17
20
21
Low 1--3
Mid High =4 ≥ 5
31
Mid High =4 ≥ 5
26
Dead
Training pressure (%)
Criteria
100%
MM
HB
zero-signal −8 pp
zero-signal −12 pp
(c) Pre-standardization reward spread widens gray: static
color: ours
Qwen
Gemma
+25%
MM-Qwen
+30%
MM-Gemma
50% HB-Qwen
+34%
HB-Gemma
+37%
0% Static
Ours
Static weight/point tier |wj|
Static
Ours
0.1
0.2
0.3
Prompt reward spread std(R)
Figure 1. Rubric-pressure diagnostic. We track each criterion’s training pressure1 across criterion signal states. A criterion is dead when no rollout passes it (p j =0), saturated when every rollout passes it (p j =1), and mixed when verdicts differ across the rollout group; dead and saturated criteria have v j =0 and give no group-relative advantage signal. (a) Criteria grouped by setting and absolute static weight/point tier (Low: |w j | ∈ {1, 2, 3}; Mid: |w j | = 4; High: |w j | ≥ 5); high-weight criteria still carry substantial dead and saturated mass, so human importance is not the same as current learnability. (b) Within-category pressure before and after policy-aware reweighting, averaged over the two base policies per setting; pressure on zero-signal criteria drops by 8–12 pp. (c) Prompt-level reward spread (rollout std( R) before GRPO standardization), static (gray) vs policy-aware (colored). Mean spread widens by 25–37%, so fewer groups collapse to the tied std( R)=0 regime where every advantage is zero. MM/HB denote our multimodal dataset and HealthBench.
Many important behaviors do not collapse cleanly onto one outcome score. Long-form medical advice, scientific writing, coding help, and visually grounded reasoning are inherently multi-dimensional: a good answer must be factually correct, complete, faithful to evidence, well-formatted, and on-instruction at the same time. Expert rubric grading exposes this finer structure where exact-answer scoring is silent [6], and recent multimodal work documents that final-answer rewards can leave perception and grounding undertrained, with models sometimes “reasoning past the image” rather than from it [7, 8]. Open-ended quality is, in short, vector-valued; pushing post-training beyond strictly verifiable domains requires rewards that expose that vector rather than collapse it. Rubrics provide that structure. A rubric decomposes response quality into prompt-specific criteria, each independently scored by an LLM judge, and rubric-based rewards have become a practical way to extend RL post-training beyond strictly verifiable domains [9], with text-only and multimodal rubric pipelines both growing rapidly [10–12]. Rubrics, however, change the nature of reward design: it is no longer a verification problem but an aggregation problem, since GRPO still requires a single scalar reward per rollout [1] and every rubric criterion must eventually be folded into one number. The common operational answer is a static weighted sum across rubric items [6, 9]. This is convenient but contains a hidden assumption: that the human-assigned weight of a criterion expresses both its desired importance in the final answer and its current usefulness as a training signal. The two are not the same. Under group-relative RL with outcome supervision, a criterion that every rollout passes, or that no rollout passes, adds the same constant to every reward and cancels out of the advantage; only criteria whose pass rate sits between the extremes can teach the current policy. A high-weight criterion can therefore be important for evaluation while still producing no gradient signal right now. This is 1We define criterion j’s training pressure as its within-category reward-weight share: w /W ( q ) under the static reward, κj j
(t) (t) w̃ j /W̃κ j (q) under POW3R (Eqs. (3) and (8)).
2
Scale AI Research
the rubric-level form of a broader fixed-scalarization issue in multi-reward RL, where static weights preserve a target preference but route learning effort poorly across objectives [13]. We test this assumption directly. Using two frozen base policies, Qwen3-VL-4B-Instruct [14] and Gemma 3 12B-IT [15], we sample rollout groups on 1,300 prompts drawn from our multimodal dataset (MM; Section 5) and HealthBench English (HB) [6], judging every rubric criterion on every rollout with GPT-5.4-mini [16].2 For each criterion j we record its absolute weight |w j |, pass rate p j , variance v j = p j (1− p j ), and training pressure. The pattern is consistent across both policies and both settings (Fig. 1). Roughly half of all rubric criteria are non-contrastive for a fresh policy: 17–26% are saturated and 20–33% are dead, leaving only the remaining half able to produce a contrastive gradient. The static aggregation therefore routes 45–51% of within-category training pressure to criteria that cannot move the policy, and the problem is not confined to low-importance criteria: human weight and rollout variance are essentially uncorrelated, and roughly half of the highest-weight criteria already carry v j =0. These shares change by only a few points across the four (model, dataset) combinations, so what we are seeing is a property of static aggregation, not of any single base policy or domain. Static weights tell us what should matter in the final answer, not which criteria can teach the current model. More useful pressure → clearer rewards
300
Reward spread gain (%)
The diagnostic gives a direct design rule: preserve the evaluation rubric as the target, but route within-category pressure toward criteria that currently distinguish rollouts. This follows the multi-objective view that scalarization can be a training-time choice rather than only a fixed preference statement [17], and it complements multi-reward GRPO work showing that naive normalization can erase objective-specific signal [18]. Our Policy-Aware Rubric Reward framework, POW3R, implements this rule on top of the standard rubric reward: (i) it measures each criterion’s rollout contrastiveness from the smoothed standard deviation of its judge verdicts, (ii) blends and clips this signal into a bounded factor so saturated and uniformly failed criteria keep a learning floor while contrastive ones receive more pressure, and (iii) renormalizes within each rubric category so that the human weight prior and category mass remain intact. Offline replay confirms the local mechanism: POW3R moves pressure off dead and saturated criteria and widens the prestandardization rollout reward spread (Fig. 1b,c), and Fig. 2 verifies the same effect prompt by prompt.
200
100
0 0
20
40
60
Weight moved off no-signal criteria (%)
Figure 2. Mechanism check. Each gray point is one prompt under one diagnostic run: Qwen3-VL-4B-Instruct or Gemma 3 12B-IT on our multimodal dataset (MM) or HealthBench English (HB). Moving weight off criteria that all rollouts pass or fail separates the rollout rewards more; reward spread is the standard deviation of rollouts before GRPO standardization. The trend holds prompt-by-prompt in every run.
Key contributions. (i) We introduce a rubric-pressure diagnostic that exposes how static rubric aggregation routes training pressure, and use it to show that human-assigned importance and current policy learnability decouple in rubric RL. (ii) We propose POW3R, a policy-aware rubric reward that preserves human weights and category balance while reallocating within-category training pressure to currently informative criteria. (iii) Under the GRPO recipe across three base policies on each of MM and HealthBench, POW3R beats binary, static-scalar, and category-balanced rewards on 24 of 30 comparisons, matches them in 2.5–4× fewer steps, and preserves external VLM benchmark scores. 2 The judge–effort combination is selected from a cost–quality calibration against a high-effort reference judge; the full
agreement table is in Section B. 3
Scale AI Research
2 Related work Rubric-based rewards and the policy-aware view. Rubric-based rewards extend RL post-training beyond deterministic verifiers by decomposing response quality into prompt-specific criteria scored by an LLM judge [9]. Expert-written rubric benchmarks scale this signal in medicine [6] and have begun to reach other modalities such as multi-turn spoken dialogue [19], while synthetic or semi-automatic pipelines reduce rubric-authoring cost [10, 11]. Other work modifies the rubric set during training: Rezaei et al. [20] elicit rubrics from pairwise comparisons, Shao et al. [21] co-evolve rubrics with the policy for long-horizon generation, and Jia et al. [12] generate multimodal rubric rewards from successful trajectories. Closest to ours is Chen et al. [22], which stratifies generalized rubrics into a perception-toreasoning curriculum and dynamically reweights them across training; we share the diagnosis that not all criteria are equally learnable at every stage, but our rubrics are prompt-specific and human-authored, we preserve human-assigned importance via static within-category weights, and we derive dynamic factors per prompt from the current policy’s rollout variance rather than from a global capability schedule. Multi-reward RL and multimodal RLVR. Several lines treat alignment as multi-objective rather than scalar optimization [13, 17, 23], complementing RLHF/RLAIF recipes that compress rich human or AI feedback into a single reward target whose scalar form can hide heterogeneous values and failure modes [24–29]. Liu et al. [18] show that naively normalizing multi-reward rollouts under GRPO collapses distinct reward combinations into identical advantages. On the multimodal side, RLVR extends GRPOstyle post-training to vision-language reasoning [30–33] and adds visual perception rewards, evidence gates, dense spatial rewards, or token-level reweighting when final-answer signals underfit perception [7, 34–36], while complementary benchmarks evaluate tool-enabled image perception, transformation, and reasoning under a unified protocol [37]. RLVR diagnostics in parallel show that observed gains can reflect spurious signals rather than newly learned capabilities, motivating inspection at the criterion level [4, 38].
3 Preliminaries 3.1 Group relative policy optimization We post-train policies with Group Relative Policy Optimization (GRPO) [1], the algorithm underlying recent reasoning-RL recipes [2, 3]. For each prompt q ∼ P( Q), GRPO samples a group of G outputs {o1 , . . . , oG } from the old policy πθold and optimizes the policy by maximizing "
JGRPO (θ ) = Eq, {oi }∼πθ
|o | 1 G 1 i n min ri,t Âi,t , ∑ ∑ (·| q ) old G i =1 | o i | t =1
clip(ri,t , 1−ε, 1+ε) Âi,t − β DKL πθ ∥ πref
o
# , (1)
where ri,t (θ ) = πθ (oi,t | q, oi,<t )/πθold (oi,t | q, oi,<t ) is the per-token probability ratio. Writing ui,t = πref (oi,t | q, oi,<t )/πθ (oi,t | q, oi,<t ) for the reference-to-policy ratio, the per-token Schulman k3 estimator is DKL πθ πref = ui,t − log ui,t − 1. (2) We use outcome supervision: a scalar reward R(oi ; q) is assigned to each output oi , standardized within the group, and the resulting Âi,t = ( R(oi ; q) − mean(R))/std(R) with R = { R(o j ; q)}G j=1 is broadcast to 4
Scale AI Research
every token in oi . When std(R)=0 (all rollouts tied) we set Âi,t =0, so the group contributes no gradient that step – a regime the Section 1 diagnostic shows is reached on a non-trivial share of prompts under static aggregation. The construction of R(oi ; q) is the focus of this paper.
3.2 Rubric-based rewards A rubric-based reward decomposes response quality into prompt-specific criteria scored by an LLM N (q) judge [6, 9]. For a prompt q we write its rubric set as R(q) = {(c j , w j , κ j )} j=1 , where each criterion c j has a static human weight w j ∈ N>0 and a category label κ j ∈ {1, . . . , K }; let Ck (q) = { j : κ j = k}. The grader produces s j (o, q) ∈ [0, 1], with s j (o, q) = 1 meaning response o satisfies c j . The standard rubric N (q)
reward used in prior work is the static weighted sum Rscalar (o; q) = ∑ j=1 w j s j (o, q). This sum bakes in three implicit assumptions: (i) categories contain comparable numbers of criteria; (ii) criteria within a category are similarly informative under the current policy; and (iii) each w j expresses both end-state importance and current training usefulness. The next section relaxes (i)–(iii) while leaving s j (·, ·) and the human weights w j unchanged.
4 Method POW3R changes only the reward aggregation before GRPO standardization, keeping the rubric, judge scores, and human weights fixed while reallocating within-category pressure toward criteria that distinguish the current rollout group. Category-normalized baseline. For each prompt, let Wk (q) = ∑ j∈Ck (q) w j and Kq be the number of populated categories. Define Rcat (o; q) =
1 1 w j s j (o, q). ∑ Kq k: C (q)̸=∅ Wk (q) j∈C∑(q) k
(3)
k
(t)
Policy-aware factors. Each prompt-rubric factor α j starts at 1 and is applied to all G rollouts in epoch t; after the epoch, judge calls yield each criterion’s pass rate and variance, (t)
pj
=
1 s j ( o i , q ), n j i∑ ∈V
(t)
vj
1 ( t ) 2 . s j ( oi , q ) − p j ∑ n j i∈V
=
j
(4)
j
with V j the valid-verdict set and n j =|V j | (criteria with < ⌈0.75G ⌉ valid verdicts retain their previous factor). POW3R then smooths the variance, category-normalizes it, blends toward 1, clips, and EMAupdates: (t) gj
r
=
(t)
(t) v j + ϵ,
(t) ḡk
=
∑ j′ ∈C valid (q) w j′ g j′ k
∑ j′ ∈C valid (q) w j′
,
(5)
k
(t)
gj
=
( t +1)
(t) (t) = clip (1 − β ema )α j + β ema α̂ j , αmin , αmax .
αj
, (t)
ḡκ j
(t)
(t) = clip (1 − λ) + λρ j , αmin , αmax ,
(t) ρj
α̂ j
(t)
(6) (7)
If all valid signals in a category vanish, POW3R sets α̂ j =1; λ trades off prior vs. rollout contrast, β ema sets response speed, [αmin , αmax ] bound deviation from 1, and ϵ>0 stabilizes ratios. 5
Scale AI Research
(t)
(t)
(t)
(t)
POW3R reward. At epoch t, set w̃ j = w j α j and W̃k (q) = ∑ j∈Ck (q) w̃ j , then compute (t)
Rdyn (o; q) =
1 1 (t) w̃ s j (o, q). ∑ Kq k: C (q)̸=∅ W̃ (t) (q) j∈C∑(q) j k
k
(8)
k
Equation (8) keeps category mass uniform and uses w j as prior: if factors in a category are equal, we get (t)
Eq. (3). The same {α j } are used for all G rollouts and fed into GRPO, requiring no optimizer change.
5 Experimental setup Datasets. We choose datasets that expose criterion-level categories and static importance weights rather than only a single outcome label [6, 9]. HB is HealthBench [6] restricted to English-language prompts, with native physician-authored point-valued criteria; we use HealthBench’s 500-task hard subset as the test split and a separate 10% slice of the remaining English training prompts as the dev split. More details on HealthBench in Section A.4. MM is our 10k-task multimodal dataset, selected from a contributor-authored prompt pool because existing rubric-RL datasets do not simultaneously provide complex images, prompt-specific categories, static weights, and enough scale for generalisability. Each MM task pairs an image with a prompt and a rubric set spanning six quality categories (Table 1); the images span charts, diagrams, photos, screenshots, and natural scenes, and each rubric criterion is anchored to specific visual elements or prompt instruction during authoring. Fig. 3 illustrates the shared rubric-RL setting, and Section A.1 gives annotation details.
Table 1. Multimodal rubric-RL dataset statistics. The six rubric categories are the highlevel quality dimensions used by the reward. Tasks Total / train / dev / test Single / Multi turn
10K / 8K / 1K / 1K 65% / 35%
Rubrics Total criteria Mean / median per task Human weight w j
84,403 8.4 / 7 integer in {1, . . . , 5}
Rubric category (share, w̄) Visual perception Visual reasoning Content completeness Instruction following Truthfulness Style / presentation
32.4%, 3.40 26.0%, 3.74 16.9%, 3.10 11.4%, 3.77 10.5%, 3.58 2.8%, 2.30
Models. On MM we post-train three vision-language base policies: Qwen3-VL-4B-Instruct and Qwen3VL-8B-Instruct [14]3 , and Gemma 3 4B-IT [15]4 . On HB we post-train three text-only base policies: Qwen3-4B-Instruct-2507 and Qwen3-8B [39], and Gemma 3 4B-IT [15]. The diagnostic of Section 1 additionally uses Qwen3-VL-4B-Instruct and the larger Gemma 3 12B-IT to check that the findings are not specific to a single base model. Reward judging. The reward judge is queried per rubric criterion: every (prompt, rollout, criterion) triple gets a reasoning-then-verdict call, returning a one-sentence rationale and a binary {0, 1} judgment for aggregation. Training rewards use GPT-5.4-nano with medium-effort reasoning and explanations; held-out evaluation responses are re-scored by GPT-5.4-mini with the same reasoning setting to reduce judge–training entanglement [16]. Section B gives the cost–agreement calibration and shows why verdictonly and per-category batched judges were not used. Both judges run at temperature T =1.0 with up to 2048 completion tokens; the system prompt is reproduced in Section C. Baselines. We use POW3R for the framework and Rdyn for the scalar reward it sends to GRPO. We compare five post-training settings, all using the same rubric set and judge. (i) Base model: 3 Qwen release pages: Qwen3-VL; Qwen3. 4 Gemma 3 release page: Gemma 3.
6
Scale AI Research
HealthBench Task
Custom Multimodal Task
Prompt Hey I'm taking this cough syrup every morning for a bad cough but I also have an energy drink every day I'm worried if that's bad to mix I feel a bit jittery should I be concerned.
Prompt I found this receipt today. By looking at it, where can it be assumed that the purchase was made? Also, list the visible articles from cheapest to most expensive. Analyze this consumer’s behavior based on the receipt.
Model Response Let’s break this down step by step to help you understand what might be happening and whether you need to worry. ### 1. **What’s in your cough syrup?** Most over-the-counter cough syrups contain: … ### 5. **What Should You Do?** **Check the labels!** Look at both the cough syrup and the energy drink to see if either contains caffeine or other stimulants...
Model Response Let's analyze the receipt step by step. ### 1. Where can it be assumed that the purchase was made? Looking at the receipt, we can see: **Payment Method**: … ### 2. List the visible articles from cheapest to most expensive. … ### 3. Analyze this consumer's behavior based on the receipt. …
Rubric Criteria (with weight and category)
Rubric Criteria (with weight and category)
W=7
Context Awareness
Recommends checking the exact ingredients on the cough syrup label …
Pass
W=4
Visual Reasoning
The response must attempt to determine where the purchase … geographical data.
Fail
W=9
Completeness
Advises the user to modify caffeine intake to see if it improves their sensation of jitteriness.
Pass
W=5
Visual Perception
The response must list the visible items by stating the clearly shown products …
Fail
…
…
…
…
…
…
…
Completeness
Recommends seeking medical evaluation … or severe anxiety/panic.
Instruction Following
The response must arrange the visible articles from cheapest to most expensive.
Pass
…
W=9
W=4
Fail
Figure 3. Illustrative tasks from each rubric-RL setting. Each task carries a prompt, a sampled response, and a prompt-specific rubric checklist; MM includes image input, while HealthBench is text-only. Reward judging uses this same criterion-level interface: each listed criterion receives an independent binary verdict before aggregation.
the un-trained checkpoint, used as the no-RL reference. (ii) Binary: a sparse all-or-nothing reward, Rbinary (o; q)=1[every required criterion passes] on MM and Rbinary (o; q)=1[HealthBench scorer=1] on HB; included as the exact-answer-style RLVR baseline. (iii) Static scalar: the standard prior-work weighted sum Rscalar from Section 3.2. (iv) Category-balanced: the static category-balanced reward Rcat from Eq. (3). (v) POW3R dynamic: Rdyn from Eq. (8). Each reported trained setting averages three completed runs under the same split, decoding, and evaluation protocol. The trained settings all run the same GRPO objective (Section 3.1); only the rubric aggregation changes between them. Evaluation. At evaluation time, each completed policy is decoded on held-out prompts and every response is re-scored by the held-out judge. We report mean rubric reward on the 1,000-task MM test set and HealthBench’s 500-task hard test split. For MM the rubric reward is the static weighted aggregation Rscalar from Section 3.2 normalized to 0–100%, applied uniformly across all five reward constructions so the evaluation target is held fixed; for HB we use the HealthBench’s official scoring script. We also report strict completion – the fraction of prompts whose response satisfies every criterion flagged as required in the rubric, and per-category mean pass rate. Rubric reward measures average quality under the rubric; strict completion measures all-required-criterion success with no partial credit. Transfer benchmarks (MM only). To check that POW3R does not over-fit the rubric judge, we also evaluate the trained MM policies on six external VLM benchmarks: HallusionBench [40], POPE [41], MM-IFE [42], MMVetV2 [43], MathVista [44], and RealWorldQA [45]. Configuration. GRPO runs with G =16 rollouts per prompt-group, sampling temperature T =1.0, and a maximum completion length of 3584 tokens. We use a learning rate of 3 × 10−7 , KL coefficient β=0.1, clip range ε=0.2, and max_grad_norm=0.5, with a per-device batch size of 1 and 4 gradient-accumulation steps under DeepSpeed ZeRO-3 [46] in BF16 with gradient checkpointing. All training runs use one node with 8×H100 GPUs and run for up to 664 GRPO steps. The dynamic-factor parametrization (Eqs. (5)–(7)) uses αmin =0.67, αmax =1.5, ϵ=10−4 , smoothing weight λ=0.5, EMA coefficient β ema =0.2, and minimum valid rollout fraction 0.75 for the completed POW3R dynamic run. 7
Scale AI Research
Table 2. Held-out evaluation on MM test and selected external VLM benchmarks. Entries are percentages: rubric reward is the mean held-out rubric score with partial credit, while strict completion is the fraction of prompts whose response satisfies every required criterion. External columns report HallusionBench, POPE, MM-IFE, MMVetV2, MathVista, and RealWorldQA. Best entries within each base-policy block are in bold and shaded. MM test Base policy
Setting
VLM benchmarks
Rubric Strict comp. HallusionBench POPE MM-IFE MMVetV2 MathVista RealWorldQA
Base Binary Qwen3-VL-4B Static scalar Category-balanced POW3R dynamic
42.6 42.8 47.1 47.9 48.8
14.7 14.6 17.9 18.7 20.2
62.1 62.0 62.2 62.5 62.4
88.8 88.6 89.2 89.0 89.1
11.5 11.8 12.5 13.0 13.5
49.9 49.8 51.0 51.8 52.2
71.0 71.2 72.2 72.8 73.2
71.9 71.7 72.4 72.8 73.1
Base Binary Qwen3-VL-8B Static scalar Category-balanced POW3R dynamic
46.8 47.0 49.5 50.4 51.6
18.4 18.3 20.7 21.6 22.6
62.4 62.5 62.2 62.6 62.9
88.4 88.2 88.6 88.7 88.9
12.5 12.3 13.5 14.2 14.8
49.7 49.5 51.0 51.8 52.6
73.6 73.8 74.1 74.8 74.6
68.6 69.0 69.4 69.7 70.2
Base Binary Static scalar Category-balanced POW3R dynamic
34.1 33.9 35.8 36.7 37.8
7.7 7.6 8.5 9.4 10.3
53.9 54.0 53.4 54.1 54.4
82.6 82.5 82.7 82.8 83.0
6.0 6.1 6.3 6.7 7.2
38.9 38.8 39.5 40.2 40.8
47.5 47.6 48.0 49.8 49.6
51.0 50.8 51.4 51.8 52.3
Gemma3-4B
6 Results and analysis 6.1 Main results Tables 2 and 3 compare POW3R with the base model, Table 3. HealthBench English test split. Entries are binary reward, static scalar reward, and category- percentages except ∆, which is already in pp. Best balanced reward under the same GRPO setup. Our cells within each base-policy block are shaded. key findings are: Base
Setting
1. POW3R is the strongest reward on the main Base Binary rubric objectives across both datasets. Across the two main-results tables, POW3R achieves the best Qwen3-4B Static scalar Category-balanced score on 24 of the 30 base-policy/metric comparPOW3R dynamic isons, sweeping every MM rubric-reward and strictBase completion column in Table 2 and every HealthBinary Bench overall-reward column in Table 3. The six Qwen3-8B Static scalar Category-balanced non-POW3R cells are split between external VLM POW3R dynamic benchmarks (which we do not train against directly; Base 4 of 6) and the HB strict perfect-score column, where Binary POW3R is best on Qwen3-4B but trails the static or Gemma3-4B Static scalar Category-balanced category-balanced reward by 0.1–0.2 pp on Qwen3POW3R dynamic 8B and Gemma3-4B.
Overall
∆
Strict
28.0 27.8 29.6 30.5 32.7 27.8 27.9 29.9 30.7 31.5 24.3 24.2 25.4 27.1 28.1
– −0.2 +1.6 +2.5 +4.7 – +0.1 +2.1 +2.9 +3.7 – −0.1 +1.1 +2.8 +3.8
4.2 4.0 3.8 4.6 5.4 4.4 4.4 4.5 4.8 4.6 2.4 2.2 2.8 2.6 2.7
2. Per-rubric-category analysis on MM: POW3R’s gain is consistent across categories, with the largest jumps on the contrastive ones. A separate per-category analysis (see Fig. 5 for the full trajectories) shows that on Qwen3-VL-4B, POW3R leads on every rubric category for the full training schedule. The biggest gaps over the static baselines appear on Visual Perception, Visual Reasoning, Truthfulness, Content, and Instruction Following; on Writing Style the three rewards stay within roughly a point of 8
Scale AI Research
(a) Two-objective dominance on MM
Strict pass-rate (%)
20
16
Gemma3-4B Base Binary Static scalar Category-balanced POW3R dynamic
8
Δ test rubric reward vs base (pp)
Qwen3-VL-4B
12
(b) Rubric-reward gain over base
Qwen3-VL-8B
23
MM (vision)
7
HB (text-only)
+6.2 6
+4.8
5
+4.7 +3.7
+3.8
Q8B
G4B
+3.7
4 3 2 1 0
35
40
45
50
Q4B
Q8B
G4B
Q4B
Test set rubric reward (%) Static scalar
Category-balanced
POW3R dynamic
Figure 4. Main result visual summary. (a) MM test rubric reward and strict completion for each reward construction; lines connect methods trained from the same base policy. (b) Test rubric-reward gain over the corresponding base model on MM and HB.
each other because most Writing Style criteria are already passed by the base policy. This is by design: POW3R concentrates pressure where the rollout group exposes learnable disagreement, and reduces to the static baseline on categories with no remaining contrast to exploit. 3. Two-objective dominance view separates mean quality from all-criteria success. Figure 4a places each method in the MM test rubric-reward/strict-completion plane; POW3R is the top-right endpoint of every base-policy line, so it Pareto-dominates the other four constructions on both objectives. This matters because a higher mean rubric score can still leave prompts with one failed required criterion; the strict-completion axis tests whether partial-credit gains turn into complete rubric satisfaction. 4. ∆ vs base across all six base policies shows cross-setting consistency. Figure 4b summarizes test gains in both modalities: Rdyn > Rcat > Rscalar > 0 in all six setting/base-policy combinations, and the smallest POW3R gain is still +3.7 pp. The ordering is unchanged between the multimodal and text-only settings, suggesting that POW3R is not exploiting one dataset’s rubric convention.
6.2 Training efficiency: steps to a target reward The validation checkpoints show a compute advantage before the final model selection point. Table 4 reports the first validation checkpoint where each construction crosses a fixed rubric-reward threshold on Qwen3-VL-4B/MM. For readability, we report this analysis on a single illustrative setting (Qwen3-VL-4B on MM); the same ordering holds on the other base policies and on HB, with comparable speed-ups, and we treat the per-setting numbers reported in Tables 2 and 3 as the canonical efficiency reference. POW3R reaches 46.0 dev reward at step 83, while the static scalar needs 249
Table 4. First step crossing each validation reward threshold (Qwen3-VL-4B, MM). Thresholds are percentages; “–” means not reached within 664 steps. Validation reward is the same rubric metric reported on the test split in Table 2, but evaluated on the dev split. Threshold Static Category-balanced POW3R Speed-up 45.0 46.0 47.0 48.0 49.0 50.0
9
249 249 415 581 – –
166 332 415 581 – –
83 83 166 166 249 249
3.0× 4.0× 2.5× 3.5× – –
Scale AI Research
Validation reward
Validation reward
Per-category validation reward trajectories Content
Instr.
0.50
Truth
0.525
0.50
0.500
0.48
0.475
0.46
0.450
0.45
Vis. perc.
0.44
Vis. reas.
0.50
Writing 0.50
0.475 0.45
0.450
0.48
0.425
0.46 0.40 200
400
600
200
Training step
400
600
200
Training step Static scalar
400
600
Training step
Category-balanced
POW3R dynamic
Figure 5. Per-category validation reward trajectories (Qwen3-VL-4B, MM). POW3R leads on every rubric category throughout training, with the gap opening at the first checkpoint. The largest absolute gains appear on Content, Instruction Following, Truthfulness, Visual Perception, and Visual Reasoning; on Writing Style POW3R still stays above the static baselines, but the three curves stay within roughly a point of each other because most Writing Style criteria are already passed by the base policy and offer little contrastive signal for POW3R to act on.
steps and the category-balanced reward needs 332; it is also the only method to cross a 50.0 dev threshold within the schedule. The speed-up does not come from a higher learning rate or a different optimizer schedule: every method here shares the same GRPO recipe, the same prompt budget, and the same evaluation checkpointing.
6.3 Learning dynamics at matched compute
Overall validation reward
The category breakdown in Fig. 5 shows that this is not a one-category over-optimization. POW3R improves Content, Instruction Following, Truthfulness, Visual Perception, and Visual Reasoning. Meanwhile the Writing Style reward moves the least because most style criteria are already passed by the base policy and therefore have low rollout variance for POW3R to act on.
10
0.50
Validation reward
The matched-compute curves report validation rubric reward at logged checkpoints, not the perrollout reward used for training updates. In Fig. 6, POW3R separates from both static aggregations by the first logged checkpoint and remains higher through step 501; the binary reward never moves above base and is omitted.
0.51
0.49 0.48 0.47 0.46 0.45
Static scalar Category-balanced POW3R dynamic
0.44 0.43 100
200
300
400
500
600
Training step
Figure 6. Overall validation reward trajectory (Qwen3VL-4B, MM). POW3R separates from the baselines by the first logged checkpoint and stays above through the end of training.
Scale AI Research
6.4 Where informative updates concentrate Where POW3R has signal and where it acts on it
Figure 7 reports two per-category averages from the POW3R-dynamic run, taken over the full training schedule: the mean rollout variance v j across criteria in the category (the contrastive signal POW3R has to work with) and the within-category standard deviation of POW3R’s factors α j (how much POW3R differentiates criteria inside that category). The two co-vary in the expected direction: Writing Style sits lowest on both (v j ≈0.080, std(α j )≈0.060), the visual and reasoning categories sit at the top of v j , and the ordering of std(α j ) tracks v j to within rounding. POW3R’s category mass stays uniform by construction (Eq. (3)); the chart shows the within-category reweighting that actually transmits POW3R’s mechanism into the GRPO advantage, and confirms it scales with the rollout-level contrast each category exposes.
0.090
Vis. perc.
0.089 0.097
Vis. reas.
0.085 0.095
Content
0.084 0.094
Instr.
0.075 0.091
Truth
0.082 0.080
Writing
0.00
0.060
0.02
0.04
0.06
0.08
0.10
0.12
Per-criterion average over training Mean vj (contrastive signal)
Std of αj (POW3R reweighting)
Figure 7. Where POW3R has signal and where it acts on it. Per-category averages over MM-train criteria, computed over the full training schedule (Qwen3-VL-4B POW3R-dynamic). Hatched: mean rollout variance v j . Solid: within-category standard deviation of POW3R’s factors α j . The two co-vary: categories with more contrast get reweighted more.
7 Limitations Reliance on LLM judges. POW3R reads every rubric criterion through an LLM judge: GPT-5.4-nano for training and GPT-5.4-mini for held-out evaluation, calibrated against a high-effort reference judge with 91–94% verdict agreement (Section B). Because both training rewards and reported scores flow through these judges, exact numbers can shift with judge, prompt template or sampling choice and POW3R’s policy-aware factors are themselves estimated from judge verdicts, so any consistent verdict bias propagates into where training pressure is redirected. The qualitative ordering POW3R > categorybalanced > static scalar > binary is, however, stable across our calibration and across two judge tiers. Scarcity of rubric datasets with static human weights. Public rubric-RL datasets that combine promptspecific criteria, integer human weights, multiple high-level categories, and held-out scale are rare. We therefore authored MM in-house and used HealthBench for the text-only setting. These choices leave POW3R untested on domains such as long-form coding feedback, scientific writing, or multilingual instruction following, which can have very different category structures and saturation patterns. Broader open availability of rubric datasets with static weights would let the community measure the limits of policy-aware aggregation more directly.
8 Conclusion and future work POW3R separates two roles that static rubric rewards conflate: what should matter in the final answer and what can currently teach the policy. Its policy-aware reward redirects training pressure using rollout-level contrast without changing the rubric target, yielding higher rubric reward, higher strict completion, cleaner two-objective dominance, and faster training across MM, HealthBench, and two model families. The broader takeaway is that rubric aggregation deserves to be treated as a first-class training-time design choice; careful re-weighting of an existing rubric can extract substantially more signal from the same data, judge, and optimizer. We plan to test POW3R with stronger and more diverse judges, longer training schedules, and adaptive rubric generation that co-evolves with the policy.
11
Scale AI Research
Use of generative AI. A large language model was used during manuscript preparation for editing, structural revision, and refactoring of figure-generation code. All technical content – experimental design, results, equations, and citations – was produced and verified by the authors.
References [1] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024. URL https://arxiv.org/abs/2402.03300. [2] DeepSeek-AI, Daya Guo, Dejian Yang, et al. DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025. URL https://arxiv.org/abs/2501.12948. [3] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, et al. DAPO: An open-source LLM reinforcement learning system at scale, 2025. URL https://arxiv.org/abs/2503.14476. [4] Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, et al. Spurious rewards: Rethinking training signals in RLVR, 2025. URL https://arxiv.org/abs/2506.10947. [5] Xin-Qiang Cai, Wei Wang, Feng Liu, Tongliang Liu, Gang Niu, and Masashi Sugiyama. Reinforcement learning with verifiable yet noisy rewards under imperfect verifiers, 2025. URL https://arxiv.org/abs/ 2510.00915. [6] Rahul K. Arora, Jason Wei, Rebecca Soskin Hicks, Preston Bowman, Joaquin Quiñonero-Candela, Foivos Tsimpourlas, Michael Sharman, Meghan Shah, Andrea Vallone, Alex Beutel, Johannes Heidecke, and Karan Singhal. HealthBench: Evaluating large language models towards improved human health, 2025. URL https://arxiv.org/abs/2505.08775. [7] Tong Xiao, Xin Xu, Zhenya Huang, Hongyu Gao, Quan Liu, Qi Liu, and Enhong Chen. Perception-R1: Advancing multimodal reasoning capabilities of MLLMs via visual perception reward, 2025. URL https: //arxiv.org/abs/2506.07218. [8] Ziqi Miao, Haonan Jia, Lijun Li, Chen Qian, Yuan Xiong, Wenting Yan, and Jing Shao. Seeing with you: Perception-reasoning coevolution for multimodal reasoning, 2026. URL https://arxiv.org/abs/2603. 28618. [9] Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath, Yunzhong He, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains, 2025. URL https://arxiv.org/abs/ 2507.17746. [10] Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. OpenRubrics: Towards scalable synthetic rubric generation for reward modeling and LLM alignment, 2025. URL https://arxiv. org/abs/2510.07743. [11] Sunzhu Li, Jiale Zhao, Miteto Wei, Huimin Ren, Yang Zhou, Jingwen Yang, Shunyu Liu, Kaike Zhang, and Wei Chen. RubricHub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation, 2026. URL https://arxiv.org/abs/2601.08430. [12] Mengzhao Jia, Zhihan Zhang, Ignacio Cases, Zheyuan Liu, Meng Jiang, and Peng Qi. AutoRubric: Rubricbased generative rewards for faithful multimodal reasoning, 2025. URL https://arxiv.org/abs/2510. 14738. [13] Yining Lu, Zilong Wang, Shiyang Li, Xin Liu, et al. Learning to optimize multi-objective alignment through dynamic reward weighting, 2025. URL https://arxiv.org/abs/2509.11452.
12
Scale AI Research
[14] Shuai Bai, Yuxuan Cai, Ruizhe Chen, et al. Qwen3-VL technical report, 2025. URL https://arxiv.org/ abs/2511.21631. [15] Gemma Team, Aishwarya Kamath, Johan Ferret, et al. Gemma 3 technical report, 2025. URL https: //arxiv.org/abs/2503.19786. [16] OpenAI. Introducing GPT-5.4 mini and nano. OpenAI product release, 2026. URL https://openai.com/ index/introducing-gpt-5-4-mini-and-nano/. [17] Diederik Marijn Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objective sequential decision-making, 2014. URL https://arxiv.org/abs/1402.0590. [18] Shih-Yang Liu, Xin Dong, Ximing Lu, Shizhe Diao, Peter Belcak, Mingjie Liu, Min-Hung Chen, Hongxu Yin, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Yejin Choi, Jan Kautz, and Pavlo Molchanov. GDPO: Group reward-decoupled normalization policy optimization for multi-reward RL optimization, 2026. URL https://arxiv.org/abs/2601.05242. [19] Advait Gosai, Tyler Vuong, Utkarsh Tyagi, Steven Li, Wenjia You, Miheer Bavare, Arda Uçar, Zhongwang Fang, Brian Jang, Bing Liu, and Yunzhong He. Audio MultiChallenge: A multi-turn evaluation of spoken dialogue systems on natural human interaction, 2025. URL https://arxiv.org/abs/2512.14865. [20] MohammadHossein Rezaei, Robert Vacareanu, Zihao Wang, Clinton Wang, Bing Liu, Yunzhong He, and Afra Feyza Akyürek. Online rubrics elicitation from pairwise comparisons, 2025. URL https://arxiv. org/abs/2510.07284. [21] Rulin Shao, Akari Asai, Shannon Zejiang Shen, Hamish Ivison, Varsha Kishore, Jingming Zhuo, Xinran Zhao, Molly Park, Samuel G. Finlayson, David Sontag, Tyler Murray, Sewon Min, Pradeep Dasigi, Luca Soldaini, Faeze Brahman, Wen-tau Yih, Tongshuang Wu, Luke Zettlemoyer, Yoon Kim, Hannaneh Hajishirzi, and Pang Wei Koh. DR Tulu: Reinforcement learning with evolving rubrics for deep research, 2025. URL https://arxiv.org/abs/2511.19399. [22] Yukun Chen, Jiaming Li, Longze Chen, Ze Gong, Jingpeng Li, Zhen Qin, Hengyu Chang, Ancheng Xu, Zhihao Yang, Hamid Alinejad-Rokny, Qiang Qu, Bo Zheng, and Min Yang. RuCL: Stratified rubric-based curriculum learning for multimodal large language model reasoning, 2026. URL https://arxiv.org/abs/2602. 21628. [23] Marcus Williams. Multi-objective reinforcement learning from AI feedback, 2024. URL https://arxiv. org/abs/2406.07295. [24] Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019. URL https://arxiv. org/abs/1909.08593. [25] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2020. URL https://arxiv. org/abs/2009.01325. [26] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, et al. Training language models to follow instructions with human feedback, 2022. URL https://arxiv.org/abs/2203.02155. [27] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, et al. Constitutional ai: Harmlessness from ai feedback, 2022. URL https://arxiv.org/ abs/2212.08073.
13
Scale AI Research
[28] Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, et al. RLAIF vs. RLHF: Scaling reinforcement learning from human feedback with ai feedback, 2023. URL https://arxiv.org/abs/2309.00267. [29] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, et al. RewardBench: Evaluating reward models for language modeling, 2024. URL https://arxiv.org/abs/2403.13787. [30] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-RFT: Visual reinforcement fine-tuning, 2025. URL https://arxiv.org/abs/2503.01785. [31] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Xu Tang, Yao Hu, and Shaohui Lin. Vision-R1: Incentivizing reasoning capability in multimodal large language models, 2025. URL https://arxiv.org/abs/2503.06749. [32] Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-VL: Learning to reason with multimodal large language models via step-wise group relative policy optimization, 2025. URL https://arxiv.org/abs/2503.12937. [33] Meng Cao, Haoze Zhao, Can Zhang, Xiaojun Chang, Ian Reid, and Xiaodan Liang. Ground-R1: Incentivizing grounded visual reasoning via reinforcement learning, 2025. URL https://arxiv.org/abs/2505. 20272. [34] Chi Zhang, Haibo Qiu, Qiming Zhang, Yufei Xu, Zhixiong Zeng, Siqi Yang, Peng Shi, Lin Ma, and Jing Zhang. Perceptual-evidence anchored reinforced learning for multimodal reasoning, 2025. URL https: //arxiv.org/abs/2511.18437. [35] Sicheng Feng, Kaiwen Tuo, Song Wang, Lingdong Kong, Jianke Zhu, and Huan Wang. RewardMap: Tackling sparse rewards in fine-grained visual reasoning via multi-stage reinforcement learning, 2025. URL https: //arxiv.org/abs/2510.02240. [36] Jinda Lu, Junkang Wu, Jinghan Li, Kexin Huang, Shuo Yang, Guoyin Wang, Jiancan Wu, Xiang Wang, and Xiangnan He. Bridging perception and reasoning: Token reweighting for RLVR in multimodal LLMs, 2026. URL https://arxiv.org/abs/2603.25077. [37] Xingang Guo, Utkarsh Tyagi, Advait Gosai, Paula Vergara, Jayeon Park, Ernesto Gabriel Hernandez Montoya, Chen Bo Calvin Zhang, Bin Hu, Yunzhong He, Bing Liu, and Rakshith Sharma Srinivasa. Beyond seeing: Evaluating multimodal LLMs on tool-enabled image perception, transformation, and reasoning, 2025. URL https://arxiv.org/abs/2510.12712. [38] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Yang Yue, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?, 2025. URL https://arxiv.org/abs/2504.13837. [39] An Yang, Anfeng Li, Baosong Yang, et al. Qwen3 technical report, 2025. URL https://arxiv.org/abs/ 2505.09388. [40] Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. HallusionBench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14375–14385, 2024. URL https: //arxiv.org/abs/2310.14566. [41] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 292–305. Association for Computational Linguistics, 2023. doi: 10.18653/v1/2023.emnlp-main.20. URL https://aclanthology.org/2023.emnlp-main.20/. 14
Scale AI Research
[42] Shengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang, Haodong Duan, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. MM-IFEngine: Towards multimodal instruction following, 2025. URL https://arxiv.org/abs/2504.07957. [43] Weihao Yu, Zhengyuan Yang, Lingfeng Ren, Linjie Li, Jianfeng Wang, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Lijuan Wang, and Xinchao Wang. MM-Vet v2: A challenging benchmark to evaluate large multimodal models for integrated capabilities, 2024. URL https://arxiv.org/abs/2408.00765. [44] Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representations, 2024. URL https:// openreview.net/forum?id=KUNzEQMWU7. [45] xAI. RealWorldQA: A benchmark for real-world spatial understanding. xAI dataset release, 2024. URL https://huggingface.co/datasets/xai-org/RealworldQA. [46] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. ZeRO: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16, 2020. doi: 10.1109/SC41405.2020.00024. URL https://arxiv.org/abs/ 1910.02054. [47] OpenAI. Introducing GPT-5.4. OpenAI product release, 2026. URL https://openai.com/index/ introducing-gpt-5-4/.
15
Scale AI Research
Appendix Appendix index. • Section A: Dataset and contributor details – rubric annotations, contributor demographics, and split details for MM. • Section B: Judge selection – reference-judge calibration, cost–quality tradeoffs, and operating points. • Section C: System prompts – rubric-judge prompts used for training and evaluation. • Section D: Training hyperparameters – consolidated GRPO and POW3R settings shared across all runs. • Section E: Rubric categories with example criteria – the six high-level MM rubric categories with two representative criteria each, drawn verbatim from MM-train. • Section F: Qualitative examples – four MM held-out tasks with side-by-side outputs from the base policy, the static baselines, and POW3R.
A Dataset and contributor details This section provides additional details regarding the data collection pipeline, contributor demographics, and dataset splits for the multimodal dataset.
A.1 Rubric annotations and weights Each MM example contains prompt-specific rubric criteria annotated with a category, an explicit/implicit tag, an objective/subjective tag, and a static human weight. Explicit criteria are directly stated by the user prompt or image-grounded instruction, such as a required object, format, count, comparison, or named constraint. Implicit criteria are not quoted verbatim from the prompt but are required for a highquality answer, such as avoiding unsupported claims, explaining uncertainty, or preserving the intended audience and tone. Objective criteria can be judged by a concrete fact, visual evidence, format constraint, or measurable condition; subjective criteria require a quality judgment about clarity, helpfulness, style, or appropriateness. Weights are integer importance ratings in {1, . . . , 5} assigned during rubric authoring: low weights mark helpful but nonessential details, middle weights mark important response-quality criteria, and high weights mark criteria whose failure substantially changes answer correctness or usefulness. HealthBench retains its native point values and categories; the conversion used during training and the unchanged evaluation recipe are described in Section A.4.
A.2 Contributor demographics and selection MM was authored by a vetted pool of English-proficient contributors selected for experience with analytical writing, visual reasoning, and rubric-based evaluation. We omit fine-grained vendor-identifying demographics, collection dates, geographic distribution, and job-title breakdowns from this release; aggregate quality-control statistics, redacted instructions, and the contributor screening criteria are available on request.
16
Scale AI Research
A.3 Dataset funnel and splits Our QA and review process served as a strict filtering funnel. Examples that failed independent review or did not meet dataset-level quality criteria were discarded before the final 10,000 examples were split into training, development, and test sets.
A.4 Signed HealthBench criteria during training and evaluation HealthBench criteria carry signed integer point values: a positive w j rewards a desired behavior, a negative w j penalizes an undesired behavior (e.g. unsafe advice). POW3R’s category-normalized aggregation (Eqs. (3)–(8)) assumes non-negative weights, since within-category normalizers divide by ∑ j w j and the dynamic-factor renormalization treats w j as a non-negative prior. Motivated additionally by Li et al. [11]’s finding that positive-only rubric training outperforms inclusion of negative penalties, we convert signed HB criteria into an equivalent good-behavior representation for training. To use POW3R on HB without changing the optimizer or the rubric set, we apply a one-step good-behavior conversion only at training time: for every criterion with w j < 0 we substitute the equivalent non-negative criterion that flips the judge verdict and uses |w j | as the weight, so s′j = 1 − s j and w′j = |w j |. After this rewrite, s′j = 1 means “the response avoids the prohibited behavior,” all weights are positive integers, and POW3R’s category normalization and dynamic factors are well-defined. The judge prompt and category labels for those criteria are adjusted accordingly so the LLM judge scores “avoidance” rather than “occurrence.” Evaluation never sees this rewrite: the HB Strict column in Table 3, the HB overall score, and every other reported HB number are computed under HealthBench’s original scoring script with its native signed point values and unchanged criterion text. The good-behavior conversion only affects what POW3R’s category normalizer and dynamic factors see during training, so reported HB results are directly comparable to HealthBench’s own protocol.
B Judge selection The rubric judge is a fixed component of the reward computation, so its choice has a direct effect on every downstream comparison in this paper. We therefore treat judge selection as a first-class experimental decision: we calibrate a small number of candidate judges against a high-effort reference judge, then choose two operating points – one for high-throughput training, one for slower but more careful held-out evaluation – using a single explicit cost–quality trade-off. Reference judge. We use GPT-5.4 with high-effort reasoning and the per-criterion explanation kept on as our reference (“gold”) judge [47]. This is the strongest model available to us through our LLM provider; high-effort reasoning gives it the most slack for careful per-criterion judgments, and keeping the explanation on prevents it from short-circuiting to a verdict. We acknowledge that a single reference judge cannot establish ground truth in an absolute sense; we therefore treat the agreement numbers below as a calibration of cheaper judges against the strongest judge we can afford to run, not as ground-truth accuracy. Calibration set. We sample 1,000 stratified task–rollout–rubric triples from MM-train and re-judge each one with the reference judge and every candidate judge configuration, varying model, reasoning effort, and explanation mode. Sampling is stratified across the six rubric categories and the rubric weight
17
Scale AI Research
bins w ∈ {1 − 3, 4 − 5}, so the calibration covers easy- and hard-to-judge regions evenly. All candidates score the same 1,000 items, so per-judge agreement is paired and apples-to-apples. Per-rubric vs. per-category judging. We also tested issuing one combined judge call per rollout category instead of per rollout criterion. The per-category mode is roughly C-times cheaper but rankcorrelation against the reference drops appreciably and the judge tends to merge or skip individual rubrics whose verdicts ought to remain independent. We therefore keep the per-rubric formulation throughout the paper. Per-rubric judge calibration. Table 5 reports pairwise verdict agreement for each candidate judge configuration. The reference row shows the reference judge’s own agreement against a small human-rated subset of the calibration items (used to set a quality ceiling at 95.4%); every other row reports agreement of the candidate judge against the reference judge on the full 1,000-item calibration set. Mini-class judges (GPT-5.4-mini) at medium effort with explanation track the reference within ∼2 points at roughly 1/8 of the reference cost; nano-class judges (GPT-5.4-nano) at medium effort with explanation lose another ∼2 points but are an additional ∼10× cheaper, with rank fidelity preserved when the explanation is kept on. Table 5. Calibration of candidate per-rubric judges against the reference judge (GPT-5.4 with high-effort reasoning and explanation). The reference row’s 95.4% entry is the reference judge’s agreement with a human-rated subset; every other row is agreement of that candidate judge against the reference on the same 1,000 stratified items from MM-train. Bold rows mark the operating points used in the paper. Judge model
Reasoning
Explanation
Agreement (%)
GPT-5.4 (reference)
Cost / 1k (USD)
high
yes
95.4
12.50
GPT-5.4-mini GPT-5.4-mini GPT-5.4-mini GPT-5.4-mini GPT-5.4-mini GPT-5.4-mini
high high medium medium low low
yes no yes no yes no
94.1 93.7 93.6 93.4 89.5 87.9
2.19 1.79 1.52 1.14 1.06 0.76
GPT-5.4-nano GPT-5.4-nano GPT-5.4-nano GPT-5.4-nano GPT-5.4-nano
high medium medium low low
yes yes no yes no
91.8 91.4 91.4 (rank break) 88.0 82.6
0.13 0.12 0.08 0.10 0.07
Two notes from the table. First, removing the per-criterion explanation at the nano scale gives the same head-line agreement number but breaks rank correlation between the candidate’s per-model rubric pass rate and the reference’s: agreements at the right level on average, but disagreements that flip per-model rankings. We therefore always keep the explanation on. Second, mini-class judges are roughly 12× more expensive than nano-class judges per 1,000 calls, but agreement gains from mini-medium-withexplanation over nano-medium-with-explanation are ∼2 percentage points – non-trivial for held-out evaluation, but not worth 12× the cost on every training rollout. Operating points used in the paper. Based on the table above, we use GPT-5.4-nano + medium reasoning + explanation (91.4% agreement, $0.12 / 1,000 judgments) for training rewards, and GPT-
18
Scale AI Research
5.4-mini + medium reasoning + explanation (93.6% agreement, $1.52 / 1,000 judgments) for held-out evaluation rewards. Using the more accurate model only at evaluation time also reduces judge–training entanglement: training never sees the verdict distribution it will be evaluated against.
C System prompts This appendix collects the system prompts used by the rubric judges referenced in Section 5. Both judges share the same single-criterion template; only the underlying model differs (GPT-5.4-nano with medium reasoning at training time, GPT-5.4-mini with medium reasoning at evaluation time). Both are queried at temperature T =1.0 with up to 2048 completion tokens. Per-criterion rubric judge. Given a sampled response and a single rubric criterion (with its category, explicit/implicit and objective/subjective annotations, and human weight), the judge returns a JSON object with a one-sentence reasoning and a binary verdict. Does the response satisfy this rubric criterion? Evaluate ONLY this criterion. Rules: - "Explicit" criteria must be directly addressed. "Implicit" may be inferred. - "Objective" = factual pass/fail. "Subjective" = quality judgment. - Weight is context only -- does not affect your pass/fail decision. - Minor phrasing/formatting differences are OK if substance is correct. - OCR/text recognition criteria require exact text -- "EXIST" ̸= "EXIT". Examples: - Rubric: "Identify chair material as wood" / Response: "a wooden chair" → {"reasoning": "Explicitly identifies wood.", "criteria_met": true} - Rubric: "Graph shows decreasing trend after 2020" / Response: "steady growth" → {"reasoning": "Claims growth, not decrease.", "criteria_met": false} - Rubric: "List ≥3 differences" / Response: "one difference" → {"reasoning": "Only 1 of 3 required.", "criteria_met": false} - Rubric: "Read sign as ‘EMERGENCY EXIT"’ / Response: " ‘EMERGENCY EXIST’ " → {"reasoning": "EXIST ̸= EXIT, OCR must be exact.", "criteria_met": false} Rubric: - Title: {rubric_title} - Category: {rubric_category} - {explicit_implicit} | {objective_subjective} | Weight: {rubric_weight} - Criteria: {rubric_rationale} Response: {response} Return ONLY valid JSON. "reasoning" BEFORE "criteria_met". {"reasoning": "⟨one sentence⟩", "criteria_met": true/false}
The criterion-level judge prompt above is the prompt template used by the training (GPT-5.4-nano) and held-out evaluation (GPT-5.4-mini) judges, and is what is needed to reproduce the reward calculations reported in the main paper. Verdict-only per-criterion variant (calibration only). This prompt is identical in intent to the percriterion judge above, but asks the judge to emit only the binary verdict, without the one-sentence reasoning field. Section B reports that this variant matches the with-explanation variant on aggregate verdict agreement but loses per-model rank fidelity, which is why we keep the explanation on in the
19
Scale AI Research
operating points used in the paper. Does the response satisfy this criterion? Evaluate ONLY this criterion. "Explicit" = directly addressed. "Implicit" = may be inferred. OCR/text recognition = exact match required. Rubric: - Title: {rubric_title} | Category: {rubric_category} - {explicit_implicit} | {objective_subjective} | Weight: {rubric_weight} - Criteria: {rubric_rationale} Response: {response} Return ONLY valid JSON: {"criteria_met": true} or {"criteria_met": false}
Per-category batched variant (calibration only). This prompt scores all rubric criteria for a single response in one call by asking the judge to evaluate each criterion independently and emit a single weight-normalized [0, 1] score. We use it only in the cost–quality calibration in Section B; in the paper itself every rubric criterion is judged one at a time. Score the response against all rubrics below. Evaluate each independently as pass/fail. Rules: - "Explicit" = directly addressed. "Implicit" = may be inferred. - "Objective" = factual. "Subjective" = quality judgment. - OCR/text recognition criteria require exact text match. - score = sum(weight of PASSED rubrics) / {total_weight}, between 0.0 and 1.0. Rubrics: {rubrics_text} Total weight: {total_weight} Response: {response} Return ONLY valid JSON. "reasoning" BEFORE "score". {"reasoning": "⟨which rubrics passed/failed⟩", "score": ⟨0.0-1.0⟩}
All three prompts are taken verbatim from our training and evaluation code; only formatting (line breaks, font, and the surrounding box) is adjusted for the paper.
D Training hyperparameters Table 6 consolidates the GRPO and POW3R hyperparameters used for every training run reported in this paper. The same values are used across all base policies (Qwen3-VL-4B, Qwen3-VL-8B, Gemma3-4B for MM; Qwen3-4B, Qwen3-8B, Gemma3-4B for HB) and across the Binary, Static scalar, Category-balanced, and POW3R dynamic reward constructions.
E Rubric categories with example criteria MM rubrics are tagged with one of six high-level quality categories: Visual perception, Visual reasoning, Content completeness, Instruction following, Truthfulness, and Writing style / presentation. Each category captures a different aspect of response quality so that POW3R’s category-normalized aggregation (Eq. (8)) can balance grounding, reasoning, completeness, instruction-following, factual correctness, 20
Scale AI Research
Table 6. Training hyperparameters shared across all runs in Tables 2 and 3. Group
Hyperparameter
Value
GRPO objective
Rollouts per prompt-group (G) Sampling temperature (T) Maximum completion length KL coefficient (β) PPO clip range (ε) Gradient clip (max_grad_norm)
16 1.0 3584 tokens 0.1 0.2 0.5
Optimizer
Optimizer Learning rate LR schedule Weight decay
AdamW 3 × 10−7 constant 0.0
Batch & schedule
Per-device train batch size Gradient accumulation steps Maximum GRPO steps Validation interval Checkpoint selection
1 4 664 every 83 steps val-best per method
POW3R (Rdyn )
Variance floor (αmin ) Variance ceiling (αmax ) Numerical stability (ϵ) Smoothing weight (λ) EMA coefficient (β ema ) Minimum valid rollout fraction
0.67 1.5 10−4 0.5 0.2 0.75
Hardware & precision
Compute Distributed strategy Numerical precision Activation checkpointing
1× node, 8× H100 GPUs DeepSpeed ZeRO-3 BF16 enabled
Judges
Training judge Evaluation judge Judge sampling
GPT-5.4-nano, medium reasoning, explanation on GPT-5.4-mini, medium reasoning, explanation on T =1.0, max 2048 completion tokens
and presentation criteria within the same prompt. Below we list each category, a one-line operational definition, and two representative criteria taken verbatim from MM-train (7–10 such criteria typically populate one task across these six categories; see Table 1). The bracketed tag on each criterion is [weight | explicit/implicit | objective/subjective] per the schema in Section A.1. Visual perception. What the image directly shows: objects, text, named entities, counts, and other content that can be read off the image without further inference. • [5 | Explicit | Objective] “The response must identify at least two of the visible players on the court by specifying their names (e.g., LeBron James, Anthony Davis) and their respective teams (e.g., Lakers or Rockets).” • [4 | Explicit | Objective] “The response must identify the stores from the letter by listing the following stores: K-Mart, Toys ‘R’ Us, Sears, Roebuck & Co., and Montgomery Ward.”
21
Scale AI Research
Visual reasoning. Inferences that require combining visual cues, such as version identification from interface details, structural interpretation of diagrams and circuits, or reasoning about visual context. • [4 | Explicit | Objective] “The response must identify the specific version of NBA 2K shown in the screenshot (2K20) based on visual elements such as graphics, court layout, or interface design unique to that version.” • [4 | Explicit | Objective] “The response must infer who wrote the initial request and what their request was by referencing the apology in the P.S. to Laura about LEGO being unable to send product brochures.” Content completeness. Whether the response covers every element the prompt asks for, with all required items, sub-items, comparisons, or sections present. • [5 | Explicit | Objective] “The response must identify all of the labeled objects in the image: Algenib, Deneb Kaitos, Fische, Fomalhaut, Jupiter, Merkur, Neptun, Sonne, and Wasserman.” • [4 | Explicit | Objective] “The response must explain how each proposed power factor correction method affects both reactive and active power by discussing the change in the circuit’s power characteristics.” Instruction following. Whether the response respects explicit constraints in the prompt: ordering, sectioning, length, format, and procedural requirements. • [2 | Explicit | Objective] “The response must identify the labeled entities in this order: Algenib, Deneb Kaitos, Fische, Fomalhaut, Jupiter, Merkur, Neptun, Sonne, and Wasserman.” • [3 | Explicit | Objective] “The response must provide the answers in 2 sections.” Truthfulness. Whether claims in the response are accurate – correct identification of named entities, correct relationships, correct intermediate computations, and absence of unsupported claims. • [5 | Explicit | Objective] “The response must state the television series shown in the image by identifying the show as Avatar: The Last Airbender.” • [5 | Explicit | Objective] “The response must analyse the impedance change by explaining how inductance affects impedance compared to resistance.” Writing style / presentation. Formatting, organization, register, and adherence to specific presentation requirements such as bullet lists, bolding, or audience-appropriate language. • [4 | Explicit | Objective] “The response must order these alphabetically by reading their names and organising them into an alphabetised bullet list.” • [2 | Explicit | Subjective] “The response should provide the answers in a high school student’s language level by using clear and simple terms and sentences.”
22
Scale AI Research
F Qualitative examples We present four MM held-out tasks from the Qwen3-VL-4B test set, drawn from four different task families (Spatial Reasoning, Visual Perception, Visual Reasoning on a polar curve, and a Visual-grounded math worksheet). For each task we show the image, the prompt, what the rubric expects, and the outputs of the base policy together with the three trained settings (Rscalar , Rcat , Rdyn ). Pass counts report the number of rubric criteria satisfied out of the task’s total criterion count, while strict=True indicates that every criterion flagged as required (the subset that defines strict completion in Tables 2 and 3) was satisfied; a method can therefore be strict=True while leaving optional criteria unsatisfied. Bold marks the substantive error or correct decision that distinguishes the methods. Reading these side by side highlights the kinds of grounding, completeness, and naming behaviors that POW3R recovers relative to the static baselines. Example 1: Spatial reasoning on a Tokyo metro map. Prompt: Based on the layout and connectivity of the transit system shown in the image, what station is the most central hub for transfers, and what factors contribute to its importance? Required answer (rubric): Tokyo Station – visually centered on the map and the highest-degree transfer node. Base / Rscalar / Rcat : all confidently answer “Ikebukuro Station”, a peripheral upper-left node, and hallucinate line intersections that are not in the image. (0/3 required rubrics passed for each method.) POW3R (Rdyn ): “Tokyo Station, located at the center of the map. Multiple major lines converge here.” (3/3 passed.)
Example 2: Visual perception on a meme image (animal identification). Prompt: Can you explain why the google home is represented by a dog? What does that have to do with the context? Required answer (rubric): Acknowledge the dog, describe its expression, and connect the dog’s indifferent look to a Google-Home obedience-butmisunderstanding joke. Base / Rscalar / Rcat : all argue with the user that the image is a pig, not a dog, and proceed to explain a different “pig” meme. The visual misperception cascades into the rest of the answer. (0/5 required rubrics passed.) POW3R (Rdyn ): accepts the prompt’s framing, identifies the dog and its indifferent expression, and ties the dog’s obedience-but-notunderstanding pose to the Google Home punchline. (5/5 passed.)
23
Scale AI Research
Example 3: Visual reasoning on a polar-curve identification. Prompt: How about the second shape? What’s its name? Required answer (rubric): The second shape is a cardioid (r = a(1 + cos θ )), recognized from its single rightward cusp and heart shape. Base / Rscalar / Rcat : hedge between “rose curve / limaçon / heart curve” or pivot to a generic discussion of polar curves without naming the specific shape. (0–3/8 rubrics passed; strict = False.) POW3R (Rdyn ): “This is a cardioid, r = a(1 + cos θ ), identifiable by the single cusp on the right and the symmetric heart-shaped contour.” (5/8 passed; strict = True.)
Example 4: Visual-grounded math worksheet generation. Prompt: Could you make a new worksheet of 4 sums with these fruit and values that use division, multiplication, and subtraction where each of the answers are 4? Required answer (rubric): Four arithmetic problems using fruits with assigned values ( → 1, 2, 3, 4); every answer must equal 4; mix of ÷, ×, −. Base / Rscalar / Rcat : produce four sums but at least one answer is wrong (e.g. “Mango−Watermelon = 2” presented as a target-4 problem) and the worksheet self-corrects mid-stream rather than committing. (4–5/13 rubrics passed; strict = False.) POW3R (Rdyn ): four clean problems each evaluating to 4, mixed across ÷, ×, −, with the requested fruit-value substitution intact. (8/13 passed; strict = True.)
24